The Complete Guide to eCommerce Development 2024

4 Useful Hints About robots.txt File For CS-Cart Stores

The robots.txt file is a file telling the crawling bots how to scan your website: what to crawl and what to ignore. Crawlers influence on how you may be ranked by the search engines. So, a proper completing of that file is all about SEO good practice to rank higher in search. How to enhance this tiny file to make it work for you? Here is a cheat sheet list from our tech guys.

#1 – Sitemap

Indicate the sitemap in your robots.txt file to make the site structure visible for bots to crawl and index your site better. Using the sitemap directive tells search engines where to find your XML sitemap.

#2 – Crawl Delay

Indicate the crawl delay intervals to define how frequently bots can scan your website. If the crawling period is too short, then it may overload your website making troubles for real customers to interact with your content.

#3 – Service Pages

Disallow indexing non-relevant service pages like printout information, for example. It makes sense to allow content and product pages. 

#4 – Dispatch=debugger 

Draw attention to *dispatch=debugger* agent as the default Cs-Cart’s robots.txt file does not configure that properly making visible some sensitive data (including developers’ credentials). Preventing bots from checking out your private folders will make them much harder to find and index.

Extra hints:

  • Ban bad bots! Learn more about bad bots from here.
  • Check the bot-to-visitor ratio to know how attractive your online store is in the eyes of the real people visiting your website. I the number of bots prevails, it should give you a pause of thought.

Closing

Adopt the best practices and rank higher with CS⁠-⁠Cart! Below is an example of a proper-configured robots.txt file drawn specifically for the CS⁠-⁠Cart platform. Note that this file still needs to be fine-tuned to fit your website.

And if you don’t want to tackle with tech issues, don’t hesitate to contact our developers!

Share:
We use cookies to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners who may combine it with other information that you’ve provided to them or that they’ve collected from your use of their services.