A robots.txt file is an ASCII or plain text document made up of commands specifically meant to be read by search engine crawlers. Crawlers (sometimes called bots or spiders) are autonomous programs used by search engines like Google and Bing to find and “read” web pages.
Crawlers enable search engines to understand what kind of information is stored on a page and then index that page so it can be displayed in response to user queries. During indexing, the search engine’s algorithm sorts pages into an order that directly affects their SERP ranking.
The first thing crawlers do when visiting any website downloads the native robots.txt file. This gives you a chance to communicate with the crawler, explain to it how it should read your site, and differentiate which pages are important and which pages are unimportant.
Every search engine has its own crawler, and every crawler has its own identifying “user-agent” designation.
- Google: Googlebot
- Google Images: Googlebot-Image
- Bing: Bingbot
- Yahoo: Slurp
- DuckDuckGo: DuckDuckBot
- Because crawlers are always scouring the internet for new pages, it’s important not only to have a robots.txt file but also to ensure that the file stays as up-to-date and accurate as possible. In a very real sense, robots.txt gives you the opportunity to take greater control over how your website is indexed, which has a huge impact on how your site’s pages will rank in search results.
Robots.txt is important for SEO
Allowing/Disallowing Certain Pages
A robots.txt file is an essential part of every website for a few different reasons. The first and most obvious is that they enable you to control which pages on your site do and do not get crawled.
This can be done with an “allow” or “disallow” command. In most cases, you’re going to be using the latter more than the former, with the allow command really only being useful for overwriting a disallow. Disallowing certain pages means that crawlers will exclude them when reading your website.
You might wonder why you would ever want to do that; after all, isn’t the whole point of SEO to make it easier for search engines, and therefore users, to find your pages?
Yes and no. Actually, the whole point of SEO is to make it easier for search engines and their users to find the correct pages. Virtually every website, no matter how big or small, will have pages that aren’t meant to be seen by anyone but you. Allowing crawlers to read these pages increases the likelihood of them showing up in search results in place of the pages you actually want users to visit.
Examples of pages you might want to disallow crawling include the following:
- Pages with duplicate content
- Pages that are still under construction
- Pages meant to be exclusively accessed via URL or login
- Pages used for administrative tasks
- “Pages” that are actually just multimedia resources (such as images or PDF files)
Additionally, for large websites with hundreds or even thousands of pages (for example, blogs or e-commerce sites), disallowing can also help you avoid wasting your “crawl budget.”
Since Google and other search engines can only crawl so many pages on a website, it’s important to make sure that your most important pages (i.e. the ones that drive traffic, shares, and conversions) are prioritized over less important ones.
Allowing/Disallowing Certain Crawlers
Most of the time, you’ll be allowing or disallowing all crawlers from a certain page or pages. However, there may be instances where you want to target specific crawlers instead.
For instance, if you’re trying to cut down on image theft or bandwidth abuse, instead of disallowing a long list of individual media resource URLs, it makes more sense to simply disallow Googlebot-Image and other image-centric crawlers.
Another time when you might want to disallow certain crawlers rather is if you’re receiving a lot of problematic or spammy traffic from one search engine more than another.
Spam traffic from bots and other sources isn’t likely to harm your website (although it can contribute to server overloads, a topic we’ll discuss more a little later). However, it can seriously skew your analytics, inhibiting your ability to make accurate, data-based decisions.
Directing Crawlers to the XML Sitemap
Robots.txt files aren’t the only tool you have to funnel search engine crawlers towards the most important pages on your website. XML sitemaps likewise serve a very similar function.
Additionally, XML sitemaps contain other pieces of useful information, including when pages were last updated, which pages search engines should prioritize, and how to locate important content that might otherwise be deeply buried.
All this makes having an XML sitemap an extremely potent weapon in your SEO arsenal. Of course, just as those kids in The Blair Witch Project discovered the hard way, a map is only useful as long as you can actually find it.
Enter robots.txt. Since a crawler will read your robots.txt file before it does anything else, you can use this to direct the crawler directly to your sitemap, ensuring that no time or resource is wasted.
This is especially helpful if you have a large website with tons of links per page, as without a sitemap crawlers rely primarily on links to find their way. If your website has rock-solid interlinking (or very few pages), then it might not be something you have to worry much about. Nevertheless, using robots.txt hand-in-hand with an XML sitemap is definitely recommended.
Protecting Against Web Server Overload
Okay, this one isn’t an “official” robots.txt directive, but it is one that several major search crawlers take heed of regardless. If anyone asks where you heard this, don’t tell them it was us.
By including a “crawl-delay” command in your robots.txt, you can control not only which pages crawlers read, but the speed at which they do it. Normally, search engine crawlers are remarkably fast, bouncing from page to page to page to page much more quickly than any human could manage. That makes them extremely powerful and efficient.
It also makes them a liability, at least for sites with limited hosting resources.
The more traffic a website receives, the harder the server it’s hosted on has to work to display the site’s pages. When the rate of traffic exceeds the server’s ability to accommodate it, the result is an overload. That means page speed slowing to a crawl, as well as a sharp increase in 500, 502, 503, and 504 errors. Simply put, it means disaster.
Although it’s doesn’t happen often, search engine crawlers can contribute to server overloads by pushing traffic past the tipping point. If this is something you’re concerned about, you can actually command crawlers to slow down, delaying them from moving to the next page by anywhere from 1 to 30 seconds.
.jpg)