What robots.txt does
The robots.txt file at the root of your domain tells crawlers which paths they may crawl. It does not reliably keep a page out of search results: a blocked URL can still be indexed (without its content) if other pages link to it. To keep a page out of the index, allow crawling and use noindex instead.
Good uses
- Stop crawlers wasting time on infinite filter combinations, internal search results or cart pages.
- Point crawlers to your XML sitemap with a
Sitemap:line.
Common mistakes
- Blocking CSS or JavaScript files, which prevents search engines from rendering pages properly.
- Leaving a staging rule like
Disallow: /on the live site, which blocks the whole site from crawling. - Using robots.txt to hide private content: the file is public, and anything sensitive should require a login.
Crawl budget (how much a search engine will crawl) is mainly a concern for very large sites with many thousands of URLs. Most small sites need only to avoid blocking important pages and to keep URLs clean.
Worked examples
A sensible robots.txt
User-agent: *
Disallow: /cart/
Disallow: /search/
Disallow: /*?sort=
Sitemap: https://www.example.com/sitemap.xmlBlocks low-value crawl paths and lists the sitemap. CSS and JS stay crawlable.
Dangerous leftover from staging
User-agent: *
Disallow: /This blocks crawling of the entire site.
Key takeaways
- robots.txt controls crawling, not indexing.
- Use noindex (on a crawlable page) to keep a page out of results.
- Never block CSS/JS needed for rendering, and remove staging Disallow rules at launch.
Practice: Audit your robots.txt
- Open yourdomain.com/robots.txt.
- Check every Disallow rule and confirm it blocks only low-value paths.
- Confirm CSS and JS are not blocked.
- Add a Sitemap line if missing.
Go further: Use Search Console’s robots.txt report to check fetch status.