# What is robots.txt?

> robots.txt is a small text file at the root of a website that tells search engine crawlers which parts of the site they are allowed to visit.

- Source: https://nerdsharks.com/glossary/robots-txt
- Publisher: Nerdsharks (Exeter, Devon, UK)
- Updated: 2026-09-26

Also called: robots file, robots exclusion protocol.

Every website can have a plain text file at yoursite.co.uk/robots.txt. It contains simple rules for bots, such as ‘do not crawl the admin area’, and usually points to the sitemap.

It controls crawling, not indexing. Blocking a page in robots.txt stops Google reading it, but Google can still list the address if other sites link to it. To keep a page out of search results properly, you use a ‘noindex’ instruction on the page instead.

It is also a request rather than a lock. Reputable crawlers follow it; badly behaved ones ignore it. It is not a security measure.

## Why it matters for a small business

One wrong line in robots.txt can hide an entire website from Google. This happens more often than you would think when a site moves from a test address to the live one and the ‘block everything’ rule from testing comes with it.

**Example:** A line reading ‘Disallow: /’ tells every well-behaved crawler to stay away from the whole site. If that is on your live website, your pages will slowly drop out of Google.