What is llms.txt, and does your business need one?
A plain-English explanation of llms.txt: what the file is, who proposed it, which AI tools use it, what to put in one, and whether it is worth your time.
Read →Glossary
What is robots.txt?
Also called: robots file, robots exclusion protocol.
Every website can have a plain text file at yoursite.co.uk/robots.txt. It contains simple rules for bots, such as ‘do not crawl the admin area’, and usually points to the sitemap.
It controls crawling, not indexing. Blocking a page in robots.txt stops Google reading it, but Google can still list the address if other sites link to it. To keep a page out of search results properly, you use a ‘noindex’ instruction on the page instead.
It is also a request rather than a lock. Reputable crawlers follow it; badly behaved ones ignore it. It is not a security measure.
Why it matters for your business
One wrong line in robots.txt can hide an entire website from Google. This happens more often than you would think when a site moves from a test address to the live one and the ‘block everything’ rule from testing comes with it.
Example. A line reading ‘Disallow: /’ tells every well-behaved crawler to stay away from the whole site. If that is on your live website, your pages will slowly drop out of Google.
Related terms
£299to set up
£99a month
3working days to launch
You approve it before anything goes live.
Designed from scratch, every word written for you, hosting and Google setup included. Or pay £1,795 once and £49 a month with no minimum term.
Let’s get your phone ringing.
£299 setup, then £99 a month. Live in 3 working days. Tell us what your business does - we reply the same working day.