Glossary
What is Crawling?
Also called: Googlebot, spider, web crawler, bot.
Google uses automated programs, often called crawlers, spiders or bots (Google's is called Googlebot), to visit websites. They load a page, read it, note the links on it, and follow those links to find more pages.
Crawling is discovery, not approval. After a page is crawled, Google decides separately whether to index it. A page nobody links to, and which is not in your sitemap, may never be found at all.
Crawlers can be blocked, deliberately or by accident, using a file called robots.txt. They also struggle with pages that only appear after clicking through forms or logging in.
Why it matters for your business
If Google cannot crawl your pages, they cannot rank. For a small site this is rarely complicated, but a broken menu, a misconfigured robots.txt or pages with no links pointing to them can leave parts of your site invisible to search.
Example. A groomer adds a new ‘Puppy first groom’ page but forgets to link to it from the menu or any other page. Googlebot never finds it, so it never appears in search.
Related terms
£299to set up
£99a month
3working days to launch
You approve it before anything goes live.
Designed from scratch, every word written for you, hosting and Google setup included. Or pay £1,795 once and £49 a month with no minimum term.
Let’s get your phone ringing.
£299 setup, then £99 a month. Live in 3 working days. Tell us what your business does - we reply the same working day.