# What is Crawling?

> Crawling is when Google's automated software visits your website and follows its links to discover pages, the step that comes before a page can be indexed.

- Source: https://nerdsharks.com/glossary/crawling
- Publisher: Nerdsharks (Exeter, Devon, UK)
- Updated: 2026-09-26

Also called: Googlebot, spider, web crawler, bot.

Google uses automated programs, often called crawlers, spiders or bots (Google's is called Googlebot), to visit websites. They load a page, read it, note the links on it, and follow those links to find more pages.

Crawling is discovery, not approval. After a page is crawled, Google decides separately whether to index it. A page nobody links to, and which is not in your sitemap, may never be found at all.

Crawlers can be blocked, deliberately or by accident, using a file called robots.txt. They also struggle with pages that only appear after clicking through forms or logging in.

## Why it matters for a small business

If Google cannot crawl your pages, they cannot rank. For a small site this is rarely complicated, but a broken menu, a misconfigured robots.txt or pages with no links pointing to them can leave parts of your site invisible to search.

**Example:** A groomer adds a new ‘Puppy first groom’ page but forgets to link to it from the menu or any other page. Googlebot never finds it, so it never appears in search.