Contact

Crawler (search bot)

Category
SEO, GEO & AEO
In Hungarian
Keresőrobot (crawler)
Entry
39 / 90

In short: What is Crawler (search bot)?

A crawler is an automated program that follows links across websites and collects their content for a search engine or an AI system.

Explanation

A crawler downloads a page, reads its content and outgoing links, and follows those links to discover further pages. Whatever it cannot reach or parse effectively does not exist for the search engine. Alongside traditional search engines, AI providers now operate their own crawlers too.

Crawlability is therefore the foundation of all SEO: it requires working internal links, fast server responses and content present in the initial HTML. The robots.txt file controls which crawlers may access which sections.

Example: if no internal link on a website points to an important subpage, a crawler will likely never discover it, no matter how strong the content is.

All terms from A to Z

Have an idea or an engineering challenge to solve?

Let's talk