Crawler (search bot)
In short: What is Crawler (search bot)?
A crawler is an automated program that follows links across websites and collects their content for a search engine or an AI system.
Explanation
A crawler downloads a page, reads its content and outgoing links, and follows those links to discover further pages. Whatever it cannot reach or parse effectively does not exist for the search engine. Alongside traditional search engines, AI providers now operate their own crawlers too.
Crawlability is therefore the foundation of all SEO: it requires working internal links, fast server responses and content present in the initial HTML. The robots.txt file controls which crawlers may access which sections.
Example: if no internal link on a website points to an important subpage, a crawler will likely never discover it, no matter how strong the content is.