Contact

robots.txt

Category
SEO, GEO & AEO
In Hungarian
robots.txt
Entry
71 / 90

In short: What is robots.txt?

robots.txt is a text file at a website's root that tells crawlers which parts of the site they may and may not crawl.

Explanation

The file specifies allow and disallow rules either for all crawlers or for specific user agents, and can also point to the sitemap URL. Well-behaved crawlers read and respect it before visiting pages.

Crucially, it is not a security mechanism: disallowed URLs remain publicly accessible, and blocking a crawler alone does not always prevent a URL from appearing in search results. Conversely, a mistyped rule can make the entire website invisible to search engines.

Example: an online store blocks crawlers from cart and internal search URLs while leaving product and category pages open.

All terms from A to Z

Have an idea or an engineering challenge to solve?

Let's talk