robots.txt
In short: What is robots.txt?
robots.txt is a text file at a website's root that tells crawlers which parts of the site they may and may not crawl.
Explanation
The file specifies allow and disallow rules either for all crawlers or for specific user agents, and can also point to the sitemap URL. Well-behaved crawlers read and respect it before visiting pages.
Crucially, it is not a security mechanism: disallowed URLs remain publicly accessible, and blocking a crawler alone does not always prevent a URL from appearing in search results. Conversely, a mistyped rule can make the entire website invisible to search engines.
Example: an online store blocks crawlers from cart and internal search URLs while leaving product and category pages open.