Also known as robots exclusion protocol. Robots.txt is a text file stored in a site’s root directory that tells a search engine crawler which site pages/sub-folders that should not be included in the search engine Index.