To prevent undesirable content from the lookup indexes, webmasters can instruct spiders to not crawl particular files or directories with the standard robots.txt file in the basis Listing from the domain. Additionally, a webpage may be explicitly excluded from a search engine's database by using a meta tag unique to robots (ordinarily ). Every time