To stay away from unwanted content while in the search indexes, site owners can instruct spiders to not crawl specified information or directories in the common robots.txt file in the root directory on the domain. Moreover, a site is usually explicitly excluded from a search engine's databases by using a meta tag specific to robots (ordinarily ). E