The robots.txt file at the root of a web site declares what parts of the web site has not to be used by the search engine crawlers. Robots.txt tool allows the administrator of a web site to define the disallow folders of the web site and the URL of the sitemap.
