Computer Magic Logo
Create the robots.txt

Tuesday, March 15, 2016

Published by Aristotelis Pitaridis

We expand the SEO node of the CM Umbraco Tools section and the Robots.txt node will appear. We select the Robots.txt node and a form will appear on the screen.

First of all we have a list with all the folders in our web site in order to check the folders that we want to disallow for the search engine crawlers. The first time that we open this tool some of the folders will be automatically be checked which are the recommended list to disallow.

The sitemap input control allows us to define the URL of the site so that it will be defined in the robots.txt file. If we do not define the sitemap URL, the equivalent record in the robots.txt will not appear in the file.

If we click the save button, the robots.txt file will be created and a link to the robots.txt file will appear on the screen.

If we click the link the contents of the generated file will appear on the screen.