Translations:Manual:Robots.txt/1/en

robots.txt files are part of the Robots Exclusion Standard. They tell web robots how to index a site. A robots.txt file must be placed in the web root of a domain.