Translations:Manual:Robots.txt/1/en

robots.txt files are part of the Robots Exclusion Standard, and can help with. They tell web robots how to crawl a site. A robots.txt file must be placed in the web root of a domain.