Translations:Manual:Robots.txt/40/en

If not directed otherwise, spiders may try to index thousands of similar pages, overloading the webserver.