Translations:Apache configuration/34/en

You can use a file to tell well-behaved spiders not to download dynamically generated pages (edit pages, for instance). This can reduce the load on your webserver, preserve your bandwidth, and prevent duplicate content issues with search engines. However, malicious bots could tie up your webserver and waste your bandwidth by downloading a large volume of pages extremely quickly. Request throttling can help protect against this.