Translations:Manual:Robots.txt/24/en

Some bots have a custom specification for this; Inktomi responds to a "mail>mailarchive:wikitech-l/2003-August/005712.html|Crawl-delay" line which can specify the minimum delay in seconds between hits. (Their default is 15 seconds.)