Translations:Manual:Robots.txt/24/en

Some bots have a custom specification for this; Inktomi responds to a "Crawl-delay" line, which can specify the minimum delay in seconds between hits. (Their default is 15 seconds.)