Translations:Manual:Robots.txt/26/en

Sometimes a custom-written bot isn't very smart or is outright malicious and doesn't obey robots.txt at all (or obeys the path restrictions, but spiders very fast, bogging down the site). It may be necessary to block specific user-agent strings or individual IPs of offenders.