Translations:Manual:Robots.txt/9/en

It is easy to prevent spiders from crawling non-article pages if you are using Wikipedia-style short URLs.