Translations:Manual:Robots.txt/9/en

It is easy to prevent spiders from crawling non-article pages if you are using Wikipedia-style short URLs. Assuming articles are accessible through  and everything else is available through $query: