Translations:Manual:Robots.txt/9/en

It is easy to prevent spiders from crawling non-article pages if you are using Wikipedia-style short URLs. Assuming articles are accessible through /wiki/Some_title and everything else is available through $query: