Manual:Robots.txt/tr

robots.txt dosyaları Robot Hariç Tutma Standardı'nın bir parçasıdır ve konusunda yardımcı olabilir. web robotlarına bir sitenin nasıl taranacağını söylerler. Bir etki alanının web köküne bir robots.txt dosyası yerleştirilmelidir.

Tüm taramayı önleyin
Bu kod, tüm botların sitenizdeki tüm sayfaları taramasını engeller:

Yalnızca belirli bir örümceği engellemek istiyorsanız, yıldız işaretini örümceğin kullanıcı aracısı ile değiştirin.

Madde dışı sayfaların taranmasını önleyin
MediaWiki, yalnızca canlı insanlar için yararlı olan birçok sayfa oluşturur: eski revizyonlar ve farklılıklar, maddelerde bulunan içeriği çoğaltma eğilimindedir. Düzenleme sayfaları ve çoğu özel sayfa dinamik olarak oluşturulur, bu da onları yalnızca editörler için yararlı ve sunmaları nispeten pahalı hâle getirir. Aksi belirtilmedikçe, örümcekler binlerce benzer sayfayı dizine eklemeye çalışarak web sunucusunu aşırı yükleyebilir.

Kısa URL'ler ile
Vikipedi tarzı kısa URL'ler kullanıyorsanız, örümceklerin madde olmayan sayfaları taramasını önlemek kolaydır. Maddelere  üzerinden erişilebildiğini ve diğer her şeyin   üzerinden erişilebilir olduğunu varsayarak:

Be careful, though! If you put this line by accident:

you'll block access to the /wiki directory, and search engines will drop your wiki!

Be aware that this solution will also cause CSS, JavaScript and image files to be blocked, so search engines like Google will not be able to render previews of wiki articles. To work around this, instead of blocking the entire  directory, only   need be blocked:

This works because CSS and JavaScript is retrieved via. Alternatively you could do it as it is done on the Wikimedia farm:

Without short URLs
If you are not using, restricting robots is a bit harder. If you are running PHP as CGI and you have not beautified URLs, so that articles are accessible through :

If you are running PHP as an Apache module and you have not beautified URLs, so that articles are accessible through :

The lines without the colons at the end restrict those namespaces' talk pages.

Non-English wikis may need to add various translations of the above lines.

You may wish to omit the  restriction, as this will prevent images belonging to the skin from being accessed. Search engines which render preview images, such as Google, will show articles with missing images if they cannot access the  directory.

You can also try

because some robots like Googlebot accept this wildcard extension to the robots.txt standard, which stops most of what we don't want robots sifting through, just like the /w/ solution above. This does however, suffer from the same limitations in that it blocks access to CSS, preventing search engines from correctly rendering preview images. It may be possible to solve this by adding another line  however at the time of writing this is untested.

Allow indexing of raw pages by the Internet Archiver
You may wish to allow the Internet Archiver to index raw pages so that the raw wikitext of pages will be on permanent record. This way, it will be easier, in the event the wiki goes down, for people to put the content on another wiki. You would use:

Rate control
You can only specify what paths a bot is allowed to spider. Even allowing just the plain page area can be a huge burden when two or three pages per second are being requested by one spider over two hundred thousand pages.

Some bots have a custom specification for this; Inktomi responds to a "Crawl-delay" line which can specify the minimum delay in seconds between hits. (Their default is 15 seconds.)

Evil bots
Sometimes a custom-written bot isn't very smart or is outright malicious and doesn't obey robots.txt at all (or obeys the path restrictions but spiders very fast, bogging down the site). It may be necessary to block specific user-agent strings or individual IPs of offenders.

More generally, request throttling can stop such bots without requiring your repeated intervention.

An alternative or complementary strategy is to deploy a spider trap.

Spidering vs. indexing
While robots.txt stops (non-evil) bots from downloading the URL, it does not stop them from indexing it. This means that they might still show up in the results of Google and other search engines, as long as there are external links pointing to them. (What's worse, since the bots do not download such pages, noindex meta tags placed in them will have no effect.) For single wiki pages, the  magic word might be a more reliable option for keeping them out of search results.