Extension:CrawlerProtection
 Release status: stable |
|
|---|---|
| Implementation | Hook |
| Description | Anti-crawler suite for MediaWiki |
| Author(s) | Jeffrey Wang (MyWikis-JeffreyWangtalk) |
| MediaWiki | >= 1.35 |
| License | MIT License |
| Download | Download master snapshot Note: |
The CrawlerProtection extension blocks anonymous users from performing actions and accessing special pages which are most frequently abused by AI crawler bots. The aim of this extension is to stop AI crawler bots from accessing pages which are expensive to render, thereby stymying excessive resource drain.
CrawlerProtection was introduced at MediaWiki Users and Developers Workshop Spring 2025. The video recording of the introduction can be found on YouTube.
Installation
[edit]- Download, extract and place the file(s) in a directory called
CrawlerProtectionin yourextensions/folder. - Add the following code at the bottom of your LocalSettings.php file:
wfLoadExtension( 'CrawlerProtection' );
Done – Navigate to Special:Version on your wiki to verify that the extension is successfully installed.
Configuration parameters
[edit]This extension doesn’t require any configuration. At the moment, the extension is plug-and-play, but also all-or-nothing.
There are a couple of system messages (e.g. crawlerprotection-accessdenied-text) which will be shown when access is denied[1]. You can override these system messages as desired on-wiki.
Handled features
[edit]The following wiki features are disabled for anonymous users by CrawlerProtection:
- Page diffs
?type=revision?action=history?diff=1234?oldid=1234
- Special:MobileDiff
- Special:RecentChangesLinked
- Special:WhatLinksHere
Frequently asked questions
[edit]- Does this extension harm SEO or otherwise prevent Google from crawling my wiki?
- This extension in no way prevents traditional legitimate robots.txt-respecting crawlers (such as those from Google) from accessing content pages of wikis with CrawlerProtection installed.
- Does hiding the history and diff pages from users violate the Creative Commons license?
- There is no requirement in any Creative Commons license to also show the history of a page. Therefore, this extension wouldn't violate the terms of any Creative Commons license. (This is not legal advice.)
Future plans
[edit]CrawlerProtection will continue to have features added to it with the community's support.
MediaWiki 1.43 support was added by Skizzerz in June 2025.