Wp-mirror

The Wikimedia Foundation offers wikipedias in nearly 300 languages. In addition, the WMF has several other projects (e.g. wikibooks, wiktionary, etc.) for a total of around 1000 wikis.

WP-MIRROR is a free utility for building mirrors of any desired set of WMF wikis. That is, WP-MIRROR builds a wiki farm that the user can browse locally.

Why Build a Mirror
The main use cases for a mirror are these:


 * Development. If you are technically minded and need a mirror with which to make experiments;
 * Infrastructure. If you need redundancy, or need to serve pages locally to minimize telecommunications traffic; and
 * Offline browsing. If you need off-line access, perhaps for reasons of mobility, availability, and privacy.

Key Features
WP-MIRROR is easy:


 * Complete. Builds a complete mirror with original size images;
 * Robust. Stable even in the face of: corrupt dump files, corrupt media files, incomplete downloads, Internet access interruptions, and low disk space; and uses check-pointing to resume after process interruption;
 * Easy to install. Available as a DEB package, and is available from a Debian package repository;
 * Easy to configure. The user may select any desired set of wikis by editing just one line in a configuration file;
 * Easy to use. Sets up virtual hosts such as http://simple.wikipedia.site/, http://simple.wiktionary.site/, and http://www.wikidata.site/, one for each wiki in the set, which the user may access with a web browser;

WP-MIRROR automatically configures other software:


 * Apache2. Enables the URL rewrite module, and enables virtual hosts;
 * Cron. Sets up a cron job that updates the mirrors weekly;
 * MediaWiki. Configures MediaWiki 1.23 and several dozen extensions; and
 * MySQL. WP-MIRROR configures MySQL to achieve an order-of-magnitude improvement in database performance.

WP-MIRROR is free:


 * Free software. Software is released under the GNU General Public License (GPLv3); and
 * Free documentation. Documentation is release under the GNU Free Documentation License, version 1.3.

Best of all is the page rendering:


 * Appearance. A wiki page rendered by the mirror looks very similar to the same page rendered by the WMF servers; and
 * Behavior. A wiki page rendered by the mirror behaves almost that same (e.g. edit, search, user account creation, beta features).

Out-of-the-box Experience
WP-MIRROR by default, builds the following set of mirrors:


 * simple wikipedia,
 * simple wiktionary, and
 * wikidata wiki,

where Simple English means shorter sentences, and Wikidata is a centralized collection of facts usable by all other wikis.

The default works out-of-the-box with no user configuration. It should build in 200ks (two days), occupy 100G of disk space, be served locally by virtual hosts: http://simple.wikipedia.site/, http://simple.wiktionary.site/, and http://www.wikidata.site/, and update automatically every week.

The default should be suitable for anyone who learned English as a second language (ESL).

Top Ten Wikipedias
The top ten wikipedias are: en, de, nl, fr, it, ru, es, sv, pl, and ja. Because WP-MIRROR uses original size image files, the top ten are too large to fit on a laptop with a single 500G disk, unless the user does not need the images (and this is configurable). The en wikipedia is the most demanding case. It should build in 1Ms (twelve days), occupy 3T of disk space, be served locally by virtual host http://en.wikipedia.site/, and update automatically every month.

Platforms
WP-MIRROR 0.7 is known to install out-of-the-box on the following platforms:


 * Debian 7.4 (wheezy) with backports. Tested both on a host, and on a virtual machine.

Contact
Author is reachable by e-mail with user name wpmirrordev and domain name gmail dot com.

Other websites

 * Home page of the WP-MIRROR project