Extension:WikiSync

Purpose
This extension allows an AJAX-based synchronization of revisions and files between global wiki site and its local mirror. Files download can optionally be disabled to speed up the synchronization. The process is partially automated, and progress indicator is displayed during time-consuming operations. Both global site and local mirror can be used either as source or destination for synchronization, although currently synchronization of files is implemented only from global site to local mirror (synchronization of revisions works in both sides).

Installation
The extension should be installed and properly configured both at local mirror and global site, because there are parts of extension required to run at both sides. require_once( "$IP/extensions/WikiSync/WikiSync.php" );
 * Download the latest available version and extract it to your wiki /extensions directory.
 * Place the following line to the "extensions section" of LocalSettings.php
 * Check out Special:Version page to verify the installation (WikiSync should be listed in the list of installed extensions)

Optionally, you may insert the following lines after the line above
 * To change default remote wiki root address:

WikiSyncSetup::$remote_wiki_root="http://www.mywikisite.org";
 * To change default remote user name:

WikiSyncSetup::$remote_wiki_user="User name";
 * To change default HTTP proxy address (complete URL, as in UNIX HTTP shell variable):

WikiSyncSetup::$proxy_address="http://user:pass@10.0.0.78:3128";
 * Since v0.2.1, separate lists of user groups which have access to synchronization are available:
 * To allow registered users to use synchronization from remote to local wiki (rtl). Please note that administrators belongs to this group as well:

WikiSyncSetup::$rtl_access_groups = array( 'user' );
 * To allow only sysops and bureaucrats to use synchronization from local to remote wiki (ltr). Ordinary users will be denied:

WikiSyncSetup::$ltr_access_groups = array( 'sysop', 'bureaucrat' );

Note, that changing default remote user name may be unneeded since v0.2.1, because it automatically fills the field with current local user name.

Usage
Open an address of local mirror in the web browser. Login to local mirror using user account with administrative rights, then go to Special:WikiSync page.

Special:WikiSync
Special:WikiSync special page provides a user interface for site synchronization. The page is available to users with the administrative rights only.

The page screen is divided to the following parts:
 * Top part shows current local mirror's wiki root address and remote (global) wiki root address. Wiki root address is a path to api.php on the site. To make sure the root is properly choosen, one may enter a wiki root URL into address line of web browser, then append /api.php to it. For example, http://www.mediawiki.org/w wiki root corresponds to http://www.mediawiki.org/w/api.php path. The later link should display valid MediaWiki API documentation page. Sites which use default setup (no "pretty URLs", usually do not have /w in their root address. The button in the middle allows to choose the direction of synchronization.
 * Left-middle part displays remote operations log. All of the remote API calls and local client calls will be displayed here, both successful and erroneous. Source calls, destination calls, errors are displayed in different colors for convenience. It is possible to clear log (for example, before next synchronization) by pressing button "Clear log".
 * Right-middle part contains form, used to login to remote (global) wiki site. Remote wiki root and user name fields are filled with default values taken from WikiSyncSetup variables (see Installation). Type a password and press "Log in" to login to remote site. In case remote login was successful, the button "Synchronize" will be enabled. Optionally, "Synchronize files" checkbox may be switched off, to speed-up global to local synchronization (skipping the transfer of the files). Currently, transferring of files from local to remote wiki is not implemented yet, so in such case switching off the checkbox might be required, if there were new files uploaded to repository of local version since last synchronization. Since v0.3.1 it is possible to store remote wiki password in cookie, although that is insecure and not recommended (although, one may use system level to restrict the access).
 * Since v0.3.1, just below remote operations log there is scheduler log, which shows the direction and start time of every synchronization and end time of the last successful synchronization. Below login form there is scheduler settings form, which allows to choose time interval between synchronizations in minutes and optionally automatically switch direction of synchronization. These settings are stored in cookies as well.
 * Bottom part is hidden until the synchronization begins.

Looking for site changes (differences)
When the synchronization begins, the first stage is to find new revisions at destination site. Progress bar indicator at the bottom of the page displays current status. At the end, when there are some changes at destination site found, the confirmation box is displayed. Press "Ok" to start synchronization, "Cancel" to abandon it. Bottom part of the page screen also displays first revision of the source page, which was not found at destination site.

Transferring of new revisions and files
At the second stage, revisions (and optionally files) are transferred from source to destination wiki. There are two progress bars, one displays revisions processing, second displays files being processed. Revisions are transferred in XML dump chunks, files in raw byte chunks.

Please note that only the newest versions of files are transferred. That is because files transfer is already a very slow during large changes between local and remote wikis. Revisions are transferred with their (changed) history.

Technical notes

 * Please note that the extension is not designed to completely synchronize every change of arbitrary wiki site to another wiki site. It should be used only for mirrors, where only the last revisions are changed, and there is no difference in the earlier revisions (no sparse differences).
 * API extensions are used to find similar revisions of source wiki at destination wiki, also to transfer files in chunks (otherwise, large file transfer can easily time out Apache mod_php process).
 * Local mirror can be located in non-public mirror behind proxy / firewall, while global site is generally considered to be a internet site, but the actual difference is:
 * Local mirror may be limited (although not required) to have only outbound http connection to global site, but no inbound http connection from global site;

Localization
Additional languages can be added into WikiSync_i18n.php file. If you have translated the messages, please mail me, so I'll update the archive, until an extension will be added to http://translatewiki.net/

Download
To extract tgz archive in Windows, free 7zip program can be used.

Credits
Thanks to Roan Kattouw for helping me to port client-side scripts to use ResourceLoader.

Troubleshooting

 * Please use not too old (IE8+, FF3+) browser for client interface and client scripts, as it's not been tested with old browsers.
 * HTTP proxies are supported, but discouraged to use, because in many cases they are caching AJAX requests, which is undesired behavior.
 * In case there are any problems, make sure you've followed every installation step and MediaWiki version is compatible with the extension. Please report the bugs at the talk page. Also you may try using previous version (when available).
 * make sure permissions are correct e.g. for curl relevant files like ssl certificate files
 * make sure the curl_path in the source code is set correctly when using https
 * with 0.3.1 make sure you add the compatibility changes as outlined in Extension_talk:WikiSync manually

Todo

 * Implement local to remote file upload.
 * Use Manual:$wgCrossSiteAJAXdomains when the compatibility to 1.15 will be dropped.
 * add compatibilty changes for 1.17 to 1.19 as outlined in Extension_talk:WikiSync
 * make compatible with 1.21 up
 * exchange Snoopy calls by direct curl access