Wikiscan

From mediawiki.org
Jump to navigation Jump to search
Manual on MediaWiki Tools
Crystal Clear action run.png
Wikiscan
Release status: beta
Implementation Database
Description Edit statistics for a wiki
Author(s) (Akerontalk)
Latest version 0.9.2 (3-4-2021)
MediaWiki 1.34+
PHP 7.2
License GPL v2
Download
Example Multi site: http://wikiscan.org/, single site: https://stats.wikidex.net/

Wikiscan is a tool written in PHP, external to MediaWiki, that computes and displays historical statistics about a wiki.

Wikiscan was created in 2011 on the initiative of a voluntary user for WMF wikis, available at http://wikiscan.org/. Most of the software development was done on a voluntary basis. Part of it was subsidized by the Wikimedia Foundation and the Wikimedia France association. Server hosting is currently supported by the Wikimedia France association.

The code was originally available on the wikiscan.org website for download. However, it's not available anymore. In 2011, User:Ciencia Al Poder was interested in this tool for another wiki project and downloaded a copy of the code before it was removed, and documented some parts of it and did a general review to fix some issues and make it to work on independent wikis from WMF.

This tool generates statistics by querying the wiki's database, and stored on its own database. Data displayed on the tool is not updated on real time. The statistics data must be generated on a regular basis, using cron or similar.

Download[edit]

Code can be downloaded on GitHub: https://github.com/ciencia/wikiscan/

Requisites[edit]

This tool currently needs to be installed on it's own domain or subdomain. It doesn't support installing on a subdirectory other than the web root.

  • Webserver
  • PHP 5.5 or newer, tested on PHP 7.3
    • PHP modules : mbstring, curl, gd, mysqlnd, memcached
  • MySQL
    • It uses its own database and also needs access to the wiki's database
  • Memcached
    • Used for caching results

Installation[edit]

  1. Extract Wikiscan to a directory (example /var/www/wikiscan/)
  2. Create a new Mysql database (example wikiscan_stats)
  3. Import ctrl/stats.sql to create the tables.
  4. Create or configure a Mysql user with SELECT,INSERT,UPDATE,DELETE rights to the wikiscan database (example: wikiscan_user)
  5. Create or configure a Mysql user with SELECT rights to the wiki's database (it can be the same previously created user)
  6. Go to config/ directory
  7. Copy db_conf.sample.php to db_conf.php
  8. Edit db_conf.php
    1. Configure "db" with your Mediawiki database
    2. Configure "dbs" with the new stats database
  9. Copy local_conf.sample.php to local_conf.php
  10. Edit local_conf.php to configure local paths and wiki urls
  11. Make the directories wpstats/ cache/ and img/* writable

The site should work at this stage but without any statistics yet.

Setting up rewrite rules[edit]

This tool generates some links that expect an existing set of rewrite rules for them to work.

apache[edit]

You can see code in the .htaccess file.

nginx[edit]

With alternate paths in Spanish:

map $request_uri $wikiscan_qs {
        default "";
        "~^/\?" $args;
        "~^/(?:hours|horas)/?$" "menu=live&list=pages";
        "~^/(?:hours|horas)/(\d+)/?$" "menu=live&date=$1&list=pages&sort=weight";
        "~^/(?:hours|horas)/(\d+)/([^&/]+)/?$" "menu=live&date=$1&list=$2&sort=weight";
        "~^/(?:grid|matriz)/?$" "menu=grid";
        "~^/(?:grid|matriz)/(\d+)/?$" "menu=grid&date=$1";
        "~^/(?:calendar|calendario)$" "menu=dates&list=pages";
        "~^/(?:date|fecha)/(\d+)/?$" "menu=dates&date=$1&list=pages&sort=weight";
        "~^/(?:date|fecha)/(\d+)/([^&/]+)/?$" "menu=dates&date=$1&list=$2&sort=weight";
        "~^/(?:users|usuarios)$" "menu=userstats";
        "~^/(?:users|usuarios)\?graphs_details=1$" "menu=userstats&graphs_details=1";
        "~^/(?:user|usuario)/?$" "menu=userstats";
        "~^/(?:user|usuario)/([^&]+)$" "menu=userstats&user=$1&usort=date";
        "~^/(?:user|usuario)/.+$" "menu=userstats&user=_";
        "~^/(?:userlist|listausuarios)/([^&]+)$" "menu=userstats&userlist=$1";
        "~^/ip/?$" "menu=userstats_ip";
        "~^/ip/([^&]+)$" "menu=userstats_ip&user=$1";
        "~^/ip/.+$" "menu=userstats_ip&user=_";
        "~^/pages_vues/?$" "menu=pageview";
        "~^/pages_vues/(\d+)/(.*)$" "menu=pageview&year=$1&ns=$2";
        "~^/pages_vues/(\d+)$" "menu=pageview&year=$1";
        "~^/pages_vues/.+$" "menu=pageview&year=_";
        "~^/ranges/?$" "menu=ranges";
        "~^/range/([^&]+)$" "menu=ranges&range=$1";
        "~^/range/.+$" "menu=ranges&range=_";
        "~^/Derniers_utilisateurs_actifs" "menu=active_users";
}

server {
        listen 80;
        #listen 443 ssl;
        server_name stats.mywikisite.com;
        root /home/www/wikiscan;

        disable_symlinks on;

        access_log /var/log/nginx/wikiscan-access.log;
        error_log /var/log/nginx/wikiscan-error.log;

        location = / {
                try_files $uri @wikiscan;
        }

        location / {
            if ($wikiscan_qs = "") {
                return 404;
            }
            try_files $uri @wikiscan;
        }

        location /img {
                try_files $uri =404;
        }
        location /libs {
                try_files $uri =404;
        }
        location /wikiscan {
                try_files $uri =404;
        }

        location = /gimg.php {
                fastcgi_pass unix:/run/php/php-fpm-wikiscan.sock;
                include /etc/nginx/fastcgi_params;
                fastcgi_param HTTPS "on";
                fastcgi_param SCRIPT_FILENAME $document_root/gimg.php;
                fastcgi_param QUERY_STRING $query_string;
        }

        location @wikiscan {
                fastcgi_pass unix:/run/php/php-fpm-wikiscan.sock;
                include /etc/nginx/fastcgi_params;
                fastcgi_param HTTPS "on";
                fastcgi_param SCRIPT_FILENAME $document_root/index.php;
                fastcgi_param QUERY_STRING $wikiscan_qs;
        }
}

Configuration parameters[edit]

Edit the conf/local_conf.php file, adding or updating keys of the $local_conf variable:

Key Description
root_path Local filesystem path of this wikiscan installation. Example: /home/www/wikiscan
logo URL of a small Wiki logo (80px tall recommended)
icon URL to a favicon image
wikilink_icon URL for a small icon to use as links to the wiki (defaults to WMF logo)
google_analytics Google analytics account (leave empty if you don't want to track users)
memcache_host Memcached host. Example: 127.0.0.1
memcache_port Memcached port. Example: 11211
site_language Language of the wiki. See Supported languages. This may determine some paths of the wikiscan site. See urlpath-menu-* language messages. Example: en.
interface_language Wikiscan's interface language. See Supported languages. Example: en.
menus_enabled List of menus enabled. Leave it to false to not restrict any. Implemented as security measure since there are some menus not fully developed and not reviewed. Example: ['home','live','grid','dates','userstats','userstats_ip']
mw_api API url, used to retrieve the list of namespaces, or members of a category to get users from. Example: https://en.wikipedia.org/w/api.php
link_page URL to article path (page titles will be appended to this URL). Example: https://en.wikipedia.org/wiki/
wiki Local wiki configuration (when not multi site). Array with keys:
  • url: URL to the wiki's homepage
  • site_host: Wiki's name
groups Local/custom group names/abbreviations. Use it if you have defined custom groups or you've localized existing group names.

Example:

'groups' => [
    'rollback'=>[
        'name'=>'Reversor',
        'abbr'=>'R',
    ],
],
view_about_link Display about link. Defaults to true
userlist_enabled Disable search for userlist. Defaults to false. It allows to search for user lists in the users section, by typing a page that contains a list of user links, or a category page that contains user pages.
robots_policy Robots policy (for search engines). Example: noindex,nofollow
hits_available Do we have hits (pageviews) information available?. Defaults to true.
stats_join_revision_actor_temp use the revision_actor_temp table when building stats (need to set it to true for pre-1.35 wikis)
stats_join_revision_comment_temp use the revision_comment_temp table when building stats (need to set it to true for pre-1.35 wikis)
log_sql Log sql queries to the query_log table of dbg. Set it to false to not log queries, specially if you're not using a multi wiki site
live_hours Array with the available hours in the last hours (live) section. Example: [24, 48].

Multi wiki site[edit]

This tool was created originally for a single wiki, and then expanded to serve all WMF wikis under the wikiscan.org domain. In this multi wiki setup, each wiki has its own wikiscan database, and there's an additional database with data to coordinate all wikis. Under this multi-wiki setup, this tool makes heavy use of undocumented and WMF-specific database views, and also it has connections with Wikidata.

This multi wiki behavior is controlled by the 'multi' configuration parameter, that defaults to false.

Supported languages[edit]

Currently supported languages:

  • English (en)
  • Français (fr)
  • Spanish (es)

To add more languages, copy or add a file in include/languages/ with the desired language code, translating from an existing one.

Populating statistics for the first time[edit]

To update all daily stats and user stats, execute :

php ctrl/run.php fullupdate
php ctrl/run.php fullupdate_months

To update last hours, execute :

php ctrl/run.php update_live

To update grid, execute :

php ctrl/run.php update_sum

Update statistics periodically[edit]

To update statistics regularly you can set up cron jobs to run several scripts:

  • ctrl/run.php update_live: Updates section hours and grid. Run it several times a day. It's fast.
  • ctrl/run.php update_sum: Updates calendar and users section. Run it at least once a day.
  • ctrl/run.php update_misc: Updates userstats graphs. Run it at least once a week.