Content translation/cxserver/Setup

From MediaWiki.org
Jump to: navigation, search

Installation[edit]

Get the code[edit]

Content Translation Server[edit]

If you want to do an anonymous checkout:

git clone https://gerrit.wikimedia.org/r/p/mediawiki/services/cxserver

Or if you plan to hack CXServer, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

git clone ssh://<user>@gerrit.wikimedia.org:29418/mediawiki/services/cxserver

Nodejs[edit]

Install nodejs 0.10 or higher. When you are using Ubuntu and depending on OS version you will not end up with the most recent version of nodejs please read this [1] first before you proceed:

sudo apt-get install nodejs npm
nodejs --version # should now print v0.10.x or higher

On Ubuntu Server 12.04 LTS instead do (to get v0.10.x)[1]:

sudo apt-get update
sudo apt-get install -y python-software-properties python g++ make
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs=0.10*
nodejs --version # should now print v0.10.x

On CentOS 7 instead do (for nodejs 6.x version):

curl -sL https://rpm.nodesource.com/setup_6.x | bash -  #Follow the instructions
node --version # should now print v6.x.x

For other systems, see:

JS dependencies[edit]

Install the JS dependencies. Run this command in the cxserver directory:

npm install

Dictd Server[edit]

Installation[edit]

sudo apt-get install dictd

To install language pair,

sudo apt-get install dict-freedict-xxx-yyy

Where xxx and yyy are two different languages.

On CentOS 7 instead do:

wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/Kenzy:/packages/CentOS_7/x86_64/dictd-server-1.12.0-6.4.x86_64.rpm
rpm -Uvh dictd-server-1.12.0-6.4.x86_64.rpm

RESTBase Server[edit]

Installation[edit]

git clone https://github.com/wikimedia/restbase.git

From the restbase project directory, install the Node dependencies:

npm install

Configuration[edit]

cp config.example.yaml config.yaml

Edit the config.yaml file in these parts:

...
allow:
 - pattern: http://YOUR_WIKI_API_ENDPOINT
 ...
 - pattern: http://YOUR_PARSOID_SERVER:PORT
 ...
 paths:
   /{domain:YOUR_WIKI_WEBSITE}:
 ...
 action:
   # XXX Check API URL!
   apiUriTemplate: http://YOUR_WIKI_API_ENDPOINT
 ...
 parsoid:
   # XXX Check Parsoid URL! localhost is the default configuration
   host: http://YOUR_PARSOID_SERVER:PORT
 ...

YOUR_WIKI_API_ENDPOINT is the location of your wiki's api.php. For example, MediaWiki's API endpoint is https://mediawiki.org/w/api.php. See The endpoint.

YOUR_WIKI_WEBSITE is the base URL of your wiki. For example, MediaWiki's base URL is https://mediawiki.org/.

YOUR_PARSOID_SERVER:PORT in the host: and pattern: entries must correspond to the URL and port of your Parsoid server. If your Parsoid and RESTBase servers are on the same machine, then localhost is a valid URL.

Run the server and Test[edit]

To run the RESTBase server use the same procedure to run the cxserver.

For testing execute:

curl http://YOUR_RESTBASE_SERVER:7231/YOUR_WIKI_WEBSITE/v1/page/html/Main_Page

You should see the HTML content of the Main_Page.

In a browser you can also test by navigating to

http://YOUR_RESTBASE_SERVER:7231/YOUR_WIKI_WEBSITE/v1/

and you should see a page titled Wikimedia REST API. On this page you can test all manner of RESTBase methods and variables.

For other testing refer to github.

Starting RESTBase Server automatically[edit]

For installing the RESTBase server as a service with systemd, create this file in the /lib/systemd/system/ directory (/etc/systemd/system/ for CentOS 7 and /usr/lib/systemd/system/ for openSUSE) and name it restbase.service.

[Unit]
Description=Mediawiki RESTBase Service
Documentation=https://www.mediawiki.org/wiki/RESTBase
Wants=local-fs.target network.target
After=local-fs.target network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/path_to_restbase-master
ExecStart=/usr/bin/node /path_to_restbase-master/server.js
KillMode=process
Restart=on-success
PrivateTmp=true
StandardOutput=syslog

Replace path_to_restbase-master with the path to your restbase project directory.

To automatically start the RESTBase server at system startup,

systemctl enable restbase.service

To control the RESTBase server,

sudo systemctl start|stop|restart|status restbase.service

If you ever change the restbase.service script, you need to reload it with,

sudo systemctl --system daemon-reload

Server Configuration[edit]

An example configuration file is given as config.dev.yaml and config.prod.yaml.

RESTBase configuration section[edit]

...
    restbase_req:
        method: '{{request.method}}'
        uri: http://localhost:7231/YOUR_WIKI_WEBSITE/v1/{+path}
        query: '{{ default(request.query, {}) }}'
        headers: '{{request.headers}}'
        body: '{{request.body}}'
...

Apertium configuration section[edit]

...
    apertium:
          api: http://localhost:2737
...

The server expected on localhost:2737 is then Apertium-apy . (See http://wiki.apertium.org/wiki/Apy for instructions; you'll have to install Apertium and at least one language pair too). Adjust configuration according to your requirement and restart the server.

Run the server[edit]

Using node[edit]

$ nodejs server.js

On CentOS 7:

$ node server.js

Using npm[edit]

$ npm start

As a cxserver command[edit]

Following step is to be performed only once. You might have to use 'sudo'.

$ npm link .

Subsequently, just use following to start the server.

$ cxserver

Then browse to http://localhost:8080/v1 You'll see the server playground page.

Running it in secure mode (https)[edit]

In the configuration file, provide the SSL certificate details and set value of 'secure' as true

Debugging[edit]

To run the ContentTranslation server:

$ npm run-script debug

It will open Chrome developer tools with the ContentTranslation source code. You can debug the code just like a web application. You can also edit the code and save from the debugger.

Starting cxserver automatically[edit]

There are many ways to start services automatically, consult your server's operating system documentation.

upstart[edit]

On Ubuntu and other operating systems using Upstart, one approach is

sudo ln -s /lib/init/upstart-job /etc/init.d/cxserver
sudo vi /etc/init/cxserver.conf

And, use following upstart script template:

# Upstart job configuration for cxserver

description "cxserver service"

start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [!2345]

setuid "www-data"
setgid "www-data"

env NODE_PATH="/path/to/cxserver/node_modules"

chdir "/path/to/cxserver"
exec npm start

respawn

To start cxserver,

sudo service cxserver start

To stop cxserver,

sudo service cxserver stop

Debugging[edit]

Upstart can be debug using log messages at /var/log/syslog.

systemd[edit]

On recent versions of Debian, Fedora and other operating systems using systemd, use a cxserver.service unit file similar to the following template (modify the file paths as appropriate) and place file at /lib/systemd/system/ directory.

[Unit]
Description=Mediawiki Content Translation service
Documentation=https://www.mediawiki.org/wiki/ContentTranslation
Wants=local-fs.target network.target
After=local-fs.target network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/path/to/cxserver/node_modules
ExecStart=/usr/bin/node /path/to/cxserver/server.js
KillMode=process
Restart=on-success
PrivateTmp=true
StandardOutput=syslog

To start cxserver,

sudo service cxserver start

To stop cxserver,

sudo service cxserver stop

If you change in service script, you need to reload script by,

sudo systemctl --system daemon-reload

Debugging[edit]

Systemd scripts logs messages at /var/log/syslog. It can be customized using StandardOutput= in cxserver.service script.

Backend Services[edit]

Varnish[edit]

Installation[edit]

sudo apt-get install varnish

Configuration[edit]

1. Configure varnish http accelerator port:

edit /etc/default/varnish

Example configuration to run varnish on port 8000

DAEMON_OPTS="-a :8000 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

2. Configure the varnish default backend:

edit /etc/varnish/default.vcl

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

3. Restart the varnish:

sudo service varnish restart

Make sure you are running cxserver nodejs server on port 8080 - the backend configuration of varnish

cxserver --port=8080

127.0.0.1:8000 will be your server URL with varnish 127.0.0.1:8080 will be your server URL without varnish

4. Configure cxclient's wgContentTranslationServerURL to 127.0.0.1:8000

Extensions dependencies[edit]

Content translation extension requires following extensions as dependencies.

  1. BetaFeatures[2]
  2. CLDR[3]
  3. EventLogging[4]
  4. UniversalLanguageSelector[5]

References[edit]