Content translation/Deployments/Deployment Plan

CX Deployment Plan for January 2015

 * Project: Content Translation


 * Deployment date: January 13, 2015


 * Initial Release: Released in Beta Labs environment (July 2014)


 * Past Releases: See: Archives


 * What is targeted for January 2015 release:
 * Content Translation 0.03 release: https://www.mediawiki.org/wiki/Content_translation/Roadmap#Ongoing_Release:_Content_Translation_0.03_release


 * Long-term project roadmap:
 * https://www.mediawiki.org/wiki/Content_translation/Roadmap


 * Language Pairs to be supported:


 * Release as: Beta Feature

Overall Plan
This release is targeted to Production with 0.03 feature set.

System Architecture
See:
 * https://www.mediawiki.org/wiki/Content_translation/Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation#Workflow_and_Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation

Caching Architecture
The following diagram includes the caching requirements for the CX framework:


 * https://www.mediawiki.org/wiki/Content_translation/Server_communications_workflow
 * https://commons.wikimedia.org/wiki/File:CX_ArchitectureV1.svg

cxserver
Content Translation service is puppetize and available in operations/puppet repository as "cxserver" module.

Primary setup instructions are at: https://git.wikimedia.org/markdown/mediawiki%2Fservices%2Fcxserver.git/HEAD/README.md

For detailed information about component, installation and configuation and instructions: https://www.mediawiki.org/wiki/Content_translation/Setup

Apertium
Apertium service is puppetize and available in operations/puppet repository as "apertium" module.

Apertium service installation and configuration: https://www.mediawiki.org/wiki/Content_translation/Apertium/Service

Dictd
No additional configuration needed for the planned languages. In future, when we enable more languages, dictd based dictionaries or external dictionaries may get introduced- but not planned right now. See: https://www.mediawiki.org/wiki/Content_translation/Dictionaries for details.

Backend Services

 * Varnish (See: https://www.mediawiki.org/wiki/Content_translation/Setup#Backend_Services)

Extension dependencies

 * BetaFeatures
 * CLDR
 * EventLogging
 * GuidedTour

External APIs called by CX

 * Wikidata
 * Parsoid API

Configuration Scripts

 * cxserver: init scripts are at, https://www.mediawiki.org/wiki/Content_translation/Setup
 * Apertium: init scripts are part of the "apertium-apy" package as suggested by Ops.

Provisioning Plan
a. Storage and Hardware Requirements
 * Service cluster A.


 * cxserver service hardware is ready. For ContentTranslation hardware measurement, we monitored Beta cluster at,
 * https://tools.wmflabs.org/nagf/?project=deployment-prep#h_deployment-cxserver03


 * Apertium service hardware is ready. For Apertium hardware measurement, we monitored Beta cluster at,
 * https://tools.wmflabs.org/nagf/?project=deployment-prep#h_deployment-apertium01

b. Bandwidth Requirements
 * Service cluster A.


 * Tech Ops has indicated that cxserver/apertium instances will go into "Service cluster A" (ie http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=Service+Cluster+A+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_report&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name )

d. Performance expectations
 * See, https://www.mediawiki.org/wiki/Content_translation/Performance

Monitoring and metrics

 * EventLogging activity for Content Translation.
 * Number of users enabling the feature on Beta (See: http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&list=betafeatures&bfcounts)
 * Performance of S:CX, backend calls?
 * Check for node and varnish.
 * Graph showing requests or timings for the WikiData API(s) we are calling.
 * Graph showing requests or timings for the Parsoid API(s) we are calling.

LE Team responsibilities

 * Kartik - Deployment, Engineer
 * Niklas - Engineer, Code Reviewer
 * Santhosh - Engineer, Code Reviewer
 * David - Engineer, Code Reviewer
 * Joel - Engineer, Code Reviewer
 * Runa - Team Scrum-Master / testing and communications
 * Pau - Feature UX reviewer, designer
 * Amir - Feature signoff

Current Status
CX Deployment Plan for January 2015

Done

 * Add patches for cxserver/apertium services in Production puppet (operations/puppet). ✅
 * Started: 2014-12-16
 * Ticket: https://phabricator.wikimedia.org/T78629
 * Status:
 * https://gerrit.wikimedia.org/r/180125 (cxserver) ✅
 * https://gerrit.wikimedia.org/r/180130 (apertium) ✅


 * Hardware Availability for CX and other services.
 * Started: 2014-11-10
 * cxserver hardware ✅
 * Ready as per Ops/Alex. Planned to host on Service Cluster A.
 * Apertium hardware from Ops ✅
 * Apertium instance is ready to go with Service Cluster A as per Alex/Kartik's estimation of CPU/RAM monitoring for Beta.


 * Content Translation Database/Tables setup with globally accessible database on Beta. ✅
 * Started: 2014-11-07
 * Ticket: https://phabricator.wikimedia.org/T1254
 * Status:
 * CX Tables need cx_ prefix: https://gerrit.wikimedia.org/r/175662 ✅
 * CX config changes: https://gerrit.wikimedia.org/r/175955 ✅
 * Import of cx tables into wikishared DB in Beta ✅
 * mediawiki-config need to adjust with using 'extension1' cluster DB: https://gerrit.wikimedia.org/r/175979 ✅

In Progress

 * Database config/deployment for Content Translation on Production.
 * Ticket: https://phabricator.wikimedia.org/T78775


 * Varnish setup on Beta.
 * Started: 2014-11-19
 * Ticket: https://phabricator.wikimedia.org/T76200
 * Status:
 * deployment-sca-cache01.eqiad.wmflabs instance has been created (2014-11-25) ✅
 * Alex need to setup Virtual IP/instance to finalize VCL rules for Varnish on Beta.

To be started

 * Enable Content Translation on Production (mediawiki-config)
 * Security review of Content Translation "Beta Feature"