Content translation/Deployments/Deployment Plan

CX Deployment Plan for January 2015

 * Deployment date: January 13, 2015


 * Project: Content Translation Framework


 * Initial Release: Beta Feature release July 2014 (Done)


 * What is targeted for January 2015 release:


 * Content Translation 0.03 release: https://www.mediawiki.org/wiki/Content_translation/Roadmap#Ongoing_Release:_Content_Translation_0.03_release


 * Long-term project roadmap:
 * https://www.mediawiki.org/wiki/Content_translation/Roadmap


 * Language Pairs to be supported:


 * Release as: Beta Feature

Overall Plan
This release is targeted to Production with 0.03 feature set.

System Architecture
See:
 * https://www.mediawiki.org/wiki/Content_translation/Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation#Workflow_and_Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation

Caching Architecture
The following diagram includes the caching requirements for the CX framework:


 * https://www.mediawiki.org/wiki/Content_translation/Server_communications_workflow
 * https://commons.wikimedia.org/wiki/File:CX_ArchitectureV1.svg

Components to be provisioned for production

 * cxserver:
 * Content Translation service is puppetize and available in operations/puppet repository as cxserver module.


 * cxserver installation and configuration:
 * Primary setup instructions are at: https://git.wikimedia.org/markdown/mediawiki%2Fservices%2Fcxserver.git/HEAD/README.md
 * For detailed information about component, installation and configuation and instructions: https://www.mediawiki.org/wiki/Content_translation/Setup


 * Apertium:
 * Apertium service is puppetize and available in operations/puppet repository as apertium module.


 * Apertium service installation and configuration:
 * https://www.mediawiki.org/wiki/Content_translation/Apertium/Service


 * Node.js


 * Dictd:
 * See: https://www.mediawiki.org/wiki/Content_translation/Dictionaries for details. This doesn't require separate setup and it is part of cxserver dependencies.


 * Backend Services:
 * Varnish
 * See: https://www.mediawiki.org/wiki/Content_translation/Setup#Backend_Services


 * Extension dependencies:
 * BetaFeatures
 * CLDR
 * EventLogging


 * External APIs called by CX:
 * Wikidata
 * Parsoid API


 * Configuration Scripts:
 * cxserver: init scripts are at, https://www.mediawiki.org/wiki/Content_translation/Setup
 * Apertium: init scripts are part of the "apertium-apy" package as suggested by Ops.

Provisioning Plan
a. Storage Requirements
 * In discussion with Ops.

b. Hardware Requirements
 * In discussion with Ops.


 * cxserver hardware is ready by Ops. For ContentTranslation hardware, we monitored Beta at,
 * https://tools.wmflabs.org/nagf/?project=deployment-prep#h_deployment-cxserver03


 * Apertium service hardware need to be ready by Ops. For Apertium hardware, we monitored Beta at,
 * https://tools.wmflabs.org/nagf/?project=deployment-prep#h_deployment-apertium01

c. Bandwidth Requirements
 * In discussion with Ops.


 * Alex suggested that cxserver/apertium instances will likely to go into "Service cluster A" (ie http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=Service+Cluster+A+eqiad&h=&tab=m&vn=&hide-hf=false&m=cpu_report&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name )

d. Performance expectations
 * See: https://www.mediawiki.org/wiki/Content_translation/Performance

Monitoring and metrics

 * EventLogging activity for CX
 * Number of users enabling the feature
 * Performance of S:CX, backend calls?
 * Check for node and varnish? Who to page?
 * Graph showing requests or timings for the WikiData API(s) we are calling
 * Graph showing requests or timings for the Parsoid API(s) we are calling

External Signoffs Required

 * Alex, Mark - Ops
 * Gabriel - Infrastructure architecture
 * Ori - Performance
 * Chris Steipp - Security
 * Greg G - Release engineering
 * Mark - Ops
 * Tim - Platform

LE Team responsibilities

 * Kartik - Deployment, Engineer
 * Niklas - Engineer, Code Reviewer
 * Santhosh - Engineer, Code Reviewer
 * David - Engineer, Code Reviewer
 * Joel - Engineer, Code Reviewer
 * Runa - Team Scrum-Ninja / testing and communications
 * Pau - Feature UX reviewer, designer
 * Amir - Feature signoff

Updates
Current progress, blockers and to-dos

CX Deployment Plan for November 2014

 * Deployment date: November 30, 2014


 * Project: Content Translation Framework


 * Initial Release: Beta Feature release July 2014 (Done)


 * What is targeted for November 2014 release:


 * Content Translation 0.02 release: https://www.mediawiki.org/wiki/Content_translation/Roadmap#Content_Translation_0.02_release


 * Long-term project roadmap:
 * https://www.mediawiki.org/wiki/Content_translation/Roadmap


 * Language Pairs to be supported: Spanish-Catalan, Catalan-Spanish, Spanish-Portuguese, Portuguese-Spanish, Catalan-Portuguese, Portuguese-Catalan.


 * Release as: Beta Feature

Overall Plan
This release includes adding support for Apertium as service (See: https://www.mediawiki.org/wiki/Content_translation/Apertium/Service) and Spanish-Catalan, Spanish-Portuguese, Portuguese-Spanish, Catalan-Portuguese, Portuguese-Catalan language pairs to support article content translation.

System Architecture
See:
 * https://www.mediawiki.org/wiki/Content_translation/Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation#Workflow_and_Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation

Caching Architecture
The following diagram includes the caching requirements for the CX framework:


 * https://www.mediawiki.org/wiki/Content_translation/Server_communications_workflow
 * https://commons.wikimedia.org/wiki/File:CX_ArchitectureV1.svg

Components to be provisioned for production
CX server installation and configuration: https://git.wikimedia.org/markdown/mediawiki%2Fservices%2Fcxserver.git/HEAD/README.md

See Setup: https://www.mediawiki.org/wiki/Content_translation/Setup for detailed information about component, installation and configuation and instructions.


 * Node.js


 * Dictd server (Also see: https://www.mediawiki.org/wiki/Content_translation/Dictionaries)


 * Apertium (See: https://www.mediawiki.org/wiki/Content_translation/Apertium/Service for detailed changes we need in this release)


 * Extension dependencies:
 * BetaFeatures
 * CLDR
 * EventLogging


 * Backend Services:
 * Varnish


 * External APIs called by CX:
 * Wikidata
 * Parsoid API


 * Configuration Scripts:
 * Upstart and Systemd init scripts are at: https://www.mediawiki.org/wiki/Content_translation/Setup
 * Apertium init scripts are inside apertium-apy package as suggested by Ops.

Provisioning Plan
a. Storage Requirements To be determined from discussion with ops

b. Hardware Requirements To be determined from discussion with ops

c. Bandwidth Requirements To be determined from discussion with ops

d. Performance expectations
 * MT TPS (Transactions per second)
 * User responsiveness
 * MT Round trip
 * General guidelines
 * https://www.mediawiki.org/wiki/Performance_guidelines
 * https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code

Monitoring and metrics

 * EventLogging activity for CX
 * Number of users enabling the feature
 * Performance of S:CX, backend calls?
 * Check for node and varnish? Who to page?
 * Graph showing requests or timings for the WikiData API(s) we are calling
 * Graph showing requests or timings for the Parsoid API(s) we are calling

External Signoffs Required

 * Alex, Mark - Ops
 * Gabriel - Infrastructure architecture
 * Ori - Performance
 * Chris Steipp - Security
 * Greg G - Release engineering
 * Mark - Ops
 * Tim - Platform

LE Team responsibilities

 * Kartik - Deployment, Engineer
 * Niklas - Engineer, Code Reviewer
 * Santhosh - Engineer, Code Reviewer
 * David - Engineer, Code Reviewer
 * Joel - Engineer, Code Reviewer
 * Runa - Team Scrum-Ninja / testing and communications
 * Pau - Feature UX reviewer, designer
 * Amir - Feature signoff

Updates
Current progress, blockers and to-dos

CX Deployment Plan for September 2014

 * Deployment date: Sep 30, 2014


 * Project: Content Translation Framework


 * Initial Release: Beta Feature release July 2014 (Done)


 * What is targeted for September 2014 release:


 * Content Translation 0.01 release: https://www.mediawiki.org/wiki/Content_translation/Roadmap#Content_Translation_0.01_release


 * Long-term project roadmap:
 * https://www.mediawiki.org/wiki/Content_translation/Roadmap


 * Language Pairs to be supported: Spanish-Catalan, Catalan-Spanish


 * Release as: Beta Feature

Overall Plan
This release includes adding support for Apertium as service (See: https://www.mediawiki.org/wiki/Content_translation/Apertium/Service) and Catalan-Spanish language pair to support article content translation.

System Architecture
See:
 * https://www.mediawiki.org/wiki/Content_translation/Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation#Workflow_and_Technical_Architecture
 * https://www.mediawiki.org/wiki/Content_translation

Caching Architecture
The following diagram includes the caching requirements for the CX framework:


 * https://www.mediawiki.org/wiki/Content_translation/Server_communications_workflow
 * https://commons.wikimedia.org/wiki/File:CX_ArchitectureV1.svg

Components to be provisioned for production
CX server installation and configuration: https://git.wikimedia.org/markdown/mediawiki%2Fservices%2Fcxserver.git/HEAD/README.md

See Setup: https://www.mediawiki.org/wiki/Content_translation/Setup for detailed information about component, installation and configuation and instructions.


 * Node.js


 * Dictd server (Also see: https://www.mediawiki.org/wiki/Content_translation/Dictionaries)


 * Apertium (See: https://www.mediawiki.org/wiki/Content_translation/Apertium/Service for detailed changes we need in this release)


 * Extension dependencies:
 * BetaFeatures
 * CLDR
 * EventLogging


 * Backend Services:
 * Varnish


 * External APIs called by CX:
 * Wikidata
 * Parsoid API


 * Configuration Scripts:
 * Upstart and Systemd scripts are at: https://www.mediawiki.org/wiki/Content_translation/Setup

Provisioning Plan
a. Storage Requirements To be determined from discussion with ops

b. Hardware Requirements To be determined from discussion with ops

c. Bandwidth Requirements To be determined from discussion with ops

d. Performance expectations
 * MT TPS (Transactions per second)
 * User responsiveness
 * MT Round trip
 * General guidelines
 * https://www.mediawiki.org/wiki/Performance_guidelines
 * https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code

Monitoring and metrics

 * EventLogging activity for CX
 * Number of users enabling the feature
 * Performance of S:CX, backend calls?
 * Check for node and varnish? Who to page?
 * Graph showing requests or timings for the WikiData API(s) we are calling
 * Graph showing requests or timings for the Parsoid API(s) we are calling

External Signoffs Required

 * Alex, Mark - Ops
 * Gabriel - Infrastructure architecture
 * Ori - Performance
 * Chris Steipp - Security
 * Greg G - Release engineering
 * Mark - Ops
 * Tim - Platform

LE Team responsibilities

 * Kartik - Deployment, Engineer
 * Niklas - Engineer, Code Reviewer
 * Santhosh - Engineer, Code Reviewer
 * David - Engineer, Code Reviewer
 * Joel - Engineer, Code Reviewer
 * Runa - Team Scrum-Ninja / testing and communications
 * Pau - Feature UX reviewer, designer
 * Amir - Feature signoff
 * Alolita - Engineering coordination, Eng Manager

Updates
Current progress, blockers and to-dos

CX Deployment Plan for Beta Feature July 2014
Deployment date: TBD (July 1-15 2014)

Project: Content Translation Framework

Initial Release: Beta Feature release July 2014

What is targeted for Beta Feature release July 2014

Minimum Viable Product (MVP) for v1.0: https://www.mediawiki.org/wiki/Content_translation/Roadmap#Content_Translation_Minimum_Viable_Product_.28MVP.29_release

Long-term project roadmap: https://www.mediawiki.org/wiki/Content_translation/Roadmap

Language Pair to be supported for MVP: Spanish - Catalan

Release as: Beta Feature

Overall Plan
We plan to have deploy the CX framework for only Spanish and Catalan language wikis to support article content translation for Spanish to Catalan where there are red links for parallel articles in Catalan.

System Architecture
See: https://www.mediawiki.org/wiki/Content_translation/Technical_Architecture

https://www.mediawiki.org/wiki/Content_translation#Workflow_and_Technical_Architecture

https://www.mediawiki.org/wiki/Content_translation

Caching Architecture
The following diagram includes the caching requirements for the CX framework.

https://www.mediawiki.org/wiki/Content_translation/Server_communications_workflow

https://commons.wikimedia.org/wiki/File:CX_ArchitectureV1.svg

Components to be provisioned for production
CX server installation and configuration: https://git.wikimedia.org/markdown/mediawiki%2Fservices%2Fcxserver.git/HEAD/README.md

See Setup: https://www.mediawiki.org/wiki/Content_translation/Setup for detailed information about component, installation and configuation and instructions.


 * Node.js


 * Dictd server (Also see: https://www.mediawiki.org/wiki/Content_translation/Dictionaries)


 * Apertium


 * Extension dependencies:
 * BetaFeatures
 * CLDR
 * EventLogging

Varnish:
 * Backend Services


 * External APIs called by CX
 * Wikidata
 * Parsoid API

Upstart and Systemd scripts are at: https://www.mediawiki.org/wiki/Content_translation/Setup
 * Configuration Scripts

Provisioning Plan
a. Storage Requirements To be determined from discussion with ops

b. Hardware Requirements To be determined from discussion with ops

c. Bandwidth Requirements To be determined from discussion with ops

d. Performance expectations https://www.mediawiki.org/wiki/Performance_guidelines
 * MT TPS (Transactions per second)
 * User responsiveness
 * MT Round trip
 * General guidelines

https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code

Monitoring and metrics

 * EventLogging activity for CX
 * Number of users enabling the feature
 * Performance of S:CX, backend calls?
 * Check for node and varnish? Who to page?
 * Graph showing requests or timings for the WikiData API(s) we are calling
 * Graph showing requests or timings for the Parsoid API(s) we are calling

External Signoffs Required

 * Faidon - Ops
 * Gabriel - Infrastructure architecture
 * Ori - Performance
 * Chris Steipp - Security
 * Greg G - Release engineering
 * Mark - Ops
 * Tim - Platform

LE Team responsibilities

 * Kartik - Deployment, Engineer
 * Niklas - Engineer, Code Reviewer
 * Santhosh - Engineer, Code Reviewer
 * David - Engineer, Code Reviewer
 * Runa - Team Scrum-Ninja / testing and communications
 * Pau - Feature UX reviewer, designer
 * Amir - Feature signoff
 * Alolita - Engineering coordination, Eng Manager

Updates

 * cxserver deployment repository: https://gerrit.wikimedia.org/r/mediawiki/services/cxserver/deploy ✅
 * Create cxserver instance on beta labs
 * Puppetize cxserver: https://gerrit.wikimedia.org/r/#/c/139095/ ✅