Content translation/Deployment Plan/Current Status

This document will list latest updates from Content Translation deployment process at the end of the day.


 * Beta deployment notes: http://etherpad.wikimedia.org/p/cx-deployment-notes
 * Production deployment notes: http://etherpad.wikimedia.org/p/cx-in-production

21/10/2014 (Friday)

 * Followup for Varnish setup in Beta with Alex.
 * Followup for creating CX Dashboard DB (Sean/Reedy).

31/10/2014 (Friday)

 * Updated CXServer/deploy to 6a7d012
 * Added new pairs, pt-ca/ca-pt to CX Beta.
 * https://gerrit.wikimedia.org/r/169993
 * https://gerrit.wikimedia.org/r/170006
 * Restart CXS when config.js changes: https://gerrit.wikimedia.org/r/170283

30/10/2014 (Thursday)

 * aperitum-apy is updated on Beta. Puppet doesn't update to new package version and this has to be done manually (ie in Beta, Kartik has to do this. Production, Ops will take care upon request).
 * Discussion with Alex regarding moving CX to production. Carved out story #4637 for that.

29/10/2014 (Wednesday)

 * Test and merge, https://gerrit.wikimedia.org/r/167629
 * Followup to merge apertium-apy update, https://gerrit.wikimedia.org/r/168760 with Alex. Identify easier way to do ‘new upstream packaging’ and document it.

23/10/2014 (Thursday)

 * Package of new apertium-apy based on co-ordination with upstream. See, db error we got on Log at, http://pastebin.com/6060q4Cy
 * Log at deployment-apertium instance is at, /var/log/upstart/apertium-apy.log (No longer in /var/log/apertium/)

21/10/2014 (Tuesday)

 * Testing of Apertium Beta instance.
 * Use Port 80 instead of default port to simplify:
 * https://gerrit.wikimedia.org/r/167747
 * https://gerrit.wikimedia.org/r/167749
 * Added cleanup patch for Apertium Puppet:
 * https://gerrit.wikimedia.org/r/167748
 * Patch to update CXServer code is ready:
 * https://gerrit.wikimedia.org/r/167629

20/10/2014 (Monday)

 * All packages are merged.
 * Apertium puppet and CXServer puppet patches are updated.
 * Testing of Apertium puppet patch on deployment-apertium.
 * Created new instance: deployment-apertium02
 * Apertium Puppet patch merged by Alex.
 * Created proxy for Apertium instance: apertium-beta.wmflabs.org

1710/2014 (Friday)

 * Updated apertium-apy package (Use default port)
 * Updated Apertium puppet patch based on Alex's reviews.
 * All packages except apertium-apy are built and installed on apt.wikimedia.org
 * CXServer patches are merged (See updates of 14/10)

15/10/2014 (Wednesday)

 * More packages reviewed and merged by Alex.
 * Followup on other pending patches and updates.

14/10/2014 (Tuesday)

 * Fixed CXServer on Beta:
 * https://gerrit.wikimedia.org/r/166535 : Add missing link to init with upstart-job.
 * https://gerrit.wikimedia.org/r/166542 : Define cxserver port.
 * Reused CXServer init link fix in Apertium Puppet patch.
 * Fixed missing tag for lttoolbox and changelog. Package is merged (https://gerrit.wikimedia.org/r/163548)

13/10/2014 (Monday)

 * Asked to revert, https://gerrit.wikimedia.org/r/#/c/165528/ as it was accidentally pushed to master while pushing upstream branch.
 * Cleaning up for Puppet patch, https://gerrit.wikimedia.org/r/#/c/165485

10/10/2014 (Friday)

 * Cleaned up Apertium Puppet patch to depend on Apertium version we provide, not default one. https://gerrit.wikimedia.org/r/#/c/165485
 * Updated (and lots of cleanups!) Puppet patch based on Alex's feedback (https://gerrit.wikimedia.org/r/#/c/165485).
 * Cleaned up APY package.

09/10/2014 (Thursday)

 * Updated Packaging document: https://www.mediawiki.org/wiki/Content_translation/Apertium/Packaging
 * Updated Apertium-APY package and Puppet patch for Apertium.
 * Followup for CXServer recreation.
 * Discussion with Apertium upstream to fix minor issues with APY.

08/10/2014 (Wednesday)

 * Updated CXServer Beta and Labs diagram, https://www.draw.io/#G0BztYqLvFZZVncXR1eV84bmRQQXM to get more clearity on process workflow for CXServer.
 * Packaging:
 * apertium-pt-ca
 * apertium-es-pt
 * apertium-apy
 * Requested needed repository to Alex.
 * Co-ordination for Deployment plan.
 * All packaging done.
 * Added initial Puppet patch for Apertium: https://gerrit.wikimedia.org/r/#/c/165485

07/10/2014 (Tuesday)

 * Packaging:
 * apertium-en-es (with tests from David's ticket to upstream)

06/10/2014 (Monday)

 * Updated, https://www.draw.io/#G0BztYqLvFZZVncXR1eV84bmRQQXM
 * Update: apertium-es-ca (for unresponsive ca-es pair), cg3 in Debian.
 * Tested more language pairs.

02/10/2014 (Thursday)

 * Updated further on: https://www.mediawiki.org/wiki/Content_translation/Apertium/Packaging
 * Minor updates to all packages.
 * Started pushing new updates from upstream.
 * Work on 'rejected' cg3 in Debian.
 * Followup for CXServer Beta continue.

01/10/2014 (Wednesday)

 * Followup for CXServer Beta continue.
 * Co-ordination with Apertium upstream for apertium-lex-tools issue(s). Fixed.
 * Pushed apertium-lex-tools to Debian and Gerrit.
 * Updated packages for minor issues.
 * Followup on ca-es issues, packages will be updated after upstream declares fix.

30/09/2014 (Tuesday)

 * Followup with team for CXServer Beta updates issue (old code, can't update without mt.service variable)
 * Follow up with Alex for new project creation.
 * Upstart issues with apertium-apy (WIP)
 * Rebuilt package against trusty; up for reviews.

29/09/2014 (Monday)

 * Branch push issues in all repositories are fixed.
 * Code to apertium/apertium-es-ca/cg3/lttoolbox is pushed.
 * 8 new projects are being created by Alex.
 * Apertium-APY work continue in ground.

26/09/2014 (Friday)

 * Continuous work on Apertium-APY and other language pairs based upon upstream packaging efforts (by Tino).
 * TODO: Fix other branch push issue.

25/09/2014 (Thursday)

 * Packaging:
 * lttoolbox: https://gerrit.wikimedia.org/r/#/c/162620/
 * apertium-es-ca and work on other language pairs.
 * Apertium-APY
 * Follow up with Alex for Gerrit other branch push issues for packaging (master/upstream/pristine-tar)
 * Follow up with Tino for Packaging and issues related to pcre patch in Apertium/lttoolbox.
 * TODO: Updated wikitech document for above issue.

24/09/2014 (Wednesday)

 * Packaging Apertium:
 * Uploaded lttoolbox/apertium/cg3 to Debian experimental.
 * Started packaging Apertium-APY and language paris.
 * Fixing git review issues with operations/debs/ repo with Alex.
 * Santhosh pointed that we need more CPU/RAM than current Labs instance for Apertium service. Asked Alex to provide large instance at deployment-apertium01.eqiad.wmflabs
 * Followup with Alolita.

23/09/2014 (Tuesday)

 * Work on Apertium packaging continue (lttoolbox, apertium)
 * Alex created needed projects at, operations/debs/...
 * Alolita to follow up with Mark.
 * #4552: Apertium Puppet work scheduled.
 * Co-ordination with Tino continue.

22/09/2014 (Monday)

 * Apertium packaging:
 * Apertium/cg3/lttoolbox
 * Also read more about WMF's Python packaging notes.
 * Co-ordination with Tino for Debian packaging issues. All fixed in Debian.
 * Beta updates:
 * Alex: deployment-apertium01.eqiad.wmflabs is ready to serve.
 * Labs/Beta client configs are updated (Niklas, Kartik).
 * Followups with Alex.
 * WIP: Apertium Puppet module.

19/09/2014 (Friday)

 * Apertium packaging started.
 * Environment of Trusty is up and running. Difficult to use Labs due to use of Gerrit username/authentication.
 * Started with Apertium. Yet to figure out some issues with build.
 * TODO: Separate following notes in another page!
 * WMF Debian packaging (apt.wikimedia.org) documents and process:
 * https://wikitech.wikimedia.org/wiki/APT_repository
 * https://wikitech.wikimedia.org/wiki/Reprepro
 * https://wikitech.wikimedia.org/wiki/Git-buildpackage
 * Process:
 * Package goes via Gerrit patchset.
 * Repository: operations/debs/
 * Build package using 'git-buildpackage' in a clean chroot Precise/Trusty environment. Aim for Trusty.
 * Once reviewed, Package(s) will be 'build' by Ops member (Alex in our case).

18/09/2014 (Thursday)

 * Meeting with Alex K.
 * Discussed:
 * Current status in Beta cluster with Content Translation.
 * Why we need Apertium as service?
 * Questions from Alex about:
 * What will be situation with Google/Bing in future?
 * Google/Bing will have agreement on API uptime and reliability with us if we opt for such service in future. This will make sure that we're not affected.
 * How frequent Apertium service will be updated?
 * We will have mostly update on Apertium language pairs once in a month, which are data packages. Core Apertium packages will be only updated upon critical bugs found in upstream bug tracker. Any bug from Wikimedia dev/user will be track at Apertium upstream bug tracker.
 * Blockers from LE team:
 * Apertium need to run as service before we go ahead with deployment.
 * Blockers from Ops:
 * Ops running tight with server allocation.
 * Action items:
 * Kartik will start working with Alex for setting up Debian repositories in WMF.
 * Kartik will start work on packages and pushing them to Debian repositories.
 * Alex will provide machine in Beta cluster when Packaging is done (in a week).
 * Kartik/Niklas will start working on Apertium Puppetizing with Alex.
 * Alex will talk to Mark about production server estimate date.
 * Alolita to setup meeting with Mark/Ops.
 * Kartik will document everything that comes under the hood of Apertium/Puppet/Repository and CX deployment process!
 * ETAs:
 * Beta cluster machine for Apertium service: 1 week after Packaging.
 * Production server: 3 to 4 weeks.
 * Update Deployment plan document with daily updates.