Wikimedia Release Engineering Team/Goal progress archive
Jump to navigation
Jump to search
![]() | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. Please see the team project in Phabricator and its related quarterly subprojects. |
Contents
- 1 Sept - Nov '14 Goals Progress
- 2 Sept Goal Progress
- 3 April - Aug '14 Goals Progress
- 4 Feb - Apr '14 Goal Progress
- 5 Dec-Feb Goal Progress
- 6 Previous format Check-ins
Sept - Nov '14 Goals Progress[edit]
- For past quarter's checkins, see the archives.
- See the WMF Engineering 14-15 goals page for the yearly view.
Phabricator[edit]
- Status: in-progress - Developer support for Migration from Bugzilla completed - Mukunda
- Be an example early adopter of features
- Status: Not done - Developer support for Migration from RT completed - Mukunda
metrics
- Status: Not done - Number of team migrated to Phabricator vs number of teams using Trello/Mingle right now
Deployment tooling[edit]
- Status: in-progress - scap(py) & trebuchet integration - Mukunda, Reedy
- how much and what it looks like TBD, some from the requirements doc
- Status: in-progress - increasing bus factor (specifically: Mukunda can respond to deployment tooling related issues) - Mukunda, Reedy
Jenkins[edit]
- Status: Not done - Jenkins performance improvements - Chris, Zeljko, Antoine
- Status: on-going - Maintenance and new test infrastructure requests (ongoing) - Chris, Zeljko, Antoine
Beta cluster[edit]
- Status: on-going - Add new services (-oids) - Antoine
- Status: Not done - Swift cluster (remove NFS) - Antoine
- Status: in-progress - Beta Cluster monitoring (baseline) - Antoine (and Yuvi)
- Status: in-progress - Graphite
- Status: Done - First pass Icinga alerts
- Status: in-progress - Yet Another Cluster - Antoine, Chris
metrics
- Status: in-progress - Real data and graphs from monitoring services
Browser tests[edit]
- Status: In progress - Workshops/trainings in lieu of one-to-one pair programming - Zeljko, Chris, Dan
- Meetup announcement for Exploratory Testing pending tentatively for Oct 22
- Another training announcement also pending
- Status: In progress - Improved "best practices" and "getting started" documentation - Zeljko, Chris
- Chris started overhaul of QA pages on mediawiki.org
- Status: on-going - Continued pairing with WMF Engineering teams - Zeljko, Chris, Dan
- Status: in processe - Begin pairing with the Flow team - Chris, Dan
- starting with refactoring in the Echo tests
- Status: in-progress - Environment abstraction layer in mediawiki-selenium to allow for less fragile and more advanced step definitions (strech goal) - Dan, Zeljko, Chris
metrics
- Status: Not done - tracking state of browser tests before Thursday branch cut
- Status: Not done - track days since last green build, per Jenkins job
Vagrant[edit]
- Status: in-progress - Wrap up pairing with MobileFrontend - Dan
- Status: in-progress - Investigate creating shareable vagrant- or docker-based test environments - Dan
- see work from Bryan?
- Status: Not done - Optimize memory hungry services running in the vagrant VM (reduce base memory usage) - Dan
metrics
- Status: Not done - qualitative survey of WMF teams on their use of MW-Vagrant
- Status: Not done - percentage of WMF production deployed extensions available in Vagrant
Hiring[edit]
- Status: in-progress - Complete hiring for QA Tester
Sept Goal Progress[edit]
Željko[edit]
Reedy[edit]
Rummana[edit]
Mukunda[edit]
- Status: in-progress - Phab data import, with Chase and Andre
Greg[edit]
- Status: in-progress - Hire QA Tester (jobvite announce
Dan[edit]
- Status: in-progress - MobileFrontend broswer tests in Vagrant
- Status: in-progress - Ruby style guide
- Status: in-progress - Refactor MediaWiki-Vagrant according to Ruby style guide
Chris[edit]
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: in-progress - hire new QA Tester
Antoine[edit]
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists OK
- agreement by End of June -> slipped due to discussions
POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
April - Aug '14 Goals Progress[edit]
- For past quarter's checkins, see the archives.
- See the WMF Engineering 13-14 goals page for the yearly view.
- Mid-quarter checkin (June 27th)
Deployment tooling[edit]
- Status: in-progress - Process through all (useful) pain points from the Dev/Deploy review session - (Greg)
- Status: in-progress - Product reviews of shell bugs as needed
- Status: blocked - Integrate HHVM support into our deployment systems - (Bryan, Greg, Mukunda, others from Platform)
- Waiting on HHVM packaging for Ubuntu.
- Status: Done - continue the scap(py) (stretch goal, finally prioritized since it adds a lot to our deployment workflow)
- All scap scripts have been migrated to python minus one
- Status: Not done - trebuchet integration conversation (stretch goal)
Phabricator[edit]
New project since this quarter
- Status: Done - Release engineer hired (Mukunda)
- Status: Done - SUL/OAuth integration - (Mukunda)
- Two patches in gerrit:
- Status: Done - Security component support - (Mukunda)
- Status: Done - Puppetization with ops (Chase) - (Mukunda)
- Status: in-progress - data import, with Chase and Andre - (Mukunda)
Beta cluster[edit]
- Status: blocked - Support HHVM deployment tooling and puppet configuration testing - (Bryan, Antoine, Mukunda)
- Waiting on HHVM packaging, puppet refactor for Trusty. See above.
- Status: in-progress - Swift cluster in beta (stretch goal)
- tracking bug: bug 62835
- Andrew Bogott/Filippo created preliminary puppet classes that matches labs environment
- merged on Tuesday 6/24 - https://gerrit.wikimedia.org/r/#/c/137803/
- Status: in-progress - assist language team to deploy cxserver - added mid-quarter - (Antoine) (added mid-quarter)
MediaWiki Release[edit]
- Status: Done - Successfully support the release of MediaWiki 1.23 - (Antoine, Greg)
- Status: in-progress - Kickoff/complete second RFP - (Greg)
- Status: Not done - Investigate and create useful release/deployment metrics visualizations - (Greg) - (stretch goal)
Browser tests[edit]
- Status: Not done - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki) - (Chris)
- dependent on the two below
- Status: in-progress - Retire Cloudbees Jenkins instance - (Željko)
- Status: in-progress - Retire /qa/browsertests repo by porting tests to /mediawiki/core and appropriate extension repos (Chris) (added mid-quarter)
- Status: Done - Integrate WMF Jenkins with new WMF SauceLabs account - (Željko, Antoine, Chris)
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.) - (Chris, Željko) Calling this done for the quarter, but we'll always be using it into the future
- Used by MobileFrontend
- Used by VisualEditor
- Used by smoke tests (part of quarterly goals)
- Status: on-hold - Add browsertests to new repos e.g. GettingStarted - (Chris, Željko)
Engineering Liasion Work[edit]
- Status: on-going - Pair with teams and community members to improve QA practices
- Language team
- Wikidata team
- Commuity members
- Status: in-progress - GSOC mentoring
Vagrant[edit]
New project since this quarter
- Status: Done Automation Engineer hired (Dan)
- Status: in-progress - Improve Vagrant setup process (especially re browser tests) and clean up Ruby according to style guide
- Status: Done - VE browser tests in Vagrant
- Status: in-progress - Ruby style guide
Hiring[edit]
- Status: Done - Complete hiring and train new Release Engineer (Greg, all)
- Status: Done - Complete hiring and train new Automation Engineer (Ruby) (Chris, all)
- Status: in-progress - Complete hiring of a new QA Tester (Chris, Greg) - added mid-quarter
Quarterly Review Action items[edit]
- Status: Done - create a plan for browser testing of MediaWiki 1.23 - Chris M
- Status: Done - Greg get firm requirements from Antoine, circle back to Mark who'll have an idea of Op's timelime for production Shinken (monitoring system to maybe replace Icinga)
- no deadline knows on ops side. Might not be high priority
- ops in charge: Alexandros (Greece based)
- maybe use it in Beta Cluster as a POC
- Status: Done - Figure out how to keep HHVM unit tests from delaying +2 for standard production commits (Antoine, Chris M, Zeljko)
July/Aug Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: in-progress - WMF product should be consulted on some shellbugs
- Status: Done - Kickoff/complete second RFP
- Status: in-progress - Hire QA Tester (jobvite announce
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: hold - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists OK
- agreement by End of June -> slipped due to discussions
POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
- Status: Done - assist language team to have cxserver deployed to beta cluster
- They handle the rest with ops (puppet) and deployers (mediawiki-config)
Mukunda[edit]
- Status: in-progress - read all the things
Done - WMF SUL/OAuth support T40
Done - security component support in Phab Upstream T4893, our phab T95
- custom plugin for this, installed on fab.wmflabs, code is in gerrit
- being reviewed upstream, but won't be included in mainline Phab
Done - Custom field to show mediawiki user page links on phabricator profiles
Done - puppet wmflib beginnings
Done - legalpad.mediawiki.org
Done - Build debian package for mailparse php extension
- Status: Done - Honor security policies for file uploads/attachments - (Mukunda)
- Status: in-progress - data import, with Chase and Andre
Bryan[edit]
- Status: In progress Scap refactor/python port
Done bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
- Status: Not done - Integrate HHVM support into our deployment systems
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
Reedy=[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: Done - migrate to WMF Jenkins
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Status: in-progress - MobileFrontend broswer tests in Vagrant
- Status: Done - Improve Vagrant setup process (especially re browser tests)
- Status: in-progress - Ruby style guide
- Status: in-progress - Refactor MediaWiki-Vagrant according to Ruby style guide
June Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: in-progress - WMF product should be consulted on some shellbugs
- Status: Done - Successfully support the release of MediaWiki 1.23
- Status: in-progress - Kickoff/complete second RFP
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: Not done - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: Done - Integrate WMF Jenkins with new WMF SauceLabs account
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists
- agreement by End of June
- POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
- Status: In progress - assist language team to have cxserver deployed to beta cluster
Mukunda[edit]
- Status: in-progress - read all the things
- Status: in-progress - WMF SUL/OAuth support T40
- Status: in-progress - security component support in Phab Upstream T4893, our phab T95
- local 6 line hack
- pinging evan regularly on real long term fix
- Status: in-progress - puppet wmflib beginnings
Bryan[edit]
- Status: In progress Scap refactor/python port
In progress bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
Done Gerrit change 130560 Build JSON git info caches during scap
Done Gerrit change 130498 support in GitInfo.php
In progress Gerrit change 142319 Fix GitInfo cache file path computation and storage location (Needs code review and +2)
In progress Gerrit change 142320 Set wgGitInfoCacheDirectory to point to scap managed location
Done bug 66844 Fix sync-common for hosts in wikimedia.org TLD
- Status: Not done - Integrate HHVM support into our deployment systems
Done Met with Emir from Facebook to discuss HHVM and deployment in general
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
Done Helped with deployment training hangout
Reedy[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: in-progress - migrate to WMF Jenkins
- Status: Done - create a plan for browser testing of MediaWiki 1.23
- https://bugzilla.wikimedia.org/show_bug.cgi?id=66272
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Chris Steipp had suggestions in email for further tests that map to legacy tests in /qa/browsertests repo
- After discussion with Rob Lanphier, will port appropriate tests from /qa/browsertests to /mediawiki/core with Dan Duvall over time
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.)
- Done for Mobile
- Done for VisualEditor
- Done for browser smoke tests
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Read all the things
- Status: Done - VE broswer tests in Vagrant
- Status: in-progress - MobileFrontend broswer tests in Vagrant
May Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: Done - config changes not being deployed after merge (RT ticket filed], successfully in Ops' court)
- Status: in-progress - WMF product should be consulted on some shellbugs
- email sent to WMF Product on 5/15
- Status: Done - onboard new Release Engineer
- ongoing
- Status: in-progress - Successfully support the release of MediaWiki 1.23
- Status: in-progress - Kickoff/complete second RFP
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: Not done - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Done - Figure out how to keep HHVM unit tests from delaying +2 for standard production commits
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: Not done - Integrate WMF Jenkins with new WMF SauceLabs account
- Status: In progress - RFC for Jenkins tests isolation
- Status: Not done - RFC for cross repositories testing and integration job in gate
Mukunda[edit]
- Status: in-progress - read all the things
- Status: in-progress - WMF SUL/OAuth support T40
- Status: Not done - security component support in Phab Upstream T4893, our phab T95
Bryan[edit]
- Status: In progress Scap refactor/python port
Done Setup test environment in beta
In progress bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
Done Gerrit change 130560 Build JSON git info caches during scap
Done Gerrit change 130498 support in GitInfo.php
- Code is deployed in beta (and prod) but it doesn't work as expected. See notes on bug for where I'm stuck.
Done Use trebuchet to deploy scap scripts
- Gerrit change 130211 Add scap/scap trebuchet target
- Gerrit change 129814 Provision scap scripts using trebuchet
Done Gerrit change 134272 source organization cleanup
Done Gerrit change 134730 Change default console logging verbosity (less noise for deployers)
Done Convert sync-* to python
- Gerrit change 135512 Extract common base class for sync operations
- Gerrit change 135513 Convert sync-dir and sync-file to python
- Gerrit change 135514 Convert sync-dblist to python
- Gerrit change 135515 Convert sync-docroot to python
- Status: Not done - Integrate HHVM support into our deployment systems
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
Done Created Train deploy instructions on wikitech based on noted from Sam and experience in March
Reedy[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: Done - create a plan for browser testing of MediaWiki 1.23
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Chris Steipp had suggestions in email for further tests that map to legacy tests in /qa/browsertests repo
- After discussion with Rob Lanphier, will port appropriate tests from /qa/browsertests to /mediawiki/core with Dan Duvall over time
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.)
- Done for Mobile
- Done for VisualEditor
- Done for browser smoke tests
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Read all the things
Feb - Apr '14 Goal Progress[edit]
See the WMF Engineering 13-14 goals page for the yearly view.
- Process through all (useful) pain points from the Dev/Deploy review session (Greg)
- Scap incremental improvements
- 1) Refactor existing scap scripts to enhance maintainability and reveal hidden complexity of current solution (Bryan)
- 2) create matrix of tool requirements per software stack (MW, Parsoid, ElasticSearch) (Greg)
- 3) Use above matrix to add/fix functionality in scap (or related) tooling for ONE software stack, prioritized by cross stack use (Bryan)
- Use the API to create test data for given tests at run time. (Jeff, Chris, Željko)
- Create the ability to test headless (Željko, Jeff, Chris)
- Run versions of tests compatible with target test environments (Chris, all)
- Make database(s) in Beta Cluster emulate production (set up db slaves) (Antoine)
April Goal Progress[edit]
Greg[edit]
- Status: Done - Update/create new MW Release Management RFP
- Status: in-progress - Process through pain points
- Status: Done - security patches
- Status: Done - LD SWAT team
- Status: in-progress - config changes not being deployed after merge
- Status: in-progress - Refine deployment system requirements with Bryan
Antoine[edit]
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
In progress Setup test environment in beta (carried forward from Feb & March; much harder than I'd hoped)
Done Scap functionality available from deployment-bastion including fanout rsync and l10n rebuilds
Done Jenkins job running scap after each beta-code-update job finished
Done Gerrit change 127399 Remove NFS dependency from beta for MediaWiki deploys
Done Gerrit change 126870 Convert hand-built Jenkins jobs to CI/JJB
Done Gerrit change 129822 Use trigger publisher instead of trigger-builds builder (fixed deadlock problem on Jenkins slave)
- Continue to enhance and simplify scripts
Done bug 63659 Gerrit change 124762 Compile wikiversions.json to cdb on deploy server
Done Gerrit change 125334 Allow hyphen (-) in dsh hostnames
Done Gerrit change 125359 Fix IRCSocketHandler to work from Jenkins
Done Gerrit change 125515 Make logging destinations configurable
Done Gerrit change 126924 Improve error message when scap lock fails
Done Gerrit change 127198 Exit with non-zero status on soft failure (Helps Jenkins jobs identify partial failures)
Done Gerrit change 127199 Convert scap-rebuild-cdbs to python
Done Gerrit change 129577 Convert mw-update-l10n to python
In progress Build .mw-git-info.json caches during scap and support in GitInfo.php
- Use trebuchet to deploy scap scripts
In progress Gerrit change 130211 Add scap/scap trebuchet target
In progress Gerrit change 129814 Provision scap scripts using trebuchet
- Status: In progress Next generation deployment tooling
In progress Organize requirements into wiki page (carried forward from Feb & March; pairing with Greg)
- Status: In progress Support train deploys
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
Continuing from Feb:
- Test and announce general availability of feature to check for ResourceLoader upon page load: ResourceLoader error checking more globally. Status: Done
- Test and announce general availability of feature to use the API to create wiki pages and users: create-page (and create-user) API. Status: Done
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Tracking this at https://bugzilla.wikimedia.org/show_bug.cgi?id=62509 . I have an idea about how to make this work. Status: In progress
Also:
- Update documentation on mw.o to reflect the refactoring and consolidation of recent times. Much of what exists now is long out of date. Status: In progress
Rummana[edit]
Željko[edit]
March Goal Progress[edit]
Greg[edit]
- Status: in-progress - Update/create new MW Release Management RFP
- Status: in-progress - Process through pain points
- Status: Done - security patches
- Status: Done - LD SWAT team
- Status: in-progress - config changes not being deployed after merge
- Status: in-progress - Refine deployment system requirements with Bryan
- Status: Done - create blockdiag version of flow chart
- LD SWAT team
- Status: Done - Kick off
- Status: Done - PushBot? - NO (without more dev than Greg can provide right now)
- Status: Done - quarterly post-mortem review
- See the review
Antoine[edit]
- Status: In progress Fix up VisualEditor browser tests and make it voting in Gerrit
- Status: In progress Makes CirrusSearch browser tests voting
Done Publish sphinx documentation (use jenkins jobs). Examples:
- Status: In progress Migrate beta cluster from pmtpa to eqiad!
Done Varnishes instances creation + puppet passing
Done Application servers created + puppet passing
Done Apache config in git (operations/apache-config.git branch: betacluster)
Done Bunch of files / git repos copied from pmtpa to eqiad
Done Sean Pringle to create MariaDB
Done Ariel Glenn to set up the swift emulator (copy pasted instance)
Done last minute sync of files
Done add Jenkins slaves to sync Parsoid and MW code
Done write jobs to update Parsoid and MW code on eqiad
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
Done Extract common logic for making a command line interface into a class
Done Fixed sync-wikiversions to use common dsh arguments
Done Converted sync-wikiversions to python
Done Invented scap-purge-l10n-cache script to cleanup l10n cache on unused branches
Done Documented process for retiring a branch from tin
In progress Setup test environment in beta (carried forward from Feb; harder than I'd hoped)
Done Salt master in eqiad beta project
Done Puppet master in eqiad beta project
Done logstash host in eqiad beta project
Not done scap host in eqiad beta project (did this and then nuked it)
- Continue to enhance and simplify scripts
- Status: In progress Next generation deployment tooling
In progress Organize requirements into wiki page (carried forward from Feb; pairing with Greg)
- Status: In progress Support train deploys
Done Deploy 1.23wmf16 to group1
Done Deploy 1.23wmf17 to group0
Done Deploy 1.23wmf17 to group1
Done Cleanup old l10n cache files on cluster
Done Deploy 1.23wmf18 to group0
Done Deploy 1.23wmf18 to group1
Done Deploy 1.23wmf20 to group0
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- (With Guillaume) Have an IRC office hour about Project management tools Status: Done on 2014-03-28
(With Guillaume) Set up and facilitate the community RfC about Project management tools, and bring it to a decision.Status: Not done -- postponed to April- (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Kill curated list on the wikipage; provide specific Bugzilla queries and "getting started" instructions for each area: Status: Done on 2014-03-07 (diff)
Introduce "Bug of the week" after talking to teams?Status: Not done - postponed to Q2/2014
- Better Bugzilla taxonomy: Discuss using classifications in Bugzilla and identify some initial non-controversial changes. - bugzilla:38990. Status: In progress; first smaller cleanup steps about "Tools" and deprecated stuff are
bugzilla:57738, bugzilla:54063, bugzilla:53986#c15, bugzilla:55351#c4, bugzilla:62386, bugzilla:62384; Proposal to use Bugzilla classifications on 2014-03-10 - Bugzilla setup / code:
Provide a NEEDINFO flag in Bugzilla - bugzilla:36064. Status: In progress: (porting more complicated as bmo will skip 4.4) -- might get postponed to Q2/2014- Provide a way to mark an item as NEEDINFO in Bugzilla - bugzilla:36064. Status: In progress: Created list of pros and cons of two implementations on 2014-03-10; next step is start broader discussion on wikitech-l@ and agree on which way to do
Install component watching extension to be able to receive bugmail for specific product/component for devs and triagers, plus no more manual adding of people to auto-CC by admins- bugzilla:37105. Status: Not done -- database issues when testing; not yet tested on 4.4 and upstream code is still 4.2 only. Porting more complicated as bmo will skip 4.4 -- postponed to Q2/2014
Chris[edit]
Continuing from Feb:
- Test and announce general availability of feature to check for ResourceLoader upon page load: ResourceLoader error checking more globally. Status: Done
- Test and announce general availability of feature to use the API to create wiki pages and users: create-page (and create-user) API. Status: In progress
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Tracking this at https://bugzilla.wikimedia.org/show_bug.cgi?id=62509 . I have an idea about how to make this work. Status: In progress
Also:
- Update documentation on mw.o to reflect the refactoring and consolidation of recent times. Much of what exists now is long out of date. Status: In progress
Rummana[edit]
Željko[edit]
February Goal Progress[edit]
Greg[edit]
- Process through pain points Status: in-progress
- security patches Status: Done
- Kick off LD SWAT team Status: Not done
- next...
- create blockdiag version of flow chart Status: Done
- Refine deployment system requirements with Bryan Status: in-progress
- quarterly post-mortem kickoff Status: Not done
- set up morgue?
Antoine[edit]
- Status: In progress Complete integration of browsertests for VisualEditor
- One build worked and triggered two successful scenario!!
- Parsoid is stopped by the job now, was not previously :/
Done Integration of browsertests for CirrusSearch. Cause me to slightly rethink the browsertests infra to closely match production.
Done Train Zeljkof on Jenkins Job Builder script
- Zeljkof started on it. Will pair with him to finish up.
- Zeljkof found his way through JJB arcanes \O/
Done Help migrating Cloudbees Jenkins template to JJB YAML templates
- Zeljkof started on it. Will pair with him to finish up.
- Zeljkof found his way through JJB arcanes \O/
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
Done Local test environment in a MW-Vagrant instance
Done scap converted to python
Done scap-1 converted to python
Done add detailed duration timing for scap & scap-1
Done remove external script dependencies from scap-1
Done add progress bar for dsh commands
Done Scap logs in json format for easy parsing
Done Scap logs sent to florine via udp2log
Done Scap logs sent to logstash via udp2log
Done Converted mwversionsinuse to python
Done Deleted obsolete scripts: scap-1, scap-2, find-nearest-rsync, scap-old
In progress Setup test environment in beta (will pair with Antione)
- Continue to enhance and simplify scripts
- Status: In progress Next generation deployment tooling
Done gather requirements via etherpad and Ops-l mailing list
In progress Organize requirements into wiki page (will pair with Greg)
- Status: In progress Support train deploys
Done Deploy 1.23wmf16 to group0
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - Status: Done. Outstanding steps from bugzilla:49597:
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Done on 2014-02-13
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: Done on 2014-02-13
- Finalize Bugzilla etiquette draft. Status: Done - 2014-02-05: Announcement
- Further small patches after 4.4 upgrade (License headers and template processing cleanup;
numerous contributed Gerrit patches waiting). Status: In progress - (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Cover common "getting started" questions: Status: Done added on 2014-02-18
- Improve Bugzilla queries on Annoying little bugs to show a good subset: Status: Done on 2014-02-18
- Add guidelines for triagers how to identify a bug report that is good for this audience: Status: Done by editing the "easy" keyword description on 2014-02-24
- Retriage existing bugs with easy keyword: mostly Status: Done
Chris[edit]
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Investigate using ResourceLoader error checking more globally. Status: Done
- Refactor tests to use the create-page (and create-user) API. Status: Done
Željko[edit]
Jeff[edit]
Rummana[edit]
Dec-Feb Goal Progress[edit]
See the WMF Engineering 13-14 goals page for the yearly view.
- Status: in-progress - Browser tests managed in feature repos with feature teams (Chris, Zeljko, Jeff, Rummana)
- Status: Done - Successfully managed the first release of MediaWiki in conjunction with our outside contractor (Greg, Antoine)
- Status: Not done - More comprehensive quarter assessments of postmortems (Greg)
- Status: in-progress - Create process documentation for ideal test/deployment steps (Greg, Reedy, Chris, and others)
- Automated API integration tests in important areas (Chris, Zeljko, Jeff, Rummana, Antoine, also with Mark Holmquist)
- Status: in-progress - UploadWizard
- Status: in-progress - Parsoid / VisualEditor
- Status: Not done - ResourceLoader
Monthly Goal Progress[edit]
February[edit]
Greg[edit]
- do post Dev and Deploy process review follow up Status: in-progress
- post images to Commons - Status: Done - included in commons:Category:Wikimedia_Foundation_software_development
- send one email with list of grouped red-cards Status: Not done
- send off first email on first topic (make it a good one) Status: Not done
- Create plan to evaluate progress on postmortem/retrospective actions Status: in-progress
- archive/put on wikitech missing post-mortems - Status: in-progress
- create BZ whiteboard entry to track retrospective bugs - Status: Not done
- RT something or other?
Antoine[edit]
- Status: In progress Complete integration of browsertests for VisualEditor
- Status: In progress Train Zeljkof on Jenkins Job Builder script
- Status: In progress Help migrating Cloudbees Jenkins template to JJB YAML templates
Reedy[edit]
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - Status: Done. Outstanding steps from bugzilla:49597:
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Done on 2014-02-13
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: Done on 2014-02-13
- Finalize Bugzilla etiquette draft. Status: Done - 2014-02-05: Announcement
- Further small patches after 4.4 upgrade (License headers and template processing cleanup;
numerous contributed Gerrit patches waiting). Status: In progress - (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Cover common "getting started" questions: Status: Done added on 2014-02-18
- Improve Bugzilla queries on Annoying little bugs to show a good subset: Status: Done on 2014-02-18
- Add guidelines for triagers how to identify a bug report that is good for this audience: Status: Done by editing the "easy" keyword description on 2014-02-24
- Retriage existing bugs with easy keyword: mostly Status: Done
Chris[edit]
- Refactor tests to use API article creation Status: Not done
- Continue creating a suite of tests for local environments Status: Not done
- Requires the API gem in mediawiki-selenium 1.20
- Finish headless Xvfb integration Status: In progress
- In Jenkins, investigate pulling master branch for beta labs builds but pull release branch for test2wiki builds
- Follow up on ResourceLoader error reporting Status: Not done
Željko[edit]
Jeff[edit]
- New task: Create browser test for VisualEditor availability on production wikis Status: In progress
- Add browser tests for using VisualEditor via Mobile UI Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
January[edit]
Greg[edit]
- Prep for Dev and Deploy process meeting (Jan 22nd)
- Create baseline flowchart of dev/deploy documentation for use in the January in-person meeting - Status: Done
- refinements (ie: suggestions from Chris) - Status: Done
- write up final agenda/notes, share before meeting (clean up notes from Robla) - Status: Done
- make physical version of flowchart - Status: Done
- do post-review followup - Status: Not done
- Create plan to evaluate progress on postmortem/retrospective actions Status: Not done
- archive/put on wikitech missing post-mortems - Status: Not done
- create BZ whiteboard entry to track retrospective bugs - Status: Not done
Antoine[edit]
- For January: got to prepare myself for the MediaWiki summit (two weeks left + one week summit)
- Mostly focused on CI (Zuul upgrade, new jenkins jobs..) last week.
Done Parsoid self update on beta cluster via a Jenkins job.
- See 'Parsoid update' on the CI dashboard https://integration.wikimedia.org/dashboard/
Done Parsoid job migrated to new repos mediawiki/services/parsoid and mediawiki/services/parsoid/deploy
- Status: In progress VE browser tests from Gerrit. Waiting for them to be passing with (a fresh wiki + phantomjs)
Done mediawiki/extensions.git out of sync since Jan 2nd bug 59758
Reedy[edit]
Andre[edit]
- (With Guillaume) Meet Project management tools stakeholders, determine requirements, and document this research. Status: Done by Guillaume here
- (With Daniel Zahn) Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - outstanding steps from bugzilla:49597:
- Upgrade Bugzilla on zirconium from 4.2 to 4.4. Status: Done
- Apply 4.2 custom changes to Bugzilla on zirconium from Git repository. Status: Done
- Apply Andre's 11 patches (linked here) in Gerrit to port our custom changes from 4.2 to 4.4. Status: Done on 2014-01-07
- Test Bugzilla on zirconium. Status: Done on 2014-01-15
- Make collectstats.pl work - bugzilla:29203. Status: Done by dzahn here on 2014-01-29
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Not done - tentative date: 2014-02-12
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: In progress - 2014-01-10: Patch in Gerrit; depends on 4.4 upgrade first
- Finalize Bugzilla etiquette draft once the lively discussion on its Talk page has ended. Status: In progress - 2014-01-09: Announcement that discussion will be closed soon
- Fix inline displaying of image files in Bugzilla - bugzilla:54181. Status: Done - andre successfully tested csteipp's patch on Labs on 2014-01-05; deployed on 2014-01-10
- (With Quim) Finish running Google Code-In contest. Status: Done
Chris[edit]
- Bring about monitoring uploads in production with existing test, with failure messages emailed to QA staff Status: In progress Željko and Antoine to discuss hiding the password properly.
- Create a suite of local bare-wiki tests to use PhantomJS in WMF Jenkins Status: Not done
- https://bugzilla.wikimedia.org/show_bug.cgi?id=60347
- Only one existing test under/qa/browsertests passes in a bare wiki
- Use the API to create test data (e.g. a wiki page) on a target wiki Status: In progress Jeff working on https://gerrit.wikimedia.org/r/#/c/106548/ . Jeff and Chris paired Jan 24 to make progress. Commit is pending unexpected auth issue: https://bugzilla.wikimedia.org/show_bug.cgi?id=60407
- Continue contributing code and review to Mobile tests Status: Done paired with Arthur on https://gerrit.wikimedia.org/r/#/c/106833/, etc.
- Continue contributing code and review to Flow tests Status: Done Reported/fixed upstream Selenium bug , continuing...
- Monitor fatal errors in beta labs, send email alerts Status: Done
- Status: Done merged a monitor script, needs to be cron'd/puppet'd bug 60046
Željko[edit]
Jeff[edit]
- New task: add browser tests for using VisualEditor via Mobile UI Status: In progress
- Worked with Aaron Arcos to create initial browser test for MultimediaViewer repo Status: Done
- Creating scripts for adding new wiki users and articles via Mediawiki API Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
December[edit]
Greg[edit]
- Create baseline flowchart of dev/deploy documentation for use in the January in-person meeting - Status: Done
- Create plan to evaluate progress on postmortem/retrospective actions Status: in-progress
Antoine[edit]
- jenkins job for VisualEditor / Parsoid (reporting to James F / Gabriel Wicke)
- making sure a change in VE or in Parsoid does not break the other since they are tightly coupled
Done parsoid init script doesn't play well when run over ssh
- Worked on, wrote an upstart job to wrap around parsoid server + logrotate configuration
- browser tests in Gerrit.
Done First with ULS cause it is simpler, pairing with Zeljkof and i18n team
- Status: In progress Second MobileFrontend https://gerrit.wikimedia.org/r/#/c/97497/
- Status: Not done later VisualEditor
Done polishing up the mw release tarball job (was bug 58278)
Slowdowns:
- deployed bunch of favicons for Google Code-in
- helped on gwtoolset, an extension to mass import materials from museum libraries
- bunch of CI changes to make jobs running in parralel
Reedy[edit]
Andre[edit]
- Google Code-In: Run and organize contest with Quim. Status: In progress
- Agree and finalize "etiquette" draft for behavior in Bugzilla, as discussed on teampractices@. Status: In progress - Discussion on Talk page still ongoing on 20131225 after asking for feedback on wikitech-l@.
- Evaluate Project management / issue tracking requirements and potential tools. Status: In progress - kicked off on teampractices@ and wiki on 2013-12-13
- Test Bugzilla 4.4 with our custom patches on Labs (or zirconium in eqiad if production is still on kaulen in Tampa). Status: In progress, will need help from ops. Steps discussed between dzahn and aklapper on 2013-12-06; dzahn has set up a copy of Bugzilla 4.2 on zirconium in eqiad (see RT #4783). Next steps are upgrading that machine to Bugzilla 4.4, applying our custom patches, testing, and finally switching over.
Chris[edit]
- Get everyone interested in API testing and monitoring pulling together: Multimedia + Jenkins + QA
- Status: Done Met with Mark Holmquist and Aaron Aarcos Dec 5
- Progress: https://bugzilla.wikimedia.org/show_bug.cgi?id=58555
- Test is merged and is pending adding to Jenkins https://gerrit.wikimedia.org/r/#/c/102603/
- Status: Done Met with Mark Holmquist and Aaron Aarcos Dec 5
- Move login method to shared code (Željko working on this right now), demonstrate improved pass rates pairticularly for IE
- Status: In progress In process, interim commit is https://gerrit.wikimedia.org/r/#/c/100579/
- Upgraded login method and corrected other issues affecting IE pass rate, builds are significantly more green, will pass on benefit to other repos when consolidated
- Status: In progress In process, interim commit is https://gerrit.wikimedia.org/r/#/c/100579/
- Concrete steps for Mobile QA after Michelle's departure
- Status: Done Chris to support automation effort, Chris/Jeff/Rummana to contribute as requested, e.g. post-deployment checking
- Status: Done Chris is monitoring the backlog of mingle cards for tests immediately. Next step is to collaborate with Mobile devs for regression tests like https://gerrit.wikimedia.org/r/#/c/103761/
- Browser test coverage for Flow
- Status: Done Coverage ongoing
- Status: Done Continuing working with S on refinements
Željko[edit]
Jeff[edit]
- Creating scripts for adding new wiki users and articles via Mediawiki API Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
- Regular exploration testing on betalabs and test2
- Verifying each week's VE deployment
- Verifying resolved bugs
- Verifying new copy-paste implementation
- Track the already reported bugs on bugzilla and changing their status accordingly
Previous format Check-ins[edit]
- Wikimedia_Release_and_QA_Team/Checkin-20131119
- Wikimedia_Release_and_QA_Team/Checkin-20131105
- Wikimedia_Release_and_QA_Team/Checkin-20131008
- Wikimedia_Release_and_QA_Team/Checkin-20130924
- Wikimedia_Release_and_QA_Team/Checkin-20130917
- Wikimedia_Release_and_QA_Team/Checkin-20131022
- Wikimedia_Release_and_QA_Team/Checkin-20130910
- Wikimedia_Release_and_QA_Team/Checkin-20130903