Wikimedia Release Engineering Team/Goal progress archive
Jump to navigation
Jump to search
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. Please see the team project in Phabricator and its related quarterly subprojects. |
Contents
- 1 Sept - Nov '14 Goals Progress
- 2 Sept Goal Progress
- 3 April - Aug '14 Goals Progress
- 4 Feb - Apr '14 Goal Progress
- 5 Dec-Feb Goal Progress
- 6 Previous format Check-ins
Sept - Nov '14 Goals Progress[edit]
- For past quarter's checkins, see the archives.
- See the WMF Engineering 14-15 goals page for the yearly view.
Phabricator[edit]
- Status: in-progress - Developer support for Migration from Bugzilla completed - Mukunda
- Be an example early adopter of features
- Status: Not done - Developer support for Migration from RT completed - Mukunda
metrics
- Status: Not done - Number of team migrated to Phabricator vs number of teams using Trello/Mingle right now
Deployment tooling[edit]
- Status: in-progress - scap(py) & trebuchet integration - Mukunda, Reedy
- how much and what it looks like TBD, some from the requirements doc
- Status: in-progress - increasing bus factor (specifically: Mukunda can respond to deployment tooling related issues) - Mukunda, Reedy
Jenkins[edit]
- Status: Not done - Jenkins performance improvements - Chris, Zeljko, Antoine
- Status: on-going - Maintenance and new test infrastructure requests (ongoing) - Chris, Zeljko, Antoine
Beta cluster[edit]
- Status: on-going - Add new services (-oids) - Antoine
- Status: Not done - Swift cluster (remove NFS) - Antoine
- Status: in-progress - Beta Cluster monitoring (baseline) - Antoine (and Yuvi)
- Status: in-progress - Graphite
- Status: Done - First pass Icinga alerts
- Status: in-progress - Yet Another Cluster - Antoine, Chris
metrics
- Status: in-progress - Real data and graphs from monitoring services
Browser tests[edit]
- Status: In progress - Workshops/trainings in lieu of one-to-one pair programming - Zeljko, Chris, Dan
- Meetup announcement for Exploratory Testing pending tentatively for Oct 22
- Another training announcement also pending
- Status: In progress - Improved "best practices" and "getting started" documentation - Zeljko, Chris
- Chris started overhaul of QA pages on mediawiki.org
- Status: on-going - Continued pairing with WMF Engineering teams - Zeljko, Chris, Dan
- Status: in processe - Begin pairing with the Flow team - Chris, Dan
- starting with refactoring in the Echo tests
- Status: in-progress - Environment abstraction layer in mediawiki-selenium to allow for less fragile and more advanced step definitions (strech goal) - Dan, Zeljko, Chris
metrics
- Status: Not done - tracking state of browser tests before Thursday branch cut
- Status: Not done - track days since last green build, per Jenkins job
Vagrant[edit]
- Status: in-progress - Wrap up pairing with MobileFrontend - Dan
- Status: in-progress - Investigate creating shareable vagrant- or docker-based test environments - Dan
- see work from Bryan?
- Status: Not done - Optimize memory hungry services running in the vagrant VM (reduce base memory usage) - Dan
metrics
- Status: Not done - qualitative survey of WMF teams on their use of MW-Vagrant
- Status: Not done - percentage of WMF production deployed extensions available in Vagrant
Hiring[edit]
- Status: in-progress - Complete hiring for QA Tester
Sept Goal Progress[edit]
Željko[edit]
Reedy[edit]
Rummana[edit]
Mukunda[edit]
- Status: in-progress - Phab data import, with Chase and Andre
Greg[edit]
- Status: in-progress - Hire QA Tester (jobvite announce
Dan[edit]
- Status: in-progress - MobileFrontend broswer tests in Vagrant
- Status: in-progress - Ruby style guide
- Status: in-progress - Refactor MediaWiki-Vagrant according to Ruby style guide
Chris[edit]
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: in-progress - hire new QA Tester
Antoine[edit]
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists OK
- agreement by End of June -> slipped due to discussions
POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
April - Aug '14 Goals Progress[edit]
- For past quarter's checkins, see the archives.
- See the WMF Engineering 13-14 goals page for the yearly view.
- Mid-quarter checkin (June 27th)
Deployment tooling[edit]
- Status: in-progress - Process through all (useful) pain points from the Dev/Deploy review session - (Greg)
- Status: in-progress - Product reviews of shell bugs as needed
- Status: blocked - Integrate HHVM support into our deployment systems - (Bryan, Greg, Mukunda, others from Platform)
- Waiting on HHVM packaging for Ubuntu.
- Status: Done - continue the scap(py) (stretch goal, finally prioritized since it adds a lot to our deployment workflow)
- All scap scripts have been migrated to python minus one
- Status: Not done - trebuchet integration conversation (stretch goal)
Phabricator[edit]
New project since this quarter
- Status: Done - Release engineer hired (Mukunda)
- Status: Done - SUL/OAuth integration - (Mukunda)
- Two patches in gerrit:
- Status: Done - Security component support - (Mukunda)
- Status: Done - Puppetization with ops (Chase) - (Mukunda)
- Status: in-progress - data import, with Chase and Andre - (Mukunda)
Beta cluster[edit]
- Status: blocked - Support HHVM deployment tooling and puppet configuration testing - (Bryan, Antoine, Mukunda)
- Waiting on HHVM packaging, puppet refactor for Trusty. See above.
- Status: in-progress - Swift cluster in beta (stretch goal)
- tracking bug: bug 62835
- Andrew Bogott/Filippo created preliminary puppet classes that matches labs environment
- merged on Tuesday 6/24 - https://gerrit.wikimedia.org/r/#/c/137803/
- Status: in-progress - assist language team to deploy cxserver - added mid-quarter - (Antoine) (added mid-quarter)
MediaWiki Release[edit]
- Status: Done - Successfully support the release of MediaWiki 1.23 - (Antoine, Greg)
- Status: in-progress - Kickoff/complete second RFP - (Greg)
- Status: Not done - Investigate and create useful release/deployment metrics visualizations - (Greg) - (stretch goal)
Browser tests[edit]
- Status: Not done - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki) - (Chris)
- dependent on the two below
- Status: in-progress - Retire Cloudbees Jenkins instance - (Željko)
- Status: in-progress - Retire /qa/browsertests repo by porting tests to /mediawiki/core and appropriate extension repos (Chris) (added mid-quarter)
- Status: Done - Integrate WMF Jenkins with new WMF SauceLabs account - (Željko, Antoine, Chris)
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.) - (Chris, Željko) Calling this done for the quarter, but we'll always be using it into the future
- Used by MobileFrontend
- Used by VisualEditor
- Used by smoke tests (part of quarterly goals)
- Status: on-hold - Add browsertests to new repos e.g. GettingStarted - (Chris, Željko)
Engineering Liasion Work[edit]
- Status: on-going - Pair with teams and community members to improve QA practices
- Language team
- Wikidata team
- Commuity members
- Status: in-progress - GSOC mentoring
Vagrant[edit]
New project since this quarter
- Status: Done Automation Engineer hired (Dan)
- Status: in-progress - Improve Vagrant setup process (especially re browser tests) and clean up Ruby according to style guide
- Status: Done - VE browser tests in Vagrant
- Status: in-progress - Ruby style guide
Hiring[edit]
- Status: Done - Complete hiring and train new Release Engineer (Greg, all)
- Status: Done - Complete hiring and train new Automation Engineer (Ruby) (Chris, all)
- Status: in-progress - Complete hiring of a new QA Tester (Chris, Greg) - added mid-quarter
Quarterly Review Action items[edit]
- Status: Done - create a plan for browser testing of MediaWiki 1.23 - Chris M
- Status: Done - Greg get firm requirements from Antoine, circle back to Mark who'll have an idea of Op's timelime for production Shinken (monitoring system to maybe replace Icinga)
- no deadline knows on ops side. Might not be high priority
- ops in charge: Alexandros (Greece based)
- maybe use it in Beta Cluster as a POC
- Status: Done - Figure out how to keep HHVM unit tests from delaying +2 for standard production commits (Antoine, Chris M, Zeljko)
July/Aug Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: in-progress - WMF product should be consulted on some shellbugs
- Status: Done - Kickoff/complete second RFP
- Status: in-progress - Hire QA Tester (jobvite announce
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: hold - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists OK
- agreement by End of June -> slipped due to discussions
POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
- Status: Done - assist language team to have cxserver deployed to beta cluster
- They handle the rest with ops (puppet) and deployers (mediawiki-config)
Mukunda[edit]
- Status: in-progress - read all the things
Done - WMF SUL/OAuth support T40- Done - security component support in Phab Upstream T4893, our phab T95
- custom plugin for this, installed on fab.wmflabs, code is in gerrit
- being reviewed upstream, but won't be included in mainline Phab
- Done - Custom field to show mediawiki user page links on phabricator profiles
- Done - puppet wmflib beginnings
- Done - legalpad.mediawiki.org
- Done - Build debian package for mailparse php extension
- Status: Done - Honor security policies for file uploads/attachments - (Mukunda)
- Status: in-progress - data import, with Chase and Andre
Done - WMF SUL/OAuth support Bryan[edit]
- Status: In progress Scap refactor/python port
- Done bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
- Status: Not done - Integrate HHVM support into our deployment systems
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
Reedy=[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: Done - migrate to WMF Jenkins
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Status: in-progress - MobileFrontend broswer tests in Vagrant
- Status: Done - Improve Vagrant setup process (especially re browser tests)
- Status: in-progress - Ruby style guide
- Status: in-progress - Refactor MediaWiki-Vagrant according to Ruby style guide
June Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: in-progress - WMF product should be consulted on some shellbugs
- Status: Done - Successfully support the release of MediaWiki 1.23
- Status: in-progress - Kickoff/complete second RFP
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: Not done - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: Done - Integrate WMF Jenkins with new WMF SauceLabs account
- Status: In progress - RFC for Jenkins tests isolation
- wrote and published to ops and engineering lists
- agreement by End of June
- POC by End of July
- Status: Not done - RFC for cross repositories testing and integration job in gate
- Status: In progress - implementing cross repository cloner (aka have repositories match branches/zuul ref)
- Status: In progress - assist language team to have cxserver deployed to beta cluster
Mukunda[edit]
- Status: in-progress - read all the things
- Status: in-progress - WMF SUL/OAuth support T40
- Status: in-progress - security component support in Phab Upstream T4893, our phab T95
- local 6 line hack
- pinging evan regularly on real long term fix
- Status: in-progress - puppet wmflib beginnings
Bryan[edit]
- Status: In progress Scap refactor/python port
In progress bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
- Done Gerrit change 130560 Build JSON git info caches during scap
- Done Gerrit change 130498 support in GitInfo.php
- In progress Gerrit change 142319 Fix GitInfo cache file path computation and storage location (Needs code review and +2)
- In progress Gerrit change 142320 Set wgGitInfoCacheDirectory to point to scap managed location
- Done bug 66844 Fix sync-common for hosts in wikimedia.org TLD
- Status: Not done - Integrate HHVM support into our deployment systems
- Done Met with Emir from Facebook to discuss HHVM and deployment in general
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
- Done Helped with deployment training hangout
In progress Reedy[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: in-progress - migrate to WMF Jenkins
- Status: Done - create a plan for browser testing of MediaWiki 1.23
- https://bugzilla.wikimedia.org/show_bug.cgi?id=66272
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Chris Steipp had suggestions in email for further tests that map to legacy tests in /qa/browsertests repo
- After discussion with Rob Lanphier, will port appropriate tests from /qa/browsertests to /mediawiki/core with Dan Duvall over time
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.)
- Done for Mobile
- Done for VisualEditor
- Done for browser smoke tests
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Read all the things
- Status: Done - VE broswer tests in Vagrant
- Status: in-progress - MobileFrontend broswer tests in Vagrant
May Goal Progress[edit]
Greg[edit]
- Status: in-progress - Process through pain points
- Status: Done - config changes not being deployed after merge (RT ticket filed], successfully in Ops' court)
- Status: in-progress - WMF product should be consulted on some shellbugs
- email sent to WMF Product on 5/15
- Status: Done - onboard new Release Engineer
- ongoing
- Status: in-progress - Successfully support the release of MediaWiki 1.23
- Status: in-progress - Kickoff/complete second RFP
- Status: hold - Refine Deployment_tooling/Notes/Deployment_system_requirements with Bryan
- Status: Not done - Investigate and create useful release/deployment metrics visualizations
Antoine[edit]
- Status: Done - Figure out how to keep HHVM unit tests from delaying +2 for standard production commits
- Status: Not done - Support HHVM deployment tooling and puppet configuration testing
- Status: Not done - Integrate WMF Jenkins with new WMF SauceLabs account
- Status: In progress - RFC for Jenkins tests isolation
- Status: Not done - RFC for cross repositories testing and integration job in gate
Mukunda[edit]
- Status: in-progress - read all the things
- Status: in-progress - WMF SUL/OAuth support T40
- Status: Not done - security component support in Phab Upstream T4893, our phab T95
Bryan[edit]
- Status: In progress Scap refactor/python port
- Done Setup test environment in beta
- In progress bug 53972 Repository .git is synchronised fine, but is broken for submodules because of hardcoded gitdir link
- Done Gerrit change 130560 Build JSON git info caches during scap
- Done Gerrit change 130498 support in GitInfo.php
- Code is deployed in beta (and prod) but it doesn't work as expected. See notes on bug for where I'm stuck.
- Done Use trebuchet to deploy scap scripts
- Gerrit change 130211 Add scap/scap trebuchet target
- Gerrit change 129814 Provision scap scripts using trebuchet
- Done Gerrit change 134272 source organization cleanup
- Done Gerrit change 134730 Change default console logging verbosity (less noise for deployers)
- Done Convert sync-* to python
- Gerrit change 135512 Extract common base class for sync operations
- Gerrit change 135513 Convert sync-dir and sync-file to python
- Gerrit change 135514 Convert sync-dblist to python
- Gerrit change 135515 Convert sync-docroot to python
- Status: Not done - Integrate HHVM support into our deployment systems
- Status: Not done Next generation deployment tooling
- Status: Ongoing Support train deploys
- Done Created Train deploy instructions on wikitech based on noted from Sam and experience in March
Reedy[edit]
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
- Status: Done - create a plan for browser testing of MediaWiki 1.23
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Chris Steipp had suggestions in email for further tests that map to legacy tests in /qa/browsertests repo
- After discussion with Rob Lanphier, will port appropriate tests from /qa/browsertests to /mediawiki/core with Dan Duvall over time
- See https://gerrit.wikimedia.org/r/#/c/133507/
- Status: in-progress - Use tags to run builds appropriate to released versions (e.g. don't run master build on test2wiki)
- Antoine fleshing out a start of this idea at Zurich
- Status: Done - Use API to create test data at runtime more widely (not just for MobileFrontend but also VisualEditor, Flow, local dev env etc.)
- Done for Mobile
- Done for VisualEditor
- Done for browser smoke tests
- Status: in-progress - hire new QA Tester
Rummana[edit]
Željko[edit]
Dan[edit]
- Read all the things
Feb - Apr '14 Goal Progress[edit]
See the WMF Engineering 13-14 goals page for the yearly view.
- Process through all (useful) pain points from the Dev/Deploy review session (Greg)
- Scap incremental improvements
- 1) Refactor existing scap scripts to enhance maintainability and reveal hidden complexity of current solution (Bryan)
- 2) create matrix of tool requirements per software stack (MW, Parsoid, ElasticSearch) (Greg)
- 3) Use above matrix to add/fix functionality in scap (or related) tooling for ONE software stack, prioritized by cross stack use (Bryan)
- Use the API to create test data for given tests at run time. (Jeff, Chris, Željko)
- Create the ability to test headless (Željko, Jeff, Chris)
- Run versions of tests compatible with target test environments (Chris, all)
- Make database(s) in Beta Cluster emulate production (set up db slaves) (Antoine)
April Goal Progress[edit]
Greg[edit]
- Status: Done - Update/create new MW Release Management RFP
- Status: in-progress - Process through pain points
- Status: Done - security patches
- Status: Done - LD SWAT team
- Status: in-progress - config changes not being deployed after merge
- Status: in-progress - Refine deployment system requirements with Bryan
Antoine[edit]
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
- In progress Setup test environment in beta (carried forward from Feb & March; much harder than I'd hoped)
- Done Scap functionality available from deployment-bastion including fanout rsync and l10n rebuilds
- Done Jenkins job running scap after each beta-code-update job finished
- Done Gerrit change 127399 Remove NFS dependency from beta for MediaWiki deploys
- Done Gerrit change 126870 Convert hand-built Jenkins jobs to CI/JJB
- Done Gerrit change 129822 Use trigger publisher instead of trigger-builds builder (fixed deadlock problem on Jenkins slave)
- Continue to enhance and simplify scripts
- Done bug 63659 Gerrit change 124762 Compile wikiversions.json to cdb on deploy server
- Done Gerrit change 125334 Allow hyphen (-) in dsh hostnames
- Done Gerrit change 125359 Fix IRCSocketHandler to work from Jenkins
- Done Gerrit change 125515 Make logging destinations configurable
- Done Gerrit change 126924 Improve error message when scap lock fails
- Done Gerrit change 127198 Exit with non-zero status on soft failure (Helps Jenkins jobs identify partial failures)
- Done Gerrit change 127199 Convert scap-rebuild-cdbs to python
- Done Gerrit change 129577 Convert mw-update-l10n to python
- In progress Build .mw-git-info.json caches during scap and support in GitInfo.php
- Use trebuchet to deploy scap scripts
- In progress Gerrit change 130211 Add scap/scap trebuchet target
- In progress Gerrit change 129814 Provision scap scripts using trebuchet
- Status: In progress Next generation deployment tooling
- In progress Organize requirements into wiki page (carried forward from Feb & March; pairing with Greg)
- Status: In progress Support train deploys
Andre[edit]
- (With Guillaume) (Continued from March) Set up and facilitate the community RfC about Project management tools/Review, and bring it to a decision if possible. Status: Done - RfC started on 2014-04-14
- (With Guillaume) Have another IRC office hour about Project management tools Status: Done on 2014-04-17 and 2014-04-22
- Gather more feedback on rebooted Annoying little bugs after GSoC start Status: In progress - Andre sent an email to five GCI mentors on 2014-04-24 and received three answers
Prepare introducing "Bug of the week" by talking to development teamsStatus: Not done -- postponed to May- Have a bugday in the second half of April Status: Done: on 2014-04-29 about General MediaWiki, see Bug management/Triage/20140429
Chris[edit]
Continuing from Feb:
- Test and announce general availability of feature to check for ResourceLoader upon page load: ResourceLoader error checking more globally. Status: Done
- Test and announce general availability of feature to use the API to create wiki pages and users: create-page (and create-user) API. Status: Done
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Tracking this at https://bugzilla.wikimedia.org/show_bug.cgi?id=62509 . I have an idea about how to make this work. Status: In progress
Also:
- Update documentation on mw.o to reflect the refactoring and consolidation of recent times. Much of what exists now is long out of date. Status: In progress
Rummana[edit]
Željko[edit]
March Goal Progress[edit]
Greg[edit]
- Status: in-progress - Update/create new MW Release Management RFP
- Status: in-progress - Process through pain points
- Status: Done - security patches
- Status: Done - LD SWAT team
- Status: in-progress - config changes not being deployed after merge
- Status: in-progress - Refine deployment system requirements with Bryan
- Status: Done - create blockdiag version of flow chart
- LD SWAT team
- Status: Done - Kick off
- Status: Done - PushBot? - NO (without more dev than Greg can provide right now)
- Status: Done - quarterly post-mortem review
- See the review
Antoine[edit]
- Status: In progress Fix up VisualEditor browser tests and make it voting in Gerrit
- Status: In progress Makes CirrusSearch browser tests voting
- Done Publish sphinx documentation (use jenkins jobs). Examples:
- Status: In progress Migrate beta cluster from pmtpa to eqiad!
- Done Varnishes instances creation + puppet passing
- Done Application servers created + puppet passing
- Done Apache config in git (operations/apache-config.git branch: betacluster)
- Done Bunch of files / git repos copied from pmtpa to eqiad
- Done Sean Pringle to create MariaDB
- Done Ariel Glenn to set up the swift emulator (copy pasted instance)
- Done last minute sync of files
- Done add Jenkins slaves to sync Parsoid and MW code
- Done write jobs to update Parsoid and MW code on eqiad
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
- Done Extract common logic for making a command line interface into a class
- Done Fixed sync-wikiversions to use common dsh arguments
- Done Converted sync-wikiversions to python
- Done Invented scap-purge-l10n-cache script to cleanup l10n cache on unused branches
- Done Documented process for retiring a branch from tin
- In progress Setup test environment in beta (carried forward from Feb; harder than I'd hoped)
- Done Salt master in eqiad beta project
- Done Puppet master in eqiad beta project
- Done logstash host in eqiad beta project
Not done scap host in eqiad beta project (did this and then nuked it)
- Continue to enhance and simplify scripts
- Status: In progress Next generation deployment tooling
- In progress Organize requirements into wiki page (carried forward from Feb; pairing with Greg)
- Status: In progress Support train deploys
- Done Deploy 1.23wmf16 to group1
- Done Deploy 1.23wmf17 to group0
- Done Deploy 1.23wmf17 to group1
- Done Cleanup old l10n cache files on cluster
- Done Deploy 1.23wmf18 to group0
- Done Deploy 1.23wmf18 to group1
- Done Deploy 1.23wmf20 to group0
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- (With Guillaume) Have an IRC office hour about Project management tools Status: Done on 2014-03-28
(With Guillaume) Set up and facilitate the community RfC about Project management tools, and bring it to a decision.Status: Not done -- postponed to April- (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Kill curated list on the wikipage; provide specific Bugzilla queries and "getting started" instructions for each area: Status: Done on 2014-03-07 (diff)
Introduce "Bug of the week" after talking to teams?Status: Not done - postponed to Q2/2014
- Better Bugzilla taxonomy: Discuss using classifications in Bugzilla and identify some initial non-controversial changes. - bugzilla:38990. Status: In progress; first smaller cleanup steps about "Tools" and deprecated stuff are
bugzilla:57738, bugzilla:54063, bugzilla:53986#c15, bugzilla:55351#c4, bugzilla:62386, bugzilla:62384; Proposal to use Bugzilla classifications on 2014-03-10 - Bugzilla setup / code:
Provide a NEEDINFO flag in Bugzilla - bugzilla:36064. Status: In progress: (porting more complicated as bmo will skip 4.4) -- might get postponed to Q2/2014- Provide a way to mark an item as NEEDINFO in Bugzilla - bugzilla:36064. Status: In progress: Created list of pros and cons of two implementations on 2014-03-10; next step is start broader discussion on wikitech-l@ and agree on which way to do
Install component watching extension to be able to receive bugmail for specific product/component for devs and triagers, plus no more manual adding of people to auto-CC by admins- bugzilla:37105. Status: Not done -- database issues when testing; not yet tested on 4.4 and upstream code is still 4.2 only. Porting more complicated as bmo will skip 4.4 -- postponed to Q2/2014
Chris[edit]
Continuing from Feb:
- Test and announce general availability of feature to check for ResourceLoader upon page load: ResourceLoader error checking more globally. Status: Done
- Test and announce general availability of feature to use the API to create wiki pages and users: create-page (and create-user) API. Status: In progress
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Tracking this at https://bugzilla.wikimedia.org/show_bug.cgi?id=62509 . I have an idea about how to make this work. Status: In progress
Also:
- Update documentation on mw.o to reflect the refactoring and consolidation of recent times. Much of what exists now is long out of date. Status: In progress
Rummana[edit]
Željko[edit]
February Goal Progress[edit]
Greg[edit]
- Process through pain points Status: in-progress
- security patches Status: Done
- Kick off LD SWAT team Status: Not done
- next...
- create blockdiag version of flow chart Status: Done
- Refine deployment system requirements with Bryan Status: in-progress
- quarterly post-mortem kickoff Status: Not done
- set up morgue?
Antoine[edit]
- Status: In progress Complete integration of browsertests for VisualEditor
- One build worked and triggered two successful scenario!!
- Parsoid is stopped by the job now, was not previously :/
- Done Integration of browsertests for CirrusSearch. Cause me to slightly rethink the browsertests infra to closely match production.
- Done Train Zeljkof on Jenkins Job Builder script
- Zeljkof started on it. Will pair with him to finish up.
- Zeljkof found his way through JJB arcanes \O/
- Done Help migrating Cloudbees Jenkins template to JJB YAML templates
- Zeljkof started on it. Will pair with him to finish up.
- Zeljkof found his way through JJB arcanes \O/
Reedy[edit]
Bryan[edit]
- Status: In progress Scap refactor/python port
- Done Local test environment in a MW-Vagrant instance
- Done scap converted to python
- Done scap-1 converted to python
- Done add detailed duration timing for scap & scap-1
- Done remove external script dependencies from scap-1
- Done add progress bar for dsh commands
- Done Scap logs in json format for easy parsing
- Done Scap logs sent to florine via udp2log
- Done Scap logs sent to logstash via udp2log
- Done Converted mwversionsinuse to python
- Done Deleted obsolete scripts: scap-1, scap-2, find-nearest-rsync, scap-old
- In progress Setup test environment in beta (will pair with Antione)
- Continue to enhance and simplify scripts
- Status: In progress Next generation deployment tooling
- Done gather requirements via etherpad and Ops-l mailing list
- In progress Organize requirements into wiki page (will pair with Greg)
- Status: In progress Support train deploys
- Done Deploy 1.23wmf16 to group0
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - Status: Done. Outstanding steps from bugzilla:49597:
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Done on 2014-02-13
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: Done on 2014-02-13
- Finalize Bugzilla etiquette draft. Status: Done - 2014-02-05: Announcement
- Further small patches after 4.4 upgrade (License headers and template processing cleanup;
numerous contributed Gerrit patches waiting). Status: In progress - (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Cover common "getting started" questions: Status: Done added on 2014-02-18
- Improve Bugzilla queries on Annoying little bugs to show a good subset: Status: Done on 2014-02-18
- Add guidelines for triagers how to identify a bug report that is good for this audience: Status: Done by editing the "easy" keyword description on 2014-02-24
- Retriage existing bugs with easy keyword: mostly Status: Done
Chris[edit]
- Investigate how release branches are described in make-deploy-notes Status: In progress
- Investigate using ResourceLoader error checking more globally. Status: Done
- Refactor tests to use the create-page (and create-user) API. Status: Done
Željko[edit]
Jeff[edit]
Rummana[edit]
Dec-Feb Goal Progress[edit]
See the WMF Engineering 13-14 goals page for the yearly view.
- Status: in-progress - Browser tests managed in feature repos with feature teams (Chris, Zeljko, Jeff, Rummana)
- Status: Done - Successfully managed the first release of MediaWiki in conjunction with our outside contractor (Greg, Antoine)
- Status: Not done - More comprehensive quarter assessments of postmortems (Greg)
- Status: in-progress - Create process documentation for ideal test/deployment steps (Greg, Reedy, Chris, and others)
- Automated API integration tests in important areas (Chris, Zeljko, Jeff, Rummana, Antoine, also with Mark Holmquist)
- Status: in-progress - UploadWizard
- Status: in-progress - Parsoid / VisualEditor
- Status: Not done - ResourceLoader
Monthly Goal Progress[edit]
February[edit]
Greg[edit]
- do post Dev and Deploy process review follow up Status: in-progress
- post images to Commons - Status: Done - included in commons:Category:Wikimedia_Foundation_software_development
- send one email with list of grouped red-cards Status: Not done
- send off first email on first topic (make it a good one) Status: Not done
- Create plan to evaluate progress on postmortem/retrospective actions Status: in-progress
- archive/put on wikitech missing post-mortems - Status: in-progress
- create BZ whiteboard entry to track retrospective bugs - Status: Not done
- RT something or other?
Antoine[edit]
- Status: In progress Complete integration of browsertests for VisualEditor
- Status: In progress Train Zeljkof on Jenkins Job Builder script
- Status: In progress Help migrating Cloudbees Jenkins template to JJB YAML templates
Reedy[edit]
Andre[edit]
- (With Guillaume) Lead the community discussion about the shortlist of candidates for Project management tools/Review (based on Project management tools/Review/Requirements), take the first implementation steps (possibly in Labs). Status: In progress
- Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - Status: Done. Outstanding steps from bugzilla:49597:
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Done on 2014-02-13
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: Done on 2014-02-13
- Finalize Bugzilla etiquette draft. Status: Done - 2014-02-05: Announcement
- Further small patches after 4.4 upgrade (License headers and template processing cleanup;
numerous contributed Gerrit patches waiting). Status: In progress - (With Quim) Reboot Annoying little bugs based on Google Code-In experience; Bugzilla's easy keyword; exposing an easy "bug of the week":
- Cover common "getting started" questions: Status: Done added on 2014-02-18
- Improve Bugzilla queries on Annoying little bugs to show a good subset: Status: Done on 2014-02-18
- Add guidelines for triagers how to identify a bug report that is good for this audience: Status: Done by editing the "easy" keyword description on 2014-02-24
- Retriage existing bugs with easy keyword: mostly Status: Done
Chris[edit]
- Refactor tests to use API article creation Status: Not done
- Continue creating a suite of tests for local environments Status: Not done
- Requires the API gem in mediawiki-selenium 1.20
- Finish headless Xvfb integration Status: In progress
- In Jenkins, investigate pulling master branch for beta labs builds but pull release branch for test2wiki builds
- Follow up on ResourceLoader error reporting Status: Not done
Željko[edit]
Jeff[edit]
- New task: Create browser test for VisualEditor availability on production wikis Status: In progress
- Add browser tests for using VisualEditor via Mobile UI Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
January[edit]
Greg[edit]
- Prep for Dev and Deploy process meeting (Jan 22nd)
- Create baseline flowchart of dev/deploy documentation for use in the January in-person meeting - Status: Done
- refinements (ie: suggestions from Chris) - Status: Done
- write up final agenda/notes, share before meeting (clean up notes from Robla) - Status: Done
- make physical version of flowchart - Status: Done
- do post-review followup - Status: Not done
- Create plan to evaluate progress on postmortem/retrospective actions Status: Not done
- archive/put on wikitech missing post-mortems - Status: Not done
- create BZ whiteboard entry to track retrospective bugs - Status: Not done
Antoine[edit]
- For January: got to prepare myself for the MediaWiki summit (two weeks left + one week summit)
- Mostly focused on CI (Zuul upgrade, new jenkins jobs..) last week.
- Done Parsoid self update on beta cluster via a Jenkins job.
- See 'Parsoid update' on the CI dashboard https://integration.wikimedia.org/dashboard/
- Done Parsoid job migrated to new repos mediawiki/services/parsoid and mediawiki/services/parsoid/deploy
- Status: In progress VE browser tests from Gerrit. Waiting for them to be passing with (a fresh wiki + phantomjs)
- Done mediawiki/extensions.git out of sync since Jan 2nd bug 59758
Reedy[edit]
Andre[edit]
- (With Guillaume) Meet Project management tools stakeholders, determine requirements, and document this research. Status: Done by Guillaume here
- (With Daniel Zahn) Prepare Bugzilla upgrade to 4.4 (and move to new datacenter) - outstanding steps from bugzilla:49597:
- Upgrade Bugzilla on zirconium from 4.2 to 4.4. Status: Done
- Apply 4.2 custom changes to Bugzilla on zirconium from Git repository. Status: Done
- Apply Andre's 11 patches (linked here) in Gerrit to port our custom changes from 4.2 to 4.4. Status: Done on 2014-01-07
- Test Bugzilla on zirconium. Status: Done on 2014-01-15
- Make collectstats.pl work - bugzilla:29203. Status: Done by dzahn here on 2014-01-29
- (With Daniel and Sean) Switch database and DNS from old kaulen server to new zirconium server. Status: Not done - tentative date: 2014-02-12
- Show common queries on Bugzilla frontpage - bugzilla:22170. Status: In progress - 2014-01-10: Patch in Gerrit; depends on 4.4 upgrade first
- Finalize Bugzilla etiquette draft once the lively discussion on its Talk page has ended. Status: In progress - 2014-01-09: Announcement that discussion will be closed soon
- Fix inline displaying of image files in Bugzilla - bugzilla:54181. Status: Done - andre successfully tested csteipp's patch on Labs on 2014-01-05; deployed on 2014-01-10
- (With Quim) Finish running Google Code-In contest. Status: Done
Chris[edit]
- Bring about monitoring uploads in production with existing test, with failure messages emailed to QA staff Status: In progress Željko and Antoine to discuss hiding the password properly.
- Create a suite of local bare-wiki tests to use PhantomJS in WMF Jenkins Status: Not done
- https://bugzilla.wikimedia.org/show_bug.cgi?id=60347
- Only one existing test under/qa/browsertests passes in a bare wiki
- Use the API to create test data (e.g. a wiki page) on a target wiki Status: In progress Jeff working on https://gerrit.wikimedia.org/r/#/c/106548/ . Jeff and Chris paired Jan 24 to make progress. Commit is pending unexpected auth issue: https://bugzilla.wikimedia.org/show_bug.cgi?id=60407
- Continue contributing code and review to Mobile tests Status: Done paired with Arthur on https://gerrit.wikimedia.org/r/#/c/106833/, etc.
- Continue contributing code and review to Flow tests Status: Done Reported/fixed upstream Selenium bug , continuing...
- Monitor fatal errors in beta labs, send email alerts Status: Done
- Status: Done merged a monitor script, needs to be cron'd/puppet'd bug 60046
Željko[edit]
Jeff[edit]
- New task: add browser tests for using VisualEditor via Mobile UI Status: In progress
- Worked with Aaron Arcos to create initial browser test for MultimediaViewer repo Status: Done
- Creating scripts for adding new wiki users and articles via Mediawiki API Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
December[edit]
Greg[edit]
- Create baseline flowchart of dev/deploy documentation for use in the January in-person meeting - Status: Done
- Create plan to evaluate progress on postmortem/retrospective actions Status: in-progress
Antoine[edit]
- jenkins job for VisualEditor / Parsoid (reporting to James F / Gabriel Wicke)
- making sure a change in VE or in Parsoid does not break the other since they are tightly coupled
- Done parsoid init script doesn't play well when run over ssh
- Worked on, wrote an upstart job to wrap around parsoid server + logrotate configuration
- browser tests in Gerrit.
- Done First with ULS cause it is simpler, pairing with Zeljkof and i18n team
- Status: In progress Second MobileFrontend https://gerrit.wikimedia.org/r/#/c/97497/
- Status: Not done later VisualEditor
- Done polishing up the mw release tarball job (was bug 58278)
Slowdowns:
- deployed bunch of favicons for Google Code-in
- helped on gwtoolset, an extension to mass import materials from museum libraries
- bunch of CI changes to make jobs running in parralel
Reedy[edit]
Andre[edit]
- Google Code-In: Run and organize contest with Quim. Status: In progress
- Agree and finalize "etiquette" draft for behavior in Bugzilla, as discussed on teampractices@. Status: In progress - Discussion on Talk page still ongoing on 20131225 after asking for feedback on wikitech-l@.
- Evaluate Project management / issue tracking requirements and potential tools. Status: In progress - kicked off on teampractices@ and wiki on 2013-12-13
- Test Bugzilla 4.4 with our custom patches on Labs (or zirconium in eqiad if production is still on kaulen in Tampa). Status: In progress, will need help from ops. Steps discussed between dzahn and aklapper on 2013-12-06; dzahn has set up a copy of Bugzilla 4.2 on zirconium in eqiad (see RT #4783). Next steps are upgrading that machine to Bugzilla 4.4, applying our custom patches, testing, and finally switching over.
Chris[edit]
- Get everyone interested in API testing and monitoring pulling together: Multimedia + Jenkins + QA
- Status: Done Met with Mark Holmquist and Aaron Aarcos Dec 5
- Progress: https://bugzilla.wikimedia.org/show_bug.cgi?id=58555
- Test is merged and is pending adding to Jenkins https://gerrit.wikimedia.org/r/#/c/102603/
- Status: Done Met with Mark Holmquist and Aaron Aarcos Dec 5
- Move login method to shared code (Željko working on this right now), demonstrate improved pass rates pairticularly for IE
- Status: In progress In process, interim commit is https://gerrit.wikimedia.org/r/#/c/100579/
- Upgraded login method and corrected other issues affecting IE pass rate, builds are significantly more green, will pass on benefit to other repos when consolidated
- Status: In progress In process, interim commit is https://gerrit.wikimedia.org/r/#/c/100579/
- Concrete steps for Mobile QA after Michelle's departure
- Status: Done Chris to support automation effort, Chris/Jeff/Rummana to contribute as requested, e.g. post-deployment checking
- Status: Done Chris is monitoring the backlog of mingle cards for tests immediately. Next step is to collaborate with Mobile devs for regression tests like https://gerrit.wikimedia.org/r/#/c/103761/
- Browser test coverage for Flow
- Status: Done Coverage ongoing
- Status: Done Continuing working with S on refinements
Željko[edit]
Jeff[edit]
- Creating scripts for adding new wiki users and articles via Mediawiki API Status: In progress
- Continuing to work getting Jenkins (CloudBees) versions of VE automated tests from red to green Status: In progress
- Clean-up work on browser test for the "Nearby" page for the Mobile team Status: In progress
- Working on a request from Antoine to setup Visual Editor browser test triggers Status: In progress
- Continuing work on adding browser tests for outstanding VE regression items Status: In progress
Rummana[edit]
- Regular exploration testing on betalabs and test2
- Verifying each week's VE deployment
- Verifying resolved bugs
- Verifying new copy-paste implementation
- Track the already reported bugs on bugzilla and changing their status accordingly
Previous format Check-ins[edit]
- Wikimedia_Release_and_QA_Team/Checkin-20131119
- Wikimedia_Release_and_QA_Team/Checkin-20131105
- Wikimedia_Release_and_QA_Team/Checkin-20131008
- Wikimedia_Release_and_QA_Team/Checkin-20130924
- Wikimedia_Release_and_QA_Team/Checkin-20130917
- Wikimedia_Release_and_QA_Team/Checkin-20131022
- Wikimedia_Release_and_QA_Team/Checkin-20130910
- Wikimedia_Release_and_QA_Team/Checkin-20130903
