Scrum of scrums/2018-01-10

Callouts

 * Search PF would like somebody from performance/MediaWiki team to look at: https://phabricator.wikimedia.org/T183101 we’ve got some fails on LinksUpdate which blocked search index update which makes Wikidata items invisible in search (~50 items)
 * [Perf] We've been looking at it, but we haven't identified a cause for this issue yet.
 * [SP] I guess we will just watch out for it and hope it was an one-time thing.


 * [Perf] Trying to compile a list of everything running in Prod that requires Zend PHP5. Currently only aware of the XML dump job (has syntax errors in output if run under HHVM).  Anything else, please reach out to Ian (email: imarlier@, irc: marlier)
 * Reminder: Whitelist your EventLogging schema if you want data older than 90 days to be retained, talk to the Analytics team if you're not sure what this means

iOS native app

 * Blocked by: none
 * Blocking: none
 * Updates:
 * Continuing work on 5.8 - synced reading lists

Android native app

 * Blocked by: Patch to unblock DELETE endpoints in varnish: https://gerrit.wikimedia.org/r/402433 DONE!
 * Blocking: none
 * Updates:
 * Completed work to integrate with the new Reading List service! Continuing to test and measure performance.

Reading Infrastructure

 * Blocked by: ops on https://gerrit.wikimedia.org/r/#/c/402433/ <= this is already merged since 2018-01-09
 * Blocking: none
 * Updates:
 * working on request batching in reading list service
 * continuing work on WP0 piracy
 * Summary endpoint switchover from TextExtracts to PCS implementation planned for Wednesday
 * wrapping up media and references endpoints
 * metadata and CSS endpoints to follow

Web

 * Wrapping up a rework of the mobile web beta/settings page.
 * Nothing new to report with regards to challenges/deliverables

Maps

 * Blocked by: Services team
 * Blocking: None
 * Updates
 * Everyone is back from vacation
 * Several issues waiting on service team

Multimedia

 * Nothing to report

Parsing

 * Sole update: Approaching additional wikis for replacing Tidy on those wikis on Jan 31st.

Global Collaboration

 * Blocked: Security on https://phabricator.wikimedia.org/T144467
 * Blocking: Ops on Flow dumps. Matt's back from sabbatical but has been sick off and on over the last week

Community Tech

 * Investigating new year's proposals
 * Will file for security/performance/DBA review for GlobalPreferences today

UI Standardization
Also we significantly reduced SVG file size for our interface elements by up to 21% with fine-tuned markup optimizations.
 * OOUI v0.25.0 released https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md
 * 6 breaking and 5 deprecating changes, and 16 UI related improvements, among those
 * Drop the `constructive` flag entirely
 * Remove `BookletLayout#getClosestPage`
 * Ongoing:
 * OOUI & based products:
 * icons: Unify, refine and align to WikimediaUI Style Guide https://phabricator.wikimedia.org/T177432 – first patches in: https://gerrit.wikimedia.org/r/#/c/402757/
 * More optimizations/size savings for RL embedded SVGs as data URIs

Technical Operations

 * Blocking:
 * None
 * Blocked:
 * Global Collaboration on Flow. https://phabricator.wikimedia.org/T172025
 * Updates
 * s5 => s8 migration for wikidata completed successfully. Feel free to reach out to DBAs for followup questions and/or details
 * meltdown/spectre week, ops busy with the mitigations, do expect some slower response times
 * This is a big hardware refresh quarter. Work is already decided and prioritized, but please do provide input when/if requested ASAP. Thanks

Release Engineering

 * Blocking
 * None?
 * Blocked
 * ops: zuul package update (blocks gerrit upgrade)
 * ops: node-tunnel-agent package update (blocks moving node testing to docker in ci)
 * Updates
 * 2 weeks of normal MediaWiki deploys (this and next) followed by 2 weeks of no MediaWiki but SWATs as needed (DevSummit/All Hands followed by RelEng team offsite)
 * Currently building nightlies of Mediawiki on the new “releases” (aka non-CI) Jenkins host. Working with Security on best way to handle security patches (which is the goal, to ensure security patches stay cleanly applicable).

Performance

 * Blocking: None
 * Blocked: None
 * Updates
 * As noted above, we've been investigating LinksUpdate failures, haven't been able to identify a cause at this point, haven't been able to reproduce.
 * Fixes to Thumbor implementation happening as issues are reported (no major ones at this point)
 * Published the first in a series of blog posts detailing how we measure performance: https://phabricator.wikimedia.org/phame/post/view/83/measuring_wikipedia_page_load_times/
 * Trying to get a list of everything running in Prod that requires PHP5 (Zend) -- only thing that I know of for sure is the XML dump job.

Security

 * No updates
 * Sam and Brian are traveling this week
 * Darian's last day is January 15th

Fundraising Tech

 * Blocking: None
 * Blocked: None
 * Updates:
 * Upstreaming stuff to Civi
 * Making our Amazon Pay SDK fork support TCP proxy
 * Re-starting work on new API for our main credit card processor
 * Stats projects:
 * Andrew Green's druid banner impressions lib: https://github.com/AndrewGreen/centralnotice_analytics
 * Jack Gleeson's stats collector:https://github.com/jackgleeson/stats-collector
 * More consideration of how the get banner stats to payments in realtime (talking with Analytics)
 * Also need to talk with analytics about preserving aggregated banner stats for December

Scoring Platform

 * Blocking: no
 * Blocked: no
 * Updates
 * We added Extension:ORES to in-house code coverage reporting :D
 * https://doc.wikimedia.org/cover-extensions/ORES/index.html
 * https://scrutinizer-ci.com/g/wikimedia/mediawiki-extensions-ORES/

Analytics

 * https://superset.wikimedia.org You can now create dashboards on top of druid data, no automatic user creation. Created users for PMs and analysts, let us know if you want a user. Perf to be seen.
 * Clickstream data now public. Research community has access to the paths readers follow through the site (aggreggated counts, not per user)
 * Eventlogging purging on master box almost done.
 * Working on wikistats UI bugs and small features to make project selection more friendly, once the UI tweaks are done we will leave this work on backburner until next year.
 * Work on new set of APIs to return pageviews per country continues. Privacy study of this data: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews/Pageviews_by_country
 * Meeting with Asaf for our next big project: geowiki (editors per country, quite sensitive data)


 * Mostly quiet (holidays)

Search Platform

 * Blocked by: none
 * Blocking: none
 * Updates:
 * Improving LTR training mechanisms (https://phabricator.wikimedia.org/T184547)
 * Working on fixes for completion suggester & redirects namespaces (https://phabricator.wikimedia.org/T115756)
 * Investigating ElasticSearch phonetic search (https://phabricator.wikimedia.org/T182708)
 * Working on refactoring search profiles to make them more config-like https://phabricator.wikimedia.org/T183279
 * Processing feedback for Wikidata fulltext & prefix search rankings
 * Running test for machine-learning ranking on Hebrew wiki (https://phabricator.wikimedia.org/T182616)
 * Discussing publishing data from WDQS query logs (https://phabricator.wikimedia.org/T143819)

Wikidata

 * Gaining speed after the holidays.
 * Working on Lexeme continues, now working on diffs: https://phabricator.wikimedia.org/T182424
 * Working on Quality Constraint integration continues, 1.0 rollout planned for Q1.
 * Preparing for Developer Summit.
 * Planning collaboration with Multimedia (MediaInfo) as well as Search for Q1.

MediaWiki Platform

 * Blocked by: Wikibase on https://phabricator.wikimedia.org/T183019
 * Blocking: N/A
 * Updates:
 * Tim: Spectre/Meltdown review
 * Multi-Content Revisions:
 * Actor table: 380669 needs code review
 * Working on https://phabricator.wikimedia.org/T36925 and https://phabricator.wikimedia.org/T182678
 * Comment table:
 * Schema changes are almost done. Then we can start trying to use it.
 * cleanupUsersWithNoId:
 * enwiki finished
 * Hopefully dewiki and wikidatawiki can be run this week
 * ExternalStore de-PHP-serialization: Patch done before vacation. But Jaime raised some interesting discussion on https://phabricator.wikimedia.org/T183419 that may determine how we go about updating ExternalStore
 * Code coverage:
 * Added a bunch of @covers tags in core, we also validate them pre-merge now (see giant spike in https://tools.wmflabs.org/coverage/mediawiki/chart.png )
 * Implemented coverage reports for extensions (https://phabricator.wikimedia.org/T71685, https://doc.wikimedia.org/cover-extensions/ ), still in a beta period, will announce as ready for usage end of this or next week once https://phabricator.wikimedia.org/T182749 is implemented and documented
 * Brainstorming/working on proof of concept for pre-merge coverage reports that don't take hours to generate
 * Audiences Technology Working Group planning
 * Developer Summit planning

SoS meeting

 * ggellerman has ongoing conflicts, looking for another facilitator every other week
 * Check in on goals here, talk about cross-team dependencies in this meeting, rather than waiting until the end of each quarter.