Scrum of scrums/2017-10-11

= 2017-10-11=

contact: https://www.mediawiki.org/wiki/Wikimedia_Engineering

Callouts

 * Are there any special steps to wire a MediaWiki LoggerFactory into logstash-beta? AWight is unable to find some logs, which are present on fluorine-beta.
 * CommTech would appreciate help reaching consensus regarding the way preferences are handlled: https://lists.wikimedia.org/pipermail/wikitech-l/2017-October/088980.html
 * DBAs have disabled Wikidata RC syndication on Commons and ruwiki, and are working on purging existing WD RC rows on those wikis.
 * This is because of performance and table size reasons. https://phabricator.wikimedia.org/T171027
 * MediaWiki 1.31.0-wmf.3 didn't go out yesterday, we are getting caught up today. https://phabricator.wikimedia.org/T174359

iOS native app

 * Blocked by: none
 * Blocking: none
 * Updates:
 * Continuing work on 5.7.0 ( https://phabricator.wikimedia.org/project/view/2899/ ) - Visual updates to onboarding, iOS 11 support, iPhone X support

Android native app

 * Blocked by: none
 * Blocking: none
 * Updates:
 * On schedule to release beta of E-release this week ( https://phabricator.wikimedia.org/project/view/2839/ )
 * Will start (local) testing of Reading List service.

Discovery
and cleaning up old code.
 * Continuing work on portal automation, resolving dependencies that fail to build on some systems

Maps

 * Blocked by: none
 * Blocking: none
 * Updates:
 * Got some help debugging nodejs issues
 * Community is activating on Spanish Wikipedia

Reading Infrastructure

 * ReadingList RESTBase patch under review; hoping to deploy both RESTBase and MediaWiki part to beta soon
 * MCS: accounting for Parsoid changes:  instead of for inline figures (https://phabricator.wikimedia.org/T177301)

Web

 * Turning off OCG [cc. ops]
 * Working on replacement for OCG

Parsing

 * cscott's ping to wikidata team: https://gerrit.wikimedia.org/r/#/c/379669/ -- Parsing is not blocked on this per se, but this is something that cscott has been contributing to as part of a longer term project. Using SoS to ping wikidata on a stalled review for him.
 * Updates
 * Parsing team offsite next week -- will be absent at next week's SoS.
 * Parsoid code to use the updated Templatedata format now merged (suggested by Thiemo at Wikimania 16). It is in the beta cluster and needs testing before being deployed.
 * We are likely to merge both the wrapper code as well as the  code at the same time so that we can bump the HTML version number once.
 * Looks like cxserver/CX has patches to handle section output. Is MCS ready to handle Parsoid's section output? C.Scott said he is going to add a patch for VE to handle this, but, if VE folks want to tackle that on your own, please do so.

Language

 * Compact Language Links out of Beta deployment being scheduled next week for German Wikipedia.
 * Work continue with VE-CX integration and adpatation.
 * cxserver has been updated to latest master, debugging issues with Beta cxserver.

Collaboration

 * New filters on RC now the default experience on all wikis except some FlaggedRevs wikis
 * Looking into Watchlist/ORES slow query issues, but DBAs are also working on that

UI Standardization

 * Blocked: icon feedback appreciated, see below / none
 * Updates:
 * Improved whitespace usage in OOUI HTMLForms https://phabricator.wikimedia.org/T177668
 * OOUI v0.23.4 release in preparation
 * Generalized icon/indicator padding/positioning https://phabricator.wikimedia.org/T161177 & streamlined icon/indicator visibility and layout https://phabricator.wikimedia.org/T160593
 * Ongoing:
 * OOUI:
 * icons: Work on icon set to be more harmonious and align to WikimediaUI Style Guide's guidelines https://phabricator.wikimedia.org/T177432
 * Aligning arbitrary, historically grown colors across products to WikimediaUI color palette https://phabricator.wikimedia.org/T148708

Community Tech

 * Blocked: not blocked, but see our callout
 * Blocking: none
 * Report:
 * Working on GlobalPreferences
 * Parser caches are almost populated with Unicode IDs

Analytics

 * No blocker (AFAIK)
 * Not blocked
 * Updates:
 * Wikistats 2.0 UI is alpha-level with reading metrics, still missing edit metrics because of missing backend
 * Setting up a production data cluster for druid with public data to be the edit-data public APIs backend. We have split our Druid cluster in two, with 3 hosts each.
 * Getting Druid Grafana dashboards in place.
 * Working on processing eventlogging data in hadoop instead of MySQL
 * Still purging data in MySQL eventlogging hosts, slow going
 * Work to move metrics to prometheus continues.

Scoring Platform

 * Blocked by: none
 * Blocking:
 * Still working on file handle exhaustion, which prevents us from using the new ORES cluster.
 * Updates:
 * Deploying "revscoring 2.0", which could cause some instability in Extension:ORES and in the ORES service.

Search Platform

 * Blocked by: none
 * Blocking: none
 * Updates:
 * Messaging fallbacks usage for search disabled (https://phabricator.wikimedia.org/T147959) - actual switch happens when we reindex
 * MLR assisted results test analysis published: https://wikimedia-research.github.io/Discovery-Search-Test-InterleavedLTR/, results generally positive
 * Patch for indexing wikidata statements merged, will index P31 and P279 initially after deployment
 * Working on porting Selenium tests from Ruby to JS
 * Working on upgrade to Elastic 5.5
 * Working on indexing Wikidata descriptions and adding them to fulltext search

Services

 * Blocked: none
 * Blocking: none
 * Updates:
 * Deploying cassandra 3 in beta cluster
 * Starting testing Parsoid with Cassandra 3

Technical Operations

 * Blocked:
 * none
 * Blocking:
 * none
 * Updates
 * Singapore caching DC ongoing
 * Quite a bit of puppet refactoring for varnishes
 * ganglia slowly being killed (nobody will probably even notice)

Fundraising Tech

 * Still trying to figure out db lag, have some leads: https://phabricator.wikimedia.org/T173472
 * Still trying to figure our missing data from kafkatee, getting some help: https://phabricator.wikimedia.org/T177331
 * More CiviCRM contact de-duplication improvements
 * Fixing Civi import bugs: https://phabricator.wikimedia.org/T177806, https://phabricator.wikimedia.org/T177803
 * Cleaned up contact groups in Civi to improve performance
 * Diagnosing apparent missing banner impressions (big chunks seem to be bot-related): https://phabricator.wikimedia.org/T176802

Wikidata

 * (Hopefully) unblocked QUnit 2 update: https://phabricator.wikimedia.org/T177764
 * We had to turn of RecentChanges integration on big wikis for performance reasons, looking into workarounds: https://phabricator.wikimedia.org/T177707
 * Fixed a regression we believe happened because of a jQuery update: https://phabricator.wikimedia.org/T177485
 * We turn JS-only components into pure NPM libraries: https://phabricator.wikimedia.org/T177087
 * Constraint checks also on qualifiers and references: https://phabricator.wikimedia.org/T176863
 * Added new Lua functions to the mw.wikibase library to avoid pulling entire entities: https://phabricator.wikimedia.org/T142903, https://phabricator.wikimedia.org/T173262
 * Full URIs for external identifiers will soon be in the RDF export: https://phabricator.wikimedia.org/T121274
 * Optimizing runtime of the wikidata.org dump generation: https://phabricator.wikimedia.org/T177486
 * wb_entity_per_page table is not updated any more: https://phabricator.wikimedia.org/T140890
 * wb_terms table on labs does have a term_full_entity_id column now: https://phabricator.wikimedia.org/T167114
 * As usual, we continue working on the Lexeme extension for Wiktionary: https://phabricator.wikimedia.org/T168263

RelEng

 * Blocked: none
 * Blocking: none?
 * Updates:
 * wmf.3 cut this morning, back on track today group0 and group1

Security

 * Reviews:
 * Ex:FileImporter (T160982)
 * Ex:WikipediaExtracts (T149424)
 * pdfrw