Wikimedia Discovery/Meetings/Analysis retrospective 2016-11-08

Format
Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Previous action items

 * Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
 * Include a statement of the perceived problem
 * Still needs to be done
 * Deb: Figure out what to do with the Maps Dashboard
 * ✅ - fixed and updated
 * Mikhail: Think about involving GL in sys admin stuff
 * Need to meet w/GL

What has happened since the last retro (2016-09-30)
(using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point)
 * Analysis of WDQS geographic breakdown of SPARQL queries
 * Wikipedia.org Portal Dashboard: investigate spike in languages visited
 * Wikipedia.org dashboard: minor updates
 * Bug in current data retrieval script
 * Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes
 * Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager
 * Completed analysis of the results of BM25 AB test (final report)
 * Search results page: how many visitors are on mobile vs desktop -
 * Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
 * Investigate recent spike in pageviews on wikipedia.org portal page
 * Add a PaulScore approximation to discovery.wmflabs.org (waiting on phab:T138087 to be deployed on the train)
 * Compare ZRR for query features across other search engines
 * Investigate what we'd need to do to ignore double quotes in search queries
 * Maps Dashboard: fix and update
 * Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
 * Add a PaulScore approximation to discovery.wmflabs.org

What has made you mad?

 * Interactions with Interactive team
 * e.g. we created a dashboard and have made many improvements on it since its deployment, but the Interactive team does not seem to use it and even cannot remember the URL at times
 * so being asked to learn a bunch of new technologies and do a bunch of additional work to make data available in a different data store so the team can do their own dashboarding is frustrating when current resource isn't being utilized

What has made you sad?

 * Frustration with working with the Interactive team - trying to decide what is needed for metrics
 * Lack of integration between Analysis and Interactive
 * Chelsy and Mikhail are a resource for Interactive to use, and they are being underused

What has made you glad?

 * Chelsy is a super fast learner and has done terrific work on her reports+1
 * Coordination with Search team on test metrics/analysis of tests - quick results
 * Updating the dashboards to be much better
 * M & C learning new things about R and the operations stack to get new and cool technology into our environment
 * Nice to be able to learn during work time
 * Watching this after a tough election night: http://i.imgur.com/lZEeZ8z.gifv
 * Kevin's involvement in Analysis; this team is better for having a TPG person+1+1

What has made you plaid? (aka whatever didn't fit in one of the other categories)

 * There are several IP addresses have been doing weird things to us, e.g. pinging portal million times per day, sending the same search queries thousands of times per day. I know we are generally not blocking any of them, but wondering if we should keep a black list for ourselves, in order to remember to check them when we are doing related analysis so that we can filter out polluted data.
 * Maybe a greylist rather than blacklist, just to identify for reporting?
 * Publicizing the list wouldn't be appropriate (privacy issues)
 * Some cases would be a judgment call; might look useless to us, but maybe they're getting value
 * and...this: http://i.imgur.com/sd15NVu.gifv

Discussion
Interactive team interactions
 * KS: I can see 3 areas: 1) ED-level of what metrics should be tracked, 2) tech level of whether to use grafana or other, 3) inter-team interactions and social component
 * They seem to be making progress on deciding what metrics to track
 * Interactive product folks (Deb/Yuri) should do some early thinking, then bring in analysts to review. Doesn't seem to be going that way so far
 * For tech, there's a ticket to get data into grafana. That's frustrating because we have a dashboard system that we're really happy with, but they won't use it; want us to do additional work and learning to support this other system
 * Our dashboards can do some things that grafana can't
 * Unclear WHY
 * Interactive team feels strongly about doing things their own way. That's why they chose the tech, and why they implemented their own metrics without consultations.
 * Can be frustrating to have expertise available to them that they don't take advantage of
 * Deb in semi-PM role is trying to help them with this
 * Discovery dashboard has all the vertical stuff in one place, consistent UI, etc. There are also valid reasons for having grafana.
 * Was hoping that after we updated the graphs, they would work for that team. Yuri wants direct immediately control over dashboard changes, which they get with grafana.
 * Not clear that grafana will be able to produce output that will be useful for demonstrating KPIs to management and other stakeholders
 * How big a project is grafana support?
 * Kind of a big project
 * Feels like a personal slight to not use our dashboards, but learning grafana would be a useful skill
 * Even if the interactive team might not appreciate the maps dashboards, other people will
 * The Discovery dashboards need to track the work of the whole team
 * Concern that Yuri might try to figure out KPIs and metrics in isolation
 * Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
 * Some optimism. There is a ticket out there.
 * Analysis team tries to be proactive and forward-looking; interactive team tends to be more quick with less forward planning

Action items

 * Mikhail: Discuss opsy stuff w/GL (and Chelsy)
 * Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
 * Include a statement of the perceived problem