Wikimedia Discovery/Meetings/Portal retrospective 2016-04-22

Discovery Portal Retrospective

2016-04-22

Covering whatever has happened related to the team since the last retro (2016-03-23)
 * Quarter ended; new quarter started
 * Process for deciding on new quarterly goals
 * Full month without Julien
 * Deb and Jan traveling
 * Fixed a bunch of bugs
 * Ran another test
 * Brainstorm/roadmap meeting
 * Moiz begins his transition out
 * Weeklong code freeze due to codfw migration
 * Started using new portal backlog workboard

Review action items from before

 * Jan: Document "production quality", beta site, demo sites, etc.
 * Generally in progress, but work related to this task:
 * Portal A/B Testing wiki has been updated with a link to the 'archive version' of the A/B test, as well as a link to the analytics results. https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing
 * Created a url to show 'in-progress' work to team: https://people.wikimedia.org/~jdrewniak/in-progress/index.html
 * Kevin: Try to unstick quick surveys
 * ✅, although I'm not sure if they are re-stuck or just on hold. Either way, they are no longer the responsibility of the portal team.
 * Agreed - the work done by the Portal team is done for the time being. All survey tickets have been moved to the Portal Backlog board

What went well?

 * Ran the localized top-links A/B test
 * Some delays turning it on and turning it off
 * Showed actual user impact
 * Went smoother than earlier tests
 * Working with Julien on code review is going well
 * Jan has done very well with the tasks assigned and delving into upcoming tasks and research +1+1
 * Great analysis by Mikhail - he's been busy with lots of other tests and work +1+1
 * Good conversations (that Deb had) with Legal, Comms, and User Testing teams about upcoming work +1
 * Also, Edward was very helpful with qualtrics
 * Portal work (small bug fixes) in Tech News, weekly status updates, mailing lists. (good communication) +1+1
 * Reminder: Work that seems small to you might still be worth publicizing
 * Good collaboration with User Research team
 * Portal Brainstorming meeting was good - lots of new ideas that can be acted on in the future+1+1
 * Communication between Portal engineers and analysis team+1
 * Team has gelled and supported each other well through some organizational turbulence

What could we improve?

 * A/B Test roll out process
 * We have improved, but could still be improved more
 * Checklist for all the things needed to complete a test
 * We do have a checklist which is very valuable; always room for improvement
 * Testing checklisst: https://meta.wikimedia.org/wiki/Discovery/Testing
 * Look at how to avoid past problems in the future (retro for each test?)
 * Would be cool to know how long each step took, so we would know where to optimize
 * Maybe sit down with FR team, because they run tests very quickly (e.g. 30 minutes)
 * They are less busy right now
 * Maybe talk with search team about their decision to build relevance forge, as a way of putting things in front of users to get feedback
 * The time it takes us to run an A/B Test
 * If all our work is going to be tested, we need to get really good at doing it
 * Some delays were due to developers being new; some was rework of hasty prototype quality code
 * Last test didn't run into those problems, but still seemed to take a while. Not sure why.
 * Portal doesn't have a rich framework like mediawiki does, so have to invent more
 * Reviews (code, security) does take a while. Seems to be getting shorter.
 * Need to be careful because any change affects so many people
 * Perhaps we could get people to review mocks in some cases rather than running a full test
 * Survey should've been out by now, disconnect between user research/design and product
 * UR/D came up with a survey they felt good about; when Deb saw it, it didn't align with her expectations
 * We should include product manager in meetings of what we're trying to do with a survey (earlier in the process)
 * The email thread has helped; we will probably do multiple surveys
 * Now facing some unexpected technical issues to implement what UR wants to do
 * Future portal meeting was underwhelming (i can provide more context)
 * Was hoping for more excitement about big ideas and ongoing work
 * We need to apply resources where the benefit/cost is highest, so regularly re-evaluating each area
 * Portal team lost months just getting the code into git
 * Team is hitting its stride this quarter, but we still have a lot to learn about what we should be doing
 * Design stuff has been slow last two weeks, sorryyy.. New mocks coming today :)
 * Moiz shouldn't leave!

What else should be noted?

 * Two surveys will probably be run on the Portal in the coming month(s)
 * requires close coordination with User Testing team, Legal and Portal teams
 * two surveys might be done - one to hit up all users after a period of inactivity on the page, the other one to talk directly with users that bounced previously from the portal and engage with a survey.

Action Items

 * Jan: Create phab task to wrap up doc of beta/demo/production sites
 * Jan: Do a post-mortem on what technical tasks actually took the longest time to complete.


 * Deb: create rough mocks of some of the ideas for the future work and add to page on wiki ro further discussion
 * Deb: set up meeting with Creative Tech (Megan) to go over their rapid testing process (Deb, Jan, Mikhail)
 * Deb: Create ticket for pushing successful lang links / localized phrase to production on portal
 * Kevin: Consider some light form of retro after next A/B test