Wikimedia Developer Summit/2018/Lessons Learned

The Wikimedia Developer Summit 2018 feedback survey was sent out to participants on 6 February, 2018.

We will close the survey on 20 February, 2018.

Results / summary of the feedback survey are being posted here, currently this page is incomplete.

---

Feedback survey background
rfarrand created the feedback survey. Many members of the Organizing Team made edits or suggested changes.

The feedback survey was sent to participants / wikitech-l, Engineering@ & WMFAll@ on 6 February, 2018.

A reminder about the feedback survey was sent out to all participants who had not filled out the survey (and those who choose to remain anonymous) on 13 February.

The feedback survey closed on 20 February.

The feedback survey received 47 responses

Privacy Statement for this survey

Considerations for next time
This section is based on the fill-in-the-blank/comment sections of the feedback form. Some of these comments will be contradictory. We are mostly trying to include common themes and issues that were felt by groups of people. This section is a bit more subjective than the #Data section below and suggestions may be paraphrased and combined with each other.

In this section comments at the top of each section were mentioned more often while comments at the bottom were mentioned by one person. Some comments are combined and many are shortened for easier summary. The future program committees will review full comments again in the future when they are working on specific areas of the event organization.

Questions for Participants
Please provide ideas for how we can improve the pre-event engagement of participants.

Any suggestions on how to improve session facilitation?

Any suggestions on how to improve topic leader preparation?

 * Create focus groups and make them by invitation only, if the overall goal is to have specific outcomes
 * Establish a common baseline of knowledge for every topic (what works, what hasn’t worked, what’s possible, etc)
 * Research the best practices in "how to organize a brainstorming session" and provide it to topic leaders to aid in better facilitation
 * Have a clear plan or goal or proposal to be discussed before coming to the conference, and not try to come up with something ad hoc
 * Needed to have a clearer idea of what people are interested in and have concrete topics for each leader to cover (something like - “your session will focus on answering these 3 questions”)
 * The purpose of the sessions are to select and prioritize. All attendees should be prepared for each session (read the position paper in advance) to distill the ideas into something workable while everyone is together in person
 * Go back to a more traditional format with one presenter and lots of discussion afterward
 * Don't have topic leaders; have the attendees decide what to talk about
 * Provide enough time in advance for the topic leaders to prepare and get more involvement of attendees; there wasn’t enough time this year due to holidays
 * Give topic leaders time to sync their sessions with other shared topics (and leaders)
 * Provide a standardized toolkit of best practices and methods

Process Questions
Ideas for improvement (related to 2018 participant selection process)

Please suggest topic ideas for 2019 position statements

What did you find helpful or confusing about the way we used Phabricator/wikipages/etherpads for organizing the work before and during the summit?

 * Wasn’t clear if (and how much) pre-event discussion was encouraged or desired
 * There wasn’t much discussion of topics in Phab (except by topic leaders) before the event
 * Felt that the discussion on Phab should have been to request answers to a small amount of questions and then go over that in the sessions
 * Quite a few folks said the usage was fine / good / helpful to be using our normal tools
 * Format was easy to find, get involved with, and were well structured
 * Tracking progress of the sessions and the discussions was very helpful
 * Input from the Phabricator pages didn’t always get to be part of the sessions
 * Would be great to only use a wiki instead of etherpads (one day)
 * Helpful to document everything but some session notes were too verbose
 * A recording (voice or video) would be just as good and with less effort (than taking notes in etherpad)
 * All three are useful as long as the scope is clear and are cross-linked
 * Phab wasn't as useful as it could be - many tasks weren’t filled in or had too much background reading
 * Better usage of Phabricator would be for detailing the action items
 * Lots of clicks required to find the relevant info - felt that it was too spread out
 * The tools were the least duplicative (from past years)
 * Very useful to have etherpads in advance of the conference - made taking notes easy
 * Too many phab tickets made it hard to navigate
 * It wasn’t clear how the notes would be transferred to wiki after the conference
 * Unsure what the final (after the conference) outcome would be - how would it feed into the 3-5 year strategy process
 * Have a ‘catch up on documentation’ time slot at the end of the conference
 * Wiki is great for providing information about scheduling, but the positions statements should have been all in Phabricator
 * The real-time aspect of the etherpad note taking plus the deliberative aspect of Phabricator and the wiki summarizing worked well

Please provide ideas for how we can improve the event session format

 * Sessions were too large (too many people in each)
 * Sessions should be a max of 2 hours (otherwise, split into smaller sessions)
 * There was not enough time during breaks for informal discussions
 * Social gatherings were too noisy to continue informal discussions
 * If a person’s position statement was accepted, have them lead / talk during the sessions
 * Position papers didn’t always “fit” into the session topics
 * Add lightning talks / unconferences
 * Keep the fireside chat
 * Product managers present their FY goals and their 3-5 year plan at the beginning of each session and then do hour long slots during the day to go over those plans
 * Keep keynotes to less than 30 minutes
 * Keep the wrap-up session at the end of each day for a good recap
 * ‘next steps for languages and cross project collaboration’ and ‘MediaWiki architecture’ programs were too big for one session
 * Some sessions were lacking in a clear purpose or goal and were too broad in subject
 * Base the participants selected on merit, not their position statements
 * Don’t break up the sessions over a break or for keynotes (keep them about 1 1/2 hours each)
 * No more than 2 keynotes a day
 * With fewer tracks and less participation, it felt like it resulted in less getting discussed
 * Create a small event of it’s own for discussions around machine learning
 * Better prepare the topic leaders for their session and to be in sync with other topics
 * Make sure all the AV equipment works
 * Facilitation was good

Do you have a suggestion for a specific keynote speaker for 2019? (Please provide their name, and affiliation / background)

 * Two survey respondents asked not to have keynotes at all
 * Tim Starling, Wikimedia Foundation
 * Don Knuth
 * Someone from GitHub
 * Something related to the events theme
 * Someone from Mozilla
 * Dries Buytaert, founder and lead developer of the Drupal CMS
 * Somebody technical, not politics/social justice
 * John Gilmore (EFF board member)
 * Zeynep Tufecki
 * StackOverflow people like Joel Spolsky or Jeff Atwood, people working in large opensource projects like Drupal, Symfony, PHP, Node, developers or managers from other wikis like Wikia / Wikihow, Clay Shirky or another social computing researcher.

Do you have suggestions for specific topics that you would like to hear about from WMF management at the 2019 summit?

 * some of technical challenges they had in 2018-2019 and how they addressed them
 * thoughts on some of the technical challenges ahead of us, and what we should do to overcome them.
 * how exactly are the funds distributed, and how can the developers influence in the allocation of resources for a small project. i.e. things that a single developer can do by herself
 * Integration of DevSummit outcomes with WMF Annual Plan and resource allocation. How they interact within each other and which one influence and guide the others.
 * A clear direction on third party operated MediaWiki and the appropriate degree of investment as a Wikimedia Foundation.
 * Insight into what and how we should build for interactive agents (voice, chatbots), the direction for A/V+VR and the tie-in to mobile computing and sensors/embedded.
 * A clear direction on our approach for channel expansion while ensuring growth in first party usage across WMF/WMDE-maintained channels.
 * I would like some clarity on our mission. According to T&C's cultural orientation, our technology is a means to an end. However, most people at the dev summit do not agree with this.
 * Specific examples of trade-offs we make, and why. How do we choose to focus on work?
 * How can we be more dynamic and collaborate more across teams?
 * A broad (still concrete) roadmap.
 * Implementation plans for the outcomes of the last summit.
 * How the Stewardship review process will be incorporated into your planning.
 * Less management involvement.
 * WMF management should prepare something, *anything* to the level they prepare a statement to the board.
 * How to manage tech debt
 * I would love to hear/talk about building the ecosystem and tech partnerships within and beyond the movement. But we shouldn't ask that the WMF management alone.
 * Mostly what they hope to get out of the summit.
 * What management did in 2018 to promote MW as a tool outside of WMF and inside WMF. Talk about how MW encapsulates the culture of WM in code.

Are there other things that we should be measuring that we are not asking about?

 * Post-summit impacts / effects / evaluation of outcomes / productivity
 * What changes have you seen since past Dev Summits? / How many sessions from previous years had any of the resolutions carried out, or blocked by lack of resources.
 * Interaction with other developers
 * Who was missing?
 * How can we measure success with such large generic topic areas?
 * Ask people who didn’t apply why they choose not to attend?
 * Ask me what I took away from this event -- why my attendance was important.
 * Did anything useful come out of the event itself or was it just because of the people attending?
 * Did anyone who did not attend gain anything either from reading documentation or following along?
 * Challenges that resulted from dividing staff (not allowing some at this event)

Data
Did you attend the Wikimedia Developer Summit 2018 in Berkeley?

Yes: 61.7%

No: 38.3%

Have you attended any previous Wikimedia Developer Summits and/or Architecture Summits organized by WMF in 2017 or earlier?

Yes: 87.2%

No: 12.8%

Questions for Participants
Omitted: How useful for you were the keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

Omitted: How useful for you were WMF management keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

How useful was each session for you?

Response options are ranked in order after giving 2 points for every "very useful" response, 1 point for every "useful" response, 0 points for every neutral responses, -1 points for every not useful response and dividing by the number of responses.
 * Growing the MW technical community: 17 responses
 * Supporting 3rd party use of MW: 16 reponses
 * Embracing open source software: 20 responses
 * Knowledge as a service : 16 responses
 * Research Analytics and Machines Learning: 11 responses (tied with next steps for language)
 * Next steps for language and cross project collaboration: 11 responses (tired with research and analytics)
 * Advancing the contributor experience: 15 responses
 * Evolving the MW Architecture: 22 responses

Please tell us how much you agree or disagree with the following statements

Attending this event was worth my time

Strongly Agree: 10

Agree: 10

Neither Agree or Disagree: 5

Disagree: 1

Strongly Disagree: 3

The opportunity to meet fellow developers was valuable to me

Strongly Agree: 17

Agree:

Neither Agree or Disagree: 2

Disagree: 2

Strongly Disagree: 1

I would like to attend this event again next year

Strongly Agree: 14

Agree: 7

Neither Agree or Disagree: 5

Disagree: 2

Strongly Disagree: 1

Process Questions
Based on your understanding of the process, do you think asking participants to write a Position Statement was fair?

Yes: 29.8% (14)

Indifferent: 8.5% (4)

No: 31.9% (15)

I don't know: 6.4% (3)

Other: 11 participants

Based on your understanding of the process, do you think the participant selection process was fair?

Yes: 21.3% (10)

Indifferent: 8.5% (4)

No: 14.9% (7)

I don't know: 46.8% (22)

Other: 4 participants

Documentation Questions
Please rank these tools in order of how helpful they were to you and your topics of interest in following the actives and outcomes of the Wikimedia Developer Summit 2018

Phabricator

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Etherpad

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Wiki

Very helpful: 14

Moderately Helpful: 0

Not Helpful: 5