Wikimedia Developer Summit/2018/Lessons Learned

The Wikimedia Developer Summit 2018 feedback survey was sent out to participants on 6 February, 2018.

We will close the survey on 20 February, 2018.

Results / summary of the feedback survey are being posted here, currently this page is incomplete.

---

Feedback survey background
rfarrand created the feedback survey. Many members of the Organizing Team made edits or suggested changes.

The feedback survey was sent to participants / wikitech-l, Engineering@ & WMFAll@ on 6 February, 2018.

A reminder about the feedback survey was sent out to all participants who had not filled out the survey (and those who choose to remain anonymous) on 13 February.

The feedback survey closed on 20 February.

The feedback survey received 47 responses

Privacy Statement for this survey

Considerations for next time
This section is based on the fill-in-the-blank/comment sections of the feedback form. Some of these comments will be contradictory. We are mostly trying to include common themes and issues that were felt by groups of people. This section is a bit more subjective than the #Data section below and suggestions may be paraphrased and combined with each other.

In this section comments at the top of each section were mentioned more often while comments at the bottom were mentioned by one person. Some comments are combined and many are shortened for easier summary. The future program committees will review full comments again in the future when they are working on specific areas of the event organization.

Please provide ideas for how we can improve the pre-event engagement of participants.

 * Most commented: More details / guidance / clarity on expected outcomes much earlier in the process. Discussions should also start earlier.
 * Multiple people: Everyone should feel some responsibility. Each person should have a task / stake in a presentation / distributing the burden instead of the session leader(s) doing most of the preparation on their own.
 * Multiple people liked: Small focused reading list / something to be done in limited time.
 * Two people: uniform pre-event engagement across the different sessions / framework of things to do while preparing the sessions.
 * Two people: Don’t expect pre-event engagement from participants
 * Be clear in Phab tasks on what conversation should be done on Phab and what should be saved for the event. / Are all attendees (not just WMF) on Phab?
 * Discussions should not take place on Phabricator / its not ideal for a long discussion
 * Have the slide decks ahead of the summit / everyone reads the slide decks ahead of the summit.


 * Structured debates on something like Kialo would be great. Something that clears up a debate and prevents people from re-hashing the same argument over and over. We need some technology help.
 * You definitely need experienced people to help pre-coordinate. Be careful that the leaders/experienced ones don't dictate the plan.
 * Have someone present a thesis and have a debate on it
 * The selection through essay writing felt rushed and unnatural. Anonymous character of it forced you to concentrate on writing something that will be liked as opposed to what you really think.
 * Session organizers should communicate how they wish to conduct their sessions and what they expect from participants.
 * I think as the outcome of all the sessions was somewhat standardized, a standard set of methods to get to these outcomes would have been helpful.
 * Identify 1 question that should be discussed beforehand / discussing in the phab ticket without a clear focus is not very effective
 * Send weekly emails to the participants about engaging / create expectations

Any suggestions on how to improve session facilitation?

 * Stay away from topics that provoke a great deal of impassioned debate but go nowhere
 * Consider dedicated facilitators for larger discussions so all participants can contribute to the discussion
 * Several people thought that the session facilitation went well
 * Have more facilitators like Joel
 * Don't have any session facilitators at all
 * Have a Gatekeeper that asks people who haven’t contributed (verbally) to the session to speak up (to avoid the pitfall of only a few strong voices talking)
 * A requirement that each session produce a list of recommendations for the Foundation, and that the would be a process afterward to consider each recommendation (reject, or accept, or modify)
 * Set up equipment day before before sessions starts to avoid AV problems etc
 * Establish a consistent format for all sessions up front (in communications) and during the introductions
 * Have list a few one or two word topics and the ones voted most popular are the ones discussed
 * Have a list of topics that relate to the overall session and time box each topic
 * Discuss the position papers themselves, not just the topics that the leaders decided on
 * Felt it was weird to invite "outsiders" (non-WMF employees) and not specifically ask for their ideas and feedback
 * It seemed passive and unclear how any outside perspective would be useful
 * More focused sessions and a topic leader with a concrete proposal
 * Keep session format consistent across sessions
 * Have topic "chairs" who review talk/discussion proposals to be more productive
 * It did not seem clear on how much moderation was expected from topic leaders
 * Topic leaders and facilitators should connect earlier with each other and discuss the agenda, group activity methods and their roles during the session

Any suggestions on how to improve topic leader preparation?

 * Create focus groups and make them by invitation only, if the overall goal is to have specific outcomes
 * Establish a common baseline of knowledge for every topic (what works, what hasn’t worked, what’s possible, etc)
 * Research the best practices in "how to organize a brainstorming session" and provide it to topic leaders to aid in better facilitation
 * Have a clear plan or goal or proposal to be discussed before coming to the conference, and not try to come up with something ad hoc
 * Needed to have a clearer idea of what people are interested in and have concrete topics for each leader to cover (something like - “your session will focus on answering these 3 questions”)
 * The purpose of the sessions are to select and prioritize. All attendees should be prepared for each session (read the position paper in advance) to distill the ideas into something workable while everyone is together in person
 * Go back to a more traditional format with one presenter and lots of discussion afterward
 * Don't have topic leaders; have the attendees decide what to talk about
 * Provide enough time in advance for the topic leaders to prepare and get more involvement of attendees; there wasn’t enough time this year due to holidays
 * Give topic leaders time to sync their sessions with other shared topics (and leaders)
 * Provide a standardized toolkit of best practices and methods

Process Questions
Ideas for improvement (related to 2018 participant selection process)

Please suggest topic ideas for 2019 position statements

 * Writing generally useful libraries and services (how, why, and are we doing this already?)
 * What kinds of APIs should MediaWiki and the Foundation have?
 * How to disentangle presentation from data in MediaWiki (without destroying Wikipedia)
 * Focus on the Wikimedia Platform Evolution, MediaWiki and it’s ecosystem
 * Tech debt: MediaWiki core, unmaintained extensions, gadgets, frontend and backend mechanisms, how to create and keep better documentation, code stewardship, sunsetting processes, future of Labs
 * Have specific tracks (not just discussions) for: MediaWiki, operational infrastructure, security, research and analytics, contributors and readers, developer support and community
 * Three things we are doing well and three things we are doing poorly
 * Improving our engineering culture
 * Several people (6+) did not want to have any position statements
 * Sessions to track our progress (starting from some year in the past) in Tech
 * Sessions on phase 2 of the strategy movement, it’s execution and the expected outcomes
 * Focus on the year 2030 and what the future could hold for us (not what’s going on right now)
 * Change up the format - product manager’s present a topic for 20 minutes, then 45-50 minutes for going into more detail, discussions and Q&A
 * Language or translation related topics
 * How will Wikipedia evolve to remain relevant in 15 years
 * What can the Foundation learn from 3rd-party and volunteer wiki developers with their "enterprise" use cases
 * How to contribute to UX research; teach how to do basic usability testing, review basic UX best practices, and ideas for future interfaces
 * Developer wishlist
 * Interview people that don’t contribute to Wikipedia right now to figure out why and what it would take for them to start contributing
 * Outcomes, recommendations, and action items from the 2018 sessions (focusing on the “how”)

What did you find helpful or confusing about the way we used Phabricator/wikipages/etherpads for organizing the work before and during the summit?

 * Wasn’t clear if (and how much) pre-event discussion was encouraged or desired
 * There wasn’t much discussion of topics in Phab (except by topic leaders) before the event
 * Felt that the discussion on Phab should have been to request answers to a small amount of questions and then go over that in the sessions
 * Quite a few folks said the usage was fine / good / helpful to be using our normal tools
 * Format was easy to find, get involved with, and were well structured
 * Tracking progress of the sessions and the discussions was very helpful
 * Input from the Phabricator pages didn’t always get to be part of the sessions
 * Would be great to only use a wiki instead of etherpads (one day)
 * Helpful to document everything but some session notes were too verbose
 * A recording (voice or video) would be just as good and with less effort (than taking notes in etherpad)
 * All three are useful as long as the scope is clear and are cross-linked
 * Phab wasn't as useful as it could be - many tasks weren’t filled in or had too much background reading
 * Better usage of Phabricator would be for detailing the action items
 * Lots of clicks required to find the relevant info - felt that it was too spread out
 * The tools were the least duplicative (from past years)
 * Very useful to have etherpads in advance of the conference - made taking notes easy
 * Too many phab tickets made it hard to navigate
 * It wasn’t clear how the notes would be transferred to wiki after the conference
 * Unsure what the final (after the conference) outcome would be - how would it feed into the 3-5 year strategy process
 * Have a ‘catch up on documentation’ time slot at the end of the conference
 * Wiki is great for providing information about scheduling, but the positions statements should have been all in Phabricator
 * The real-time aspect of the etherpad note taking plus the deliberative aspect of Phabricator and the wiki summarizing worked well

Please provide ideas for how we can improve the event session format

 * Sessions were too large (too many people in each)
 * Sessions should be a max of 2 hours (otherwise, split into smaller sessions)
 * There was not enough time during breaks for informal discussions
 * Social gatherings were too noisy to continue informal discussions
 * If a person’s position statement was accepted, have them lead / talk during the sessions
 * Position papers didn’t always “fit” into the session topics
 * Add lightning talks / unconferences
 * Keep the fireside chat
 * Product managers present their FY goals and their 3-5 year plan at the beginning of each session and then do hour long slots during the day to go over those plans
 * Keep keynotes to less than 30 minutes
 * Keep the wrap-up session at the end of each day for a good recap
 * ‘next steps for languages and cross project collaboration’ and ‘MediaWiki architecture’ programs were too big for one session
 * Some sessions were lacking in a clear purpose or goal and were too broad in subject
 * Base the participants selected on merit, not their position statements
 * Don’t break up the sessions over a break or for keynotes (keep them about 1 1/2 hours each)
 * No more than 2 keynotes a day
 * With fewer tracks and less participation, it felt like it resulted in less getting discussed
 * Create a small event of it’s own for discussions around machine learning
 * Better prepare the topic leaders for their session and to be in sync with other topics
 * Make sure all the AV equipment works
 * Facilitation was good

Do you have a suggestion for a specific keynote speaker for 2019? (Please provide their name, and affiliation / background)

 * Two survey respondents asked not to have keynotes at all
 * Tim Starling, Wikimedia Foundation
 * Don Knuth
 * Someone from GitHub
 * Something related to the events theme
 * Someone from Mozilla
 * Dries Buytaert, founder and lead developer of the Drupal CMS
 * Somebody technical, not politics/social justice
 * John Gilmore (EFF board member)
 * Zeynep Tufecki
 * StackOverflow people like Joel Spolsky or Jeff Atwood, people working in large open source projects like Drupal, Symfony, PHP, Node, developers or managers from other wikis like Wikia / Wikihow, Clay Shirky or another social computing researcher.

Do you have suggestions for specific topics you’d like the keynote speakers to cover for 2019?

 * Multiple people say that they should be technology based / not about social justice
 * Multiple people say wikimedia focused or tailored to our community specifically
 * Two people say: no keynotes at all
 * Two people say: People from other large open source organisations with similar goals / solving similar problems
 * We had to many keynotes
 * Theory and practice of operating a FLOSS project; Collaboration at scale; Promoting successful bottom up planning
 * Wikipedia use under authoritarian regimes
 * More Wikipedia related Machine Learning.
 * Someone who has improved the code quality and organization of the software
 * Someone who has dramatically reformed APIs or developer-test-prod VMs/rigs.
 * Knowledge equity. How do we spread knowledge to marginalized people? Can marginalized people come talk to us about it? How do they see the world, what do we need to do to meet them where they are?
 * The psychology behind different personalities, what drives different people to participate in wikis or other collaboration.
 * Software freedom in the context of the internet services era.
 * External speakers on tech debt and documentation.
 * Developer support & community
 * The role of an ecosystem for an open source software
 * Consensus-building and governance challenges of open-source software,
 * Social computing and the ways software influences community health and cultural norms,
 * An analyst / (reasonable) futurologist talking about internet and technology trends.
 * Boring technology.

Do you have suggestions for specific topics that you would like to hear about from WMF management at the 2019 summit?

 * some of technical challenges they had in 2018-2019 and how they addressed them
 * thoughts on some of the technical challenges ahead of us, and what we should do to overcome them.
 * how exactly are the funds distributed, and how can the developers influence in the allocation of resources for a small project. i.e. things that a single developer can do by herself
 * Integration of DevSummit outcomes with WMF Annual Plan and resource allocation. How they interact within each other and which one influence and guide the others.
 * A clear direction on third party operated MediaWiki and the appropriate degree of investment as a Wikimedia Foundation.
 * Insight into what and how we should build for interactive agents (voice, chatbots), the direction for A/V+VR and the tie-in to mobile computing and sensors/embedded.
 * A clear direction on our approach for channel expansion while ensuring growth in first party usage across WMF/WMDE-maintained channels.
 * I would like some clarity on our mission. According to T&C's cultural orientation, our technology is a means to an end. However, most people at the dev summit do not agree with this.
 * Specific examples of trade-offs we make, and why. How do we choose to focus on work?
 * How can we be more dynamic and collaborate more across teams?
 * A broad (still concrete) roadmap.
 * Implementation plans for the outcomes of the last summit.
 * How the Stewardship review process will be incorporated into your planning.
 * Less management involvement.
 * WMF management should prepare something, *anything* to the level they prepare a statement to the board.
 * How to manage tech debt
 * I would love to hear/talk about building the ecosystem and tech partnerships within and beyond the movement. But we shouldn't ask that the WMF management alone.
 * Mostly what they hope to get out of the summit.
 * What management did in 2018 to promote MW as a tool outside of WMF and inside WMF. Talk about how MW encapsulates the culture of WM in code.

Are there other things that we should be measuring that we are not asking about?

 * Post-summit impacts / effects / evaluation of outcomes / productivity
 * What changes have you seen since past Dev Summits? / How many sessions from previous years had any of the resolutions carried out, or blocked by lack of resources.
 * Interaction with other developers
 * Who was missing?
 * How can we measure success with such large generic topic areas?
 * Ask people who didn’t apply why they choose not to attend?
 * Ask me what I took away from this event -- why my attendance was important.
 * Did anything useful come out of the event itself or was it just because of the people attending?
 * Did anyone who did not attend gain anything either from reading documentation or following along?
 * Challenges that resulted from dividing staff (not allowing some at this event)

Data
Did you attend the Wikimedia Developer Summit 2018 in Berkeley?

Yes: 61.7%

No: 38.3%

Have you attended any previous Wikimedia Developer Summits and/or Architecture Summits organized by WMF in 2017 or earlier?

Yes: 87.2%

No: 12.8%

Questions for Participants
Omitted: How useful for you were the keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

Omitted: How useful for you were WMF management keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

How useful was each session for you?

Response options are ranked in order after giving 2 points for every "very useful" response, 1 point for every "useful" response, 0 points for every neutral responses, -1 points for every not useful response and dividing by the number of responses.
 * Growing the MW technical community: 17 responses
 * Supporting 3rd party use of MW: 16 reponses
 * Embracing open source software: 20 responses
 * Knowledge as a service : 16 responses
 * Research Analytics and Machines Learning: 11 responses (tied with next steps for language)
 * Next steps for language and cross project collaboration: 11 responses (tired with research and analytics)
 * Advancing the contributor experience: 15 responses
 * Evolving the MW Architecture: 22 responses

Please tell us how much you agree or disagree with the following statements

Attending this event was worth my time

Strongly Agree: 10

Agree: 10

Neither Agree or Disagree: 5

Disagree: 1

Strongly Disagree: 3

The opportunity to meet fellow developers was valuable to me

Strongly Agree: 17

Agree:

Neither Agree or Disagree: 2

Disagree: 2

Strongly Disagree: 1

I would like to attend this event again next year

Strongly Agree: 14

Agree: 7

Neither Agree or Disagree: 5

Disagree: 2

Strongly Disagree: 1

Process Questions
Based on your understanding of the process, do you think asking participants to write a Position Statement was fair?

Yes: 29.8% (14)

Indifferent: 8.5% (4)

No: 31.9% (15)

I don't know: 6.4% (3)

Other: 11 participants

Based on your understanding of the process, do you think the participant selection process was fair?

Yes: 21.3% (10)

Indifferent: 8.5% (4)

No: 14.9% (7)

I don't know: 46.8% (22)

Other: 4 participants

Documentation Questions
Please rank these tools in order of how helpful they were to you and your topics of interest in following the actives and outcomes of the Wikimedia Developer Summit 2018

Phabricator

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Etherpad

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Wiki

Very helpful: 14

Moderately Helpful: 0

Not Helpful: 5