Wikimedia Developer Summit/2018/Lessons Learned

The Wikimedia Developer Summit 2018 feedback survey was sent out to participants on 6 February, 2018.

We will close the survey on 20 February, 2018.

Results / summary of the feedback survey are being posted here, currently this page is incomplete.

---

Feedback survey background
rfarrand created the feedback survey. Many members of the Organizing Team made edits or suggested changes.

The feedback survey was sent to participants / wikitech-l, Engineering@ & WMFAll@ on 6 February, 2018.

A reminder about the feedback survey was sent out to all participants who had not filled out the survey (and those who choose to remain anonymous) on 13 February.

The feedback survey closed on 20 February.

The feedback survey received 47 responses

Privacy Statement for this survey

Considerations for next time
This section is based on the fill-in-the-blank/comment sections of the feedback form. Some of these comments will be contradictory. We are mostly trying to include common themes and issues that were felt by groups of people. This section is a bit more subjective than the #Data section below and suggestions may be paraphrased and combined with each other.

In this section comments at the top of each section were mentioned more often while comments at the bottom were mentioned by one person. Some comments are combined and many are shortened for easier summary. The future program committees will review full comments again in the future when they are working on specific areas of the event organization.

Please provide ideas for how we can improve the pre-event engagement of participants.

 * Most commented: More details / guidance / clarity on expected outcomes much earlier in the process. Discussions should also start earlier.
 * Multiple people: Everyone should feel some responsibility. Each person should have a task / stake in a presentation / distributing the burden instead of the session leader(s) doing most of the preparation on their own.
 * Multiple people liked: Small focused reading list / something to be done in limited time.
 * Two people: uniform pre-event engagement across the different sessions / framework of things to do while preparing the sessions.
 * Two people: Don’t expect pre-event engagement from participants
 * Be clear in Phab tasks on what conversation should be done on Phab and what should be saved for the event. / Are all attendees (not just WMF) on Phab?
 * Discussions should not take place on Phabricator / its not ideal for a long discussion
 * Have the slide decks ahead of the summit / everyone reads the slide decks ahead of the summit.


 * Structured debates on something like Kialo would be great. Something that clears up a debate and prevents people from re-hashing the same argument over and over. We need some technology help.
 * You definitely need experienced people to help pre-coordinate. Be careful that the leaders/experienced ones don't dictate the plan.
 * Have someone present a thesis and have a debate on it
 * The selection through essay writing felt rushed and unnatural. Anonymous character of it forced you to concentrate on writing something that will be liked as opposed to what you really think.
 * Session organizers should communicate how they wish to conduct their sessions and what they expect from participants.
 * I think as the outcome of all the sessions was somewhat standardized, a standard set of methods to get to these outcomes would have been helpful.
 * Identify 1 question that should be discussed beforehand / discussing in the phab ticket without a clear focus is not very effective
 * Send weekly emails to the participants about engaging / create expectations

Any suggestions on how to improve session facilitation?

 * Stay away from topics that provoke a great deal of impassioned debate but go nowhere
 * Consider dedicated facilitators for larger discussions so all participants can contribute to the discussion
 * Several people thought that the session facilitation went well
 * Have more facilitators like Joel
 * Don't have any session facilitators at all
 * Have a Gatekeeper that asks people who haven’t contributed (verbally) to the session to speak up (to avoid the pitfall of only a few strong voices talking)
 * A requirement that each session produce a list of recommendations for the Foundation, and that the would be a process afterward to consider each recommendation (reject, or accept, or modify)
 * Set up equipment day before before sessions starts to avoid AV problems etc
 * Establish a consistent format for all sessions up front (in communications) and during the introductions
 * Have list a few one or two word topics and the ones voted most popular are the ones discussed
 * Have a list of topics that relate to the overall session and time box each topic
 * Discuss the position papers themselves, not just the topics that the leaders decided on
 * Felt it was weird to invite "outsiders" (non-WMF employees) and not specifically ask for their ideas and feedback
 * It seemed passive and unclear how any outside perspective would be useful
 * More focused sessions and a topic leader with a concrete proposal
 * Keep session format consistent across sessions
 * Have topic "chairs" who review talk/discussion proposals to be more productive
 * It did not seem clear on how much moderation was expected from topic leaders
 * Topic leaders and facilitators should connect earlier with each other and discuss the agenda, group activity methods and their roles during the session

Any suggestions on how to improve topic leader preparation?

 * Create focus groups and make them by invitation only, if the overall goal is to have specific outcomes
 * Establish a common baseline of knowledge for every topic (what works, what hasn’t worked, what’s possible, etc)
 * Research the best practices in "how to organize a brainstorming session" and provide it to topic leaders to aid in better facilitation
 * Have a clear plan or goal or proposal to be discussed before coming to the conference, and not try to come up with something ad hoc
 * Needed to have a clearer idea of what people are interested in and have concrete topics for each leader to cover (something like - “your session will focus on answering these 3 questions”)
 * The purpose of the sessions are to select and prioritize. All attendees should be prepared for each session (read the position paper in advance) to distill the ideas into something workable while everyone is together in person
 * Go back to a more traditional format with one presenter and lots of discussion afterward
 * Don't have topic leaders; have the attendees decide what to talk about
 * Provide enough time in advance for the topic leaders to prepare and get more involvement of attendees; there wasn’t enough time this year due to holidays
 * Give topic leaders time to sync their sessions with other shared topics (and leaders)
 * Provide a standardized toolkit of best practices and methods

Process Questions
Ideas for improvement (related to 2018 participant selection process)

(This was the toughest section to summarize, we had many long and well thought out responses. We shortened the responses to be more readable and broke many of them into multiple parts to fit into the different sections. We are making a new proposal based on the main themes here and will make sure anyone who ends up on the program committee is able to see the full and complete responses people submitted to this section.)

Invitation Position statements unfair / bring the wrong people Current process OK Position statement process Topics Who should be there No event Other
 * A few people suggested renaming the event:  Re-titled the "Wikimedia Foundation technical planning summit" / strategy planning summit, call it a strategy planning summit rather than a "developer" summit that most developers don't get to attend.
 * Invitation only event:  (1) in practice the event would probably have mostly the same people present as this year; (2) specific invitations could help mitigate groups opting out entirely;
 * The program committee to take responsibility and define topics to be covered (maybe in a bottom-up fashion). They then invite people who run the sessions for these topics and in collaboration with these people create a list of people who will be invited to attend the event. Who will get invited will depend on the outcome desired by each session and people's expertise.
 * Liked blind review process. Need more continuity in the participants list. Invite the topic leaders of this summit to attend the next summit, so that we can ensure that we actually build upon work. / Ensure the exchange with product managers (both from wmf and wmde).
 * Position statement are unfair: expectations were very ambiguous and the position statements were essentially unused both in the actual conference and in the planning of it. I am not opposed to the essay contest format in principle, but if we force people to write them, then they should definitely be used.
 * Someone's always going to be unhappy, but maybe try and make sure that it's not always the same person. Position papers are fine, but select for the type of person that's adept at writing position papers.
 * Position papers seemed like writing an essay for a college course, but with a competitive nature. So I had to both figure out what to say, and how to say it well. I failed to do either. I've never been good at English, and this is supposed to be a *developer* summit. Your quintessential developer sits in a dark corner and codes. Communication is not our thing, especially when presented as a competition, of sorts. If we don't understand the strategy, we're not welcome, or at least that's how it felt.
 * The summit as conceived this year was basically business meetings to determine future direction (tactical implementation of strategy). This is a very poor fit for an essay contest as there's no guarantee the right people will be in the room to discuss these things as you need stakeholders (i.e. people with decision making power) to have a fruitful discussion on tactics.
 * If we're going to limit the number of participants to be smaller than our core developer community, we should stop thinking of this as a "developer summit" and instead call it something like an "insiders-only summit". There seemed to be a pressure this year to enforce structure and control on the event that was unnecessary. We ended up having a few dead-end conversations rather than many more conversations where only some of which lead to a dead-end.
 * For an anonymous selection process, I don't think I can come up with a fairer solution.
 * Position statements and anonymous ranking is great. Limit the length of the statements for two reasons: Help scale the event to hundreds of applicants without crushing the committee with paperwork. Make it easier to read the position statements of all the participants as a participant.
 * The position paper process was great / I liked how you attempted to group them.
 * Position statement were good but the summit should provided the space to talk about them. Strategy will not be relevant at the next summit.
 * I think position statements worked. There are a fair number of senior people in the organization who did not submit position statements. I feel like diversity was given importance which is GREAT. It's really important to have diverse voices in these important conversations.
 * Position statement process needs to be clear, transparent, and valuable.
 * Make it clearer what we expect in the position statements (ie not a 5 page paper)
 * It would be useful to have better clarity as to how position statements  are going to be matter. Also, there should be some mechanisms for non-attendees to follow / watch / participate or weigh into the discussions. There are engineers and others that can contribute very useful input and feedback and context even if they don't or cannot write position statements.
 * Allow everyone to vote on participants, or just those who submitted the position statements.
 * Similar process to before but first identify voices that should be represented. Position statements should be about what perspectives are being brought to the table, who are you representing, etc.
 * I think we should have a mixed process: One that gives seats to those who are making the decisions right now (within the WMF and WMDE) in addition to an open process for others to apply. Bring all Annual Plan program owners from Audiences and Technology (or their delegates). Add on to that a "tell us why you should be there" process.("I'm someone/my team is implicated heavily by the topics discussed and decisions being made and I have expertise that would be helpful but I don't have a specific position on a topic that I'd like to shepherd at the event") persona. That persona was notably absent from this year and it shows.
 * A submission system may be fine, but it should somehow be more technically oriented, and not a structured essay. Maybe a Q&A system, through Google Forms?
 * Time preparing the position statements but very little factored into actual session discussion. Some position statements were high level, and others had detail. Themes and topics were picked based on position statements, but other they were not used.
 * Turn the summit a drafting space for annual plan proposals. More open proposal process. People could be asked to write (possibly in groups) a summary of the proposals beforehand / select the ones that generate the most interest.
 * Do not ask for position papers, ask for presentation abstracts on topics people are field experts in. If the number of participants has to be reduced that much, have a scientific committee pick people based on relevance.
 * If you want to keep the size limited, I suppose continuing make it more C/architect-level friendly will reduce size, as the average developer sees little value proposition in attending.
 * Topics would be separate from position statements chosen by participants and leadership.
 * If you're looking for more ideological, 15+ year ideas, or more practical implementation ideas then the summit needs to change to support broader ideological discussions that might challenge some long and strongly held beliefs.
 * Solicit arguments for and against certain hot topics. One of the overall problems I saw in this meeting was that everyone had good ideas, but there were few rebuttals. When there were, it still didn't seem like we had made progress on down-selection.
 * The focus should be towards targeted groups. Who needs to attend and what positive outcome should come be analysed. Creating a small competition or hackathon before one month can be more helpful as there will be more contributions.
 * Less WMF people attending which is overall a good thing
 * I suggest making attendance compulsory for all product management and all direct reports of Victoria and Toby.
 * Having some product managers in these conversations would be nice too.
 * Include *not just senior technologists* but people with a varied level of expertise. Best outcomes emerge when the juniors also got to have a say in their framing, or they are just allowed to be present as listeners.
 * Each WMF team pre-recruits folks from their team who get to apply for the position statement and that way we ensure that there is a good mix of people from various projects/teams who land up at the summit.
 * Allowing the participation of only those who submitted a position statement might limit the success of the DevSummit if that means that key people (based again on scope and goal) are missing because of the process.
 * If you want technologists to talk only among themselves—which is perfectly reasonable—then your process is a good one. But if you'd also like discussions to influence future product developments then an additional qualifying route would be desirable.
 * People more involved with our development process (long term contributors, prominent volunteers, staffers working on important projects) should be given importance because these people can actually implement outcomes from these sessions.
 * I think my one solution would be to scrap the event outright.
 * I strongly suggest against having a dev summit in such conditions
 * People with Opinions (capital O intended), and they view the Dev Summit as a chance to share those and basically get validation of them. A chance to start with an unknown and bash out a plan is rare--critique of existing plans seems to fall on deaf ears.
 * I think the developer summit would be much better if (1) it can be as big as the community it is bringing together and (2) the topics can be controlled by the participants. (2) can be achieved with track chairs who manager general topic spaces and mentor potential speakers/facilitators.
 * The heart of a *Developer* Summit, where we share our ideas and think things through *technically*. Those who do understand long-term goals need the developers who understand technical challenges, best practices, etc., and together they can figure out the best approach.
 * Maybe the selection criteria could be made more transparent. But I guess it's ok like it was done this year.
 * The summit could be a place for brainstorming and kicking off new projects  (worked well in the past, e.g. the developer wishlist and the Platform team proposal) More flexible schedule. People to propose projects beforehand (and then checking their activity on Phabricator) could be used for selection.
 * If along-side all-hands: (1) Have the whole Foundation in one place (the office?), (2) Hold exclusive dev summit sessions totalling ~9 hours over two days (the same as in 2018), (3) Not separate participants and non-participants for the remaining time.
 * This event seemed to eliminate the so-called "hallway track" which eliminated much of the most useful discussions from previous summits.

Please suggest topic ideas for 2019 position statements

 * Writing generally useful libraries and services (how, why, and are we doing this already?)
 * What kinds of APIs should MediaWiki and the Foundation have?
 * How to disentangle presentation from data in MediaWiki (without destroying Wikipedia)
 * Focus on the Wikimedia Platform Evolution, MediaWiki and it’s ecosystem
 * Tech debt: MediaWiki core, unmaintained extensions, gadgets, frontend and backend mechanisms, how to create and keep better documentation, code stewardship, sunsetting processes, future of Labs
 * Have specific tracks (not just discussions) for: MediaWiki, operational infrastructure, security, research and analytics, contributors and readers, developer support and community
 * Three things we are doing well and three things we are doing poorly
 * Improving our engineering culture
 * Several people (6+) did not want to have any position statements
 * Sessions to track our progress (starting from some year in the past) in Tech
 * Sessions on phase 2 of the strategy movement, it’s execution and the expected outcomes
 * Focus on the year 2030 and what the future could hold for us (not what’s going on right now)
 * Change up the format - product manager’s present a topic for 20 minutes, then 45-50 minutes for going into more detail, discussions and Q&A
 * Language or translation related topics
 * How will Wikipedia evolve to remain relevant in 15 years
 * What can the Foundation learn from 3rd-party and volunteer wiki developers with their "enterprise" use cases
 * How to contribute to UX research; teach how to do basic usability testing, review basic UX best practices, and ideas for future interfaces
 * Developer wishlist
 * Interview people that don’t contribute to Wikipedia right now to figure out why and what it would take for them to start contributing
 * Outcomes, recommendations, and action items from the 2018 sessions (focusing on the “how”)

What did you find helpful or confusing about the way we used Phabricator/wikipages/etherpads for organizing the work before and during the summit?

 * Wasn’t clear if (and how much) pre-event discussion was encouraged or desired
 * There wasn’t much discussion of topics in Phab (except by topic leaders) before the event
 * Felt that the discussion on Phab should have been to request answers to a small amount of questions and then go over that in the sessions
 * Quite a few folks said the usage was fine / good / helpful to be using our normal tools
 * Format was easy to find, get involved with, and were well structured
 * Tracking progress of the sessions and the discussions was very helpful
 * Input from the Phabricator pages didn’t always get to be part of the sessions
 * Would be great to only use a wiki instead of etherpads (one day)
 * Helpful to document everything but some session notes were too verbose
 * A recording (voice or video) would be just as good and with less effort (than taking notes in etherpad)
 * All three are useful as long as the scope is clear and are cross-linked
 * Phab wasn't as useful as it could be - many tasks weren’t filled in or had too much background reading
 * Better usage of Phabricator would be for detailing the action items
 * Lots of clicks required to find the relevant info - felt that it was too spread out
 * The tools were the least duplicative (from past years)
 * Very useful to have etherpads in advance of the conference - made taking notes easy
 * Too many phab tickets made it hard to navigate
 * It wasn’t clear how the notes would be transferred to wiki after the conference
 * Unsure what the final (after the conference) outcome would be - how would it feed into the 3-5 year strategy process
 * Have a ‘catch up on documentation’ time slot at the end of the conference
 * Wiki is great for providing information about scheduling, but the positions statements should have been all in Phabricator
 * The real-time aspect of the etherpad note taking plus the deliberative aspect of Phabricator and the wiki summarizing worked well

Please provide ideas for how we can improve the event session format

 * Sessions were too large (too many people in each)
 * Sessions should be a max of 2 hours (otherwise, split into smaller sessions)
 * There was not enough time during breaks for informal discussions
 * Social gatherings were too noisy to continue informal discussions
 * If a person’s position statement was accepted, have them lead / talk during the sessions
 * Position papers didn’t always “fit” into the session topics
 * Add lightning talks / unconferences
 * Keep the fireside chat
 * Product managers present their FY goals and their 3-5 year plan at the beginning of each session and then do hour long slots during the day to go over those plans
 * Keep keynotes to less than 30 minutes
 * Keep the wrap-up session at the end of each day for a good recap
 * ‘next steps for languages and cross project collaboration’ and ‘MediaWiki architecture’ programs were too big for one session
 * Some sessions were lacking in a clear purpose or goal and were too broad in subject
 * Base the participants selected on merit, not their position statements
 * Don’t break up the sessions over a break or for keynotes (keep them about 1 1/2 hours each)
 * No more than 2 keynotes a day
 * With fewer tracks and less participation, it felt like it resulted in less getting discussed
 * Create a small event of it’s own for discussions around machine learning
 * Better prepare the topic leaders for their session and to be in sync with other topics
 * Make sure all the AV equipment works
 * Facilitation was good

Do you have a suggestion for a specific keynote speaker for 2019? (Please provide their name, and affiliation / background)

 * Two survey respondents asked not to have keynotes at all
 * Tim Starling, Wikimedia Foundation
 * Don Knuth
 * Someone from GitHub
 * Something related to the events theme
 * Someone from Mozilla
 * Dries Buytaert, founder and lead developer of the Drupal CMS
 * Somebody technical, not politics/social justice
 * John Gilmore (EFF board member)
 * Zeynep Tufecki
 * StackOverflow people like Joel Spolsky or Jeff Atwood, people working in large open source projects like Drupal, Symfony, PHP, Node, developers or managers from other wikis like Wikia / Wikihow, Clay Shirky or another social computing researcher.

Do you have suggestions for specific topics you’d like the keynote speakers to cover for 2019?

 * Multiple people say that they should be technology based / not about social justice
 * Multiple people say wikimedia focused or tailored to our community specifically
 * Two people say: no keynotes at all
 * Two people say: People from other large open source organisations with similar goals / solving similar problems
 * We had to many keynotes
 * Theory and practice of operating a FLOSS project; Collaboration at scale; Promoting successful bottom up planning
 * Wikipedia use under authoritarian regimes
 * More Wikipedia related Machine Learning.
 * Someone who has improved the code quality and organization of the software
 * Someone who has dramatically reformed APIs or developer-test-prod VMs/rigs.
 * Knowledge equity. How do we spread knowledge to marginalized people? Can marginalized people come talk to us about it? How do they see the world, what do we need to do to meet them where they are?
 * The psychology behind different personalities, what drives different people to participate in wikis or other collaboration.
 * Software freedom in the context of the internet services era.
 * External speakers on tech debt and documentation.
 * Developer support & community
 * The role of an ecosystem for an open source software
 * Consensus-building and governance challenges of open-source software,
 * Social computing and the ways software influences community health and cultural norms,
 * An analyst / (reasonable) futurologist talking about internet and technology trends.
 * Boring technology.

Do you have suggestions for specific topics that you would like to hear about from WMF management at the 2019 summit?

 * some of technical challenges they had in 2018-2019 and how they addressed them
 * thoughts on some of the technical challenges ahead of us, and what we should do to overcome them.
 * how exactly are the funds distributed, and how can the developers influence in the allocation of resources for a small project. i.e. things that a single developer can do by herself
 * Integration of DevSummit outcomes with WMF Annual Plan and resource allocation. How they interact within each other and which one influence and guide the others.
 * A clear direction on third party operated MediaWiki and the appropriate degree of investment as a Wikimedia Foundation.
 * Insight into what and how we should build for interactive agents (voice, chatbots), the direction for A/V+VR and the tie-in to mobile computing and sensors/embedded.
 * A clear direction on our approach for channel expansion while ensuring growth in first party usage across WMF/WMDE-maintained channels.
 * I would like some clarity on our mission. According to T&C's cultural orientation, our technology is a means to an end. However, most people at the dev summit do not agree with this.
 * Specific examples of trade-offs we make, and why. How do we choose to focus on work?
 * How can we be more dynamic and collaborate more across teams?
 * A broad (still concrete) roadmap.
 * Implementation plans for the outcomes of the last summit.
 * How the Stewardship review process will be incorporated into your planning.
 * Less management involvement.
 * WMF management should prepare something, *anything* to the level they prepare a statement to the board.
 * How to manage tech debt
 * I would love to hear/talk about building the ecosystem and tech partnerships within and beyond the movement. But we shouldn't ask that the WMF management alone.
 * Mostly what they hope to get out of the summit.
 * What management did in 2018 to promote MW as a tool outside of WMF and inside WMF. Talk about how MW encapsulates the culture of WM in code.

Are there other things that we should be measuring that we are not asking about?

 * Post-summit impacts / effects / evaluation of outcomes / productivity
 * What changes have you seen since past Dev Summits? / How many sessions from previous years had any of the resolutions carried out, or blocked by lack of resources.
 * Interaction with other developers
 * Who was missing?
 * How can we measure success with such large generic topic areas?
 * Ask people who didn’t apply why they choose not to attend?
 * Ask me what I took away from this event -- why my attendance was important.
 * Did anything useful come out of the event itself or was it just because of the people attending?
 * Did anyone who did not attend gain anything either from reading documentation or following along?
 * Challenges that resulted from dividing staff (not allowing some at this event)

Data
Did you attend the Wikimedia Developer Summit 2018 in Berkeley?

Yes: 61.7%

No: 38.3%

Have you attended any previous Wikimedia Developer Summits and/or Architecture Summits organized by WMF in 2017 or earlier?

Yes: 87.2%

No: 12.8%

Questions for Participants
Omitted: How useful for you were the keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

Omitted: How useful for you were WMF management keynotes? Because these answers might negatively reflect the contribution of one individual they are not included. The answers were helpful to the organizers.

How useful was each session for you?

Response options are ranked in order after giving 2 points for every "very useful" response, 1 point for every "useful" response, 0 points for every neutral responses, -1 points for every not useful response and dividing by the number of responses.
 * Growing the MW technical community: 17 responses
 * Supporting 3rd party use of MW: 16 reponses
 * Embracing open source software: 20 responses
 * Knowledge as a service : 16 responses
 * Research Analytics and Machines Learning: 11 responses (tied with next steps for language)
 * Next steps for language and cross project collaboration: 11 responses (tired with research and analytics)
 * Advancing the contributor experience: 15 responses
 * Evolving the MW Architecture: 22 responses

Please tell us how much you agree or disagree with the following statements

Attending this event was worth my time

Strongly Agree: 10

Agree: 10

Neither Agree or Disagree: 5

Disagree: 1

Strongly Disagree: 3

The opportunity to meet fellow developers was valuable to me

Strongly Agree: 17

Agree:

Neither Agree or Disagree: 2

Disagree: 2

Strongly Disagree: 1

I would like to attend this event again next year

Strongly Agree: 14

Agree: 7

Neither Agree or Disagree: 5

Disagree: 2

Strongly Disagree: 1

Process Questions
Based on your understanding of the process, do you think asking participants to write a Position Statement was fair?

Yes: 29.8% (14)

Indifferent: 8.5% (4)

No: 31.9% (15)

I don't know: 6.4% (3)

Other: 11 participants

Based on your understanding of the process, do you think the participant selection process was fair?

Yes: 21.3% (10)

Indifferent: 8.5% (4)

No: 14.9% (7)

I don't know: 46.8% (22)

Other: 4 participants

Documentation Questions
Please rank these tools in order of how helpful they were to you and your topics of interest in following the actives and outcomes of the Wikimedia Developer Summit 2018

Phabricator

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Etherpad

Very helpful: 18

Moderately Helpful: 0

Not Helpful : 4

Wiki

Very helpful: 14

Moderately Helpful: 0

Not Helpful: 5