New Developers/Quarterly/2017-10

This first edition of the New Developers Quarterly Report, covers a timeline of our developer outreach activities between July–September 2017, metrics and trends, key findings, and recommendations for next steps. It also includes results from the survey completed by new developers who contributed code to Wikimedia projects in the previous quarter. This report is one of the outcomes of the Onboarding New Developers annual program. Throughout the program, we plan to release a report every quarter highlighting results from the previous one, that will help inform all our work related to this program.

Timeline
Newcomer-focused events, programs, and activities between July–September 2017:


 * September 25th: Wikimedia Austria won the Open Minds Award in the diversity category for the mentoring program at Wikimedia Hackathon in Vienna
 * July–September: Developer Relations team worked on revising newcomer documentation. This project is still ongoing.
 * September 7th: Application period for working with Wikimedia in Outreachy Round 15 opens
 * September 6th: Google Summer of Code 2017 and Outreachy Round 14 ended and concluded with a project showcase
 * August 23rd: First weekly Technical Advice IRC Meeting
 * August 9th–10th: Mentoring program at Wikimania Hackathon
 * July 8th: MediaWiki workshop in Buea, Cameroon by Africa Wikimedia Developers Project

Key findings
1. Regular trends in attraction, and stagnating in retention of new developers

On an average, we are attracting around 54 developers per quarter and retaining 8% of them. Two noticings around this: 1) number of new developers joining are more than usual around the time outreach programs, and events are taking place, 2) the retention percentages in 2016 were much higher than they were in the current year. We will check on this in the future to reach a conclusion.

2. New Developers tend to become involved in contributing code to Wikimedia software projects when they learn about it through their peers

A few of our survey respondents mentioned that they heard about Wikimedia through a friend or someone who they met at an event. This fact holds true for a few of our developer outreach activities. For example, through the extended network of the Wikimedia outreach program participants, we draw more candidates from their geographic location for the next rounds. Note: This is preliminary data based on few responses, we will check on this for accuracy at the end of the next quarter.

3. New developers tend to be working professionals with over five years of experience developing software

New developers want to contribute to Wikimedia in their spare time for different reasons; they are: intrigued by a project, want to bring an impact through their valuable contribution, or grow their learning. While a majority of developers in our survey results accounted for those who are working professionals, it also indicates that it's likely that other than the outreach programs, we are missing on reaching to student volunteer developers more broadly. Note: This is preliminary data based on few responses, we will check on this for accuracy at the end of the next quarter.

4. Code review is less of a problem for new developers than we thought

Wikimedia technical community seems to be active in reviewing changesets of new developers. In the previous quarter, we merged 50.70%, abandoned 26.06% and haven’t taken a decision yet on 23.24% of changesets.

5. Continued guidance offered by the local community, support in easy onboarding of new developers

Africa Wikimedia Developers project kicked-off earlier this year in June with a goal to involve developers from Africa in the Wikimedia movement. So far, they have reached a community of over 100 people, conducted MediaWiki development workshops in two countries: Ghana and Cameroon, and their ongoing efforts of engaging students after these events by mentoring them on tasks has led to onboarding of three active developers.

6. Three projects that received most contributions from new developers in the previous quarter are Extension:Example, Wikipedia Android app, and MediaWiki Core

After investigating further the commits received by these projects, it is apparent why they are in the list of top three. Extension:Example receives a vast number of commits; mostly test commits (as this repository is explicitly mentioned in our Gerrit documentation) and through them, new developers try to develop their understanding of the Wikimedia code contribution process. MediaWiki core receives bug fixes which is an easy way for new developers to get started, and Wikipedia Android app looks like a newcomer-friendly project.

7. New developers struggle with the Wikimedia code review process

A few new developers mentioned in the survey that they are either developing their understanding of Wikimedia code contribution process or they are struggling with it. One thing they struggle most while contributing to Wikimedia projects is understanding conventions around Gerrit code review.

8. Dedicated spaces, platforms, and learning environments for new developers both online and offline empower them and contribute to an engaging experience

New developers who participated in the mentoring program at Wikimedia/Wikimania Hackathon in Vienna/Montreal seemed to have a satisfying experience, have ideas for projects they could contribute to, and guidance to work on them. We leveraged communication platforms like Zulip (for Outreach programs) and Telegram (for Hackathons) that helped in keeping the new developers engaged and connected with community members.

9. Featuring newcomer-friendly projects helps in attracting developers

New developers in the past have reported being directed to Phabricator workboard for task or project hunting as challenging. We tried to change this approach by presenting projects with clear goals and expectations, and a listing of skills required in MediaWiki (see New Developers). In Wikimedia's participation in the ongoing round of Outreachy, this approach helped draw 39 potential candidates to seven projects in one-month timeframe who expressed their interest in working on a project by commenting on the corresponding Phabricator task.

New developers metrics and trends
All numbers refer to volunteer developers only.

Trends in attraction and retention
These are the KPIs of the Onboarding New Developers program.

Developers contributing patches for review
180 developers contributed between July–September 2017. (Source)

QoQ : +11.11%. YoY : +20.81%.

New developers attracted
49 new developers contributed between July–September 2017. (Source)

QoQ : -19.67%. YoY : -12.50%.

New developers retained
Percentage of developers active one year (±3months) after their first contribution, out of all new developers attracted one year ago. (Source: Calculation on data)

QoQ : -26.5%. YoY : -60.0%.

Review of changesets by new developers
142 changesets were contributed by new developers. (Source: Calculation on data . Data as of October 06, 2017.)

Projects with most received contributions by new developers
49 new developers contributed to 24 repositories. (Source: "Repos by New Authors")

Outreach events

 * Wikimania Hackathon 2017 (August 2017):
 * Attracted 47 new developers.
 * Wikimedia Hackathon 2017 (May 2017):
 * Attracted 37 new developers.
 * Retained 3 new developers (8.1%) who were still active in the quarter after the event (July–September 2017). (Source)

Outreach programs

 * Google Summer of Code 2017 (May–August 2017):
 * Attracted 8 new developers out of which 7 students completed the internship
 * Outreachy Round 14 (May–August 2017):
 * Attracted 3 new developers. All students completed the internship

Survey analysis
View detailed analysis of survey or read through below for a quick summary.

We sent a survey to 35 new developers who submitted code for the first time between July–August 2017. The survey was designed using the Qualtrics platform, with a goal of understanding more clearly demographics and background, motivations, challenges and needs of our new developers.

Out of the 35 new developers to whom we reached out, 9 (25.7%) completed the survey. Four people started filling out the survey but didn’t finish, indicating that the completion rate of the survey was 69%.

Note: Since we got quite a few responses, the analysis of this survey may not be entirely justified, and actions for next steps and recommendations indicated in this report (coming soon! ) may not have many conclusions from the survey.

Below are the results from the survey:

Demographics and background information

 * 33.33% of respondents of the survey were from United States, and 11.11% from Canada, Cote d'Ivoire, Germany, India, Sweden, and Turkey
 * 44.44% said they belong to the age group between 25-34
 * 77.78% identified themselves as male
 * 66.67% were working professionals and had more than five years of experience developing software

Contributions
Types of contributions

Participants could select more than one option

Participants (n) = 9 { "version": 2, "width": 300, "height": 250, "data": [ { "name": "table", "values": [ {"x": 1, "y": 6}, {"x": 2,  "y": 5}, {"x": 3,  "y": 5}, {"x": 4,  "y": 4}, {"x": 5,  "y": 3}, {"x": 6,  "y": 2}, {"x": 7,  "y": 2}, {"x": 8,  "y": 1}, {"x": 9,  "y": 1}, {"x": 10, "y": 1}, {"x": 11, "y": 1} ] } ], "scales": [ { "name": "x", "type": "ordinal", "range": "width", "domain": {"data": "table", "field": "x"} }, { "name": "y", "range": "height", "domain": {"data": "table", "field": "y"} } ], "axes": [ {"type": "x", "scale": "x", "title":"Types of contributions"}, {"type": "y", "scale": "y", "title": "No of developers"} ], "marks": [ { "type": "rect", "from": {"data": "table"}, "properties": { "enter": { "x": {"scale": "x", "field": "x"}, "width": {"scale": "x", "band": true, "offset": -1}, "y": {"scale": "y", "field": "y"}, "y2": {"scale": "y", "value": 0} }, "update": { "fill": {"value": "steelblue"} } } } ] }


 * Types of contributions
 * 1 – Fixing bugs (19.35%)
 * 2 – Reporting bugs (16.13%)
 * 3 – Editing articles (16.13%)
 * 4 – Uploading media to Wikimedia Commons (12.90%)
 * 5 – Task discussion (9.68%)
 * 6 – Feature development (6.45%)
 * 7 – Documentation (6.45%)
 * 8 – Organizing meetups (3.23%)
 * 9 – Planning technical implementation (3.23%)
 * 10 – Answering developer queries (3.23%)
 * 11 – Other (3.23%)

35.48% of respondents of this survey said they fix or report bugs

Motivations
A few new developers indicated that they first heard about Wikimedia as a place to contributing code through a friend, Wikimedia member or at an event. New developers decided to work on a project as they:
 * Found it interesting
 * Project was a good fit for their graduate studies
 * Could share their research by working on the project
 * Wanted to improve the project
 * Wanted to get started with fixing bugs, submitting patches, etc.
 * Were being asked to do so at work
 * Wanted to learn new things and to minimize burden on Wikimedia developers
 * Heard about it at a hackathon

Challenges

 * 55.65% of respondents of this survey said they are developing their understanding of Wikimedia's code contribution process and 33.33% said they struggle with it.
 * Some challenges that new developers face are: finding time, connectivity issues, documentation, git-review process and understanding conventions when working with Gerrit, ignorance to patch for a bug fix, not being able to access Phabricator due to IP ban, poor communication, lack of agile practices, poor requirements, project vision, etc.
 * Learning resources and channels that new developers refer to when they get stuck: Help pages on Wikitech, IRC, Google, Github, Gerrit, other developers, video tutorials, books, Stack overflow, documentation on MediaWiki, extension documentation page, etc.

Experience contributing to Wikimedia
50.00% of respondents of this survey said they are extremely satisfied with their experience contributing to Wikimedia. 33.33% said that they are extremely likely to recommend Wikimedia for code contribution to friends and colleagues.

Suggestions for improvement
Here are some suggestions made by new developers (responses below have been only slightly edited):
 * Standard toolbox over in-house or unusual systems (Gerrit, Phragile, ...)
 * More autonomy for development teams: build a trust-based DevOps culture allowing teams to adopt Continuous Integration/Continuous Delivery practices and deliver value in small iterations (current deployment guide mentions a lead time of over six weeks for new services!).
 * Breaking focused services out of the monolithic MediaWiki core.
 * Be bold with code changes. Approve more patches in a timely fashion. Enlarge the group of developers who are authorized to approve patches.
 * Surveying and documenting conventions on how to interact with other developers through Phabricator and Gerrit.
 * Git commit message guidelines suggests "To express that a commit resolves a bug, add Bug: Txxx in the footer at the end of the commit message." What do you do when the commit doesn't resolve the bug because the bug has multiple items? The "Bug: Txxx" footer should _always_ be there, because that automatically adds a change notice to Phabricator. That convention isn't made particularly clear from the guidelines.
 * Fix the IP ban that devoid access to Phabricator
 * Be better on follow up issues
 * Better guides. Make more documentation on Wikitech. So that we can contribute more.

Recommendations for next steps
From this report's key findings, detailed survey analysis, metrics, read below recommendations for next steps. Note: these are just ideas for next steps, Developer Relations will discuss them in depth and generate action items from it.

Shorten the New Developers Survey and improve the survey design Review tutorials explaining the code contribution process and investigate what we can improve further
 * Overall response rate for the survey was quite low (26.4%). In the next iteration, we will shorten the survey, with a goal to obtain a higher percentage of responses. We hope that these responses will then solidify our understanding of new developers and help plan for next steps.
 * The reason to include contributors from July–September 2017 was to keep the timeframe similar to the timeline of this report. However, we could not pilot the survey to developers who contributed code in September as we initially planned. It would have been challenging to compile responses and include them in the report before its scheduled release. For the next report, we are planning to pilot the survey among new developers who contributed code between September–November 2017.

New developers struggle with code contribution process and understanding conventions around Gerrit/ and git-review. They contribute test commits to the Example extension to understand Gerrit (as instructed in the tutorial) which do not necessarily land in the codebase. We could consider reviewing and improving the tutorial, including a list of easy tasks requiring one-line fixes, etc. This might bring some value to the first contribution made by newcomers.

Better tracking information of new developers who participate in Wikimedia international developer events

We currently struggle to keep track of information about new developers such as project they worked on, their contact details (e.g., Phabricator, Gerrit, and Github username), etc. We’ve tried to collect this information during the events manually, but it's challenging. This information is crucial for: a) our follow-up planning of engaging new developers afterward b) to measure retention levels. We will investigate how we could do better.

Consider getting Wikipedia Android Apps project listed as a featured project

Wikipedia Android Apps is among top three projects that receives newcomer contributions and we would like to get it listed as a featured project on the New Developers page.