MediaWiki-Vagrant/Surveys/2014Q4
MediaWiki-Vagrant Survey, 2014Q4[edit]
Below is a broad summary of the MW-Vagrant 2014Q4 survey results. For the anonymized raw data and a more complete summary of responses, please see Links.
Background[edit]
MW-Vagrant has become an indispensable tool for MediaWiki engineers and non-engineers alike, facilitating dozens of use cases that require a reproducible Linux environment running MediaWiki core, along with its extensions and supporting services. In an effort to better serve MW-Vagrantâs growing user base and explore its potential for a wider one, we conducted a survey across willing WMF staff and Wikitech community members.
We sought to measure the awareness of certain features, usage patterns, the perceived importance of MW-Vagrantâs different aspects (flexibility, performance, stability, etc.) relative to one another, and the satisfaction with those aspects, both current and trending.
Methodology[edit]
The survey was conducted through Qualtrics, published online, and disseminated via the wmfall and wikitech-l mailing lists. Both engineers and non-engineers were asked to participate whether they currently used MW-Vagrant or notâthough the majority of survey questions regarding feature awareness and satisfaction were conditional on whether the participant was indeed a current user.
Responses were exported and further analyzed using Google Sheets.
Summary[edit]
General Awareness[edit]
One objective of the survey was to gauge general awareness of MW-Vagrant and its core features, and how that awareness spreads.
Of the 49 participants who responded to whether they had heard of MW-Vagrant before the announcement of the survey (Q3), 93.9% chose yes. It should be noted, however, that although previously unaware participants were sought in the survey announcement, there may have been a significant selection bias in the way the announcement email was crafted and disseminated or inherent in oneâs inclination to even respond to such a request to participate. We canât really take much away from this number alone.
A more useful finding is perhaps how people are made aware of MW-Vagrant. According to responses to "how did you hear about it" (Q4), 39.4% had learned of it by word of mouth, 18.3% through IRC discussion, and 16.9% by email.
When we consider the generally high level of awareness, these passive communication channels can seem quite effective. By holding more participatory events such as workshops and talks, we can foster these more viral channels and further raise awareness of MW-Vagrant.
Awareness of Core Features[edit]
In addition to general awareness of MW-Vagrant itself, we wanted to know whether users were aware of its core features and, among those aware, to what degree each was being used (Q8).
Responses show that among users of MW-Vagrant, 100% are aware of the automated setup and the roles system, not that surprising given the former is part of the installation process and the latter is the most prominent feature MW-Vagrant. The most significant and actionable finding is the relatively low awareness of advanced customization (44%), wiki import (58.3%), and Vagrant sharing (36%). We should plan to discuss these features in future talks and workshops.
In addition to gauging awareness, we wanted to know to what degree these features were being used. The following chart shows the levels of feature use among those that were at least aware of each.
Again, automated setup and roles (both 96%) are highly used features among those aware of them, with custom wiki configuration getting quite a high rate of depend of it responses relative to those aware of it (41.7%) which amounts to an even higher rate relative to its users (62.5%).
Also significant is the high level of total use among those aware of advanced customization (54.5%) and wiki import (71.4%), underscoring the need to educate users on these features.
Overall Satisfaction[edit]
All in all, users of MW-Vagrant consider it an extremely useful tool and are satisfied with its overall utility. When asked "how satisfied are you with MediaWiki-Vagrant overall" (Q23), 27.9% responded they were very satisfied, and 50% responded satisfied. In a similar question that asked "would you recommend MWV over developing without" (Q12), 92.3% responded yes.
Satisfaction with Individual Aspects[edit]
Despite the overall level of satisfaction, there were specific aspects of MW-Vagrant that participants not only seemed somewhat dissatisfied with but were perceived to have worsened over time.
Participants were first asked to rank the relative importance of six different aspects of MW-Vagrant (Q10): provisioning speed, performance, flexibility, ease of use, stability, and added utility. Respondents ranked stability above all else, with ease of use coming in second.
Next, participants were asked to rate their satisfaction with each aspect and whether each had improved or worsened over time (Q25, Q26).
The responses to these questions indicate a growing dissatisfaction with stability, performance, and provisioning speed, in order from worst to least worst. All three can be seen as areas for potential improvement, particularly stability considering it was ranked highest in terms of importance.
We shouldnât discount, however, the majority of respondents who did feel satisfied with each aspect and the significant number that felt things were getting betterâwe can feel good about that fact.
Actionable Conclusions[edit]
Current users may benefit from further education on features that have a low level of awareness but are found to be very useful: customization, wiki content/template import, and Vagrant sharing. Since word of mouth seems to be a powerful channel for MW-Vagrant users, tech talks and workshops may be the best way to spread knowledge of these features and of MW-Vagrant in general. (The November 2014Q4 "What's new with MediaWiki-Vagrant?" tech talk was a good start in raising such awareness.)
Development goals for the upcoming quarters should focus on problem areas such as stability, provisioning speed, and VM performance, stability being perhaps the primary focus.
Improving stability issues that are due to MediaWiki, extensions, and supporting services being developed within the VM may be tricky, as they are inherently moving targets (and we want developers to keep up-to-date with master branches), but we can at least address the stability of MW-Vagrant itself by establishing unit- and acceptance-test suites to protect against regression, and tagging releases to better communicate breaking changes. More cross-platform testing of each release could also help.
Although provisioning speed wasn't rated as important as stability, there's enough dissatisfaction with it to warrant a concerted improvement effort. Anonymous usage statistics could help us to target resources that are provisioned most often and could potentially include Puppet performance metrics to help us target big offenders.
All in all, we can take pride in the fact that users of MW-Vagrant find it extremely valuable.