MediaWiki-Vagrant/Surveys/2014Q4

= MW-Vagrant Survey, 2014Q4 =

Background
MW-Vagrant has become an indispensable tool for MediaWiki engineers and non-engineers alike, facilitating dozens of use cases that require a reproducible Linux environment running MediaWiki core, along with its extensions and supporting services. In an effort to better serve MW-Vagrant’s growing user base and explore its potential for a wider one, we conducted a survey across willing WMF staff and Wikitech community members.

We sought to measure the awareness of certain features, usage patterns, the perceived importance of MW-Vagrant’s different aspects (flexibility, performance, stability, etc.) relative to one another, and the satisfaction with those aspects, both current and trending.

Methodology
The survey was conducted through Qualtrics, published online, and disseminated via the wmfall and wikitech-l mailing lists. Both engineers and non-engineers were asked to participate whether they currently used MW-Vagrant or not—though the majority of survey questions regarding feature awareness and satisfaction were conditional on whether the participant was indeed a current user.

Responses were exported and further analyzed using Google Sheets.

General Awareness
One objective of the survey was to gauge general awareness of MW-Vagrant and its core features, and how that awareness spreads.

Of the 49 participants who responded to whether they had heard of MW-Vagrant before the announcement of the survey (Q3), 93.9% chose yes. It should be noted, however, that although previously unaware participants were sought in the survey announcement, there may have been a significant selection bias in the way the announcement email was crafted and disseminated or inherent in one’s inclination to even respond to such a request to participate. We can’t really take much away from this number alone.

A more useful finding is perhaps how people are made aware of MW-Vagrant. According to responses to "how did you hear about it" (Q4), 39.4% had learned of it by word of mouth, 18.3% through IRC discussion, and 16.9% by email.



When we consider the generally high level of awareness, these passive communication channels can seem quite effective. By holding more participatory events such as workshops and talks, we can foster these more viral channels and further raise awareness of MW-Vagrant.

Awareness of Core Features
In addition to general awareness of MW-Vagrant itself, we wanted to know whether users were aware of its core features and, among those aware, to what degree each was being used (Q8).



Responses show that among users of MW-Vagrant, 100% are aware of the automated setup and the roles system, not that surprising given the former is part of the installation process and the latter is the most prominent feature MW-Vagrant. The most significant and actionable finding is the relatively low awareness of advanced customization (44%), wiki import (58.3%), and Vagrant sharing (36%). We should plan to discuss these features in future talks and workshops.

In addition to gauging awareness, we wanted to know to what degree these features were being used. The following chart shows the levels of feature use among those that were at least aware of each.



Again, automated setup and roles (both 96%) are highly used features among those aware of them, with custom wiki configuration getting quite a high rate of depend of it responses relative to those aware of it (41.7%) which amounts to an even higher rate relative to its users (62.5%).

Also significant is the high level of total use among those aware of advanced customization (54.5%) and wiki import (71.4%), underscoring the need to educate users on these features.

Overall Satisfaction
All in all, users of MW-Vagrant consider it an extremely useful tool and are satisfied with its overall utility. When asked "how satisfied are you with MediaWiki-Vagrant overall" (Q23), 27.9% responded they were very satisfied, and 50% responded satisfied. In a similar question that asked "would you recommend MWV over developing without" (Q12), 92.3% responded yes.



Satisfaction with Individual Aspects
Despite the overall level of satisfaction, there were specific aspects of MW-Vagrant that participants not only seemed somewhat dissatisfied with but were perceived to have worsened over time.

Participants were first asked to rank the relative importance of six different aspects of MW-Vagrant (Q10): provisioning speed, performance, flexibility, ease of use, stability, and added utility. Respondents ranked stability above all else, with ease of use coming in second.



Next, participants were asked to rate their satisfaction with each aspect and whether each had improved or worsened over time (Q25, Q26).





The responses to these questions indicate a growing dissatisfaction with stability, performance, and provisioning speed, in order from worst to least worst. All three can be seen as areas for potential improvement, particularly stability considering it was ranked highest in terms of importance.

We shouldn’t discount, however, the majority of respondents who did feel satisfied with each aspect and the significant number that felt things were getting better—we can feel good about that fact.