Topic on Talk:Growth/Discussing potential Growth ideas/2023

Tangentially Related:

https://phabricator.wikimedia.org/T90666 (Past test of making Visual Editor available, results were generally neutral or negative)

https://phabricator.wikimedia.org/T135478 (Potential future A/B test for IP users, with no clear purpose or metrics. We can't track IP retention or other metrics over time.)

This is potentially high-impact and low-effort.

Visual Editor was built with the intent to make editing easier for new users, to bring in more people as contributors, and to increase new user retention. Single Edit Tab has been developed and deployed to some wikis, providing a configurable option for which editor is given as the default first-editor when new users click the one-and-only edit link. Some wikis are configured to provide Visual Editor as the initial default, and some wikis are configured to provide wikitext as the initial default. Progress on this has stalled, due to controversy and lack of hard data on which editing environment produces the best outcomes.

I propose gathering the required data. A sample population should be divided in to two equal groups, with the only difference being the initial default value for primary editor. When they first click the EDIT link, half will initially load Visual Editor and half will initially load the wikitext editor. After that point, new users might or might not discover and use the opposite editor on their own. I propose compiling at least three metrics, preferably over a period of at least six months to evaluate long term outcomes.

1. A graph of total number edits from each group over time.
2. A graph of user retention for each group. If a user creates an account at time T, how many of them are retained and make at least one edit at time > T+x?
3. A graph of users actively switching away from the assigned default editor. For each group, consider all edits up to time x. What percentage of edits up to time x are made with the non-default editor?

1. When we talk about "new users", how do we intend to identify them? I am pretty diligent with my watchlist, and so I see a lot of edits from "new" user accounts and new IPs that exhibit a level of editing skill and knowledge of policies etc that suggests to me they are not in fact "new" (I run training classes so I see genuine new user edits all the time). So you need to distinguish between new account and new to Wikipedia editing as these are not the same thing. Existing users with a new account are likely to be already skilled in one editor (probably the source editor) and they are therefore likely to switch to their preferred editor if given the other. So gathering statistics from such folk will not give you good data. You really want "truly new" users for this experiment.
2. As for the metrics, what evidence is there that number of edits in VE should be the same as the number of edits in source editor to do the same task. In VE you always edit the whole article but in source editor, you can edit individual sections, so if someone is making a couple of changes to an article, they might do it in one edit in VE and two edits in source. Also in source editing, it is easy for the newbie to break the syntax in ways not really possible in the VE, so a newbie who saves a breaking-syntax edit may then come back for a 2nd edit to fix it (and a 3rd and a 4th if they don't how to fix it) which doesn't occur in the VE. ALso in the VE, if you do something wrong, you will see the error in front of you most of the time and so you will fix it before Saving. Different tools lead to different user behaviours and hence different numbers of edits.

Also, you might want to talk to @Halfak (WMF) as he has done a similar experiment as he may have some useful advice. ~~~~

@Alsee -- thanks for weighing in and thinking about this. I think it's always good to get more data, and your comments made me go back and review what we know about Visual Editor. In addition to the study that @Kerry Raymond mentioned, I also found a presentation from the monthly activities meeting from March of this year discussing edit success rates with different editors. While that doesn't answer all questions, I wanted to make sure you had seen it.

Though the editing experience itself probably has an important impact on newcomer retention, our team is focusing on helping newcomers learn what they need to know to be successful, regardless of which editor they are using. We learned from the New Editors Experiences research that newcomers struggle with many things: technology, wiki concepts, and wiki culture. Our team's mission is to help newcomers to be productive no matter what technology is in place in the wikis now or in the future. They will always have questions or confusion, and we want them to have avenues to get answers. That's why we'll be focusing on features that allow them to ask questions, discover how the wiki works, and learn as they go along. I hope this makes sense in terms of our team's mission.

I've watched the presentation, and it only confirms how badly we need this research to be done.

To quote the rigorous controlled study May 2015 study#Edit completion rates: the rate at which editors attempted and successfully saved edits was significantly lower when VE was enabled. Look at how much lower all the figures are for VE:

	users	intra-edit sessions	"visualeditor" (prop)	attempted save (prop)	successful save (prop)
control (Wikitext as the only available editor)	3421	4677	51 (0.011)	3204 (0.685)	2977 (0.637)
experimental (Primary Edit link set to VE, wikitext offered on second link)	3471	4245	2396 (0.564)	2669 (0.629)	2449 (0.577)

The biggest problem here is the presentation is selectively citing irrelevant figures. Our goal is to help more people to contribute, and for them to stay and contribute more. It doesn't actually matter what the 'successful save rate' is, we care if people are actually staying to make edits.

The second problem is that the presentation is claiming the opposite as the controlled study. The presentation is citing junk data. We know why it's junk data. It is a common part of learning and workflow in wikitext to open additional tabs to view or copy wikitext. That presentation is improperly counting wikitext-viewing and wikitext-copying as if it were an editing failure. That's clearly wrong.

The mission of the growth team is: Growth. To quote the objective: engaging new contributors in Wikimedia projects... The Growth Team's objective is to address this problem through software changes that help retain new contributors." While the team had certain projects in mind, there's an explicit call here for additional ideas. And the the various software changes relating to the editor(s) clearly has an important and direct impact on new contributors. This research is in-scope for the team, if management decides that it is in-scope.

I'd would say that while the technology can be an initial barrier to entry, but I know from doing training that you can get people over that hurdle with the VE (some people never reach competency using source editing). However, once the basic technology skill is acquired, the next thing that happens to a new user is that they fall foul of one of our many policies including the Manual of Style, encounter an "owner'/"gatekeeper" who implements their own" I-don't-like-it " policy, or any of the zillion rules of the encyclopedia whose 5th pillar is "no firm rules" (talk about misleading advertiing!). Their edit gets reverted, the explanation is often done by a tool (like Twinkle) so is quite non-specific, or if hand-written contains mention of (or links to) one of our vast army of policies which are probably not written with the new user as the target readership. The new user really needs more specific and more explanatory help.

Now, as someone who is diligent with my watchlist, I feel this tension myself in dealing with problematic but probably good faith edits by new users. I do try to help by being more fullsome in my explanation of the revert or by trying to modify their edit to make it conformant, but sometimes I just don't have the time to do that due to other demands on me from either other Wikimedia activities or the demands of real life. The other constraint on doing a better job of helping newbies is technological. When I am away from home, I usually am on a mobile device from which I can process my watchlist, but the only actions I can easily take are the click-the-button actions like "revert, tick box for reason" or "thanks" or "welcome". I don't have the ability to edit on a mobile device which providing good assistance to a newbie requires. When real life causes me to be away from home for days or weeks on end, I just have to do revert the more problematic edits and move on. So even as someone who tries to reach out and help new people, I just can't always do what I think needs to be done, and of course a lot of people just don't even try to help in the first place. Many editors will revert an edit without explanation or if they do write something on the User Talk page (with a tool or manually), many don't appear to watch that page for a reply. So a combination of our strict rules and abrasive culture loses us a lot of new contributors at this point

I think the revert of a new contributor needs to be drawn to the attention of someone other than the reverting editor as this is obviously a point at which we are likely to lose the new contributor if they are not helped over this hurdle. (I am not referring here to outright vandalism but anything that appears may be good faith). Having observed new contributors in training and events, I know they do not always notice notifications/messages from the tool bar; their attention is very focussed on the article text area of the screen. So, if they make an edit, it's get very quickly reverted, they often don't realise what has happened. What happens is that they don't see their change in their article and so they assume that they didn't make their change correctly (from a technological perspective) so go back into edit mode and repeat their edit (or a minor variation of it) and save it again. And they will do it again and again, and they are quickly at 3RR without even realising anyone was objecting to their edit. Of course the reverting editors don't see the situation like that, they see the persistent repetition of an edit that has been explained is unacceptable one, twice or thrice as defiance and are happy to see them blocked. New user behaviour is easily misinterpreted because most people never get to observe new users in real life to see how they misunderstand/misinterpret the whole experience.