Topic on VisualEditor/Feedback

Can someone please update VisualEditor/Portal/Why?

m:Research:VisualEditor's_effect_on_newly_registered_editors/May_2015_study#Results

You've had hard data for a year now that everything on the Portal/Why page is incorrect. Visual Editor doesn't help any additional people edit. Visual Editor doesn't increase editor retention. Zero and zero.

Why are you deploying Visual Editor as default for all new users? Why did this bug have to be opened at all?

It would be equally accurate to describe that data as "the visual editor doesn't prevent any additional people from editing or hurt editor retention". That's what "no difference" means: neither help nor hurt. You are also omitting the fact that, in a sea of "no effect", there was one area in which the visual editor performed better, and no areas in which the wikitext editor performed better. This one area was statistically significant, but the practical effect is small. (It's sort of the difference between a whole country seeing a BMI improvement of just 0.1, versus an individual doing the same.)

So, based on that single study, the data is "no effect, no effect, no effect, very small help". An unbiased person would probably be thinking, "It's probably not important, but if you have to choose, then why not go with the thing that is 0.1% better instead of the thing that is 0.1% worse?"

"there was one area in which the visual editor performed better, and no areas in which the wikitext editor performed better"

Here's the first and primary question of the research:

Research Questions & Hypotheses

RQ1: Does VisualEditor (VE) make editing easier?

• H1.1: edit time will be reduced for VE user due to the increased ease of editing
• H1.2: edit completion rate will be increased for VE users

H1.1: The results were bad. Editors who used Wikitext to make their edits usually took about 35 seconds, while users of VisualEditor usually took 2 minutes.

The density of time-to-save is plotted for intra-edit sessions during the VisualEditor A/B test by editor within the experimental condition.

H1.2: The results were bad. the rate at which editors attempted and successfully saved edits was significantly lower when VE was enabled

	users	intra-edit sessions	"visualeditor" (prop)	attempted save (prop)	successful save (prop)
control	3421	4677	51 (0.011)	3204 (0.685)	2977 (0.637)
experimental	3471	4245	2396 (0.564)	2669 (0.629)	2449 (0.577)

Result: VE made editing harder. It took a lot longer for new users to figure out how to make an edit, they had a lot more trouble creating a successful edit to save, a lot more edits had to be abandoned and restarted from scratch.

The research gives no details for the one area you cite where VE is supposedly better... but apparently a very slight reduction in reverts. That of course is completely outweighed by the edits needed to fix the messes that VE leaves behind. For example the majority of these VE edits need cleanup of one sort or another. The cost of cleaning up after VE doesn't show up in the research results.

I'll also note that the WMF never A/B tested making VE default for new users. When people were first given two links, they could see there were two editors and look in wikitext to see how things worked, it didn't reduce the number of new editors. But dropping people into VE likely DOES result in a loss new editors. At a minimum, you have no basis to claim it doesn't cause any harm.

• I don't think that you have fairly described the results of @Halfak (WMF)'s study. For example, it is probably wrong to assume that "spent longer on an edit in a completely unfamiliar editing environment" means "less easy to use". That claim would require checking things like whether the edit made larger changes (e.g., including sources).
• An A/B test has been discussed for several months. I don't know when or whether it will happen.
• The burden of cleaning up after VE edits (e.g., unwanted nowiki tags, which are invisible to the reader) is lower than the burden of cleaning up after wikitext edits (e.g., mismatched brackets, which break links for readers, and nowiki tags added during test edits). This is why somewhat more newbies have their edits reverted if they use the wikitext editor than if they use the visual editor.
• I would be happy to see any evidence that you can provide that suggests that having the visual editor as the first editing environment results in a loss of new editors. So far, I've found none, and I have looked.

• How can you call my comments unfair? The WMF itself asserted "edit time" and "edit completion rate" as metrics for "Does VisualEditor (VE) make editing easier". I agree that some new accounts have previously edited and it may somewhat affect the results, but that hardly invalidates a nearly 4-to-1 result for all new users. If the research had demonstrated positive results for VE then the WMF certainly would have expected critics to face that data. It's biased for the WMF to disregard it's own data whenever the data contradicts ideology.
• An A/B test would be good..
• Useless invisible nowiki tags are annoying, but I'm primary talking about issues that damage or break content for readers. Did you try reviewing a batch of hits at the link I gave? BTW, reverts are a poor metric for "burden" on the established community. I suggested an additional metric, but the analysis was never done. Edits needing cleanup may attract a followup edit from watchers. Those cleanup edits will tend to reduce the usual interval between different editors. It would be hard to convert that to a quantitative burden value, but if there is a measurable difference then it should show which editor is better for new users.
• The data we have shows that adding/removing the VE tab has no measured effect for bringing in new editors. I have no access to any data on whether removing the wikitext link reduces new editor retention. I said I believe it likely that would have a negative impact, and there's no solid basis to claim it doesn't. An A/B test would either establish this is a real problem, show it has no effect, or show VE is beneficial. Determining reality is better than arguing theory vs theory. It seems unusual that the WMF skipped the typical A/B step here.

And back to the original subject:

1. I asked if VisualEditor/Portal/Why could be updated. I was tempted to add the negative research results there, but the page would be incoherent without a substantial rewrite. The page argues "Why VE" is to increase the number of new editors and improve retention of those editors. It seems(?) that you have basically agreed that VE does not have that effect. Not only is the page misleading to readers, I think it is a reflection of WMF-culture having a hard time integrating the negative VE research results. The vague atmosphere seems to be that VE is an "obvious benefit for new editors" and that any disagreement is irrational change averse obstructionism.
2. Why did this bug have to be opened at all, and is it going to be addressed? The WMF knew this was a problem nine days ago, Forrester won't respond on his talk page despite editing on four different days, and the bug is still listed as untriaged.

It's biased for the WMF to disregard it's own data whenever the data contradicts ideology.

I don't think that's fair. If we had seen lower completion rates and negative results in usability sessions along with the longer time-to-save, then that would be consistent with VE being harder to use. As it stands the time-to-save results is the sole contradiction to other evidence so I think it is reasonable (though I value your questioning it) to consider whether the metric really captures the en:hidden variable we are trying to measure.

Generally, I think that the "reading the page in edit mode", "doing more substantial edits", or "exploring the interface" hypotheses seem to correspond to the data more than the "VE is hard to use" hypothesis. As always, more data would be more better. Regretfully, a new study is expensive time-wise, so we haven't prioritized it. Really, I think that the lab studies and productivity measurements speak for themselves. In the end, the measurements are not the reality of what is measured and it's up to use to interpret.

If you think I'm disregarding data that is otherwise legitimately pointing in a specific direction, then I challenge you to hypothesize why this one result points into a different direction than the rest.

 As it stands the time-to-save results is the sole contradiction to other evidence

Huh? The completion rates were also worse, and the other results were neutral.

H1.2: the rate at which editors attempted and successfully saved edits was significantly lower when VE was enabled

	users	intra-edit sessions	"visualeditor" (prop)	attempted save (prop)	successful save (prop)
control	3421	4677	51 (0.011)	3204 (0.685)	2977 (0.637)
experimental	3471	4245	2396 (0.564)	2669 (0.629)	2449 (0.577)

"Visual editor is better" is at best unsupported, and at worst contradicted. There are of course critically important aspects that aren't captured by that study, but I'm trying to stick to the hard data we have rather than get into that spaghetti-debate.

The original subject here, the VisualEditor/Portal/Why page, argues the reason for VE is to bring in more editors and to increase retention. VE helped an additional 0% of new users make a first edit, and increased retention by 0%. The VE-Why page is 100% misinformation.

One more thing that I think I should mention: I have posted lots of negative results about WMF products (VE, AFT, Echo, Growth experiments, etc.). I have a history of getting myself in trouble with WMF Product teams for findings that contradict desired outcomes. For example, I advocated against the first deployment of VE because of the negative results of that study. The Product team working on VE has no power over me. I work in a whole separate department. My job is to try to figure out what is really going on. My supervisor evaluates me on that competency alone.