I'd like to know whether draftquality is the same for new accounts that create new articles in the visual editor as it is for new accounts that create new articles in the older wikitext editors (at the English Wikipedia).
@Nettrom, I'm assuming that this is outside the scope of your current projects. @Neil P. Quinn-WMF, is this something that you could do? I'm not sure how much work this would be, but I assume that it's not very difficult.
I'm very interested in all VE research results. Please ping me if this project goes forwards.
Confounding variables of using different self-selected populations would produce unreliable results, especially because some percentage of those "new accounts" actually represent experienced editors. (Paid editors in particular abuse throw-away accounts for each new article.) However I have a fix. You can do a retroactive controlled study. You have to ignore whether an article was created using VE, and look for any difference in draftquality between the experimental and control groups of the May 2015 study of VisualEditor's effect on newly registered editors.
Comparing control group wikitext articles against experimental group wikitext+VE articles will cut your signal strength in half, but it's the only way to avoid junk data due to skewed population selection.
Your idea might measure the added value for VisualEditor's contribution (e.g., if you were trying to show that a new editor using VisualEditor is more likely to properly format a citation), but that's not actually my goal. I'm thinking that the chosen editing environment might be a useful marker.
There's obviously no value in collecting data showing that experienced editors produce higher quality drafts than new users.
That is likely what we would get if we ran your proposed data-collection without modification. An experienced user with a new-account is more likely to know how to switch to the secondary editor. This can introduce an experienced-user bias in the study's population-selection.
(Collecting reliable data from the wild isn't easy.)
My proposal is to study an objective, non-speculative condition: "new accounts that create new articles".
What value would that have, if it merely establishes that experienced users produce higher quality drafts than new users?
We can get much more valuable results by re-examining data from the controlled study.
You are welcome to re-examine that old data if you want to.
You are also welcome to study whether new accounts that you believe to be experienced editors actually produce higher quality drafts. However, it sounds circular to me: How will you divide the brand-new accounts into "experienced" and "new" editors? By looking at the quality of the draft. What are you going to study? Whether the ones that you labeled "experienced", on the basis of their higher quality drafts, produced higher quality drafts than the ones that you labeled "new", on the basis of their lower quality drafts. If you did not find a perfect correlation in such a study, then you would probably want to look for an arithmetic error.
I do not want to discourage you from researching whatever interests you, but your question does not interest me.
What??? Do you understand why you're going to get junk data?
For new accounts, you can't distinguish experienced editors from new editors. It's a confounding factor. You're proposing to use biased populations.
I also don't understand why you seem actively-averse to looking at high quality data.
Again, I'm not trying to distinguish experienced editors from new editors.
I'm trying to find out whether new accounts (=an objective, unbiased, machine-identifiable state that is only partially correlated with the actual experience level of the humans who are using those accounts) and either use, or don't use, the newer editing software, produce the same or different results on the specific measure of ORES draftquality.
As a side note, it sounds like you're assuming that experienced editors are more likely to switch to visual editing than new editors. I don't think that there is any data to support your assumption.
Hypothesis 1: Experienced users are more likely to know how to switch to VisualEditor. Draftquality for new-accounts using VisualEditor will skew high, because you're measuring more experienced editors in VE vs newbies in wikitext.
Hypothesis 2: Experienced editors overwhelmingly prefer wikitext. Draftquality for new-accounts using VisualEditor will skew towards 'suck', because you're measuring more experienced editors in wikitext vs newbies in VE.
I find it hard to imagine any valid use for the results when you don't know what you're measuring. I can however imagine some invalid uses for a collection of random numbers.
Edit: Perhaps it would aid my understanding if you identified how you wanted to use the data, rather defining the data to be collected. It's the intent here, which will help me understand if I'm mistaken.
