Account creation user experience/Launch

Notes for the account creation user experience work by editor engagement experiments regarding which test version to launch permanently.

Comparing A/B test results

Summary:

• ACUX-1: increase in conversion by 1-2%, which would lead to ~150 new accounts a week depending on seasonality.
• ACUX-2: The purpose of this test was to measure how much we gain or lose from adding the benefits section. We expect no more than a 5% difference either way, but we could be surprised.
• ACUX-3: While the 1-2% improvement of ACUX-1 was good enough to integrate the design improvements, the amount of effort that must go in to making the client-side validation and associated API is much larger, and therefore we expect much better gains if we're going to expend that effort.

Integration options

We're not going to leave the old form, however, which version of the new form we integrate will depend on the results, compared above. Here are our two options, based on consensus in the team.

Option 1: full integration incl. clientside validation and API

If we see an increase of 10% or better, then we will integrate the version of our account creation improvements which include client-side validation and an API. This would result in 300-400 additional registrations a day, or 2,000-3,000 a week (depending on seasonality).

Option 2: new design, no validation updates, hand-off API

If we see conversion gains less than 10%, we will integrate the version with reduced scope, which modernizes the site's design, but doesn't change the validation. This will still give us improved numbers of registrations per week, but will require far less work to integrate.

Results

For the results from the third round of testing, we saw a 4% gain in conversion. Though this is good, it clearly does not meet the target set above.