VisualEditor on mobile report

This page is a report on mobile editing using the visual editor that is being produced as part of the Editing team's work from the annual plan, as described at m:Wikimedia Foundation Annual Plan/2018-2019/Audiences#Outcome 3: Mobile Contribution.

Strategic direction
The Wikimedia Foundation 2030 strategic direction calls on the Foundation to invest in knowledge equity, which “means focusing on the knowledge and communities that have been left out by structures of power and privilege, and welcoming people from every background to build strong and diverse communities."

The | Annual Plan for FY2018/2019 lays out how the Foundation aims to achieve these goals by focusing on the diversity of contributors and content as well as the need to ensure that mobile devices are fully supported in order to empower people in underrepresented regions and languages.

The Audiences department is focused on improving the mobile editing experience. Within the department, the Editing team is working to improve the experience of visual-based mobile editing by: Conducting research to understand the current visual-based mobile editing experience and define areas for improvement (Q1) Working with the community to determine a strategic approach to building tools (Q2-Q4) Developing and testing solutions to ensure pain points have been lessened or removed (Q1-Q4)

Objective of this report
The objective of this report is to document the current state of visual-based mobile web editing, and to identify any challenges that exist. This report will:


 * explore and document the end-to-end workflow for current users
 * define pain points in the workflow and larger scale problems outside the workflow
 * make proposals for addressing identified pain points
 * validate thinking and stimulate discussion with both senior management and the communities

This report is intended to define the current visual-based mobile editing experience and identify areas of challenge. The report will be updated based on feedback, but ultimately it’s intended to be static documentation of the current understanding of mobile editing (as of Q1 2018), and not to be a moving target to document of ALL the things. The Audience team will build upon this work and ultimately all action items will go into Phabricator tasks.

Definitions

 * mobile editing – Mobile Editing refers to any kind of contribution made to Wikimedia through a phone or tablet. Mobile content creators and editors access Wikimedia through native or web apps. The number of people who are using mobile devices to access Wikimedia projects is constantly increasing. Many users access Wikimedia sites exclusively this way. In particular, emerging Wikimedia communities have many more primary and exclusive mobile users compared to existing communities.


 * visual editor – the current technical implementation in the browser. Editing using wikitext is a barrier for many mobile contributors, due to the unfamiliar markup system and the difficulties created by the use of mobile devices in general. For example, smaller screens allow contributors to see and preview only a fraction of a page at a time, and the absence of common markup characters, such as  and , on the devices' main keyboard views raise the bar for technical know-how required to easily contribute. The visual editor within the mobile web experience solves many of the pain points that contributors have with wikitext on mobile, however, it has usability, user experience, and performance issues. ..


 * visual-based mobile editing – An experience that allows an editor to see what the end result will look like while the edit is being created. This is the general term for the approach to creating this kind of edit.


 * mobile web – The use of the internet through a handheld mobile device such as a phone or tablet.

History of visual editing on the mobile web
Visual editing on Wikimedia has evolved significantly over the years. Some highlights include:
 * Summer 2012: Wikitext editing on mobile Web was developed initially as a beta, and then was released for all logged-in users in July 2013.
 * Autumn 2013: Visual editing on mobile Web was developed alongside the desktop release. It was initially released to tablet-based logged-in beta users in October 2013, with a "switcher" to let users move back and forth between editors; pressing the edit pencil would always initially open the wikitext editor.
 * 2014: Visual editing was rolled out more widely, and has been opt-in available to all users on all devices for years up to September 2018.


 * November 2014: Mobile Web wikitext and visual editing was expanded to logged-out editors, initially on the Italian Wikipedia


 * April 2015: Logged-out visual editing on mobile Web was expanded to all wikis.

Quantitative metrics
See T202132.

One way that product teams gather an understanding of how their products are being used is to use quantitative metrics.

Below are some metrics that the Editing team is interested in, as well as a brief explanation of each one.

EventLogging metrics
The metrics in this section were calculated from the Edit event stream, which uses EventLogging to track various events that happen as users move through the process of making a edit. This means that these metrics share certain caveats:


 * This event stream has a sampling rate of 6.25%, meaning that roughly speaking only one of every sixteen edit sessions sends data for analysis.
 * EventLogging does not send events if the user has enabled Do Not Track in their browser.
 * These metrics are based on the events recorded during May 2018, so they may not reflect the long-term average.
 * A session's platform (mobile or desktop) was identified based on whether the user was using the desktop website or MobileFrontend. Since a user can choose to switch to the desktop website on a mobile device or vice versa, the desktop platform includes some edits made on mobile phones or vice versa.
 * This data is observational rather than experimental, so metric differences between the editing interfaces may be due to the type of users who choose that interface rather than the interfaces themselves.
 * Since users must opt in to using the mobile visual editor, its metrics are based on a small number of session (see table below) from users who are likely better informed than average.

Usage of common editing features
See T202133 and T202148.

Explanation: understanding which common editing features people use (or do not use) may affect our direction in terms of possibly building stripped-down experiences with fewer features.

Unfortunately, it was not possible to calculate these metrics, as the relevant data was not being collected. The Editing team set out to begin recording this data, and as of the publication of this report, the relevant data is now being captured in Schema:VisualEditorFeatureUse, but there is not yet enough data to perform a rigorous analysis or fill in the tables below; the tables are kept for posterity, to record the information we wanted to get, and hope to have in the future.

Edit completion rate
See T202134.

An edit session starts when a user clicks the edit button, but most sessions don't result in a saved edit. The edit completion rate gives the proportion of sessions which do result in a save, out of all those where the editor has a chance to fully load. This metric can help us figure out which platforms are more difficult for users to use.

On desktop, the visual editor has a lower completion rate, but this is due to fact that the visual editor tends to be used by less experienced users. When the rate is broken down by the user's experience (measured in edit count) at the time of attempting the edit, the visual editor has a higher completion rate within all groups except for IP editors. On mobile, the situation is reversed: the visual editor has a higher overall completion rate, due to the fact that as an opt-in feature it tends to be used by better-informed editors. In each experience group except IP editors, it has a lower completion rate than the wikitext editor.

Drop-off along the edit funnel
See T202147.

To better understand the majority of edit sessions where the user does not save an edit, we can look at where in the the process users tend to drop off. This can help us target our improvements on those steps.

Caveat: this overall chart does not include  or   since not all editors log those events. Almost 40% of users drop off before the editing interface is ready to be used; these may be cases where users clicked on the button accidentally or where the interface took too long to load. These sessions were excluded from calculating the edit completion rate above.

Caveat: Some editors do not log certain events, so those steps are blocked off.

Time to save
See T202137.

This metric tracks how long it takes for users to complete their edits (in technical terms, how long it takes for completed sessions to get from the  event to the   event). Understanding this can help us optimise the experience to make it more efficient for editors.

Caveat: it can be difficult to interpret the meaning of a higher time to save, as it can mean either that users are making larger edits because they're more comfortable with the editor (then higher is better), or it could also mean that usability issues slow down their editing (then higher is worse).

Overall, users take longer to save on either platform when using the visual editor.



The median time to save is similar on mobile and on desktop, but the distribution is tighter on mobile. This suggests that simple edits take longer to make on mobile, and users also tend not to make long, complex edits on mobile.

Use of the visual and wikitext editing interfaces
See T202138.

The edit history data uses edit tags to track which edits are made using the mobile site or the visual editor, but it can be difficult to determine which of the remaining edits were made using the wikitext editor and which were made with the rollback tool or third-party tools using the editing API since all these edits are left untagged.

Since the edit event logging does not run for API edits or rollbacks, we can look at the breakdown of  events to get a better sense of the relative use of the two editing interfaces.

Caveat: Some first-party editing tools like undo and some third-party tools still involve loading the wikitext editor even when user rarely uses it. These tools do generate edit events and so are included as wikitext edits in the breakdown below; whether this is correct or not is open to interpretation.

Edit history metrics
The metrics in this sections are calculated from edit history preserved by MediaWiki. The dataset included edits from the past year (October 2017 to September 2018), so unlike the EventLogging metrics above, they are not likely to be impacted by one-time events.

They also share some caveats:


 * As with the EventLogging metrics, a edit's platform (mobile or desktop) was identified based on whether it was made with the desktop website or MobileFrontend. Since a user can choose to switch to the desktop website on a mobile device or vice versa, the desktop platform includes some edits made on mobile phones or vice versa.
 * Because grouping edits made by a single unregistered (IP) editor is unreliable, these metrics look only at registered editors only.

Per-interface retention rates
See T202146.

Analyzing how likely a user is to continue using a given editing interface can help us to understand how appealing or useful it is and to better target experiences to increase those qualities.

The following analysis calculates per-interface retention by looking at a six-month period and taking each user's first edit with each distinct interface during the period as a starting point. Then, it looks at whether the user made more edits using the same interface in each of the first 26 weeks after that starting point—that is, whether they were retained in each of those weeks.



Per-user platform mix
See T202135.

Some users edit only using the desktop site, some edit only using the mobile site, and some do both. Understanding the exact breakdown can help us understand to what extent the mobile editing experience is currently serving as a users' only method of editing. Overall, 25% of editors make some mobile editors. Among low-experience editors, these mobile editors are primarily mobile only, but among highly-experienced editors almost all mobile editors make some desktop edits as well.



Explore the end-to-end workflow for current users
The editing process begins when the user taps edit. Upon tapping edit, the user is taken to a modal dialogue that presents them with three options: edit anonymously, create account, or sign up.

Upon choosing one of the actions and completing it, the user is immediately taken to the source editor, which shows them the wikitext for the article. On many wikis, the user's viewport is filled with template syntax for the infobox, so a new user can't immediately understand how what they're seeing now relates to what they were reading.

The visual editor is available on mobile web, but it is not exposed to the user in a clear way. The edit pencil at the top of the viewport allows users to switch to visual editing, and when they do so, they are presented with the mobile web version of the visual editor. This version of the visual editor has stripped down functionality compared to the desktop version:


 * certain things have been disabled, such as image editing
 * some editing tasks are only available in a simpler form, e.g. math formula editing

Define pain points in the workflow and larger scale problems outside the workflow
An important part of improving products is understanding where problems and pain points currently exist—once the problems are understood, improvements can be made. One of the traps it is easy to fall into is to assume that the problems you have with the product are the problems that everyone has—whilst this is a useful first approximation, product teams carry out processes in order to gather data that is more rigorous, more comprehensive, and more reliable.

Process
Visual Editor Expert Review Framework – This framework was developed by the team designer in conjunction with the Audiences Design team. It's based on The 10 Heuristics for User Interface Design, the Wikimedia Foundation values, and design principles.

Methods

 * 1) Expert review – UX, UI, Accessibility, Language and Engineering professionals will evaluate the tasks provided against the visual editor expert review framework.
 * 2) Usability test usertesting.com research method + demographics screener + protocol – Mobile Web Contributors will attempt to complete the tasks provided.

Personas


Although we strive to design features for all humans, in the scope of this project we have identified a type of editor that represents the "persona" of users who we believe that we are in the best position to support and delight.

These users have one or more of the following attributes:


 * are using the browser on their mobile device to edit
 * are using the visual editor within Wikipedia.org to edit
 * have interest/motivation to fix or improve existing pages (not create new pages or perform administrative functions)

This focus allows the product team to craft a rubric for prioritizing attention to contributor attributes.

The prioritization is:


 * 1) Experienced editors who are experience on mobile
 * 2) Experienced editors who are inexperienced on mobile
 * 3) Inexperienced editors who are experienced on mobile
 * 4) Inexperienced editors who are inexperienced on mobile

Usability test


Visual Editor on Mobile Web Usability Testing via Usertesting.com:

Of the 6 tasks tested (getting to edit mode, adding and formatting text, saving edits, adding internal links and adding citations), only the first 3 were performed without much difficulty or confusion. Though the participants (who hadn’t previously edited Wikipedia) gave generally positive feedback about the editing process, one participant gave up on the last task in frustration. Not only did participants experience bugs and various feature elements needing improvement, they were sometimes entirely unaware that they didn’t complete a task or completed it incorrectly. Given that the tasks were relatively basic/MVP-level, further design/research iteration is recommended.

Heuristic analysis
In addition to performing a usability study, a study performed by internal (within the Wikimedia Foundation) and industry experts provided insight into the quality of the tool. A dozen experts were chosen in the fields of product, engineering, accessibility, design and RTL scripts. The experts took the usability test and then rated the tool against a values-based rubric that the product designer created based off of Jakob Neilsen's 10 Heuristics for User Interface.

Overall, the product did not perform well against the criteria in the rubric. Here is a report-card themed summary of that information. Each report card mirrors a value presented in the Wikimedia Mission.



Report Card 1: We Welcome and Cherish Our Differences
* Letter case is only relevant for languages that are written in Latin, Cyrillic, Greek, and Armenian alphabet.

Make proposals for addressing identified pain points
Two broad categories for ways to address the problem:*

1. Look at the UX issues in the end-to-end workflow and improve (and, possibly, remove) steps: e.g. is it correct to have the first step be a full-screen three option modal? do we have evidence that it's bad, or just intuition?

2. Come up with alternative visual-based editing experiences, and expose those to users'''

Technical research

 * What are the technical constraints that make the current VE mobile experience difficult to improve or change?


 * What could we do to make that easier?


 * What things have been persistent challenges due to those technical constraints?


 * Would a different visual-based system have different technical constraints?

Current software constraints

This section includes issues that exist due to the current state of visual editing software and have the potential to impact mobile editing.

Platform


 * Mobile Safari does strange viewport scrolling when the keyboard is opened, mostly based around the assumption that you are focussing a short input field. MobileFrontend has some hacks to work around these, but they may need more tweaking, especially to work with the OOUI dialog system
 * Current Action: improve viewport scrolling (Phabricator ticket to be added shortly)
 * Mobile OSes often unload pages from memory if you switch tab or application, especially iOS. This would result in the editor being reloaded when you return to that tab. Fortunately with auto-save this is somewhat less catastrophic, but there may be more work to remember other user state (e.g. if the user was in the middle of creating a citation)
 * Current Action: research the rate at which this happens by adding logging for autosave restoration, which will help us determine how high of a priority it is to address (T202656 and T202148)
 * Screen space constraints affect certain editing tasks, e.g. gallery editing: currently thumbnails and editing toolbars are squashed onto a small screen; difficult to solve without rethinking the workflow
 * Current Action: improve consistency of mobile dialogs (T202575)

Code Repositories

 * The existing mobile wikitext editor code is spread between two repositories (Minerva and MobileFrontend), which makes making and reviewing changes more difficult. Some code specific to mobile visual editor is in yet another repo, VisualEditor. Streamlining the repositories will improve performance for all users
 * Current Action: move mobile editor code (T198765)

Performance

 * Mobile web is much slower than web apps, so we may need to anticipate actions not happening immediately and guard against it in the UI (e.g. prevent buttons from being clicked twice)
 * Current Action: current user testing will determine whether this is a concern we need to address

Input and language support
Types of input method can vary significantly


 * Input via hardware keyboard / software keyboard / full-screen handwriting / voice / ...
 * Candidates inserted as single code units / grapheme clusters / words / phrases
 * Different paradigms for text correction

Current Action: see VisualEditor mobile IME analysis for more information on research to better understand these constraints

Mobile development limitations
These limitations are concerns we must bear in mind as we make design and product decisions.

Platform-specific limitations

 * Mobile Safari (iOS) does not track the visible viewport size, which makes it extremely challenging to attach toolbars to the bottom of the viewport (which the Google Docs App does, for example)
 * Some browsers don't use read/write from/to the clipboard programmatically, so we can't make our own copy/paste buttons in the UI
 * Safari's webkit engine in general is considered much buggier than Chrome's blink
 * We have less control over text input than in apps. For example in Mobile Safari a context menu with [B/I/U/Copy/Paste/...] is shown on text selections, which we can't suppress

Performance

 * Devices in the wide variety of targeted markets are extremely variable in form both factor and performance. We will need to account for that variability in planning
 * Bandwidth, latency and intermittent connections will vary greatly across target audiences. Variability in performance means that certain intensive processes (e.g. visual diffs of long documents) will have particularly variable user experience

Target wikis
To measure and quantify the impact of the interventions that are better being made, the team has identified a set of target wikis where we will focus our efforts.

The criteria are Wikipedias sorted by majority-mobile editors percentage with over 200 monthly editors. From this list, some larger wikis have been removed in order to target medium-sized wikis that are likely to benefit from the changes we hope to make. The list of target wikis is below.

For our purposes, these can be broken down into 3 groups based on the attributes of their language:

Validate thinking and stimulate discussion with both senior management and the communities
We are asking for thoughts and reactions on these pain points from as many communities as possible:


 * Have you experienced similar frustrations with the visual editor on the mobile web?
 * What do you think works well with the visual editor on the mobile web?
 * Did we miss anything?

Please join the conversation on the the talk page.

Next steps

 * 1) Remote team brainstorms
 * 2) *The Editing team will split into small groups of around four people (breakout groups), to brainstorming and discuss items and problems that were raised in the heuristic and usability studies.
 * 3) Team shareout 1
 * 4) *The breakout groups will share their brainstorming with the other breakout groups.
 * 5) Design intervention planning
 * 6) *The product manager and designer will create some proposals for how the problems can be addressed.
 * 7) Feasibility audit
 * 8) *The proposals will be audited for their feasibility from multiple perspectives, including technical difficulty, design complexity, social blockers, and more.
 * 9) Team shareout 2
 * 10) *The team will share the findings of the audits and discuss further.
 * 11) Call for community and management input
 * 12) *A call for input will be circulated so that members of our communities and Wikimedia Foundation management can give input on the proposals and thoughts on the next steps. (Feedback is, of course, welcome at any stage in this process, as well as in this specific step.)
 * 13) Triage Phabricator to reflect current thinking