Beta Features

This is the project page for Beta Features. For a quick overview of this project, visit 'About Beta Features'.

Please let us know what you think of this program on this discussion page.

Purpose
The primary purpose of Beta Features is to allow for Wikimedia designers and engineers (from the Foundation and community alike) to roll out technical improvements in an environment where a large numbers of users can test, give feedback, and use these features in real-world settings. The secondary purpose of Beta Features is to provide a path so that helpful, well-designed gadgets can be integrated into core after vetting, testing, and review from Wikimedia Design and Engineering.

Current Version
The code is being worked on as an extension, Extension:BetaFeatures; you can use it on a test wiki here (note: you will need to create an account).

Access
Beta Features' preferences will be available in a Special:Preferences tab, accessible via the personal bar at the top of the page for logged-in users.

Appearance
The layout is purposefully different, to both inspire interest and propose a departure from the standard layout and complexity of the existing user preferences.

Here is a screenshot showing what the current version of Beta Features looks like for its first release:


 * BetaFeatures-Screenshot.png

See also the mockups below for examples of how this design may evolve over time.

Functionality
For our first minimum viable product, users will be able to:
 * opt-in to individual features (manually)
 * enroll in all new features as they are released (automatically)

To use either function, users will select the appropriate checkbox, then click 'Save', which will automatically save their BetaFeatures preferences.

In future releases, as new features are made available to users, other functions will be added, as outlined below. For example, notifications can be sent out via "Echo" when new features are added, as outlined in the Future releases section below.

Features
At this time, we are considering these features for our first release:


 * Media Viewer
 * VisualEditor - Basic Opt-in Version (1)
 * VisualEditor - Extra Features
 * Typography Update

(1) This option would only be for sites that require opt-in. It would not be available on sites that provide the VE tool on an opt-out basis.

We invite product teams that want to test features through this project to write up detailed descriptions on this New Features page.

Schedule
Our goal is to release a first barebones version of Beta Features in November, with a couple features to start with.

Here are our target dates for the first release:
 * 1 Nov - Release to test.wikipedia.org and test extensively
 * 4 Nov - Release to MediaWiki.org and test extensively
 * 5 Nov - Pre-announce Beta Features on all wikis
 * 7 Nov - Release on Commons + Meta, test extensively
 * 8 Nov - IRC office hours chat + start Commons/Meta discussions
 * 19 Nov - Review all final features before launch
 * 20 Nov - Pre-release last minute bug fixes and features
 * 21 Nov - Release to all Wikis (via platform train)
 * 21 Nov - Publish blog post, socialize
 * 22 Nov - Start discussions on all large wikis

The current goal is to try to ride the platform train, if possible (typically on Thursdays). However, special deploy windows may be needed along the way.

The extension would first be deployed on mw.org and test sites on a Thursday, then the following Monday on sister sites, and Thursday on all wikis.

We may adjust this schedule as needed, based on our progress with deliverables, and subject to team review.

Contents
Here are the proposed headers and descriptions for the first features at launch (this is placeholder copy, to be edited collaboratively):

Top header: Here are some new features we're considering for <$wgSitename>. Please try them out and give us your thoughts, so we can improve them based on your feedback. About Beta Features | Leave feedback.


 * ☐ Enable new beta features as they are released


 * Media Viewer
 * Improve your multimedia viewing experience with this new tool. It displays images in larger size on pages that have thumbnails. Images are shown in a nicer lightbox overlay, and can also be viewed in full-size.

Links: Info | Discussion


 * Typography Update
 * Try out this new look that makes text more readable, to benefit readers and editors alike. This beta feature uses serif headings for sections and sans serif for body copy to clearly differentiate them from body paragraphs, for a better visual experience that works consistently across desktop and mobile platforms.

Links: Info | Discussion


 * VisualEditor
 * Try out VisualEditor for a better editing experience. This is the basic version of this new tool. (1)

Links: Info | Discussion

''(1) This VisualEditor - Basic Opt-in Version would only be for sites that require opt-in for VE. It would not be available on sites that provide the VE tool on an opt-out basis. ''


 * VisualEditor Formulæ
 * With this opt-in feature, you can add support inside VisualEditor for creating and editing mathematical formulæ like algebra or equations on pages you're working on.

Links: Info | Discussion

''In coming days, we will edit this placeholder copy and add graphics, as well as links to overview and discussion pages for each feature. To see a preliminary version of this tool, create an account on this prototype site], then log in and click on 'Preferences > Beta features'.''

Creating Your Own
Beta Features help Wikimedia designers and engineers test out new features or significant changes to existing features.

To add your project to Beta Features if must meet the following basic requirements:
 * Not significantly degrade site performance;
 * Not noticeably degrade perceived performance of the site, or the user's system;
 * Not crash the user's browser;
 * Not cause data loss, or corruption; and
 * Contribute positively to the user's experience of the site, and be additive in nature. e.g. Beta Features cannot be used to remove site features or functionality without adding features meant to replace what was removed.

To check that these requirements are met, be sure to test your new feature on this WMF beta server for at least one week before deploying to production. This testing period is intended to catch any serious bugs before jeopardizing users on production.

Once you've met these requirements, follow the next steps for preparing your feature and required asset.

Tasks
Here are some of the development tasks we are working on for the first release of BetaFeatures.

Browser and Technical Compatibility
As a user, I want to know if a beta feature doesn't work for my browser or technical configuration. (See Mingle ticket #35)

Browser and technology compatibility can vary from one feature to another (VisualEditor in particular isn't available for IE yet). Other features may require Javascript, CSS or other technical configurations as well, that could cause them to malfunction if not enabled.

So each beta feature extension needs to have a blacklist of platforms that it cannot support, stored in Javascript, JSON or other machine-readable format that can be sent to Beta Features.

The Beta Features (or the extension) then needs to check whether the user's configuration matches one of these restricted platforms, so it can let the user know if their platform is not compatible with that feature.

Last but not least, we also need to inform the user about features that are not compatible with their particular browser or configuration. For example, we may want to display a message like: "This feature is not compatible with your browser yet."

This is required so the user understands why they cannot enable that feature (even though it's still checked in their preferences -- so they can use it on other platforms).

That message could be bolded and/or colored in red, to provide a visual cue that this feature will not work. Maybe the entire feature could be grayed out, to make it even more explicit?

Here are the types of notes we may need to display, when a feature is not compatible for the user's configuration:
 * Notes about browser compatibility
 * Text displayed to user "This feature is not compatible with your browser yet."
 * Notes about JS compatibility
 * Text displayed to user "You must enable JavaScript in your browser to use this feature."
 * Notes about CSS compatibility
 * Text displayed to user "Your site skin or custom CSS is not compatible with this feature, to enable it use the default site skin."

Only one error message should be displayed at any given time, the order of display should be as follows e.g. if there is both a browser and JS compatibility issues the user would only see the JS issue.
 * 1) CSS/Skin compatibility
 * 2) JS compatibility
 * 3) Browser compatibility

VisualEditor Opt-In
As a user, I want the option to enable Visual Editor, if it is not available by default on this site. (See Mingle ticket #43)

Some sites require user opt-in for the VisualEditor (the English and German Wikipedias in particular aren't ready yet to make this feature available by default). So we would like to make the basic version of VisualEditor available as a prominent BetaFeature -- but for those sites only. For example, we could handle this issue by having a site whitelist/blacklist for each beta feature -- or there may also be other technical solutions that work just as effectively to achieve the same results.

Note that on sites where the basic version of VisualEditor is available by default, some users will want the option to opt-out and may need some guidance on how to do this. Because BetaFeatures will also have an option to enable 'Visual Editor - Extra Features', some users may get confused. So we may want to include a sentence at the end of the 'Extra' description to let users know about that basic version, with a link to a page that tells them more, including how to disable it.

Finalize VisualEditor Extra Features
Determine which 'safe' features to include in VE Extras and update extension accordingly. Add feature list on New Features Page (with thumbnail & description).

Popup Guide
As a user, I want to know what the 'Beta' label in my personal menu means. (See Mingle ticket #49)

Show a pop-up guide that explains what Beta Features is, the first time you load a page that contains the 'Beta' label in your personal menu.

It would show a couple lines explaining what this 'Beta' label does and point to it (between 'Preferences' and 'Watchlist'), as so:

'Introducing $Sitename Beta Features! Beta Features lets you test and give feedback about new features before everyone else gets to see them. Actions: Maybe later     Try Now'


 * 'Try Now' would link to the Beta Features page
 * 'Maybe later' would simply close the popup guide

See mockup in Mingle ticket #49.

More tasks may be added here, as needed -- as well as on our Multimedia team's Mingle project management site.

Research
Here is a preliminary list of research questions we hope to answer about the Beta Features tool, for discussion purposes.

Here are our goals for researching Beta Features adoption. (See also this Trello ticket, as well as Mingle ticket #55).
 * Beta Features Adoption


 * How many users are opting into beta features?
 * What's the breakdown of adoption by user tenure?
 * What are the most popular features by number of enrolled users?
 * How many users are selecting the auto-enroll option for new features?
 * What proportion of users is disabling beta features after trying them out?

No ad-hoc instrumentation seems needed to measure adoption, as all data required to obtain descriptive usage stats are already available from these sources:
 * the PrefUpdate log
 * the user_properties table (core)
 * the betafeatures_user_counts table (extension)

For this product, we would like to build a set of LIMN dashboards to track basic usage patterns, so that research can inform our next steps. This development may need to be phased over time, based on available resources. Key questions are listed below in order of priority.

Here are some other metrics we might want to track at a future stage.
 * Beta Features Usage


 * Overall Stats
 * How many times did users click on the  'Beta' link in the personal bar?
 * How many times did users view the 'Beta features' preference page? (impressions)
 * What is the clickthrough rate for enabling feature x based on total impressions for the Beta preferences page?

(repeat for each feature x, z, z)
 * Auto-enroll New Features
 * How many users have enabled all new beta features?
 * How many users have disabled new beta features after trying it out?
 * Individual Features
 * How many unique users enabled feature x?
 * How many unique users disabled feature x after trying it out?


 * Info & Discussion Pages
 * How many times did users click on the info page icon for feature x?
 * How many times did users click on the discussion page icon for feature x?

These metrics could be collected through a combination of EventLogging and related technologies, and visualized with LIMN dashboards (see example). We would track this data on a daily basis, as well as on a cumulative basis.

Future Releases
Here are some ideas we're considering for future releases. They are listed below in order of priority, from a product perspective.

Beta Link Preference
Some users may object to seeing the 'Beta' link in the personal menu (between 'Preferences' and 'Watchlist'). To prepare for this potential issue, we may want to consider a preference to let users remove that link, if they don't want it.

This could be a simple checkbox at the bottom of the 'Beta Features' preferences page, that would say something like:

"[ x ] Show 'Beta' link in my personal menu."

There may be better ways to do this than preferences, since we are trying the overall number of preferences to simplify the user experience. However, this is a relatively easy task that could be implemented quickly, without too much development; this could be the most practical approach for now, if we have to move fast to respond to a large number of community requests. We will update this section if we decide to implement this proposed feature.

Beta Link When There are New Features
Another more sophisticated way to address complaints about the 'Beta' link in the personal menu would be to only show that link if there are new features you haven't seen yet (perhaps with a little number next to the 'Beta' label. Once you have seen these features, the Beta link would disappear until there are more new features -- or feature updates.

This may be a bit more elegant than implementing the 'Beta Link Preference' described above. However, this would require some new designs and more development than our current limited resources can support. So it may not be as practical or expedient as the Beta Link Preference at this time. We will update this section if we decide to implement this proposed feature.

Automatically enable all 'current and' new beta features
A number of users have told us they are confused by the current implementation of the first tickbox, "Automatically enable all new beta features". They are puzzled as to why it works that way and believe that it should enable all beta features right away, not just future ones.

As a result, we recommend changing both the functionality and the wording to say something like 'Automatically enable all current and new beta features'. This would match user expectations more closely than the current solution, which is disappointing.

Sorting Options
As more beta features are added, we may need to provide affordances for sorting the list by importance, by name, or by date. For example, we will want the option to feature an important feature like Visual Editor at the top of the page (because of its significance for the movement's strategic goals), but also give users the option to change that default sorting to list features instead by date (e.g. most recent first) or by name (e.g. alphabetically). It may also become necessary to make the page design more compact, if we are beta testing dozens of different features at the same time.

Echo Notifications
Beta Features will trigger Echo notifications at different times, as soon as practical. This is to increase visibility and get additional feedback, so the product teams can react, update and improve features before making a decision to integrate into core, iterate the idea further, or abandon the experiment.

Here's a preliminary list of notifications that seem desirable, subject to feasibility and team discussion:
 * New Feature Available: Sent to all logged-in users who have signed up for Beta features and enabled this notification category, when a new feature is available to test
 * New Feature Enabled: Sent to logged-in users who have enabled automatic feature enrollment  and enabled this notification category
 * Feature Updated: When major changes are made to a feature, a notification may be sent to enrolled users who have signed up for Beta features and enabled this notification category.
 * Feature Graduation (Future): When features have achieved a high level of stability, acceptance and polish, they may be chosen to be integrated by default as an extension on a project-by-project basis. 
 * Back to the Drawing Board (Future): Experiments that don't pass muster may be discontinued, and removed from the Beta Experiments program to be re-evaluated. 

Local Configuration
From time to time, it may be necessary to exclude a beta feature from a particular project, either because it may not work on that project, or at the request of its community. Typically, this use case can be supported by not deploying the particular extension for that beta feature on that project. However, there may be cases where the beta feature is not an extension, but part of core, which would cause it to appear automatically. For such cases, it may be necessary to create a local configuration file or hook that could check for dependencies or community requests for features.

Unsaved Changes
Right now, if I select a feature and leave the page without clicking 'Save', the preference isn't saved (understandably).

This feature would showing a message letting users know they have unsaved changes, something like:

"Your changes have not been saved. Discard changes? [ OK ] [ Cancel ]"

This feature has been partly implemented across all preferences, which all share the same basic issue, since the save button is often below the fold and hard to see. However, the current implementation uses confusing language that is redundant and a bit unclear, compared to the simpler copy proposed above. But it may not be technically feasible to implement the simpler copy, due to some browser limitations.

Guided Tour
As a user, I want to know how 'Beta' features work.

Show a guided tour that explains how Beta Features work the first time you load a page that contains the 'Beta' label in your personal menu.

Here is a proposed sequence of popups that would point to different parts of the Beta Features UI:

1) Introducing Beta Features! 

Beta Features let's you test and give feedback about new features before everyone else gets to see them.

Actions: Maybe later     [Try Now]

2) You can try out each feature separately

From here you can see what each feature does, learn more, or join the discussion about what you like about it, what you'd change, and if you encounter any issues.

Actions: Leave Tour     [Next]

3) To turn on a feature check this box

You can always come back here to turn it off, once you try a feature be sure to go to its discussion page to let us know your thoughts on it.

Actions: Leave Tour     [Next]

3) If you want new features to be enabled by default check here

New features will be turned on for you as they are released, you'll always be looking at the most up-to-date version. Have fun, and don't forget to let us know what you think.

Actions:    [Done!]

Note that some team members think this guided tour seems like overkill for such a simple tool as Beta Feature, which is pretty self-explanatory. If we implement this feature, we would disable the Popup Guide described above.

Microsurvey when opting-out of a Beta Feature
A microsurvey would be created to capture reasons why people decide to opt out of specific beta features, using a similar way to how we collect gender data for account registrations. See Trello ticket for more info about this proposal. This seems nice to have, but implementing micro-surveys could be a pretty labor-intensive task, which could be outside the scope of this project.