Beta Features

Purpose/Goals
The primary purpose of Beta Features is to allow for Wikimedia designers and engineers (from the Foundation and community alike) to roll out technical improvements in an environment where a large numbers of users can test, give feedback, and use these features in real-world settings. The secondary purpose of Beta Features is to provide a path so that helpful, well-designed gadgets can be integrated into core after vetting, testing, and review from Wikimedia Design and Engineering.

Current Implementation (in progress)
The code is being worked on as an extension, Extension:BetaFeatures; you can use it on a test wiki here (note: you will need to create an account).

Access
Beta Features' preferences will be available in a Special:Preferences tab, accessible via the personal bar at the top of the page for logged-in users.

Appearance
The layout is purposefully different, to both inspire interest and propose a departure from the standard layout and complexity of the existing user preferences.



Functionality
For our first minimum viable product, users will be able to:
 * opt-in to individual features (manually)
 * enroll in all new features as they are released (automatically)

To use either function, users will select the appropriate checkbox, then click 'Save', which will automatically save their BetaFeatures preferences.

In future releases, as new features are made available to users, other functions will be added, as outlined below. For example, notifications can be sent out via "Echo" when new features are added, as outlined in the Future releases section below.

Tasks
Here are some of the development tasks we are working on for the first release of BetaFeatures.

Browser and Technical Compatibility
As a user, I want to know if a beta feature doesn't work for my browser or technical configuration. (See Mingle ticket #35)

Browser and technology compatibility can vary from one feature to another (VisualEditor in particular isn't available for IE yet). Other features may require Javascript, CSS or other technical configurations as well, that could cause them to malfunction if not enabled.

So each beta feature extension needs to have a blacklist of platforms that it cannot support, stored in Javascript, JSON or other machine-readable format that can be sent to Beta Features.

The Beta Features (or the extension) then needs to check whether the user's configuration matches one of these restricted platforms, so it can let the user know if their platform is not compatible with that feature.

Last but not least, we also need to inform the user about features that are not compatible with their particular browser or configuration. For example, we may want to display a message like: "This feature is not compatible with your browser yet."

This is required so the user understands why they cannot enable that feature (even though it's still checked in their preferences -- so they can use it on other platforms).

That message could be bolded and/or colored in red, to provide a visual cue that this feature will not work. Maybe the entire feature could be grayed out, to make it even more explicit?

Here are the types of notes we may need to display, when a feature is not compatible for the user's configuration:
 * Notes about browser compatibility
 * Text displayed to user "This feature is not compatible with your browser yet."
 * Notes about JS compatibility
 * Text displayed to user "You must enable JavaScript in your browser to use this feature."
 * Notes about CSS compatibility
 * Text displayed to user "Your site skin or custom CSS is not compatible with this feature, to enable it use the default site skin."

Only one error message should be displayed at any given time, the order of display should be as follows e.g. if there is both a browser and JS compatibility issues the user would only see the JS issue.
 * 1) CSS/Skin compatibility
 * 2) JS compatibility
 * 3) Browser compatibility

VisualEditor Opt-In
As a user, I want the option to enable Visual Editor, if it is not available by default on this site. (See Mingle ticket #43)

Some sites require user opt-in for the VisualEditor (the English and German Wikipedias in particular aren't ready yet to make this feature available by default). So we would like to make the basic version of VisualEditor available as a prominent BetaFeature -- but for those sites only. For example, we could handle this issue by having a site whitelist/blacklist for each beta feature -- or there may also be other technical solutions that work just as effectively to achieve the same results.

Note that on sites where the basic version of VisualEditor is available by default, some users will want the option to opt-out and may need some guidance on how to do this. Because BetaFeatures will also have an option to enable 'Visual Editor - Extra Features', some users may get confused. So we may want to include a sentence at the end of the 'Extra' description to let users know about that basic version, with a link to a page that tells them more, including how to disable it.

Finalize VisualEditor Extra Features
Determine which 'safe' features to include in VE Extras and update extension accordingly. Add feature list on New Features Page (with thumbnail & description).

Popup Guide
As a user, I want to know what the 'Beta' label in my personal menu means. (See Mingle ticket #49)

Show a pop-up guide that explains what Beta Features is, the first time you load a page that contains the 'Beta' label in your personal menu.

It would show a couple lines explaining what this 'Beta' label does and point to it (between 'Preferences' and 'Watchlist'), as so:

'Introducing $Sitename Beta Features! Beta Features lets you test and give feedback about new features before everyone else gets to see them. Actions: Maybe later     Try Now'


 * 'Try Now' would link to the Beta Features page
 * 'Maybe later' would simply close the popup guide

See mockup in Mingle ticket #49.

More tasks may be added here, as needed -- as well as on our Multimedia team's Mingle project management site.

Features
At this time, we are considering these features for our first release:


 * VisualEditor - Basic Opt-in Version (1)
 * VisualEditor - Extra Features
 * Typography Update
 * Media Viewer (if ready)

(1) This option would only be for sites that require opt-in. It would not be available on sites that provide the VE tool on an opt-out basis.

We invite product teams that want to test features through this project to write up detailed descriptions on this New Features page.

Schedule
Our goal is to release a first barebones version of Beta Features in early November, with at least a couple features to start with.

Here are our target dates for the first release:
 * Oct. 23 - Code reviewed and merged for remaining tasks (e.g. Browser Compatibility, VE Opt-in)
 * Oct. 24 - First features staged for early testing on beta site (VE Opt-in, VE Extras, Typography, Media Viewer)
 * Oct. 28 - All feature packages ready, info pages complete on MediaWiki.org
 * Oct. 30 - Publish blog post, linking to all feature info pages
 * Oct. 31 - Release to MediaWiki.org and test extensively
 * Nov 1 - Pre-announce Beta Features on all large wikis (with community team)
 * Nov 4 - Release to sister projects (non-Wikipedia)
 * Nov 7 - Release to all Wikipedias

The current goal is to try to ride the platform train, if possible (typically on Thursdays). However, special deploy windows may be needed along the way.

The extension would first be deployed on mw.org and test sites on a Thursday, then the following Monday on sister sites, and Thursday on all wikis.

We may adjust this schedule as needed, based on our progress with deliverables, and subject to team review.

Contents
Here are the proposed headers and descriptions for the first features at launch (this is placeholder copy, to be edited collaboratively):

Top header: Here are some new features we're considering for <$wgSitename>. Please try them out and give us your thoughts, so we can improve them based on your feedback.


 * ☐ Enable new beta features as they are released


 * VisualEditor
 * Try out VisualEditor for a better editing experience. This is the basic version of this new tool. (1)

Links: Info | Discussion

''(1) This VisualEditor - Basic Opt-in Version would only be for sites that require opt-in for VE. It would not be available on sites that provide the VE tool on an opt-out basis. ''


 * VisualEditor - Extra Features
 * Get the latest features inside VisualEditor. As we develop tools and make changes for the editor, we make them available for testing ahead of general release. Please remember to always review your changes before saving.

Links: Info | Discussion


 * Typography Update
 * Try out this new look that makes text more readable, to benefit readers and editors alike. This beta feature uses serif headings for sections and sans serif for body copy to clearly differentiate them from body paragraphs, for a better visual experience that works consistently across desktop and mobile platforms.

Links: Info | Discussion


 * Media Viewer
 * Improve your multimedia viewing experience with this new tool. It displays images in larger size on pages that have thumbnails. Images are shown in a nicer lightbox overlay, and can also be viewed in full-size.

Links: Info | Discussion

''In coming days, we will edit this placeholder copy and add graphics, as well as links to overview and discussion pages for each feature. To see a preliminary version of this tool, create an account on this prototype site], then log in and click on 'Preferences > Beta features'.''

Creating Your Own Beta Features
Beta Features help Wikimedia designers and engineers test out new features or significant changes to existing features.

To add your project to Beta Features if must meet the following basic requirements: Once you've met these requirements, follow the next steps for preparing your feature and required asset.
 * Not significantly degrade site performance;
 * Not noticeably degrade perceived performance of the the site, or the user's system;
 * Not crash the user's browser;
 * Not cause data loss, or corruption; and
 * Contribute positively to the user's experience of the site, and be additive in nature. e.g. Beta Features cannot be used to remove site features or functionality without adding features meant to replace what was removed.

Research
Here is a preliminary list of research questions we hope to answer about the Beta Features tool, for discussion purposes.

Beta Features Adoption
Here are our goals for researching Beta Features adoption. (See also this Trello ticket, as well as Mingle ticket #55).


 * How many users are opting into beta features?
 * What's the breakdown of adoption by user tenure?
 * What are the most popular features by number of enrolled users?
 * How many users are selecting the auto-enroll option for new features?
 * What proportion of users is disabling beta features after trying them out?

No ad-hoc instrumentation seems needed to measure adoption, as all data required to obtain descriptive usage stats are already available from these sources:
 * the PrefUpdate log
 * the user_properties table (core)
 * the betafeatures_user_counts table (extension)

For this product, we would like to build a set of LIMN dashboards to track basic usage patterns, so that research can inform our next steps. This development may need to be phased over time, based on available resources. Key questions are listed below in order of priority.

Beta Features Usage
Here are some other metrics we might want to track at a future stage.


 * Overall Stats
 * How many times did users click on the  'Beta' link in the personal bar?
 * How many times did users view the 'Beta features' preference page? (impressions)
 * What is the clickthrough rate for enabling feature x based on total impressions for the Beta preferences page?

(repeat for each feature x, z, z)
 * Auto-enroll New Features
 * How many users have enabled all new beta features?
 * How many users have disabled new beta features after trying it out?
 * Individual Features
 * How many unique users enabled feature x?
 * How many unique users disabled feature x after trying it out?


 * Info & Discussion Pages
 * How many times did users click on the info page icon for feature x?
 * How many times did users click on the discussion page icon for feature x?

These metrics could be collected through a combination of EventLogging and related technologies, and visualized with LIMN dashboards (see example). We would track this data on a daily basis, as well as on a cumulative basis.

Possible Future Changes or Additions
Here are some ideas we're considering for future releases.

Guided Tour
As a user, I want to know how 'Beta' features work.

Show a guided tour that explains how Beta Features work the first time you load a page that contains the 'Beta' label in your personal menu.

Here is a proposed sequence of popups that would point to different parts of the Beta Features UI:

1) Introducing $Sitename Beta Features! 

Beta Features let's you test and give feedback about new features before everyone else gets to see them.

Actions: Maybe later     [Try Now]

2) You can try out each feature separately

From here you can see what each feature does, learn more, or join the discussion about what you like about it, what you'd change, and if you encounter any issues.

Actions: Leave Tour     [Next]

3) To turn on a feature check this box

You can always come back here to turn it off, once you try a feature be sure to go to its discussion page to let us know your thoughts on it.

Actions: Leave Tour     [Next]

3) If you want new features to be enabled by default check here

New features will be turned on for you as they are released, you'll always be looking at the most up-to-date version. Have fun, and don't forget to let us know what you think.

Actions:    [Done!]

If we implement this feature, we would not need to do the Popup Guide described above. This would be implemented using the Guided Tour technology already in use on Wikipedia.

Unsaved Changes
Right now, if I select a feature and leave the page without clicking 'Save', the preference isn't saved (understandably).

This feature would showing a message letting users know they have unsaved changes, something like:

"Your changes have not been saved. Discard changes? [ OK ] [ Cancel ]"

This would be nice to have, especially since the save button is below the fold.

Beta Link Preference
Some users may object to seeing the 'Beta' link in the personal menu (between 'Preferences' and 'Watchlist'). To prepare for this potential issue, we may want to consider a preference to let users remove that link, if they don't want it.

This could be a simple checkbox at the bottom of the 'Beta Features' preferences page, that would say something like:

"[ x ] Show 'Beta' link in my personal menu."

There may be better ways to do this than preferences. We will update this section if we decide to implement this proposed feature.

Microsurvey when opting-out of a Beta Feature
A microsurvey would be created to capture reasons why people decide to opt out of specific beta features, using a similar way to how we collect gender data for account registrations. See Trello ticket for more info about this proposal.

Echo Notifications
Beta Features will trigger Echo notifications at different times, as soon as practical. This is to increase visibility and get additional feedback, so the product teams can react, update and improve features before making a decision to integrate into core, iterate the idea further, or abandon the experiment.

Here's a preliminary list of notifications that seem desirable, subject to feasibility and team discussion:
 * New Feature Available: Sent to all logged-in users who have signed up for Beta features and enabled this notification category, when a new feature is available to test
 * New Feature Enabled: Sent to logged-in users who have enabled automatic feature enrollment  and enabled this notification category
 * Feature Updated: When major changes are made to a feature, a notification may be sent to enrolled users who have signed up for Beta features and enabled this notification category.
 * Feature Graduation (Future): When features have achieved a high level of stability, acceptance and polish, they may be chosen to be integrated by default as an extension on a project-by-project basis. 
 * Back to the Drawing Board (Future): Experiments that don't pass muster may be discontinued, and removed from the Beta Experiments program to be re-evaluated. 

Local Configuration
From time to time, it may be necessary to exclude a beta feature from a particular project, either because it may not work on that project, or at the request of its community. Typically, this use case can be supported by not deploying the particular extension for that beta feature on that project. However, there may be cases where the beta feature is not an extension, but part of core, which would cause it to appear automatically. For such cases, it may be necessary to create a local configuration file or hook that could check for dependencies or community requests for features.

Sorting Options
As more beta features are added, we may need to provide affordances for sorting the list by importance, by name, or by date. For example, we will want the option to feature an important feature like Visual Editor at the top of the page (because of its significance for the movement's strategic goals), but also give users the option to change that default sorting to list features instead by date (e.g. most recent first) or by name (e.g. alphabetically). It may also become necessary to make the page design more compact, if we are beta testing dozens of different features at the same time.