Wikipedia.org Portal A/B testing

Background
We want to ensure that the Wikipedia.org Portal page is easy to use and understand while promoting further discovery into Wikipedia and it's various sister projects.

The Portal page has around 10 million daily pageviews. This is awesome. However, a majority of our visitors simply leave the page without doing a search, selecting a language link, or clicking into a sister project page. We want to change that by presenting intriguing and interesting content with a more modern look and feel to the page.

The Discovery Department's Portal Team recently updated the search box. The search box is now more prominent and displays meta data in the dynamic type-ahead search results to help visitors choose exactly what they were looking for.

Now, it's time to refresh the page to be more dynamic and visually stimulating, so that more people can easily access the content available in the Wikimedia projects. To do this, starting in Q4 2015/2016 (April - June 2016), the Discovery Department's Portal Team will conduct a number of A/B tests and analyze the data from our visitors. Once testing and analysis is done for each test, the Portal Team will publish the results, and if the test was successful then we'll look at pushing the experiment to production.

Below is a list of possible A/B tests that the Portal Team may perform. Feedback and input is welcomed on the talk page. The mockups represent an idea of what we want to do, and may not represent the final product.

Rationale
"Why this test?" The Portal Team is curious as to how many of the visitors to the Portal page do not have English set as their preferred browser language.

Description
"What is this test?" In this test, the portal code will detect what the visitor's browser's preferred language(s) are and then re-sort the Top Ten language links around the globe to match those language preferences.

Also included in this test, will be a localized language phrase positioned below the Wikipedia work mark: The Free Encyclopedia. The phrase will be based on the visitor's first preferred browser language. If there isn't a translation for the preferred language, the English phrase will be displayed. The same phrase that is currently underneath all the top 10 language links will be removed to provide more white space and a more cleaner look and feel to the page.

To view the visual treatment of this test that the B Group will see, visit this page. The A Control group will see this page, which is the current implementation of the Portal page.

Rationale
"Why this test?" The Portal team wants to find out if the listing of languages by article count could be displayed in a more streamlined way without significantly increasing the bounce rate or decreasing the clickthrough rate.

Description
"What is this test?" In this test, there will be a selection dropdown added directly below the search box on the Wikipedia.org portal page. When a visitor clicks on the dropdown, they will see the familiar look and feel of the existing languages by article count treatment, with the top most languages by article listed first.

By inserting the article counts into a dropdown, the page will be much simpler and streamlined with additional easy-on-the-eyes white space. Included in the dropdown, will be all the languages that Wikipedia supports. All the language links will be clickable, enabling the visitor to quickly go to that language specific wiki site.

This test is not meant to affect the click-through rate of the language by article count links, but to put the sometimes confusing information into a simpler to understand interface.

Rationale


"Why this test?" The Portal team wants to see if collapsing the top 10 language links increases the click-through rate to those specific wiki sites.

Description
"What is this test?" The top 10 language links will be collapsed into a single (or possibly two) lines of links above or below the search box. There will be an additional link added to include a dropdown selection of all language wiki links.

In this test, we want to see if the different treatment will affect the average click-through rate on those primary language links. The top ten language links will in the order of the visitor's browser's language preference (if they have one) from left to right.

Rationale


"Why this test?" Once the Portal Team has done the testing listed above and the analysis shows measureable successful results, we'd like to take the next step of combining the successes into one page for another test. This test will be similar to the previous tests, in that we don't want to have a significant increase in the bounce rate on the Portal page or a significant decrease in the click-through rate for any of the links that will be displayed in a different manner. To determine the success of this test, we'll be comparing it to the baseline currently established on the dashboard for the Portal page. Ultimately, this test would go into production, once analysis is done and code is finalized.

Description
"What is this test?" This test will combine several tests that at this point, the Portal Team have successfully tested. It would entail displaying the:
 * preferred browser language(s) in order,
 * localizing The Free Encyclopedia phrase,
 * putting the top 10 language links into a new horizontal visual treatment, and
 * showing all languages (ordered by article count) in an easy to use dropdown.

Rationale


"Why this test?" Usage of the other (sister) project links is quite small in comparison to Wikipedia and the Portal Team would like to find out if adding short descriptive text to each link would create investigation into those projects by the portal visitors.

Description
"What is this test?" This test will keep the page in a state consistent with any previous successful test / production release and add in a few words to each sister project link. The few words added to the project links are meant to be short but descriptive of what the visitor to those project can expect to find.

The exact wording of the descriptive text will be determined, at some time in the future, but the accompanying draft mocked image shows one way we might want to display them. The test will be to see if the analysis shows increased click-through to the sister project sites with the addition of the descriptive text.

Rationale


"Why this test?" As a way to show more interesting articles to our portal visitors, the Portal Team will add trending and/or most viewed articles to the Wikipedia.org portal page.

Description
"What is this test?" This test will keep the page in a state consistent with any previous successful test / production release and add articles to the page to entice visitors to find out more information without having to specifically search for them. The method of determining the trending (or most viewed) articles will be very similar to how the Wikipedia mobile app determines which articles to display.

Exactly how and what articles to be displayed is still to be determined, as well as if the articles displayed will be localized to the visitor's preferred language and / or location. The test will be to see if the analysis shows increases click-through by visitors to the articles that are displayed.