Reading/Web/Desktop Improvements/Repository/First Prototype Feedback Report

= Introduction = In December we published a prototype of the first few features of the desktop improvements project for community feedback. The prototype presented a collapsible version of the sidebar, a fixed-width layout, and a new, more prominent location for the language switcher. We received detailed, thoughtful feedback from over 200 logged-in users, across five languages. The feedback is mostly positive,  with the majority of users seeing the changes as an improvement over the current design. However there were also some areas of concern. Many of the issues raised were due to bugs in the prototype (particularly with the language switching menu), while others exposed areas for improvement that we will iterate on and/or keep an eye on during development. This report highlights the main points raised by the feedback, both positive and negative, and our plans going forward in response to this feedback.

= Feedback method = The prototype was shown to logged-in users using central notice on the following wikis:


 * Basque Wikipedia
 * French Wikipedia
 * French Wiktionary
 * Hebrew Wikipedia
 * Persian Wikipedia
 * Polish Wikipedia
 * English Wikipedia
 * Portuguese Wikiversity

Replies were collected using a pre-populated form with specialized questions around the changes that were presented. We also added questions around general thoughts and opinions around the changes the prototype presents. Responders were also given the option to submit their thoughts via email. The results presented contain both the feedback we received on the page as well as via email.

We would like to note that we received a lot of comments related to bugs in the prototype, especially around the language switching functionality and the Universal Language Selector (ULS). We apologize for not providing enough information on the constraints of the prototype itself and for not catching these earlier.

= Summary of results = A total of 222 logged-in users gave us feedback on the prototype. The majority of the responses were in English (#?), followed by French (#?) and Farsi (#?).

Positive


 * The majority of the logged-in users who tested the prototype preferred the new location of the language selector over the current implementation in the sidebar
 * The majority of editors liked the ability to collapse the sidebar; specifically the more focused reading view that results from the sidebar being collapsed with the content at a fixed-width

Room for improvement


 * There were a number of concerns raised with the language switcher, most of which were related to prototype bugs as well as issues relating to language switching requiring two clicks instead of one, the new language switcher being too prominent, and questions around internationalizing the languages names in the list
 * There were some concerns around the amount of white space introduced with a collapsible sidebar and fixed-width layout

Feature requests


 * Dark/night mode
 * Make the table of contents always available on the page

= Results =

Positive
Overall, most users preferred the new location of the language selector - they reported it was easier to find than the previous location. People who used the language selector often reported that it would be faster for them to switch languages. A number of users also reported that the new location is more intuitive as it follows a pattern that other multilingual sites use.

Room for improvement and iteration ideas

 * One-click language switching: currently you are able to switch to a different language Wiki with a single click (and potentially some scrolling depending on your screen height). In the prototype it takes two clicks to switch languages (plus potentially needing to scroll as well). While many people noted that the new position of the language switcher was easier to find, they also expressed concern about having an extra click in order to switch languages, especially for cases where people expect  to switch frequently. There are two ways we are thinking about this:
 * We will be collecting usage data and performing A/B tests on our test wikis to determine whether language switching increases, decreases, or stays relatively the same. It is our hope that despite the extra click, the increased prominence of the language switcher will lead to greater discoverability and ultimately more language switching.
 * We’ve sketched out an option that includes links for switching languages directly on the page, adjacent to the general language selector as shown below.04_-_DIP_-_Language_links_in_article_title_bar_alt.png

However since language switching is not relevant to all people we would have to figure out some logic that handles when to show these direct links, otherwise they could be distracting. Perhaps they could appear once someone has switched languages once, or if we think they are likely to switch languages (e.g. due to a discrepancy between their operating system language and the language of the Wiki they are reading). We have not yet settled on an exact approach and plan on revisiting this idea once the initial implementation of the language switching is in place. Any feedback on this or the above mockups would be greatly appreciated


 * Issues with the finding a given language within the Universal Language Selector (ULS): A lot of users liked the position of the new language selector but expressed concerns with the language selector itself.
 * Some of these concerns were around the relative difficulty of finding a given language within the selector and were in part due to the following bugs within the prototype itself:  the list of frequently used and suggested languages was missing, the language search was broken, and the menu was being rendered too small.
 * There were also concerns raised around the ordering of languages by region, and the spacing between the items within the selector leading to too much scrolling.

We will prioritize the suggestions and issues and are hoping to work with the Language team to make some improvements to the universal language selector. We will update more on this soon.


 * Preference for the current location over the new location with varying reasons: The main theme seemed to be the difficulty of using the universal language selector and the loss of a one-click access, both of which are addressed below.  A few also commented on the increased prominence of the language button itself. We believe that encouraging users to switch languages is very important and that, due to the current location, many are unaware this is a possibility.  That said, we agree that it might be an inconvenience to people that do not often switch languages.  We plan on discussing this further, however we are currently prioritizing former over the latter.


 * Location & prominence: The following concerns were expressed about the location of the language switcher in the prototype:
 * The current location is good and people are used to it there — Some people preferred the current location over the new location with varying reasons.  The main theme seemed to be the difficulty of using the universal language selector and the loss of a one-click access, both of which are addressed below.
 * The new location is too prominent since many people won’t ever switch languages. A main goal of this project is to increase the prominence of frequently used functionality.  Currently, the language switching links are the most frequently used links within the sidebar (link to data). In addition, we believe that allowing multilingual users to switch languages is a crucial way of promoting knowledge equity and, due to the current location, many are unaware this is a possibility.
 * The new location is good but the button is too big — we have sketched out several other options for the button treatment, shown belowShowing four treatments for the interlanguage links menu.png


 * Looks like a translation: A couple of users mentioned that placing the language selector within the article view might cause casual readers to think that when they switch languages they are merely viewing a translation of the article, rather than switching to a different wiki.  Clarifying the relationship between different language Wikis is important and we are currently conducting user testing to investigate this and understand if this confusion is occuring, and how much of an issue this would present to readers.

Other notes

 * There were some requests on displaying the English names of languages in addition to the native name (i.e. displaying both Bengali and বাংলা, rather than just বাংলা).
 * Translate the word “language” A number of users mentioned that the word “language” should be translated to the language of the wiki.  This was a mistake of the prototype. The interface will be translated into the language of the wiki in the final version.
 * Some people mentioned adding some kind of indicator in the sidebar that points people towards the new location of the language switcher — we hope to explore this.
 * Some people were wondering what the language switcher will look like on articles with no other languages available — sketch coming soon.

Positive
The majority of users who provided feedback liked the collapsed sidebar for personal use, and especially for the purposes of reading. People mentioned that it removes distractions and makes reading a more pleasant experience. One user with reading disabilities mentioned that removing the sidebar makes it easier for them to focus on the content.

Room for improvement and iteration ideas

 * Do not collapse by default for logged-out users: Some users did not like the idea of the sidebar being collapsed by default for logged-out users, and expressed a preference to the current status quo.  Their reasoning was that the items in the sidebar could raise interest in editing and the inner workings of Wikimedia projects by readers. We acknowledge that with the sidebar collapsed there are fewer total entry points to editing/contributing related pages (e.g. Recent changes, etc.), however our hope is that by reducing the amount of links on the page the ones that remain (Edit, Talk, History, etc.) will be more noticeable and hopefully see more engagement. While we don’t plan on iterating the current design based on this feedback, we will be testing A/B clicks on links to the sidebar to compare how a collapsible sidebar compares to a non-collapsible one.  We will also be monitoring whether the sidebar changes have any effects on account creation (as a proxy for conversion from readers to editors).


 * Collapse by default to everyone: Conversely, a number of users mentioned that they would prefer the collapsed sidebar to be default for readers and editors, or to remove the sidebar altogether.  Currently, we don’t have plans to pursue either of these ideas, but we welcome individual communities to discuss sidebar links and their usefulness further.


 * Refine and/or relocate the open/close trigger for the menu: While many people commented that the hamburger icon wasn’t an intuitive way to open/close the menu, only one of the 220+ users were unable to open the sidebar when prompted (link to feedback). The two most common suggestions in terms of possible improvements were: 1) when the sidebar is open the icon should change to something like an “X” or “<<”, and 2) try moving the icon closer to the menu itself, rather than having it in the site header. Sketches for these two ideas are below.

[add sketch]


 * Don’t hide random: A few users mentioned that the random page link is important to readers and a core part of the Wikimedia wiki experience.  We agree that this is an interesting idea and are exploring moving the link outside of the sidebar and into the search bar.

[add mock]


 * Collapse the sections within the sidebar as well, or reduce the amount of links in the sidebar: Currently the sidebar contains many links that are not frequently used. By making the sidebar collapsible some people worry that we are ignoring the bigger challenge which would be to remove the unused links in the sidebar. Some have suggested making the sections within the sidebar collapsible. Others have suggested cleaning up the links. The links in the sidebar are determined on a per-Wiki basis, so while we agree that folks should remove unnecessary links from the sidebar this type of effort is outside the scope of our project.

Open questions, new ideas, and other notes

 * Where did the edit button and categories go? A few users mentioned that they were not able to access the edit button and list of categories within the prototype.  This was a bug of the prototype. We apologize for not mentioning it in the directions.


 * Use a different icon: Some users mentioned that they were confused by the usage of the hamburger icon.  A few expected that this icon would open a different sidebar and others had trouble finding the icon itself.  As mentioned above, all 220+ users but one found the icon when prompted and were able to understand how collapsing and uncollapsing of the sidebar works.  We are also currently testing this icon with readers in a separate study to make sure that the purpose of the icon is understood well by that audience as well.  We are also exploring providing different states for collapsing and uncollapsing the sidebar to make it easier for users to understand.

[mockup goes here]

Context
One of the bugs in the prototype was that the content was limited to a fixed-width only when the sidebar was closed. Many people commented on how this was odd, and how much the content jumped across the page when you closed the sidebar. This was a mistake, not how we intended it to be — please see this updated prototype to understand how the fixed-width should work.

Positive
The majority of people responded positively to the fixed-width layout, noting that it resulted in a more comfortable reading experience. Several people mentioned that having margins around the content makes it easier to focus, and having shorter line-lengths is a more optimal experience for reading.

[add side-by-side of current & fixed-width]

Our rationale
There are two reasons why we believe a fixed-width layout will be beneficial to readers and editors:

1. Readability
The primary reason for a fixed-width layout is to improve readability of Wikimedia wiki pages. So perhaps the first question is: how can we know what the optimal size for readability is? There are research-based recommendations regarding optimal line lengths for readability of text, but then again Wikimedia wiki articles aren’t like most articles or web pages. They are unique (long pages, many different elements, lack of uniformity), and as such people read them in different ways than they might read a book or another online article. Our design must take into account these distinctions. Below we explain how we’ve arrived at our design recommendation for a fixed-width.

Without studying readability of Wikimedia wiki pages directly we can’t know what is optimal, but in an attempt to make an educated guess the research on optimal line length for readable text seems like a good starting point. The research and recommendations in this area seem to be well established and hardly contested. The Wikipedia page on Line Length provides a good overview, as does the essay Size Matters: Balancing Line Length And Font Size In Responsive Web Design by Professor Laura Franz. The popular recommendation is that there should be between 40 and 75 characters per line. In comparison, a Wikimedia wiki page on a browser window at 1280px* has a character count of ~170 characters per line, and that’s at the small end of the screen size spectrum. (*The most common computer screen size, accounting for 22% of users, is 1366px wide according to StatCounter; imagining a browser window at nearly full width you end up with ~1280px). Contrast that with an online New York Times article at ~64 characters per line, a BBC article at ~82 characters per line, or a Reddit post or comment at ~87 characters per line. Then factor in that on Wikimedia wiki the character count per line grows as the screen width grows (whereas on the other sites mentioned the character count per line remains the same, the result of fixed-width layouts). So on the second most popular screen-size, 1920px (21% of users), the character count per line is ~262 (again assuming a browser window at nearly full-width), more than three times the recommended value. So as a starting point we know that for paragraphs of uninterrupted text we are well over the recommended limit.

The question then becomes: why not limit the width of Wikimedia content such that we achieve the recommended line length, as other online content sites seem to? The short answer is: because our pages are different, and therefore people read them differently. Wikimedia wiki pages contain large amounts of information and multiple media types (text, tables, images, graphs, infoboxes, etc.), and they are not uniform from one page to the next. As a result, people have a greater need to skim and search within pages than they would when reading a typical online article or book (this is supported by our research around reading time on Wikipedia – link). So while the line length recommendations provide a good starting point, we can’t necessarily follow them directly due to the uniqueness of content and reading behavior ok Wikimedia wikis (for more information regarding different types of online reading please see this 2006 study conducted by the Nielsen Norman Group: link). Additionally, because Wikimedia wiki pages contain many floating elements aside from text, it is not straightforward to achieve a specific number of text characters per line.

[add mock of what an article would look like if we aimed for the 40–75 character recommendation]

So how do we find a width for the content that accommodates both focused/engaged reading (shorter line lengths, less density) and scanning/searching/skimming (longer line lengths, more density)? Based on the above information it seems safe to assume we should limit the width by some amount, while still enabling readers to skim and search around, obtaining a visual map of the page. As stated earlier, without studying this directly (which we hope to be able to do during the course of this project) it is impossible to know what the optimal width is, but we can make an educated guess. While Wikimedia wiki pages are not uniform and contain a huge amount of variation in terms of layout, there are two common experiences we might want to anchor around when making this consideration.


 * 1) The top of an article with a paragraph of text situated next to an infobox [add image]
 * 2) The middle of an article, a paragraph with no elements interrupting it [add image]

We can consider these two experiences at various widths, counting the character length per line for each: Based on the recommended line length it seems like somewhere around 700px is a reasonable width. However we also may want to consider that there are other common inline elements that are wider than infoboxes (resulting in narrower text columns), so it might benefit us to be a bit more forgiving (i.e. slightly wider). On the other side of the spectrum at 1000px wide an uninterrupted paragraph of text is ~154 characters long, just about double the upper limit of the recommended range.

...talk about the possible benefit of basing the width off a multiple of the infobox width in order to achieve some visual harmony

...arrive at something within the range of 845px–1100px

2. Establishing a common reading experience
The second reason we think a fixed-width could be beneficial to the reading experience is because it would establish a common size for editors to anchor on when making decisions about how wiki page contents are arranged. Currently an editor might be editing a page at a width of 1500px, while a reader reads it at a width of 1200px. By implementing a fixed-width we don’t remove this discrepancy completely (because anyone with a screen smaller than the fixed-width would still see a “non-standard” version of the article), however we would be greatly limiting the range of variation. Instead of there being a discrepancy of X, there would be a discrepancy of Y [add image to communicate this point]. Our hypothesis is that the more similar the editing experience is to the reading experience, the more connected editors will be with the readers of the article. What you see (as an editor) is what you get (as a reader).

Breaking templates / content / special pages / etc.

...add a paragraph here about issues with limiting the width of wiki pages, such as panoramic images, wide tables, etc.

Overall feedback and other feature ideas

 * Build a night mode A significant amount of users made requests for a dark/night mode to be added to the scope of the project. While we were not considering this in the initial scope, we realize this is a request we’ve seen multiple times in the past as well.  We are currently in discussions on whether we can potentially add dark mode to the scope of the project and how this might affect our timeline. We will update once we know more.


 * The table of contents should always remain in view: Many people commented that having access to the table of contents regardless of how far down the page they had scrolled would be, in one person’s words, “extremely useful”. We are planning to explore this functionality as part of this project.


 * Build a sticky header A lot of users expressed interest in having constant access for commonly used actions at the top of the page.  Users requested to have access to commonly used pages and actions (talk, history, edit) as well as other functionality such as the search bar.  A similar request was the need to return to the top of the page and/or to switch sections in a quick manner. We plan on exploring this option as a part of this project.  Our current thinking is that we would like to introduce a sticky header with commonly used actions as well as a persistent table of contents.  Below are some examples of our ideas so far.  We’d appreciate any feedback around this you might have.

[mockups here]


 * Clean up the top bar There were some thoughts and ideas around cleaning up the links at the top of the page by collapsing them or moving them under dropdowns.  This is also something we’re considering for the project

[mockup here]


 * The logo is too small.