Reading/Web/Desktop Improvements/Repository/Collapsible Sidebar Report

In July 2020, we deployed the new Collapsible Sidebar feature to the early adopter wikis. The goal of the feature is to make it easier for people to focus on the content itself during reading or editing. Also, we wanted to highlight important functionality, such as the edit and history buttons, language switching, and search, while decreasing the prominence of less-frequently used functionality.

Since making the change, we were able to observe the patterns of behavior of anonymous and logged-in users while using the sidebar. This report represents our findings so far. We hope to use this data in future iterations of the collapsible sidebar functionality.

Study Questions
When measuring the usage of the sidebar, we wanted to consider the following set of questions:


 * What is the frequency that users collapse and uncollapse the sidebar
 * What is the difference in interaction with sidebar links between people with the sidebar collapsed (the majority of the time) versus uncollapsed (the majority of the time). i.e. Once the sidebar is collapsed, do users open it when they need to, or do they just ignore it altogether?
 * What is the overall difference in interaction with sidebar links before and after the change
 * In particular, what is the overall difference in interaction with the donate link?

What is the frequency that users collapse and uncollapse the sidebar?

 * Between July 22 and August 31, 2020, there were 28,745 unique sessions that included a click to collapse and/or uncollapse the sidebar. This represents only about 0.13% of all recorded sessions using the modern vector skin and 13.67% of all modern vectors sessions with a click recorded.
 * Since the sidebar was set as uncollapsed by default, there are more total clicks to collapse the sidebar; however, there are roughly the same average number of clicks per session to either collapse or uncollapse. Average Clicks Per Session to Collapse: 1.37; Average Clicks Per Session to Uncollapse: 1.38.


 * French Wiktionary had the lowest average sidebar clicks per session (1.87) while Persian Wikipedia had the highest (2.94).
 * Logged-out users have more than twice the number of collapse events compared to uncollapse events, likely because the state of the sidebar for logged out users is not preserved.
 * Most sessions overall included just one-click to either collapse or uncollapse the sidebar. Logged out users have a higher frequency of one sidebar clicks per session while logged in users have a higher frequency of two clicks per session.



What are the different interaction types with the sidebar?

 * For the majority of sessions (81.88%), the sidebar was collapsed and/or uncollapsed just once. For sessions where the sidebar was collapsed only once, 34.5% sessions kept the sidebar collapsed the majority of their session time and 65.4% of sessions had the sidebar uncollapsed the majority of their session time (did not collapse it until towards the end of their session).

Sidebar Interactions Summary [1] Clicks were more than than 1 second apart

What is the overall difference in interaction with sidebar links before and after the change?
Note re approach:

Since this was deployed as opt-out to all users on the test wikis, there is not a random, equal split of users between the two vector skins so a direct comparison cannot be made as there may be some bias (the legacy skin version was only available to those that explicitly opted out) and also there more users that had the default experience. However, I compared the average interactions per session for each skin type to control for the different number of users and confirm if there were any significant differences. Note: Data in these comparisons may still be impacted by self-selection bias between the two groups. I also reviewed data from a two-week period before and after each test wiki's deployment date to confirm if there were any significant changes.


 * The legacy skin has a slightly higher number of total sidebar link clicks per session (4.68) compared to modern (3.63). This is also true on a per wiki basis except for on French Wiktionary and Portuguese Wikiversity, which have a slightly higher average number of total sidebar link clicks on the modern skin compared to legacy.
 * Clicks to the donate link do not appear to be impacted by the new skin version. There is a slightly higher number of average clicks per session on the modern skin (1.11) compared to the legacy skin (1.06).
 * There was a decrease (-9.15%) in the overall total number of sidebar link interactions comparing the two weeks prior and two week following the deployment of the collapsible sidebar across all test wikis; however, this percentage is likely inflated due to seasonality effects (previous year's data was not available using webrequest to complete a year over year analysis).

Full Report
The full report is available in this Jupyter Notebook.

Discussion and Next Steps
Based on the results above, we have learned that logged-in and logged-out users have very different behavior while using the sidebar. Logged-in users tended to, overall, collapse the sidebar and leave it collapsed, leading to the conclusion that not many of the items in the sidebar were useful to them. However, the rate of uncollapsing also indicated that users are aware that, were they to need to navigate to an item in the sidebar, that option was available to them.

Logged-out users, on the other hand, tended to collapse and uncollapse as needed. This indicates that they had a greater need of the items in the sidebar and that their behavior was more task-based, rather than indicating an overall pattern.

Based on the above findings, we plan on taking the following actions:


 * Making the sidebar collapsed by default for logged-out users
 * Keeping the sidebar uncollapsed by default for logged-in users

Once this change is made, we will continue to monitor the above patterns, in particular the decreases in clicks to links in the sidebar, and will iterate again based on those findings. These changes will take place after our work on language switching is completed. Switching languages is important to many users and it is our priority to ensure that that functionality is easy to find at all times.