Wikimedia Research/Design Research/Contributors Team UX Research/2016.09-11 Recent Changes Filters, Round 1

Recent Changes Filters, Round 1

Daisy Chen

= Abstract/Background = The Special:Recent Changes page is a central starting point for many types of edit review. A main goal of the Edit Review Improvements project is to provide a more supportive new-user review process. To that end, we user-tested a suite of new tools Collaboration Team is planning for the RC page—intended for a beta option initially— that should make edit-reviewing more efficient generally while providing features that will help newcomers.

A prototype page was developed for usability testing, and during this first round of testing, an updated prototype was iterated and tested based on ongoing research findings. The goal of this research study is to evaluate the usability of the new filtering capabilities on the Recent Changes page and gather general feedback about users’ related experiences using the current Recent Changes page or other similar tools.

= Research Questions =
 * Concept and Intent
 * Do users understand what the filter categories represent, specifically the Quality, Intent, and Experience filters?
 * Do subjects understand what the different ORES (Objective Revision Evaluation Service) levels mean?
 * Do users understand the difference between filtering and highlighting, and how they might use these singly or in combination?


 * Usability
 * Do users notice and/or understand the default filters?
 * Are users able to find the correct filters and use them in combination effectively?
 * Do users find and learn to effectively use the highlighting to isolate targets?
 * Can they manage and interpret the highlight colors?
 * Do they find the “ignore highlights” button and understand its use?*
 * removed after first iteration


 * Effectiveness
 * How do users feel the new tools will affect their work and workflows?
 * What are users’ reactions regarding the new interface and its functionalities?
 * Do users feel any differently about the new interface as compared to the current Recent changes page?
 * Do they see that the new interface as being functionally neutral, inferior or superior?

= Methodology =
 * Participant recruiting
 * 9 participants
 * Youtube Live / Google Hangout on Air sessions
 * Test environments
 * prototype 1
 * prototype 2

= Participants = Existing Wikipedia editors; ideally active users of recent changes and/or watchlist.

= Protocol =

Introduction

 * 1) Tell us a little about yourself / what you do / interests.
 * 2) As a [vandal fighter / new user helper], could you take a minute or two to explain your current workflows and the tools you use?
 * 3) Can you describe how is your activity connected with the following: helping newcomers, prevent vandalism, and encourage users to keep making good contributions?
 * 4) Do you use the Recent Changes page? How?
 * 5) You might notice that this prototype page doesn’t exactly match the existing RC page. Please keep in mind that this is a prototype testing page, and not all elements on this page may match, work in general, or work as expected. When this happens, let us know what you’re trying to do, and what you expect to happen when you interact with the page.
 * 6) Take a look around the page and take a little time to explore. What do you think this page does?

Evaluating tool understanding

 * 1) This is the reimagined recent changes page. Can you tell me what types of edits/contributions are being displayed? How can you tell?
 * 2) Can you describe in your own words some of these filtering tools that you’ve found? What do they represent?
 * 3) What do you think so far?
 * 4) Let’s look at the first section (‘Contribution quality’). What do these filters mean to you?
 * 5) Could you explain the differences between these filters?
 * 6) Based on your past experiences working with RC and/or Twinkle, how might you use these filters?
 * 7) Play around with these filters for a minute or so. What do you think is happening when you select one or more filters in this section?

Use Cases

 * 1) What kinds of edits you would like to focus on for your reviewing activities? How would you do so with the prototype?
 * 2) Could you show me how you’d find what types of edits you’re looking for?
 * 3) How would you increase your chances of finding what you're looking for?
 * 4) Thanking users for their good contributions encourages them to contribute more. Imagine you want to find those users that are new to the community and are making positive contributions, in order to thank them.
 * 5) Two scenarios:
 * 6) Since you have plenty of time you want to target as many of those as possible so that no one is left without a "thanks" when they deserve it.
 * 7) Since you have very little time to review their contributions in detail you want to focus only on cases where it's pretty clear the contributions are good.
 * 8) Can you imagine how you or someone pursuing different goals from you might use these tools?
 * 9) Imagine that, as a vandalism fighter, you’d like to find all the recent edits that are potentially damaging. Show me how you’d do that.
 * 10) Now that you’ve gotten your results, you feel like there are too many to work with in one sitting. How would you narrow your search results to find only the most clearly damaging?
 * 11) Imagine you want to find all the edits that may be bad, but you want to be able to tell which of those are the most likely to need your attention.
 * 12) You want to focus on reviewing vandalism, but you also want to be aware of whether users are new in order to review them with extra care and provide them more guidance. Can you show me if it is possible to view all kinds of vandalism by all kinds of users while distinguishing which users are new and which are not? (if the user just filters, insist: are contributions by non newcomers there?)
 * 13) You are interested in reviewing all potentially damaging edits. You are want to review them for all types of users, but you want to pay extra attention to the relatively new users because you want to let them know about the problem with a softer message. How would you set up this filters page to show what you need?
 * Now, let’s imagine that you want to visually distinguish possibly damaging edits, the most damaging edits and also whether newcomers were involved in making them. How might you display these edits?

Wrap-up feedback

 * 1) Tell me about your experience with the prototype page today. What elements worked or didn’t work for you?
 * 2) How do you feel about the filtering tools and the overall interface?
 * 3) Did you encounter any areas of confusion and/or lack of functionality you wished were present?
 * 4) Please take a little time to check out the current Recent Changes page.
 * 5) How do you think this prototype interface would fit into your current work on the wikis?
 * 6) Any other final feedback?
 * 7) Review filter (if time allows)
 * 8) How would you feel about a feature which could assist in preventing duplicate work - something to indicate that someone else trusted has already reviewed an edit. Either the original "Patrolled edits" feature, or an improvement thereof.

= Findings =