Wikimedia Research/Design Research/Contributors Team UX Research/2017.05 Recent Changes Extended Filters (Round 2) Usability/Generative Testing
Recent Changes Extended Filters (Round 2)
After introducing the new filters for edit review to positive feedback, the Collaboration team’s next iteration aimed to include support for advanced functions.
In these sessions, we utilized a prototype testing environment and lead users through a series of tasks. Through completing said tasks, we are able to evaluate if research participants were aware of the functionalities, found them intuitive to use, and whether they found utility in them.
- What elements of the extended filters are most and least useful for users?
- Do users notice the new capabilities?
- If they find them, are they easy to understand and use?
New page elements
- Namespaces, tag, users, and categories filters
- Filter set bookmarking
- Display options (size and pagination of results)
- Navigating and updating results
- Participant recruiting
- 5 participants
- Youtube Live / Google Hangout on Air moderated sessions
- Test environments
- Users of Recent Changes, Watchlist and Contributions pages.
- If time allows, users of the RC filter beta.
- If time further allows, non-users.
Protocol and Researcher Notes
- Keep in mind that you’ll be working with prototypes, and not everything will function as expected.
- Tell us a little about yourself / what you do / interests.
- How long have you been editing on Wikipedia?
- What Editing Review (Recent Changes, Watchlist, Contributions, etc) page do you work on primarily?
- What are your typical actions on RC, Watchlist and how much time do you spend on these pages?
- What (external) tools do you use?
- Based on your experience with edit review, what limitations did you have in finding what you needed?
- Do you use any filtering capabilities on RC/Watchlist/ext tools?
- Take a minute or two to walk me through what you’re seeing on this page. As you’re exploring the page, narrate a bit about what you’re seeing and and thinking. (BRIEF, nothing in depth)
- Imagine you’d like to find recent edits made by relatively new editors so you can send them welcome messages. (Tests user’s ability to use the basic menu and to exclude)
- Now let’s say want to distinguish which edits are probably good edits and which are possibly problematic, so you can send them slightly differing messages. You want to see both types of edit at the same time, instead of filtering one or the other type out. How would you set up filters to show this combination of results? (prompt with visually distinguish if no highlighter) [tests highlighting]
- Let’s go through some of these other filters, and tell me a little more about what you think they represent and whether you’d find utility in them.
- (have they noticed advanced settings?) What did you expect to find in this section? What do you think of the content in this section? (quick, not comprehensive)
- Address last revision/watchlist sets… are they useful?
Namespaces / Exclude
- Now, let’s say you’d like to find newly created pages made by logged-in editors only. (tests ability to exclude all but new pages and newcomers. Sets up to find out whether they are later thrown by the Exclude function.)
- As you look through the results, you notice that there are quite a few that are from talk pages that you’re not interested in reviewing. How would you focus on just the edits from this list that affect articles and user pages only? (namespace menu. Note: if they try to use just the “Page edits” filter) OK, start over with the default filters. What would you do if, for some reason, you wanted to see all edits except edits to articles. Is there a way to do that? (testing exclude in namespace)
- What did you think of these two experiences using the filters?
- (did users notice bookmark yet?) Let’s start over on this page, and have you set up a few filters that allow you to review edits of interest. Perhaps you want to review these same edits every morning… is there a way to keep this set of filters easily accessible for the next time you arrive on this page? Did you notice this before?
- In your work on RC and watchlist type pages or other tools, have you had the need to access the same filters repeatedly?
- What other combinations of filters might you need in your work if at all? Can you show me how you would add them if they exist?
- Can you think of a combination of settings you might want for this page to load whenever you first come to it? Do you see a way to make that happen? How would this capability fit into your usage or routine on RC/watchlist type pages?
- Let’s take a look at the options in this list that you didn’t put there. Explain briefly what you think they are, and whether you use these? What do you think of them? If this were live, would you take any actions in regard to the items on this list?
- Imagine that there are a couple of items on the list you think you might use frequently. Is there a way to make using these in the future easier? [If needed] What do you think the icon to their left means?
Display options / size, paginations || View newest changes / newer, older
- (have they noticed?) Let’s go ahead and clear the current set of filters/refresh the page. How many changes are usually shown on the current Recent Changes page? What do you think of this? How does this compare to the prototype?
- In your experience with this and similar pages, have you ever felt the need to adjust the options for the number of results shown? Why/why not? (page load time, having smaller # of results to go thru at once, other?)
- Let’s say just for the sake of this test that you wanted to work with a smaller number of results at one time. How would you do this? How do you feel about the way you access the controls for this? (should it be advanced function or should it be more visible)
- Now, imagine that you’ve been reviewing the changes on the page for a while. You want to see new changes. How would you do that? [Explain that this part of the simulation isn’t functional.]
- (Whatever the user picks) What would you expect to happen when you click that?
- Do you see other options that might bring up new results as well?
- (Whatever the user picks) What do you think that would do? How might be different from what you just picked?
- (Prompt user to explore other options and speculate on what they do and how they differ. Draw the users attention to them if need be.)
- We’ve just gone through 4 different ways to get new results. It’s not a given that all will be built. Which, in the last month, would you have wanted to use in your work if you’d had it?
- Let’s say you’ve taken a 3 day break from wiki work, and now want to check on the changes made since, with the oldest changes displayed first. How might you show those?
- What if you were on vacation in the first week of April and wanted to take a look at the changes from that week?
- Have you used these types of controls before on Wikipedia or on the web? Would you use them on RC/watchlist if they were available?
- Let’s say that you have a friend (Ss112) who just started editing and might be making not-so-good edits. How would you search for that specific user’s edits on Article, User, and User Talk namespaces? What if you wanted to view all edits by this user except for those in the File and File Talk namespaces?
- How might you find the changes made by another user (Gatoclass) who’s been vandalizing content about Geography using a mobile device? Assume there are results if there are none... how would you describe those results to a person who is not familiar with Wikipedia? How would you distinguish the results that are from mobile apps and those that are from the mobile website?
- Have you had cause to search for the edits of specific users and within categories in the past? How do you think having this feature will impact your work on the RC/watchlist pages? What do you think of having this feature available in general? (concerns around targeted harassment?)
- How often have you had to search for category-specific changes in the past? Would you use this feature in your future work on RC/watchlist type pages? Why/why not?
(do they notice prefix during?)
- If you select the display option of ‘show live updates’, what do you expect will happen?
- How do you feel about this option vs what you saw with the ‘view newest changes’ vs how the RC and watchlist pages currently work?
- How might the availability of this feature factor in your work?
- How would you compare your experience with the current RC/Watchlist experience to your experience with the prototype(s) today? (visual, layout, elements, experience)
- Imagine you logged in tomorrow and your watchlist page looks like the watchlist prototype page you just saw today. How would you feel about that and why? (would this interface be helpful on watchlist)
- Think about the next month or more after that first day you see the changes. Which of these features or filters or elements on this page do you see yourself utilizing the most? Please explain which and how. (not everything will be implemented; which do users find most useful?)
- Of the new features and elements on the prototype(s), which would you like to see implemented and why? Do these differ from the features you mentioned in the last question and if so, why? (not everything will be implemented; which do users find most useful?)
- Are there any elements or features or specific filters that weren’t on the prototype or currently exist that you’d like to see included?