Wikimedia Research/Design Research/Contributors Team UX Research/2017.10 Visual Diffs Usertesting
Visual Diffs Usertesting
Daisy Chen
Visual Diffs[edit]
Research Questions[edit]
- How does the visual diff compare to the wikitext diff for new/non-editors?
- Do users understand what theyâre seeing when they arrive at the visual diff page?
- Can users easily and effectively identify particular changes that have been made?
- Does the diff page communicate the type of change with appropriate amount of context?
- What feedback do users have about the feature functionality, layout, interaction, visual elements?
Findings[edit]
Task Success[edit]
- Fix a typo - 10/10 participants succeeded
- Adding text - 10/10 participants succeeded
- Removing text - 10/10 participants succeeded
- Text formatting - 10/10 participants succeeded. 2/10 tried right clicking first. 1/10 changed formatting with a keyboard shortcut.
- Add link - 9/10 participants succeeded in adding a link of some kind. 1/10 added an external link, thinking it was an internal link.
- Remove link - 8/10 participants succeeded.
- Add citation - 7/10 participants succeeded. 2/10 added external link. 1/10 skipped.
- Reuse citation - 3/10 participants succeeded. 4/10 added another citation. 2/10 added an external link. 1/10 skipped.
Wikitext/Visual Diff, Participant Reactions[edit]
Table View[edit]
Wikitext Diff | Visual Diff | |
---|---|---|
P1 | âtrying to figure out what it's showing meâ | âoh, so much better. so much easier to understand.â |
P2 | âa lot of information here, i have to intuit what the changes are, look at what was highlighted.â | âthis is what i thought it would look like. very simple display of what i changed. looks like they grayed out the section i didn't change. i can skip the stuff not relevant to what i'm reviewing. relief. lot easier to look at and review.â |
P3 | âeasy to understandâ | âdifferent format. looks interesting.â |
P4 | âit's a bit much. it's coded. a bit techie. insults your eyes with lots of words.â | âooh! much better. much easier to see what you've done, basically.â |
P5 | âscans through my inputâ | âhere, the view is a lot easier. you can actually see the bold vs quotation marks (re text formatting change.â |
P6 | n/a | âcrossing out original and adding my edits.â |
P7 | âlooks like mumbo jumbo clutter. hurts my eyes to look at this. don't know where to beginâ | âeasier for me to see, know what i'm looking atâ |
P8 | âthis can be tricky, even impossible, to read. i think it is html code for edits.â | âextremely easier to realize what you changed.â noted text formatting change he didnât see in wikitext diff |
P9 | âit's showing what i have changed. compares new with previous.â | ânice, showing what i have changed and added. easy for me to see, betterâ |
P10 | âlike the parallel viewsâ | n/a |
Raw notes[edit]
- Wikitext Diff
- P1 - âtrying to figure out what it's showing meâ
- P2 - âa lot of information here, i have to intuit what the changes are, look at what was highlighted.â
- P3 - âeasy to understandâ
- P4 - âit's a bit much. it's coded. a bit techie. insults your eyes with lots of words.â
- P5 - âscans through my inputâ
- P6 - no reaction, just reviews changes
- P7 - âlooks like mumbo jumbo clutter. hurts my eyes to look at this. don't know where to beginâ
- P8 - âthis can be tricky, even impossible, to read. i think it is html code for edits.â
- P9 - âit's showing what i have changed. compares new with previous.â
- P10 - âlike the parallel viewsâ
- Visual Diff
- P1 - âoh, so much better. so much easier to understand.â
- P2 - âthis is what i thought it would look like. very simple display of what i changed. looks like they grayed out the section i didn't change. i can skip the stuff not relevant to what i'm reviewing. relief. lot easier to look at and review.â
- P3 - âdifferent format. looks interesting.â
- P4 - âooh! much better. much easier to see what you've done, basically.â
- P5 - âhere, the view is a lot easier. you can actually see the bold vs quotation marks (re text formatting change.â
- P6 - âcrossing out original and adding my edits.â
- P7 - âeasier for me to see, know what i'm looking atâ
- P8 - âextremely easier to realize what you changed.â noted text formatting change he didnât see in wikitext diff
- P9 - ânice, showing what i have changed and added. easy for me to see, betterâ
- P10 - no reaction, just reviews changes
Questionnaire Responses[edit]
Table View[edit]
Which option better represented changes more clearly? | Which would you prefer to see when editing on the wikis? | Other comments? | |
---|---|---|---|
P1 | visual | visual | n/a |
P2 | wikitext | wikitext | both might be nice |
P3 | visual | visual | it was easy to switch modes |
P4 | visual | visual | more hints and tips when hovering/using icons |
P5 | visual | visual | both viewers should be used together |
P6 | visual | visual | |
P7 | visual | visual | |
P8 | visual | visual | |
P9 | visual | visual | |
P10 | wikitext | wikitext |
Raw Notes[edit]
- P1 - visual, visual
- P2 - wikitext, wikitext, both might be nice
- P3 - visual, visual, it was easy to switch modes
- P4 - visual, visual, more hints and tips when hovering/using icons
- P5 - visual, visual, both viewers should be used together
- P6 - visual, visual
- P7 - visual, visual
- P8 - visual, visual
- P9 - visual, visual
- P10 - wikitext, wikitext
Conclusion[edit]
8/10 participants prefer visual over wikitext diffs.
1 of the 2 participants who preferred wikitext mentioned that visual diffs also has its advantages, and would like to have both modes available.
There is a possibility that the 2 participants who preferred wikitext may have conflated the mode with the '2-column' or 'side-by-side' view.
There were no red flags observed as to the participants' understanding of the edits they made while using the visual diff beta. Relatively more hesitation in identifying changes was observed when participants viewed the wikitext diff; a couple participants did not notice the text formatting edit they made.
Overall, the visual diff view is well-received.
Recommendations for further testing:
- Visual vs wikitext diffs with experienced editors cohort
- Prototype views of more involved and complicated edits within visual diff to determine ideal representations
Researcher Notes[edit]
- Searching for terms in the link inspector is not always consistent. âDogsâ will find âdogâ, âdog(s)â, and other pages including âdogâ, but âdichromatsâ will not find âdichromatâ or âdichromacyâ. Can search find more matching/relevant terms including singular/plural versions, noun/verb forms, etc.?
- Clicking somewhere on the word(s) to remove a link was not intuitive for many users. 8/10 highlighted the link to remove, and experienced some level of confusion. The delete button and its hover text (âclear stylingâ) was also not intuitive for some.
- Even though this cohort is comprised of new/non-editors, they all had difficulty with reusing a citation and over the course of testing, the tabs in the Cite menu did not clearly resonate. It is unclear if the issue is with the tabs' labels, visual styling/lack of contrast for the tabs on the hover window, or both.
Testing Methodology / Raw Notes[edit]
Set-up[edit]
- âUser will log in with test account credentials to the 'Dog' article on wmflabs beta.
- Users are directed to use Firefox, Chrome, or Safari for the test.
- 10 total participants
- Usertesting.com user panel
- New or non-editors
- 5 men, 5 women
- 2 US/UK/CAN and 3 non-US/UK/CAN per group
Screener (Bold indicates accepted answer):
- How many edits have you made on Wikipedia? (0-25, 26-50, 51-100, 100+)
- Do you know the difference between the wikitext and the visual editors? (Y or N)
Protocol[edit]
Introduction
- Keep in mind that though this test wiki is meant to replicate the live Wikipedia site as much as possible, there could be glitches that you might experience using this site. If you try something and it doesnât work, try it once more before moving on to the next task. Please go to the next task.
- Go to https://en.wikipedia.beta.wmflabs.org/wiki/Dog if not already there. Make sure to first log in with the credentials [redacted].
Tasks
- After coming to this Wikipedia article, imagine you wanted to improve some things on the page. First thing you notice is that thereâs a typo in the first sentence. Go ahead and fix that. (click âEditâ, may need to click âStart Editingâ on a pop-up, and ignore any other error messages)
- You noticed that the âBiologyâ section is missing some information you know, that dogs are highly variable in height and weight. Can you expand this section by adding this fact to the second paragraph?
- You feel the last sentence in the Taxonomy section doesnât add too much relevant information. Can you remove that sentence?
- Currently, the latin name for dog (in the first paragraph) is not formatted in line with the Wikipedia Manual of Style (itâs in normal text). Change it so that it is in bold instead.
- In the âSensesâ and âVisionâ section, you think itâd be helpful to add a link to the Wikipedia page on âdichromatsâ in the first sentence. Go ahead and give that a try.
- In that same paragraph, you notice that âdogsâ is linked multiple times, and you donât think itâd be useful for a page to link to itself. Go ahead and remove those repetitive links.
- You think the first sentence needs one more citation. Please cite that sentence to this article: http://www.sciencedirect.com/science/article/pii/S1055790398905139
- In that same sentence, there is a citation to the http://onlinelibrary.wiley.com/doi/10.1002/zoo.20302/abstract source. Re-use this same citation on the last sentence of the first paragraph.
- Click on âSave Changesâ, and then on the next screen, the button âReview your changesâ. What is your first impression of this page?
- You made about 8 types of changes to the page. Based on what youâre seeing here, can you identify those changes without going back and reviewing your previous tasks? Take a look and talk through which edits you can identify here.
- Now, click on âVISUAL (beta)â near the top right part of that same viewer window (you may need to scroll back up). What is your first impression of what you see here?
- Again, which of the changes you made do you recall by reviewing this page? Talk us through them.
Post-task Questionnaire[edit]
- Which of the two options do you feel represent the changes more clearly? Why?
- Which would you prefer to see if you were editing on Wikipedia? Is this preferred option different than the one you indicated in the last question? If so, why?
- If there is anything else youâd like to mention?
- Thank you and please log out of the account.
Notes Spreadsheet[edit]
10 participants (5 male, 5 female) completed the editing tasks and answered post-test questions through UserTesting.com.
Raw notes on the 10 sessions can be found here.
Due to privacy concerns, the notes are shared internal to Wikimedia Foundation staff.