Extension:WikiGrok/Version B Guerrilla Testing, November 2014

Guerrilla Testing for Wikigrok Version B
The goal of this research was to observe people interact with the wikigrok version b micro-contributions beta feature and determine:
 * Goal
 * 1) do users notice the call to action
 * 2) how do users respond to the call to action
 * 3) how do users’ expectations of what is being prompted match up to our intentions for this feature
 * 4) did users understand the overall purpose of the feature and also the contents/purpose of the ‘tell me more’ link
 * 5) do users have any particular reaction to articles with fewer/more available answer tags

To get started, we asked people a few basic questions first:
 * Questions
 * 1) What kind of phone do you use/what OS?
 * 2) Do you use a tablet, and if so, what model/OS?
 * 3) Do you use a computer, and if so, what model/OS?
 * 4) Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not. If they answer, "Yes, I use Wikipedia", we ask "how do you use it?" If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?”

We presented users with either an iPod Touch (for iOS users) or a Nexus (for Android/other users).
 * Task

The first four users were prompted to search for one of Arun Chaudhary/Rudolph von Delbruck/Peter Mate (fewer available tags) and then one of Sheryl Sandberg/Meg Whitman (more available tags) and interact with the beta feature.

The fifth user was provided the wmflabs beta page for Anne Dallas Dudley (one tag) and then Princess Diana (four tags) for interacting with the beta feature.

The sixth user was provided the wmflabs beta page for Anne Dallas Dudley (one tag) and then Princess Diana (four tags) for interacting with the beta feature. However, the beta feature seemed to break down (she used our iOS test device; the fifth user was using an Android device) and the wikigrok feature reverted to wikigrok A functionality.

Friday, November 7 & 14, 2014: We went to Yerba Buena.
 * When/Where

Findings: Patterns Observed

 * 1) When asked to describe the page, users do not actively describe or mention the wikigrok feature.
 * 2) Users did not have issues interacting with the feature, but a few had issues with the specifics of the phrases/words on the tags ("there could be a better descriptor than just ‘American'”)
 * 3) A few users expressed confusion at having only one tag available on those particular pages.
 * 4) A few users expressed some degree of concern about what happens when an incorrect tag is chosen and submitted (and the resulting positive message).
 * 5) A few users expressed confusion about the tell me more link and the explanation of WikiGame.
 * 6) A few users suggested gamifying the wikigrok feature further.
 * 7) Some users said they would want a way to scroll through more of them and tag more, or have them in other sections

Bugs and/or Suggestions

 * 1) May be helpful to add an option after the first inquiry is completed for user to answer more questions, if they'd like to continue the wikigame.
 * 2) Accessing this game will be located in the sidebar on future iterations.
 * 3) It may be helpful to inform potential interactions how long answering a 'tag' question might take on the front end (e.g., take five seconds to help improve this page).
 * 4) As users continue to mention the issue of incorrect tags, we should try to add a brief explanation perhaps in the Tell me more section.
 * 5) Get aligned on the message we want on the end page of the tag feature. Do we want to: make the user feel great about participating, explain structured data, or do something else?

Test A

 * Male (26-35)
 * iPhone, no tablet, Mac computer
 * Reads Wikipedia, didn’t know he could edit

Task:
 * User starts searching for Peter Mate (not familiar) in the browser, not the wiki page itself. later recommends that the search is in blue or another clearer color
 * Does not mention noticing the feature
 * Thinks it is for tagging information for the page
 * Answers (tags musician), mentions that he wouldn’t normally do this if encountered randomly - too lazy
 * Does not notice the ‘tell me more’ link
 * Searches for Sheryl Sandberg (not familiar), then checks entrepreneur and CEO (technically incorrect). Asks, ‘should I choose more than one?’

Test B

 * Male (26-35)
 * Samsung feature phone, no tablet, Mac computer
 * Reads Wikipedia, knows about editing and would like to attend an editathon soon

Task:
 * User searches for Arun Chaudhary (not familiar). Explains that the one tag available, that he is from the United States, could be of a better description, such as ‘documentary filmmaker’
 * Doesn’t realize immediately that the feature is a call to action (‘Oh, it’s asking me something?’)
 * Thinks tagging is for refining search results; confused that there is only one tag option
 * Mentions that he would probably ignore this feature; likes consuming content on wiki, not contributing. generally does not like stuff the solicits the user for participation
 * Does not notice tell me more link
 * Searches for Sheryl Sandberg (vaguely familiar), selects the COO tag. ‘I suppose she’s a manager…’ but doesn’t select that tag.
 * Wouldn’t want to do more of these

Test C

 * Male (26-35)
 * iPhone & Android, iPad, Mac/Lenovo computers
 * Reads Wikipedia, knows about editing but doesn’t. ‘If I felt like I could, I would'

Task:
 * User searches for Arun Chaudhary (not familiar). Notices the call to action, thinks that they’re tags that users could attribute to the page that would help search functionality, and the user could also validate if the tags are accurate.
 * Mentions that we should state the specific benefit of this exercise before users engage, to create more motivation for users
 * Prompted to look at the tell me more link; user ‘doesn’t see how it’s a game’, and thinks it would be still unclear for some users
 * Suggests gamification, users can get points, more emotional/incentive-driven draw to contributing this way
 * Searches Meg Whitman (familiar). Still confused, notices there are more tags on this page. 'Is this like a quiz? This could be telling OR asking me’
 * Clicks ‘engineer’. ‘Is she?’ Asks if it says if the tag is incorrect, but notices that it is a ‘you improved wikipedia' screen regardless. ‘if it’s wrong, i would still get positive validation'
 * Realizes a bit later that more than one tag can be selected
 * Suggests we add something where users can tag more of these. Also adds that perhaps we could load more somewhere so people can do more if so inclined

Test D

 * Male (15-35)
 * iPhone, no tablet, Mac computer
 * Reads Wikipedia, knows about editing but does not feel confident enough to do so

Task:
 * User pulls up menu bar first to search. Navigates eventually to Arun Chaudhary (not familiar), notices the call to action, understands that he needs to tag and thinks it is for refining search, but says he doesn’t know enough about the person yet and that he’d need to read more. Mentions that he’d probably skip it
 * Reads that Arun’s from NYC, tags quickly
 * Thinks the tell me more link is confusing. ‘Only on mobile devices? it didn’t make much sense’. Does realize it is another way to contribute
 * Searches for Sheryl Sandberg (familiar). Selects COO, muses on whether she is an entrepreneur as well (‘broad term’), but moves on
 * Says it is redundant to have more than one ‘profession’ term tag
 * Mentions that he may want to do more of these; suggests that they appear in other sections as well

Test E (tested with beta wmflabs pages on Android)

 * Male (26-35)
 * Android, iPad, PC computer
 * Reads Wikipedia, knows about editing. Wants to edit, but hasn’t yet. Has logged in once previously to do so, but had a mental block

Task:
 * User is presented with the Anne Dudley (not familiar) page, and describes all page features except for wikigrok
 * Confused by ‘improve information by tagging’. Goes to tell me more, says that game sounds like fun
 * Asks ‘do i hit this?’ about the one tag on Anne’s page. Doesn’t know her, and there’s only one option, muses that maybe other pages have more tags.
 * Acknowledges the thanks message, but wonders if it was actually true? Realizes that she wasn’t an insurance person; ‘so I made Wikipedia worse…’
 * User is presented with the Princess Diana (familiar) page, and notices additional tags. Checks ambassador.
 * Suggests that this is more gamified; get points, lose points for wrong answers
 * Concerned that anyone can edit and input anything they want

Test F (tested with beta wmflabs pages on iOS, reverted to wikigrok A)

 * Female (15-25)
 * iPhone, no tablet, Mac computer
 * Reads Wikipedia, knows about editing but hasn’t felt doing it

Task:
 * User is presented with the Anne Dudley (not familiar) page, and describes all page features except for wikigrok
 * Mentions that she would probably choose not to engage; she’s more into reading wikipedia as opposed to input
 * Not sure what this feature does, but clicks no quickly on the wikigrok a question. Acknowledges the thanks message
 * User is presented with the Princess Diana (familiar) page, and isn’t sure about the wikigrok A question
 * Is prompted to look at the tell me more page; asks whether the feature is meant to test users’ knowledge