Wikimedia Research/Guerilla Testing for iOS App

Guerrilla Testing for iOS App

In order to get some quick feedback on the edit work flow, we decided to go out into the city and find a few people to test with. We approached people in pairs, one person facilitating and the other taking notes and observing. After several observations, the group of us came together to debrief and refine notes. Then, we went back to find a few more people to test with. In the end we spoke with 7 people: 4 men (around 20-30 years old) and 3 women (2 in their 20s-30s one in her 40-50s)]

Where

We went to a coffee shop first, but it was too noisy and people were busy meeting and eating lunch. So, we went to Yerba Buena park and found 6 people willing to spend a few minutes with us running through the edit flow for the iOS App. 

 Tasks Tested  

 Making an edit using Visual Editor. Specifically, I tested making text edits and not inserting anything, adding cites, links or tables.  

Observations: Hypothesis to be further tested

1. Some people are not seeing the "how did you improve the page" call to action. (5 of 7)

2. the navigation through the edit flow (and the icons: arrow and x) are not intuitive (3 of 7)

3. Specifically, what the arrow will do is not clear (3 of 7) 4. Though, to some, the edit flow is intuitive. (3 of 7)

Raw Notes

Kaity's notes

Man (20s)

• Couldn't figure out how to edit section at the bottom
• Expected to highlight text and be able to edit right there.
• Does this go live right away? Or does someone have to check it first?
• I want to see a change log and see why things were changed.
• (Showed history)
• This isn't very interesting, I can't see why they were changed, doesn't mean anything to me

Man (20s)

• Found toc quickly
• Explored menu
• Scroll seems slow

Woman (20s)

• Scrolled to navigate
• Guessed it was a menu
• I like that toc scrolls article
• Finds edit right away
• Finds next right away
• How do you get back?
• I don't know what this arrow means

Woman (50 or so)

• Scrolled to navigate
• I know about all these!
• Menu -info about the topic
• I read wiki on my iPad or scrolls easier
• Don't know how to edit
• Pencil could be a highlighter
• Tries to hit x at first
• What's that arrow for, how do you save?
• Confused after preview
• Tell us what you changed she found
• Selected fix a typo

Maryana's notes:

Marketing app lady (20s-30s):

• Didn't know you can edit Wikipedia
• Doesn't want to edit -- when asked how she would fix the typo, says "I wouldn't"
• But finds edit button quickly when prompted, so def. gets the iconography
• Goes through proceed to preview & proceed to save pretty quickly -- finds those top right icons easily & naturally
• Doesn't see "how did you improve the page"

Young kid (high school/college age?):

• Knows you can edit
• Finds edit button & dives right into editing without any hesitation
• Finds proceed to preview but doesn't see that there's another step from there -- hesitates & asks if done
• Finds save & IDs the two options right away "login & save and save anonymously", saves anon
• Doesn't see "how did you improve the page"

Developer guy (20s)

• Knows you can edit
• Recognizes pencil as edit button but wants some fancy gesture thing to activate editing (savvy tech user, clearly)
• Says he'd be scared if he didn't know wikimarkup, which he does
• Gets proceed to preview & proceed to save buttons very easily
• Confused on the preview screen that he's taken to the top of the section instead of to his edit -- same after he saves
• Wants clearer indication that he saved something
• Doesn't see "how did you improve the page"