Wikimedia Research/Guerilla Testing for iOS App

Guerilla Testing for iOS App
In order to get some quick feedback on the edit work flow, we decided to go out into the city and find a few people to test with. We approached people in pairs, one person facilitating and the other taking notes and observing. After several observations, the group of us came together to debrief and refine notes. Then, we went back to find a few more people to test with. In the end we spoke with 7 people: 4 men (around 20-30 years old) and 3 women (2 in their 20s-30s one in her 40-50s)]

Where
We went to a coffee shop first, but it was too noisy and people were busy meeting and eating lunch. So, we went to Yerba Buena park and found 6 people willing to spend a few minutes with us running through the edit flow for the iOS App.

=== Tasks Tested === Making an edit using Visual Editor. Specifically, I tested making text edits and not inserting anything, adding cites, links or tables.

=== Observations of patterns - Hypothesis to be further tested === 1. Some people are not seeing the "how did you improve the page" call to action. (5 of 7)

2. the navigation through the edit flow (and the icons: arrow and x) are not intuitive (3 of 7)

3. Specifically, what the arrow will do is not clear (3 of 7) 4. Though, to some, the edit flow is intuitive. (3 of 7)

Man (20s)
Couldn't figure out how to edit section at the bottom

Expected to highlight text and be able to edit right there.

Does this go live right away? Or does someone have to check it first?

I want to see a change log and see why things were changed.

(Showed history)

This isn't very interesting, I can't see why they were changed, doesn't mean anything to me
Man (20s)

Found toc quickly

Explored menu

Scroll seems slow

Woman (20s)
Scrolled to navigate

Guessed it was a menu

I like that toc scrolls article

Finds edit right away

Finds next right away

How do you get back?

I don't know what this arrow means

Woman (50 or so)
Scrolled to navigate

I know about all these!

Menu -info about the topic

I read wiki on my iPad or scrolls easier

Don't know how to edit

Pencil could be a highlighter

Tries to hit x at first

What's that arrow for, how do you save?

Confused after preview

Tell us what you changed she found

Selected fix a typo

Marketing app lady (20s-30s):

 * Didn't know you can edit Wikipedia
 * Doesn't want to edit -- when asked how she would fix the typo, says "I wouldn't"
 * But finds edit button quickly when prompted, so def. gets the iconography
 * Goes through proceed to preview & proceed to save pretty quickly -- finds those top right icons easily & naturally
 * Doesn't see "how did you improve the page"

Young kid (high school/college age?):

 * Knows you can edit
 * Finds edit button & dives right into editing without any hesitation
 * Finds proceed to preview but doesn't see that there's another step from there -- hesitates & asks if done
 * Finds save & IDs the two options right away "login & save and save anonymously", saves anon
 * Doesn't see "how did you improve the page"

Developer guy (20s)

 * Knows you can edit
 * Recognizes pencil as edit button but wants some fancy gesture thing to activate editing (savvy tech user, clearly)
 * Says he'd be scared if he didn't know wikimarkup, which he does
 * Gets proceed to preview & proceed to save buttons very easily
 * Confused on the preview screen that he's taken to the top of the section instead of to his edit -- same after he saves
 * Wants clearer indication that he saved something
 * Doesn't see "how did you improve the page"