Quality Assurance/Browser testing/Search features
10min video with basic instructions: youtu.be/Kad3EUM4GdM Etherpad to document progress: http://etherpad.wmflabs.org/pad/p/Browser-automation
Feature: Search Scenario: Search suggestions Given I am at random page When I search for: main Then a list of suggested pages should appear And Main Page should be the first result
WHEN: We will start with 30min demo streamed on Wednesday, March 13, 2013 at 17h UTC and we will be helping volunteers during the rest of the week. This is an ongoing activity: you can arrive / leave at any time.
WHO: Anybody interested, including you! The only requirement is basic level of plain English in order to describe the features to be tested automatically. It is that simple.
See also How to contribute.
Focus[edit | edit source]
Here is a list of open bugs we are likely to fix soon that we want to test automatically:
- Search for simple text misses some results
- Incorrect UTF-8 processing
- Search ignores user settings
- Search does not find pages created with <pages index...> tag. (Could this be testing in English?)
- Search results should always be shown in the same order
- Accents should be disregarded in Search
We could use regression tests for Search features also:
- Umlauts and accents are disregarded in search
- Some ligatures match the separate letters. For example, a search for aeroskobing will find pages containing Ærøskøbing (ae = Æ)
- It is not possible to search for the string |LT| (letters "LT" between two vertical bars); all articles with "lt" will be returned.
- Ignore parentheses except for exact matches: Credit (finance)" will return articles with the words "credit" and "finance", ignoring the parentheses, unless an article with exact title "Credit (finance)" exists
- A phrase in double quotes returns exact matches
There are more examples in Bugzilla.
Both desktop and mobile browsers need Search tests.
Evaluation[edit | edit source]
- Summary: It was our first Browser testing activity. The preparation work was significant but the external participation was small: 3 people. Still, we broke the ice and organizing a second edition will be a lot easier. Promoting the event to our interns and group members was useful.
- Results: 9 new scenarios related with Search in the Test backlog (the goal was 5).
- Participants: 7 in total. 4 WMF employees (Chris, Zeljko, Quim, Sumana), 2 OPW interns (Valerie, Sucheta), 1 volunteer (Rachel99). 3 external when the goal was 5. All women!
- New: 4 people wrote their first scenario ever: Valerie, Sucheta, Sumana, Quim.
- Repeating: Rachel99 had volunteered previously.
- Mood: people seemed to have fun.
- Documentation: we created a highly reusable weekly goal page (this one), a 10min video (youtu.be/Kad3EUM4GdM, move to Commons pending), one new page (Examples) and significant improvements improvements to Browser testing, Group Browser testing and Test backlog.
- Promotion: it wasn't easy because the concept is quite novel for most people. We did broadcast through social media, wikitech-l, wikitech-ambassadors, en.wiki tech Village Pump, Chris & Zelko's blog posts. We sent calendar invitations to 9 WMF employees and 6 OPW interns. We contacted the 4 members of the Browser testing group (ecp declined because of travel).
- Developer team: Munagala "xyvram" Ramanath is the only Search developer currently. He was available during the activity, pointing to details about bug reports.
- Organizers: It took a significant amount of time and work to get Chris, Zeljko and Quim in the same page and to prepare the documentation. But we did it and next time it should be a lot simpler.
- Lessons learned:
- There must be one topic well defined in advance.
- Pointers to bug reports or scenarios are a must.
- Organizing a synchronous activity is good and fun, but we need a clear workflow for DIY contributors landing at other times.
- Encouraging pair programming was interesting for the couples formed, but maybe that left some individuals out? Needs more thought.