User:CKoerner (WMF)/Discernatron

Introduction

You can help improve search relevance

To make changes to search at a quicker pace, the Search team needs to be able to test changes before making them available on-wiki. Discernatron is a tool that allows participants to judge the relevance of search results. When evaluating potential changes to the Wikimedia search the team will use these judgements to help rate potential changes by how much better they are at putting the most relevant articles at the top of the search results page.

Get Started  »

(login with your Wikimedia account)

What queries am I rating?

Every month the Discovery department loads approximately 500 randomly selected search queries from the English Wikipedia into Discernatron for grading. These queries represent around 0.0001% of all full text searches on English Wikipedia. This sample is incredibly small, but still represents a wide swath of the types of queries received. Before being released to Discernatron, WMF engineers review the sampled set of queries and remove anything that could be considered personally identifiable information (PII). Initially only queries for English Wikipedia are being used but Discernatron will expand to other languages—such as French, Spanish and Russian—as time goes by.

So someone is looking at all my searches?

No. When reviewing queries there is no additional meta data, such as user name, location, or IP address. Additionally due to the sample size it is very unlikely that the sample contains more than one query from any single user. See Discovery's Data access guidelines for more information on how the department manages user data.

What kinds of queries are removed?

Anything potentially personally identifiable. This means any kind of phone number, serial number, or non-notable address. We remove searches for specific URLs, and names of non-notable companies and non-notable people (those that don't have wiki articles and aren't mentioned prominently in any other article). For the benefit of participants most non-English searches are also removed, as it would be hard to judge the quality of results. Finally "junk" queries, such as "Ikohoyugc", are removed. (These junk queries make up one to two percent of total query volume.)

Instructions

How do I score queries?

You will be presented with a page containing the query at the very top and a list of results that could be relevant to the query. Tapping (or clicking) on the result will cycle the relevance ranking among the following options.

• Unrated
• Irrelevant [red]
• Maybe Relevant [orange]
• Probably Relevant [yellow]
• Relevant [white on green]

Tapping once more after Relevant (green background) will bring the result back to Unrated. You must rate at least 80% of the results for a query for the results to be saved. If you aren't sure select 'Skip this query' and you will be taken to a new query to rate. Skipped queries will not be shown to you again.

Snippets

Along with each potential search result there is a snippet of the Wikipedia article; all snippets are shown by default. Use the "Hide/Show all snippets" button to open or close all the snippets at once, or click the up (↑) or down (↓) arrow before the article name to hide or view a particular snippet.

What differentiates Relevant from Probably Relevant?

A result is Relevant if you would expect to find it in the top 5 results for a query. If something is related and possibly the answer to the query, but not certainly, use Probably Relevant. When grading please keep in mind that the top of results page is limited in space; 10 results that are all the "best answer" to a query would be impossible to show. Try to pick the best results as Relevant, and set the others to Probably Relevant. Probably Relevant results are ones you would expect to find in the bottom two thirds of the first result page.

Maybe Relevant?

The Maybe Relevant ranking is reserved for items that aren't completely irrelevant, but also aren't great answers to the query. Maybe Relevant results could show up on the results page, but wouldn't be particularly desirable. The main difference between Maybe Relevant and Irrelevant is that Irrelevant queries have no relationship to the query.

What about disambiguation pages, lists, talk pages, categories, etc.?

We are not sure yet if these are good pages to appear in results or not. Use your best judgement as to the quality of a result with respect to the given query and we will compare inter-judge rankings to try to determine what people expect.