User:CKoerner (WMF)/Discernatron

You can help improve search relevance
To make changes to search at a quicker pace, the Search team needs to be able to test changes before making them available on-wiki. Discernatron is a tool that allows participants to judge the relevance of search results. When evaluating potential changes to the Wikimedia search the team will use these judgements to help rate potential changes by how much closer they are to putting the most relevant articles at the top of the search results page.

Get Started »

(login with your Wikimedia account)

What queries am I rating?
Every month the Discovery department loads approximately 500 randomly selected search queries from the English Wikipedia into Discernatron for grading. These queries represent around 0.0001% of the total full text searches on English Wikipedia. This sample is incredibly small, but still represents a wide swath of the types of queries received. Before being released to Discernatron, WMF engineers review the sampled set of queries and remove anything that could be considered personally identifiable information (PII). Initially only queries for English Wikipedia are being used but Discernatron will expand to other languages - such as French, Spanish and Russian - as time goes by.

So someone is looking at all my searches?
No. When reviewing queries there is no additional meta data, such as user name, location, or IP address. Additionally due to the sample size it is very unlikely that the sample contains more than one query from any single user. See Discovery's Data access guidelines for more information on how the department manages user data.

What kinds of queries are removed?
Anything potentially personally identifiable. This means any kind of phone number, serial number, or non-notable address. We remove searches for specific URLs and non-notable companies. Additionally names of non-notable people (those that don't have wiki articles and aren't mentioned prominently in any other article). For the benefit of participants most non-English searches are also removed, as it would be hard to judge the quality of results. Finally "junk" queries, such as "Ikohoyugc", are removed (These junk queries make up one to two percent of total query volume).

=Instructions=

How do I score queries?
Participants will be presented with a page containing the query at the very top and a list of results that could be relevant to the query. Tapping (or clicking) on the result will cycle the relevance ranking between the following options. Tapping once more after Relevant (green background) will bring the result back to unrated. You must rate at least 80% of the results to a query for the results to be saved. If you aren't sure select 'Skip this query' and you will be taken to a new query to rate. Skipped queries will not be shown to you again.
 * None to Irrelevant
 * Maybe Relevant
 * Probably Relevant
 * Relevant

Snippets
Along with each potential search result there is a snippet of the Wikipedia article available. Clicking on the down arrow (↓) will allow participants to view the snippet for a given result.

What differentiates Relevant from Probably Relevant?
A result is relevant if you would expect to find it in the top 5 results to a query. If something is related and possibly the answer to the query, but not certainly, use "probably" (relevant). When grading please keep in mind that the top of results page is limited in space; having 10 results that are all the best answer to a query is impossible to show. Try and pick the best results as relevant, and set the others to probably relevant. Probably relevant results are ones you would expect to find in the bottom two thirds of the first result page.

Maybe Relevant?
The maybe relevant ranking is reserved for items that aren't completely irrelevant, but also aren't great answers to the query. Maybe relevant results could show up on the results page, but wouldn't be particularly desirable. The main difference between maybe relevant and irrelevant is that irrelevant queries have no relationship to the query.

What about disambiguation pages, lists, talk pages, categories, etc.?
We are not sure yet if these are good pages to appear in results or not. Use your best judgement as to the quality of a result with respect to the given query and we will compare inter-judge rankings to try to determine what people expect.