User:TJones (WMF)/Notes/Survey of Zero-Results Queries

Introduction
In an effort to reduce the rate of Wikipedia search queries that produce no results (see the Discovery team's proposal), I've undertaken a manual review of three batches of 500,000 full-text queries that returned no results (taken from the top 52 wiki's, with 100K+ articles).

Samples
My first sample ("7/24") is the first 500,000 zero-result full-text queries from the 2015-07-24 Cirrus Search Request log. The queries are time-stamped from 2015-07-23 07:51:29 to 2015-07-23 10:11:29. (The time zone is not indicated, but I assume it is consistent from file to file.) I reviewed this sample the most extensively, reviewing patterns I had previously found in a sample of 100,000 similar queries restricted to enwiki, as well as looking for new patterns.

I also reviewed similar samples of 500,000 zero-result full-text queries from the Cirrus Search Request logs dated 2015-07-10 ("7/10", time-stamped from 2015-07-09 07:43:26 and 2015-07-09 10:20:40) and 2015-07-17 ("7/17", 2015-07-16 07:42:37 to 2015-07-16 09:52:23). In these samples I only looked for the patterns I had previously identified in the 7/24 sample.

Caveats and Limitations
An important part of this process has been looking for patterns of queries that would not show up when listing the individual top zero-result queries. However, I could not manually review every unique query individually in a timely fashion, so I have resorted to heuristics (mostly grep patterns) for counting instances of the various patterns. The numbers are not exact, but they sometimes vary significantly from sample to sample anyway. I do believe that large systematic query patterns have been identified.

The samples were limited to the first 500K relevant queries in each file, and do not represent the full 24-hour day.

This review is necessarily subjective and also limited by my familiarity with the languages and writing systems involved. (Hence, there's often more detail in enwiki and the top wikis in various Romance languages.)

Recurring Patterns
I've sorted the patterns by maximum frequency of occurrence in my three samples. Unless otherwise noted, we haven't yet tracked down a source for the queries, and their intent is generally unclear.

DOI
Examples: We had between 15,393 and 96,998 such queries in our samples, representing 3.08% to 19.40% of all zero-results queries. And while there were more in enwiki, there were many in these wikis as well: nl, ja, zh, war, vi, uk, sv, pt, pl, no, ko, it, id, hu, fr, fi, fa, de, cs, ceb, ca, ar, es, ru.
 * "10.3897/zookeys.457.6760" OR "http://zookeys.pensoft.net/articles.php?id=4267"
 * "10.1371/journal.pntd.0003900" OR "http://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0003900"
 * "10.3332/ecancer.2013.301"
 * "10.7821/naer.1.1.2-6"

We were able to track the source of these queries down to a software package called Lagotto, used to track references to articles made online and elsewhere.

These queries are well-formed and can return results, but many do not because not all published academic articles are referenced in Wikipedia.

Unix Timestamps
Examples: We had between 26,351 and 42,650 such queries in our samples, representing 5.27% to 8.53% of all zero-results queries. They were spread across many wikis, with substantial numbers in en, it, ru, ja, fa, tr, nl, he, ar, id, cs, hi, vi, ro, hu, and uk.
 * 1431786835781:بيت لحم
 * 1436436482196:Илюзия
 * 1432198699732:Meryl Streep

Though the leading number looks like a Unix timestamp in epoch seconds, the numbers don't make sense. The time indicated spans from last year to next month.

These queries do not return results, though removing the number and colon from the beginning of the query results in a title match on the relevant wiki in all the examples tested.

"Article_title" AND "title of link taken from article"
Examples: We had an estimated 8,174 to 16,657 such queries in our samples, representing 1.63% to 3.33% of all zero-results queries. These are generally restricted to enwiki.
 * "Argentine_football_league_system" AND "Football in Moldova"
 * "Argentine_football_league_system" AND "Football in Mongolia"
 * "Argentine_football_league_system" AND "Football in Mozambique"
 * "Argentine_football_league_system" AND "Football in Papua New Guinea"

These queries seem to consist of a quoted article title (with underscores) ANDed with the quoted title of an article (without underscores) linked to in the first article. There can be hundreds of different queries with the same first component.

Variants of the queries with spaces instead of underscores or with neither quotes nor underscores do not return results. Searching for just the first component does give results.

TV Episodes / Movies—"..." film
Examples: We had between 7,878 and 8,794 such queries in our samples, representing 1.58% to 1.76% of all zero-results queries. These were common in the following wikis: en, nl, de, fr, ja.
 * "88 Minutos" film
 * "Castle S1E1" film
 * "Como treinar o seu Dragão 2 Filmes Completos Dublados" film
 * "30 Rock - Season 3 S3E18" film

These queries consist of quoted material (generally a movie or TV show title, often followed by a season/episode number, e.g. S3E18), followed by the word film, even if the quoted material is not a film.

Many of the titles used in the queries (88 Minutos, Como treinar o seu Dragão 2) return results in the appropriate language wiki.

I believe that these queries are intended to find websites to download these films or TV episodes.

quot
Examples: We had between 5,888 and 7,768 such queries in our samples, representing 1.18% to 1.55% of all zero-results queries. These were generally found in enwiki.
 * quot Anesthesia quot
 * quot Albert Payne quot
 * Canberra AND quot Andy Fisher quot
 * Moira East quot

These queries contain the word quot in them. It appears that quotation marks in the original query were converted to entities (&amp;quot;) and then sanitized (removing & and ; from the query), leaving quot.

Many of the queries without the quot's are exact matches for article titles, and many match with the the quot's either dropped or converted to straight quotation marks (").

term+term+term country
Examples: We had an estimated 3,437 to 6,725 such queries in our samples, representing 0.69% to 1.35% of all zero-results queries. These were generally found in eswiki, with some in enwiki. The country names are generally in Spanish or English, though some were in other languages.
 * ópera+del+estado+de+hamburgo Bangladés
 * zanthoxylum+thomasianum Bélgica
 * finance+and+revenue+f+c Germany
 * finance+and+revenue+f+c Ecuador

The countries included in the query don't seem to necessarily have any relationship with the other search terms.

Many of the queries do return results is the country name is excluded, in both enwiki and eswiki.