User:TJones (WMF)/Notes/Survey of Zero-Results Queries

=Review of a multilingual 500K sample=

Introduction
In an effort to reduce the rate of Wikipedia search queries that produce no results (see the Discovery team's proposal), I've undertaken a manual review of three batches of 500,000 full-text queries that returned no results (taken from the top 52 wiki's, with 100K+ articles—for future reference, at this time that's en, sv, de, nl, fr, war, ru, it, ceb, es, vi, pl, ja, pt, zh, uk, ca, fa, sh, no, ar, fi, id, ro, hu, cs, sr, ko, ms, tr, min, eo, kk, eu, da, sk, bg, hy, he, lt, hr, sl, et, uz, gl, nn, vo, la, simple, el, hi, & ce).

Samples
My first sample ("7/24") is the first 500,000 zero-result full-text queries from the 2015-07-24 Cirrus Search Request log. The queries are time-stamped from 2015-07-23 07:51:29 to 2015-07-23 10:11:29. (The time zone is not indicated, but I assume it is consistent from file to file.) I reviewed this sample the most extensively, reviewing patterns I had previously found in a sample of 100,000 similar queries restricted to enwiki, as well as looking for new patterns.

I also reviewed similar samples of 500,000 zero-result full-text queries from the Cirrus Search Request logs dated 2015-07-10 ("7/10", time-stamped from 2015-07-09 07:43:26 and 2015-07-09 10:20:40) and 2015-07-17 ("7/17", 2015-07-16 07:42:37 to 2015-07-16 09:52:23). In these samples I only looked for the patterns I had previously identified in the 7/24 sample.

Caveats and Limitations
An important part of this process has been looking for patterns of queries that would not show up when listing the individual top zero-result queries. However, I could not manually review every unique query individually in a timely fashion, so I have resorted to heuristics (mostly grep patterns) for counting instances of the various patterns. The numbers are not exact, but they sometimes vary significantly from sample to sample anyway. I do believe that large systematic query patterns have been identified.

The samples were limited to the first 500K relevant queries in each log file, and do not represent the full 24-hour day.

This review is necessarily subjective and also limited by my familiarity with the languages and writing systems involved. (Hence, there's often more detail in enwiki and the top wikis in various Romance languages.)

Recurring Patterns
I've sorted the patterns by maximum frequency of occurrence in my three samples. Unless otherwise noted, we haven't yet tracked down a source for the queries, and their intent is generally unclear.

DOI
Examples: We had between 15,393 and 96,998 such queries in our samples, representing 3.08% to 19.40% of all zero-results queries. And while there were more in enwiki, there were many in these wikis as well: nl, ja, zh, war, vi, uk, sv, pt, pl, no, ko, it, id, hu, fr, fi, fa, de, cs, ceb, ca, ar, es, ru.
 * "10.3897/zookeys.457.6760" OR "http://zookeys.pensoft.net/articles.php?id=4267"
 * "10.1371/journal.pntd.0003900" OR "http://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0003900"
 * "10.3332/ecancer.2013.301"
 * "10.7821/naer.1.1.2-6"

We were able to track the source of these queries down to a software package called Lagotto, used to track references to articles made online and elsewhere.

These queries are well-formed and can return results, but many do not because not all published academic articles are referenced in Wikipedia.

The Lagotto developer indicated that some traffic may the result of an error in a demo, and so it may decrease, but will not disappear.

Unix Timestamps
Examples: We had between 26,351 and 42,650 such queries in our samples, representing 5.27% to 8.53% of all zero-results queries. They were spread across many wikis, with substantial numbers in en, it, ru, ja, fa, tr, nl, he, ar, id, cs, hi, vi, ro, hu, and uk.
 * 1431786835781:بيت لحم
 * 1436436482196:Илюзия
 * 1432198699732:Meryl Streep

Though the leading number looks like a Unix timestamp in epoch seconds, the numbers don't make sense. The time indicated spans from last year to next month.

These queries do not return results, though removing the number and colon from the beginning of the query results in a title match on the relevant wiki in all the examples tested.

We tracked down these queries to a bug in the Wikipedia Mobile App, which has since been fixed.

"Article_title" AND "title of link taken from article"
Examples: We had an estimated 8,174 to 16,657 such queries in our samples, representing 1.63% to 3.33% of all zero-results queries. These are generally restricted to enwiki.
 * "Argentine_football_league_system" AND "Football in Moldova"
 * "Argentine_football_league_system" AND "Football in Mongolia"
 * "Argentine_football_league_system" AND "Football in Mozambique"
 * "Argentine_football_league_system" AND "Football in Papua New Guinea"

These queries seem to consist of a quoted article title (with underscores) ANDed with the quoted title of an article (without underscores) linked to in the first article. There can be hundreds of different queries with the same first component.

Variants of the queries with spaces instead of underscores or with neither quotes nor underscores do not return results. Searching for just the first component does give results.

All of these queries are coming from a single IP address. We are trying to contact the owner. We have at least temporarily blocked the IP for failing to adhere to our terms of use and failing to contact us.

TV Episodes / Movies—"..." film
Examples: We had between 7,878 and 8,794 such queries in our samples, representing 1.58% to 1.76% of all zero-results queries. These were common in the following wikis: en, nl, de, fr, ja.
 * "88 Minutos" film
 * "Castle S1E1" film
 * "Como treinar o seu Dragão 2 Filmes Completos Dublados" film
 * "30 Rock - Season 3 S3E18" film

These queries consist of quoted material (generally a movie or TV show title, often followed by a season/episode number, e.g. S3E18), followed by the word film, even if the quoted material is not a film.

Many of the titles used in the queries (88 Minutos, Como treinar o seu Dragão 2) return results in the appropriate language wiki.

We tracked down these queries to a media player, which searches the title of a the file being played.

quot
Examples: We had between 5,888 and 7,768 such queries in our samples, representing 1.18% to 1.55% of all zero-results queries. These were generally found in enwiki.
 * quot Anesthesia quot
 * quot Albert Payne quot
 * Canberra AND quot Andy Fisher quot
 * Moira East quot

These queries contain the word quot in them. It appears that quotation marks in the original query were converted to entities (&amp;quot;) and then sanitized (removing & and ; from the query), leaving quot.

Many of the queries without the quot's are exact matches for article titles, and many match with the the quot's either dropped or converted to straight quotation marks (").

These are coming from National Library of Australia. They have been contacted.

term+term+term country
Examples: We had an estimated 3,437 to 6,725 such queries in our samples, representing 0.69% to 1.35% of all zero-results queries. These were generally found in eswiki, with some in enwiki. The country names are generally in Spanish or English, though some were in other languages.
 * ópera+del+estado+de+hamburgo Bangladés
 * zanthoxylum+thomasianum Bélgica
 * finance+and+revenue+f+c Germany
 * finance+and+revenue+f+c Ecuador

The countries included in the query don't seem to necessarily have any relationship with the other search terms.

Many of the queries do return results is the country name is excluded, in both enwiki and eswiki.

term+term+term
(This pattern is out of order to be grouped with the previous one.)

Examples: We had an estimated 1,382 to 2,536 such queries in our samples, representing 0.28% to 0.51% of all zero-results queries. These were generally found in eswiki, with some in enwiki.
 * accountable+care+organizations+and+evidence+based+payment+reform
 * android+phone+gone+awol+just+google
 * como+afilar+una+batidora+picadora+electrica+recupera+batidoras
 * el+peine+de+las+sirenas

These queries are characterized by being run together with pluses instead os spaces.

paint
Examples: We had an estimated 1,094 to 3,554 such queries in our samples, representing 0.22% to 0.71% of all zero-results queries. These were generally found in enwiki.
 * ""abel boulineau"" paint
 * ""wilhelmina k. lagerholm"" paint
 * Carl Friedrich Schulz - Zeitungslektüre am Biertisch (1851) paint
 * Karr par Breton d'après Petit paint

These queries are characterized by ending with the word paint. They come in two formats: seems to be the name of an artist, double quoted twice. is the name of a file on Wikimedia Commons, without the file suffix (e.g., .jpg). The artist names I tried did not return results, though Google found them in various art galleries. The Commons files generally return results when searched on Commons.
 * "" "" paint
 * paint

Highly repeated searches
These are idiosyncratic searches, but can be repeated up to hundred of times per hour, indicating that they are probably not driven by a human typing in the search over and over.

Examples: These are hard to quantify, but I looked for queries that were repeated more than 50 times in my samples, didn't fall into other categories, and were unique enough not to be driven by the day's events or random searching (e.g., one word searches).
 * ou as I can get tonight without being detected, and Tuck and Clay will be there too, along with an undercover team. You’ll have an earpiece no one will be able to see, so we
 * Google books says this is a snippet from a novel. In a two-week sample of the top 100 queries per day, this came up 12 times. It was probably there the other two days, but didn't make the top 100. There are up to 964 queries in a day.
 * Iamlookingfornodethree
 * There is a weird pattern here of iamlookingfornode, where  can be a few different things.
 * form 1+ 3dprinter
 * Just found one day, but 668 times in less than three hours.
 * Dounload feer game
 * Just found one day, but 248 times in less than three hours.

We had an estimated 892 to 3,019 such queries in our samples, representing 0.18% to 0.60% of all zero-results queries.

{searchTerms}
Examples: We had between 1,909 and 2,314 such queries in our samples, representing 0.38% to 0.46% of all zero-results queries. These were generally found in ruwiki.
 * {searchTerms}
 * %7bsearchTerms%7d
 * Liste der {searchTerms} Episoden
 * {searchTerms}'||'

This is likely developer error in an app or other automated search.

Similarly, we had a number of examples of search_suggest_query (440 to 509, 0.09% to 0.10%, en, de, fr) and \{@} (148 to 205, 0.03% to 0.04%, en, de, ru).

## tel fax
Examples: We had between 33 and 1,293 such queries in our samples, representing 0.01% to 0.26% of all zero-results queries. These were generally found in dewiki.
 * Aluminum Bracket 44 uk tel fax
 * diecast aluminum housing 31 nl tel fax
 * plastic injection mold makers 34 es tel fax

These queries seem to be manufacturing terms, a two digit number, a country code, and "tel fax".

Chinese product descriptions .xyz
Examples: We had an estimated 172 to 989 such queries in our samples, representing 0.03% to 0.20% of all zero-results queries. These were generally found in enwiki.
 * 花店用品/蓝色妖姬着色剂 根部吸水浅宝蓝玫瑰*13230782866*QQ34040316座机03177896222*psuddf.zhijieranliao.xyz
 * 直接染料/直接深棕GTL *13230782866*QQ34040316座机03177896222*zkdact.baohuabanranliao.xyz

These appear to be product description in Chinese, along with additional information. They all end in .xyz. Note that 座机 means landline, indicating contact info.

Online searches for parts of these reveal a similar pattern on Chinese-language business/manufacturing sites.

Massive snippets
This particular category is fairly rare, but may incur significant computational cost, so it is worth noting. These are searches that are 500 characters in length or more (up to more than 5,000 characters). Many look like snippets from larger texts, such as books or articles, and are in several different languages.

We had between 183 and 261 such queries in our samples, representing 0.04% to 0.05% of all zero-results queries. These were generally found in en, fr, de, and ru.

Miscellany
Below are some very general impressions from the larger collections of zero-results (10K+ from a given wiki). My ability to analyze languages I don't know is limited, but here is what I noticed:
 * dewiki has a few hundred OR'd together wildcard searches, some of which seem to be trying handle variations in declension.
 * jawiki has lots of "..." film searches.
 * ruwiki has a few non-cyrillic searches
 * itwiki has lots of queries that are multi-word phrases with underscores instead of spaces
 * eswiki and frwiki have a fair number of build up searches and searches in Arabic, and frwiki has a fair number of searches in Chinese
 * zhwiki has lots of non-Chinese searches in various languages
 * plwiki has a fair number of queries of the form * * AND (muzyk* OR Dyskografia) (with asterisks) where seems to be an artist, band, album, or something similar.

Summary Table
= One Month Followup = I've conducted a followup study to track changes to these patterns over time. Since my investigation is more focused this time (on these queries) I was able to process a lot more data.

As before, I've limited my investigation to full text queries against the 52 wikis with 100K+ articles (see above).

My new samples are taken from the entire day (i.e., one log file, labeled from ~8am to ~8am) for each day queried. For zero-result queries, I sampled 1-in-15, to get a 300K-400K query sample. For all queries, I sampled 1-in-45 to get a 600K to 800K query sample.

I focused on the largest patterns above, and included some of the easier to automatically detect patterns as well.

In particular: DOI, Unix Timestamp, "Title_1" AND "Title 2", "..." film, quot, "..." paint, tel fax, .xyz, {searchTerms}, \{@}, search_suggest_query, term+term+term Country, and long lines (150+ characters).

I counted the frequency and ratio of these kinds of queries for each day of one recent week (8/19-8/25) and one week around the original investigation date (7/22-7/28); both are Wednesday to Tuesday (since I was investigating on a Wednesday, 8/26).

Last Month (July 2015)
Below are the results for a week in July. Keep in mind that Zero Queries are sampled 1-in-15 and All Queries are sampled 1-in-45. The percentage to the right of the sample size for the Zero Queries is the zero results rate (zero_queries*15/all_queries*45).

This Month (August 2015)
Below are the results for a week in August. Keep in mind that Zero Queries are sampled 1-in-15 and All Queries are sampled 1-in-45. The percentage to the right of the sample size for the Zero Queries is the zero results rate (zero_queries*15/all_queries*45).

Analysis
The overall zero results rate hasn't changed much from July to August. Friday to Tuesday track pretty closely in the graph below, and Wednesday and Thursday have higher zero results rates in August than July.

The decrease in some queries, such as the "Title_1" AND "Title 2" queries, have little overall effect, because the zero results rate is only slightly higher than the overall query rate (usually within 1% above), so removing both the zero results queries and the queries that do return results has little net effect.

"Title_1" AND "Title 2"
This IP was banned (apparently around 8/19), and queries dropped to zero, both overall and zero results queries.

Unix Timestamps
The fix to the Wikipedia Mobile App was deployed, and rates have dropped significantly, though not to 0%, because not everyone updates their apps right away.

DOI
DOI rates have always fluctuated considerably across various samples (3% to 19%, see above), so it's hard to be sure, but in the samples for these weeks, the overall rate seems to be down.

quot and "..." film
Not much has happened with the quot and "..." film queries. We've contacted the respective sources, but it isn't clear anything will be done soon. The media player is behaving as designed, so these queries probably won't stop. The NLA may eventually fix their quot queries.

Comparison to Dashboards
Despite the large dip in "Title_1" AND "Title 2" and Unix Timestamp queries, we haven't seen any corresponding movement in the Discovery KPI dashboard.

Part of the reason for this is that this analysis and the dashboards are measuring different things: the dashboards look at all zero results (including prefix searches), while I'm looking only at full text queries to large wikis (to limit the scope of my initial investigation). I'm going to try to confirm this (and see if anything else pops out!) by quickly looking at everything I didn't look at here.

Everything Else (full text)
I took a 1-in-45 sample of full text queries from 8/25 for wikis not in the 52 100K+ wikis, and found some interesting stuff.

There were 269,924 samples (compared to 695,853 in the "big" wikis)—so, roughly 28% of traffic.

There were 152,726 zero result queries, for a 56.6% zero results rate (compared to ~20% in the "big" wikis).

The usual suspects do not show up as much, other than DOI: Other interesting facts:

There were only about 3K non-latin searches (lots of Greek, Cyrillic, Hebrew, Arabic, Devanagari, Thai, Japanese, Chinese, emoji, and more).

A quick skim of the zero results show lots of names.. but not a lot else stuck out.

The top 12 wikis are responsible for 145,325 of the zero results queries (95.2%) (with Wikidata on top and most of the rest coming from itwiki...):
 * 20699 wikidatawiki_content (13.5%)
 * 19798 itwiktionary_content (13.0%)
 * 18753 itwikiversity_content
 * 18080 itwikinews_content
 * 16587 itwikivoyage_content
 * 15504 itwikibooks_content
 * 13460 itwikisource_content
 * 12148 itwikiquote_content
 * 3773 eswiki
 * 3764 nlwiktionary_content
 * 1419 sewiki_content
 * 1340 thwiki_content

=Full manual review of a 1K enwiki sample=

Introduction
Looking for large recurring patterns of searches will not reveal the frequency of idiosyncratic erorrs (like typos, gibberish, and queries in foreign languages) that can't be easily be recognized automatically. So I undertook a manual review of 1047 randomly sampled queries from from one day's worth of logs of enwiki full-text searches, starting on 2015-07-29. This was a random sample (or as random as pseudorandom can be) rather than an every-N sample.

Caveats
All categories are at least somewhat subjective, and depend in part on my ability to recognize (or uncover) user intent. Many items could have been put in multiple categories, but I chose just one each time (some comments about this are included below). I know others would disagree on some categories, and I know that there are errors in categorization (i.e., I'd disagree with myself), but the overall trends are still illustrative.

Requestors
Overall, 67.2% were requested via API, 32.8% via web.

Typos
I broke typos up into two categories: apparent mistakes, and incomplete words or phrases. Note that the vast majority in the incomplete category come via API, hinting that some app may be sending incomplete queries. 153     14.6%  TYPO,    54.2% web,  45.8% api 93       8.9%  INCOMP,  97.8% api,   2.2% web Total: 23.5%

Typo autosuggestions
I took a random sample of 40 of the queries in the TYPO category and searched for them in enwiki via the web. The results were oddly evenly distributed: Half had clearly correct suggestions, a quarter had no results, and a quarter had some non-zero results that were either wrong, or not clearly correct.
 * 10/40 = 25.0% zero results
 * 10/40 = 25.0% some results
 * 20/40 = 50.0% correct results

Typo reverse index
By manual inspection, 20 of the 153 TYPOs (13.1%) had an error in the first two characters of one of the search terms, and thus might benefit from a reverse index.

Previously seen categories
These are the previously discussed large categories of zero-return queries. The distributions are different from previous samples, sometimes drastically different. This could be attributed to the current small sample size, the time skew in the earlier sample, day-of-the-week effects, and random vagaries of millions of people searching—though that last can account for almost any variance! 125     11.9%  AND,   100.0% api	"Article_title" AND "title of link taken from article" 91       8.7%  UNIX,  100.0% api	Unix Timestamps 29       2.8%  FILM,  100.0% web	TV Episodes / Movies—"..." film 28       2.7%  QUOT,  100.0% api	quot 20       1.9%  DOI,   100.0% api	DOI 3        0.3%  TERM+, 100.0% api	term+term+term / term+term+term country Total: 28.3%

Foreign languages
Though enwiki has many pages with titles in other languages, these searches didn't get any results. I didn't dig too deeply into most of them. However, numerous entires filed under MOVIES, MUSIC, and YOUTUBE are in Spanish, Portuguese, Turkish, or other languages. 13       1.2%  CHINESE,    61.5% api,  38.5% web 12       1.1%  ARABIC,     58.3% web,  41.7% api 7        0.7%  CYRILLIC,   57.1% web,  42.9% api 7        0.7%  TAGALOG,    85.7% web,  14.3% api 6        0.6%  SPANISH,    83.3% api,  16.7% web 5        0.5%  MALAY,      80.0% web,  20.0% api 3        0.3%  GERMAN,     66.7% api,  33.3% web 3        0.3%  DEVTRANS,  100.0% api 2        0.2%  NORWEGIAN, 100.0% api 2        0.2%  SWAHILI,   100.0% web 1 each (0.1%, 100% api) for GREEK, THAI, TAMIL 1 each (0.1%, 100% web) for PORTUGUESE, DUTCH, FRENCH, LATIN, SWEDISH, ITALIAN, CROATIAN, HINDI, ESTONIAN, HMONG, KANNADA, FINNISH Total: 7.2%

One or Two Word Queries
How many of these foreign language queries are one or two words, and thus, for example, more likely to get a match in Wikitionary or Wikivoyage?


 * Arabic: 0/12, 3/12
 * Chinese: hard to judge, but apparently 0/13; not sure on "two words"
 * Cyrillic: 4/7, 1/7
 * Dev-Trans: 3/3
 * German: 1/3, 1/3
 * Malay: 0/5, 1/5
 * Norwegian: 0/2, 2/2
 * Spanish: 1/6, 2/6
 * Swahili: 0/2, 0/2
 * Tagalog: 0/7
 * Croatian, Estonian, Swedish, Tamil: 1/1
 * Dutch, Greek, Hindi, Hmong, Italian, Portuguese, Thai: 0/1, 0/1
 * Finnish, French, Kannada, Latin: 0/1, 1/1

Totals:
 * One word: 13/75
 * Two words: 14/75

Mystery queries
These are ones that I just couldn't figure out. They weren't clearly junk. They could be typos, but often they are are too ambiguous to be sure. 66       6.3%  ??,  71.2% api,  28.8%

Not encyclopedic
These categories are potentially problematic, since they may depend in part on what you think should or should not be in Wikipedia.

Total: 28.0%

Wrong website
57       5.4%  PROD,      66.7% api,  33.3% web 45       4.3%  QUESTION,  57.8% web,  42.2% api 32       3.1%  URL,       62.5% web,  37.5% api 3        0.3%  NEWS,      66.7% web,  33.3% api 2        0.2%  TWITTER,  100.0% api Total: 13.3%
 * PROD: These are queries that appear to be about general or specific products, including video games, clothing brands, drugs, decorations, laptop replacement parts, etc.
 * QUESTION: These are queries that seem to be asking for non-encyclopedic information, including advice on romance, study habits, and home furnishing, job searches, making travel arrangements, celebrity facts, scholarly research, etc.
 * URL: These look like they are or tried to be URLs.
 * NEWS: Questions about current events.
 * TWITTER: Actual tweets or parts of tweets.

Content
These appear to be searches for particular content, including particular songs, albums, music by a particular artist, movies, TV episodes, books, or scholarly articles. The YOUTUBE category are queries that exactly match the titles of individual YouTube videos. ISBNs are specific, plain ISBN numbers. 42       4.0%  MUSIC,    78.6% api,  21.4% web 25       2.4%  YOUTUBE,  92.0% api,   8.0% web 10       1.0%  MOVIE,    50.0% api,  50.0% web 7        0.7%  ARTICLE,  85.7% web,  14.3% api 7        0.7%  ISBN,     71.4% web,  28.6% api Total: 8.8%

People and places
These are queries for particular people or places that are not in Wikipedia, including named individuals or parts of names (PERSON), online usernames (USER), addresses of business (ADDRESS—note that all are in Las Vegas) or email addresses.

The LINKED category are searches in this form: Both LINKEDIN and VIADEO (professional social networking sites) were used. 26       2.5%  PERSON,   53.8% web,  46.2% api 16       1.5%  USER,     56.2% api,  43.8% web 10       1.0%  ADDRESS, 100.0% api 4        0.4%  LINKED,  100.0% api 2        0.2%  EMAIL,   100.0% web Total: 5.6%
 * "SURNAME" "FIRST MIDDLE" "COMPANY" LINKEDIN

Misc
Stuff I could at least partly identify, but couldn't categorize elsewhere. One was just a number, one a slightly mangled Wikimedia Commons file name, the other a Wikimedia Commons category name. 2        0.2%  COMMONS,  50.0% api,  50.0% web 1        0.1%  NUMBER,  100.0% web Total: 0.3%

Junk
34       3.2%  JUNK,   85.3% web,  14.7% api 9        0.9%  OCR,    55.6% api,  44.4% web 3        0.3%  ERROR,  66.7% api,  33.3% web 3        0.3%  EMOJI, 100.0% web 2        0.2%  SPAM,   50.0% api,  50.0% web 1        0.1%  NODE,  100.0% web Total: 5.0%
 * JUNK includes snippets of larger texts, multiply repeated letters, keyboard banging, and the like.
 * OCR are "words" that seem to appear primarily as OCR errors in Google Books.
 * ERRORs include "search_suggest_query" and the like.
 * EMOJI are strings of emoji characters.
 * SPAM includes an actual advertisement and what looks like a hacking probe attempt.
 * NODE is a pattern that's come up before: Iamlookingfor[...]node[...]

Misses
Actual contentful queries that there are no entries for.

12       1.1%  MISS,     50.0% api,  50.0% web 4        0.4%  SPECIES,  75.0% web,  25.0% api 5        0.5%  DICT,     80.0% api,  20.0% web Total: 2.0%
 * DICT: mostly obscure words, but they are in Wiktionary.
 * SPECIES: Latin species names.
 * MISS: other misses that seem to describe reasonable things that could be in Wikipedia, but either aren't or weren't found.