User:TJones (WMF)/Notes/Survey of Zero-Results Queries

July/August 2015 — See TJones_(WMF)/Notes for other projects.

Review of a multilingual 500K sample

Introduction

In an effort to reduce the rate of Wikipedia search queries that produce no results (see the Discovery team's proposal), I've undertaken a manual review of three batches of 500,000 full-text queries that returned no results (taken from the top 52 wikis, with 100K+ articles—for future reference, at this time that's en, sv, de, nl, fr, war, ru, it, ceb, es, vi, pl, ja, pt, zh, uk, ca, fa, sh, no, ar, fi, id, ro, hu, cs, sr, ko, ms, tr, min, eo, kk, eu, da, sk, bg, hy, he, lt, hr, sl, et, uz, gl, nn, vo, la, simple, el, hi, & ce).

Samples

My first sample ("7/24") is the first 500,000 zero-result full-text queries from the 2015-07-24 Cirrus Search Request log. The queries are time-stamped from 2015-07-23 07:51:29 to 2015-07-23 10:11:29. (The time zone is not indicated, but I assume it is consistent from file to file.) I reviewed this sample the most extensively, reviewing patterns I had previously found in a sample of 100,000 similar queries restricted to enwiki, as well as looking for new patterns.

I also reviewed similar samples of 500,000 zero-result full-text queries from the Cirrus Search Request logs dated 2015-07-10 ("7/10", time-stamped from 2015-07-09 07:43:26 and 2015-07-09 10:20:40) and 2015-07-17 ("7/17", 2015-07-16 07:42:37 to 2015-07-16 09:52:23). In these samples I only looked for the patterns I had previously identified in the 7/24 sample.

Caveats and Limitations

An important part of this process has been looking for patterns of queries that would not show up when listing the individual top zero-result queries. However, I could not manually review every unique query individually in a timely fashion, so I have resorted to heuristics (mostly grep patterns) for counting instances of the various patterns. The numbers are not exact, but they sometimes vary significantly from sample to sample anyway. I do believe that large systematic query patterns have been identified.

The samples were limited to the first 500K relevant queries in each log file, and do not represent the full 24-hour day.

This review is necessarily subjective and also limited by my familiarity with the languages and writing systems involved. (Hence, there's often more detail in enwiki and the top wikis in various Romance languages.)

Recurring Patterns

I've sorted the patterns by maximum frequency of occurrence in my three samples. Unless otherwise noted, we haven't yet tracked down a source for the queries, and their intent is generally unclear.

DOI

Examples:

• "10.3897/zookeys.457.6760" OR "http://zookeys.pensoft.net/articles.php?id=4267"
• "10.1371/journal.pntd.0003900" OR "http://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0003900"
• "10.3332/ecancer.2013.301"
• "10.7821/naer.1.1.2-6"

We had between 15,393 and 96,998 such queries in our samples, representing 3.08% to 19.40% of all zero-results queries. And while there were more in enwiki, there were many in these wikis as well: nl, ja, zh, war, vi, uk, sv, pt, pl, no, ko, it, id, hu, fr, fi, fa, de, cs, ceb, ca, ar, es, ru.

We were able to track the source of these queries down to a software package called Lagotto, used to track references to articles made online and elsewhere.

These queries are well-formed and can return results, but many do not because not all published academic articles are referenced in Wikipedia.

The Lagotto developer indicated that some traffic may the result of an error in a demo, and so it may decrease, but will not disappear.

Unix Timestamps

Examples:

• 1431786835781:بيت لحم
• 1436436482196:Илюзия
• 1432198699732:Meryl Streep

We had between 26,351 and 42,650 such queries in our samples, representing 5.27% to 8.53% of all zero-results queries. They were spread across many wikis, with substantial numbers in en, it, ru, ja, fa, tr, nl, he, ar, id, cs, hi, vi, ro, hu, and uk.

Though the leading number looks like a Unix timestamp in epoch seconds, the numbers don't make sense. The time indicated spans from last year to next month.

These queries do not return results, though removing the number and colon from the beginning of the query results in a title match on the relevant wiki in all the examples tested.

We tracked down these queries to a bug in the Wikipedia Mobile App, which has since been fixed.

"Article_title" AND "title of link taken from article"

Examples:

• "Argentine_football_league_system" AND "Football in Moldova"
• "Argentine_football_league_system" AND "Football in Mongolia"
• "Argentine_football_league_system" AND "Football in Mozambique"
• "Argentine_football_league_system" AND "Football in Papua New Guinea"

We had an estimated 8,174 to 16,657 such queries in our samples, representing 1.63% to 3.33% of all zero-results queries. These are generally restricted to enwiki.

These queries seem to consist of a quoted article title (with underscores) ANDed with the quoted title of an article (without underscores) linked to in the first article. There can be hundreds of different queries with the same first component.

Variants of the queries with spaces instead of underscores or with neither quotes nor underscores do not return results. Searching for just the first component does give results.

All of these queries are coming from a single IP address. We are trying to contact the owner. We have at least temporarily blocked the IP for failing to adhere to our terms of use and failing to contact us.

TV Episodes / Movies—"..." film

Examples:

• "88 Minutos" film
• "Castle S1E1" film
• "Como treinar o seu Dragão 2 Filmes Completos Dublados" film
• "30 Rock - Season 3 S3E18" film

We had between 7,878 and 8,794 such queries in our samples, representing 1.58% to 1.76% of all zero-results queries. These were common in the following wikis: en, nl, de, fr, ja.

These queries consist of quoted material (generally a movie or TV show title, often followed by a season/episode number, e.g. S3E18), followed by the word film, even if the quoted material is not a film.

Many of the titles used in the queries (88 Minutos, Como treinar o seu Dragão 2) return results in the appropriate language wiki.

We tracked down these queries to a media player, which searches the title of a the file being played.

quot

Examples:

• quot Anesthesia quot
• quot Albert Payne quot
• Canberra AND quot Andy Fisher quot
• Moira East quot

We had between 5,888 and 7,768 such queries in our samples, representing 1.18% to 1.55% of all zero-results queries. These were generally found in enwiki.

These queries contain the word quot in them. It appears that quotation marks in the original query were converted to entities (") and then sanitized (removing & and ; from the query), leaving quot.

Many of the queries without the quot's are exact matches for article titles, and many match with the the quot's either dropped or converted to straight quotation marks (").

These are coming from National Library of Australia. They have been contacted.

term+term+term country

Examples:

• ópera+del+estado+de+hamburgo Bangladés
• zanthoxylum+thomasianum Bélgica
• finance+and+revenue+f+c Germany
• finance+and+revenue+f+c Ecuador

We had an estimated 3,437 to 6,725 such queries in our samples, representing 0.69% to 1.35% of all zero-results queries. These were generally found in eswiki, with some in enwiki. The country names are generally in Spanish or English, though some were in other languages.

The countries included in the query don't seem to necessarily have any relationship with the other search terms.

Many of the queries do return results if the country name is excluded, in both enwiki and eswiki.

term+term+term

(This pattern is out of order to be grouped with the previous one.)

Examples:

• accountable+care+organizations+and+evidence+based+payment+reform
• android+phone+gone+awol+just+google
• como+afilar+una+batidora+picadora+electrica+recupera+batidoras
• el+peine+de+las+sirenas

We had an estimated 1,382 to 2,536 such queries in our samples, representing 0.28% to 0.51% of all zero-results queries. These were generally found in eswiki, with some in enwiki.

These queries are characterized by being run together with pluses instead of spaces.

paint

Examples:

• ""abel boulineau"" paint
• ""wilhelmina k. lagerholm"" paint
• Carl Friedrich Schulz - Zeitungslektüre am Biertisch (1851) paint
• Karr par Breton d'après Petit paint

We had an estimated 1,094 to 3,554 such queries in our samples, representing 0.22% to 0.71% of all zero-results queries. These were generally found in enwiki.

These queries are characterized by ending with the word paint. They come in two formats:

• ""<artist>"" paint
• <commons file> paint

<artist> seems to be the name of an artist, double quoted twice. <commons file> is the name of a file on Wikimedia Commons, without the file suffix (e.g., .jpg). The artist names I tried did not return results, though Google found them in various art galleries. The Commons files generally return results when searched on Commons.

Highly repeated searches

These are idiosyncratic searches, but can be repeated up to hundred of times per hour, indicating that they are probably not driven by a human typing in the search over and over.

Examples:

• ou as I can get tonight without being detected, and Tuck and Clay will be there too, along with an undercover team. You’ll have an earpiece no one will be able to see, so we
  • Google books says this is a snippet from a novel. In a two-week sample of the top 100 queries per day, this came up 12 times. It was probably there the other two days, but didn't make the top 100. There are up to 964 queries in a day.
  • Upon further investigation, the user agent for these queries (which have since stopped) appear to be from a Nook tablet e-reader, which does support searching Wikipedia. Perhaps someone set their Nook down in a way that the "search Wikipedia" button was continuously pressed (for a couple of weeks!)
• Iamlookingfornodethree
  • There is a weird pattern here of iamlookingfornode<x>, where <x> can be a few different things.
• form 1+ 3dprinter
  • Just found one day, but 668 times in less than three hours.
• Dounload feer game
  • Just found one day, but 248 times in less than three hours.

These are hard to quantify, but I looked for queries that were repeated more than 50 times in my samples, didn't fall into other categories, and were unique enough not to be driven by the day's events or random searching (e.g., one word searches).

We had an estimated 892 to 3,019 such queries in our samples, representing 0.18% to 0.60% of all zero-results queries.

{searchTerms}

Examples:

• {searchTerms}
• %7bsearchTerms%7d
• Liste der {searchTerms} Episoden
• {searchTerms}'||'

We had between 1,909 and 2,314 such queries in our samples, representing 0.38% to 0.46% of all zero-results queries. These were generally found in ruwiki.

This is likely developer error in an app or other automated search.

Similarly, we had a number of examples of search_suggest_query (440 to 509, 0.09% to 0.10%, en, de, fr) and \{@} (148 to 205, 0.03% to 0.04%, en, de, ru).

## <countrycode> tel fax

Examples:

• Aluminum Bracket 44 uk tel fax
• diecast aluminum housing 31 nl tel fax
• plastic injection mold makers 34 es tel fax

We had between 33 and 1,293 such queries in our samples, representing 0.01% to 0.26% of all zero-results queries. These were generally found in dewiki.

These queries seem to be manufacturing terms, a two digit number, a country code, and "tel fax".

Chinese product descriptions .xyz

Examples:

• 花店用品/蓝色妖姬着色剂 根部吸水浅宝蓝玫瑰*13230782866*QQ34040316座机03177896222*psuddf.zhijieranliao.xyz
• 直接染料/直接深棕GTL *13230782866*QQ34040316座机03177896222*zkdact.baohuabanranliao.xyz

We had an estimated 172 to 989 such queries in our samples, representing 0.03% to 0.20% of all zero-results queries. These were generally found in enwiki.

These appear to be product description in Chinese, along with additional information. They all end in .xyz. Note that 座机 means landline, indicating contact info.

Online searches for parts of these reveal a similar pattern on Chinese-language business/manufacturing sites.

Massive snippets

This particular category is fairly rare, but may incur significant computational cost, so it is worth noting. These are searches that are 500 characters in length or more (up to more than 5,000 characters). Many look like snippets from larger texts, such as books or articles, and are in several different languages.

We had between 183 and 261 such queries in our samples, representing 0.04% to 0.05% of all zero-results queries. These were generally found in en, fr, de, and ru.

Miscellany

Below are some very general impressions from the larger collections of zero-results (10K+ from a given wiki). My ability to analyze languages I don't know is limited, but here is what I noticed:

• dewiki has a few hundred OR'd together wildcard searches, some of which seem to be trying handle variations in declension.
• jawiki has lots of "..." film searches.
• ruwiki has a few non-cyrillic searches
• itwiki has lots of queries that are multi-word phrases with underscores instead of spaces
• eswiki and frwiki have a fair number of build up searches and searches in Arabic, and frwiki has a fair number of searches in Chinese
• zhwiki has lots of non-Chinese searches in various languages
• plwiki has a fair number of queries of the form *<musical thing>* AND (muzyk* OR Dyskografia) (with asterisks) where <musical thing> seems to be an artist, band, album, or something similar.

Summary Table

Query type	Sample 7/10	Sample 7/17	Sample 7/24	% of zero rate (min / max) of 500K samples		Most affected wikis
DOI	15393	96998	50181	3.08%	19.40%	en, nl, ja, zh, war, vi, uk, sv, pt, pl, no, ko, it, id, hu, fr, fi, fa, de, cs, ceb, ca, ar, es, ru
Unix timestamps	42650	26351	28089	5.27%	8.53%	en, it, ru, ja, fa, tr, nl, he, ar, id, cs, hi, vi, ro, hu, uk, etc.
"Article_title" AND "title of link taken from article"	10524	8174	16657	1.63%	3.33%	en
TV Episodes / Movies—"..." film	7989	7878	8794	1.58%	1.76%	en, nl, de, fr, ja
quot	7768	5888	6297	1.18%	1.55%	en
term+term+term country	6725	3437	5645	0.69%	1.35%	es, en
paint	3554	1917	1094	0.22%	0.71%	en
Highly repeated searches	892	1186	3019	0.18%	0.60%	?
term+term+term	2247	1382	2536	0.28%	0.51%	es, en
{searchTerms}	2314	1909	1997	0.38%	0.46%	ru
## <countrycode> tel fax	572	33	1293	0.01%	0.26%	de
Chinese product descriptions .xyz	172	221	989	0.03%	0.20%	en
search_suggest_query	449	509	440	0.09%	0.10%	en, de, fr
Massive snippets	258	183	261	0.04%	0.05%	en, fr, de, ru
\{@}	205	150	148	0.03%	0.04%	en, de, ru

One Month Followup

I've conducted a followup study to track changes to these patterns over time. Since my investigation is more focused this time (on these queries) I was able to process a lot more data.

As before, I've limited my investigation to full text queries against the 52 wikis with 100K+ articles (see above).

My new samples are taken from the entire day (i.e., one log file, labeled from ~8am to ~8am) for each day queried. For zero-result queries, I sampled 1-in-15, to get a 300K-400K query sample. For all queries, I sampled 1-in-45 to get a 600K to 800K query sample.

I focused on the largest patterns above, and included some of the easier to automatically detect patterns as well.

In particular: DOI, Unix Timestamp, "Title_1" AND "Title 2", "..." film, quot, "..." paint, tel fax, .xyz, {searchTerms}, \{@}, search_suggest_query, term+term+term Country, and long lines (150+ characters).

I counted the frequency and ratio of these kinds of queries for each day of one recent week (8/19-8/25) and one week around the original investigation date (7/22-7/28); both are Wednesday to Tuesday (since I was investigating on a Wednesday, 8/26).

Last Month (July 2015)

Below are the results for a week in July. Keep in mind that Zero Queries are sampled 1-in-15 and All Queries are sampled 1-in-45. The percentage to the right of the sample size for the Zero Queries is the zero results rate (zero_queries*15/all_queries*45).

	Wednesday 07-22				Thursday 07-23				Friday 07-24
	Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries
samples / zero%	304845	14.86%	683966		310079	15.92%	649420		354738	17.63%	670675
DOI	11753	3.86%	3868	0.57%	15410	4.97%	5092	0.78%	44656	12.59%	15022	2.24%
Unix Timestamp	23611	7.75%	8033	1.17%	23220	7.49%	7526	1.16%	23119	6.52%	7792	1.16%
"Title_1" AND "Title 2"	16226	5.32%	32664	4.78%	11029	3.56%	21975	3.38%	14793	4.17%	25581	3.81%
"..." film	5490	1.80%	2219	0.32%	6454	2.08%	2471	0.38%	5199	1.47%	1983	0.30%
quot	3886	1.27%	1896	0.28%	3868	1.25%	2005	0.31%	3809	1.07%	1887	0.28%
paint	563	0.18%	261	0.04%	684	0.22%	292	0.04%	576	0.16%	281	0.04%
tel fax	270	0.09%	85	0.01%	130	0.04%	45	0.01%	109	0.03%	51	0.01%
.xyz	80	0.03%	28	0.00%	363	0.12%	101	0.02%	406	0.11%	137	0.02%
{searchTerms}	1026	0.34%	757	0.11%	1026	0.33%	750	0.12%	1045	0.29%	759	0.11%
\{@}	74	0.02%	20	0.00%	74	0.02%	25	0.00%	67	0.02%	26	0.00%
search_suggest_query	365	0.12%	127	0.02%	394	0.13%	113	0.02%	374	0.11%	131	0.02%
term+term Country	2063	0.68%	849	0.12%	2674	0.86%	1297	0.20%	2354	0.66%	1249	0.19%
long lines (150+)	1347	0.44%	493	0.07%	1572	0.51%	566	0.09%	1197	0.34%	416	0.06%

	Saturday 07-25				Sunday 07-26				Monday 07-27				Tuesday 07-28
	Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries
samples / zero%	333763	16.87%	659440		379430	17.70%	714432		404075	16.99%	792621		385144	18.18%	706337
DOI	31388	9.40%	10309	1.56%	65501	17.26%	21910	3.07%	70647	17.48%	23619	2.98%	70225	18.23%	23551	3.33%
Unix Timestamp	23019	6.90%	7691	1.17%	25149	6.63%	8336	1.17%	27026	6.69%	8980	1.13%	23915	6.21%	8077	1.14%
"Title_1" AND "Title 2"	16639	4.99%	24661	3.74%	9516	2.51%	16438	2.30%	15114	3.74%	24345	3.07%	16195	4.20%	24692	3.50%
"..." film	5885	1.76%	2237	0.34%	7177	1.89%	2820	0.39%	6554	1.62%	2623	0.33%	5636	1.46%	2258	0.32%
quot	3599	1.08%	1708	0.26%	3850	1.01%	1865	0.26%	3954	0.98%	1919	0.24%	3807	0.99%	1893	0.27%
paint	226	0.07%	136	0.02%	243	0.06%	110	0.02%	218	0.05%	151	0.02%	221	0.06%	96	0.01%
tel fax	181	0.05%	62	0.01%	204	0.05%	54	0.01%	82	0.02%	25	0.00%	264	0.07%	76	0.01%
.xyz	281	0.08%	103	0.02%	557	0.15%	179	0.03%	450	0.11%	140	0.02%	417	0.11%	137	0.02%
{searchTerms}	987	0.30%	690	0.10%	959	0.25%	653	0.09%	956	0.24%	680	0.09%	1039	0.27%	720	0.10%
\{@}	65	0.02%	22	0.00%	50	0.01%	11	0.00%	53	0.01%	20	0.00%	70	0.02%	25	0.00%
search_suggest_query	381	0.11%	127	0.02%	474	0.12%	171	0.02%	528	0.13%	176	0.02%	457	0.12%	134	0.02%
term+term Country	2223	0.67%	1143	0.17%	2220	0.59%	1071	0.15%	1940	0.48%	1030	0.13%	1971	0.51%	1031	0.15%
long lines (150+)	1375	0.41%	496	0.08%	1298	0.34%	459	0.06%	1264	0.31%	462	0.06%	1799	0.47%	589	0.08%

This Month (August 2015)

Below are the results for a week in August. Keep in mind that Zero Queries are sampled 1-in-15 and All Queries are sampled 1-in-45. The percentage to the right of the sample size for the Zero Queries is the zero results rate (zero_queries*15/all_queries*45).

	Wednesday 08-19				Thursday 08-20				Friday 08-21
	Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries
samples/ zero%	401789	19.29%	694139		393384	19.56%	670250		354555	18.12%	652231
DOI	30369	7.56%	9992	1.44%	45115	11.47%	15167	2.26%	16416	4.63%	5599	0.86%
Unix Timestamp	5045	1.26%	1710	0.25%	4831	1.23%	1656	0.25%	4618	1.30%	1474	0.23%
"Title_1" AND "Title 2"	4834	1.20%	6768	0.98%	0	0.00%	0	0.00%	0	0.00%	0	0.00%
"..." film	6065	1.51%	2466	0.36%	5682	1.44%	2275	0.34%	5851	1.65%	2385	0.37%
quot	3566	0.89%	1751	0.25%	3879	0.99%	2036	0.30%	3983	1.12%	2008	0.31%
paint	194	0.05%	90	0.01%	119	0.03%	55	0.01%	198	0.06%	78	0.01%
tel fax	49	0.01%	8	0.00%	233	0.06%	87	0.01%	120	0.03%	43	0.01%
.xyz	265	0.07%	85	0.01%	283	0.07%	106	0.02%	462	0.13%	166	0.03%
{searchTerms}	1047	0.26%	752	0.11%	1064	0.27%	746	0.11%	1060	0.30%	716	0.11%
\{@}	86	0.02%	31	0.00%	53	0.01%	28	0.00%	52	0.01%	26	0.00%
search_suggest_query	349	0.09%	126	0.02%	377	0.10%	127	0.02%	400	0.11%	125	0.02%
term+term Country	0	0.00%	1	0.00%	0	0.00%	0	0.00%	0	0.00%	0	0.00%
long lines (150+)	1420	0.35%	545	0.08%	1487	0.38%	0	0.00%	1657	0.47%	576	0.09%

	Saturday 08-22				Sunday 08-23				Monday 08-24				Tuesday 08-25
	Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries		Zero Queries		All Queries
samples/ zero%	341387	17.86%	637122		355099	17.47%	677356		391179	17.50%	744908		394403	18.89%	695853
DOI	17008	4.98%	5681	0.89%	18768	5.29%	6295	0.93%	26501	6.77%	8890	1.19%	39322	9.97%	13236	1.90%
Unix Timestamp	4412	1.29%	1529	0.24%	4542	1.28%	1579	0.23%	4566	1.17%	1560	0.21%	4206	1.07%	1378	0.20%
"Title_1" AND "Title 2"	1	0.00%	0	0.00%	0	0.00%	0	0.00%	0	0.00%	0	0.00%	0	0.00%	1	0.00%
"..." film	6578	1.93%	2690	0.42%	8623	2.43%	3387	0.50%	7442	1.90%	3160	0.42%	5174	1.31%	2138	0.31%
quot	3801	1.11%	1998	0.31%	3896	1.10%	1869	0.28%	4028	1.03%	1869	0.25%	4219	1.07%	2064	0.30%
paint	136	0.04%	78	0.01%	129	0.04%	50	0.01%	84	0.02%	30	0.00%	61	0.02%	28	0.00%
tel fax	41	0.01%	15	0.00%	0	0.00%	0	0.00%	252	0.06%	69	0.01%	490	0.12%	143	0.02%
.xyz	302	0.09%	92	0.01%	381	0.11%	123	0.02%	527	0.13%	187	0.03%	455	0.12%	160	0.02%
{searchTerms}	1002	0.29%	666	0.10%	980	0.28%	582	0.09%	1043	0.27%	622	0.08%	949	0.24%	580	0.08%
\{@}	57	0.02%	16	0.00%	58	0.02%	20	0.00%	41	0.01%	19	0.00%	68	0.02%	29	0.00%
search_suggest_query	388	0.11%	119	0.02%	423	0.12%	160	0.02%	464	0.12%	150	0.02%	354	0.09%	142	0.02%
term+term Country	0	0.00%	1	0.00%	1	0.00%	1	0.00%	923	0.24%	434	0.06%	2095	0.53%	1066	0.15%
long lines (150+)	1163	0.34%	400	0.06%	1359	0.38%	498	0.07%	1666	0.43%	601	0.08%	1222	0.31%	452	0.06%

Analysis

The overall zero results rate hasn't changed much from July to August. Friday to Tuesday track pretty closely in the graph below, and Wednesday and Thursday have higher zero results rates in August than July.

Overall Zero Results Rate

The decrease in some queries, such as the "Title_1" AND "Title 2" queries, have little overall effect, because the zero results rate is only slightly higher than the overall query rate (usually within 1% above), so removing both the zero results queries and the queries that do return results has little net effect.

"Title_1" AND "Title 2"

This IP was banned (apparently around 8/19), and queries dropped to zero, both overall and zero results queries.

"Title_1" AND "Title 2" query rates in Zero Results Queries

"Title_1" AND "Title 2" query rates in All Queries

Unix Timestamps

The fix to the Wikipedia Mobile App was deployed, and rates have dropped significantly, though not to 0%, because not everyone updates their apps right away.

Unix Timestamp query rates in Zero Results Queries

DOI

DOI rates have always fluctuated considerably across various samples (3% to 19%, see above), so it's hard to be sure, but in the samples for these weeks, the overall rate seems to be down.

DOI query rates in Zero Results Queries

quot and "..." film

Not much has happened with the quot and "..." film queries. We've contacted the respective sources, but it isn't clear anything will be done soon. The media player is behaving as designed, so these queries probably won't stop. The NLA may eventually fix their quot queries.

quot query rates in Zero Results Queries

"..." film query rates in Zero Results Queries

Comparison to Dashboards

Despite the large dip in "Title_1" AND "Title 2" and Unix Timestamp queries, we haven't seen any corresponding movement in the Discovery KPI dashboard.

Part of the reason for this is that this analysis and the dashboards are measuring different things: the dashboards look at all zero results (including prefix searches), while I'm looking only at full text queries to large wikis (to limit the scope of my initial investigation). I'm going to try to confirm this (and see if anything else pops out!) by quickly looking at everything I didn't look at here.

Everything Else (full text)

I took a 1-in-45 sample of full text queries from 8/25 for wikis not in the 52 100K+ wikis, and found some interesting stuff.

There were 269,924 samples (compared to 695,853 in the "big" wikis)—so, roughly 28% of traffic.

There were 152,726 zero result queries, for a 56.6% zero results rate (compared to ~20% in the "big" wikis).

The usual suspects do not show up as much, other than DOI:

samples/ zero%	152726	56.6%
DOI	3967	2.60%
Unix Timestamp	961	0.63%
"Title_1" AND "Title 2"	0	0.00%
"..." film	152	0.10%
quot	73	0.05%
paint	0	0.00%
tel fax	166	0.11%
.xyz	0	0.00%
{searchTerms}	30	0.02%
\{@}	3	0.00%
search_suggest_query	7	0.00%
term+term Country	0	0.00%
long lines (150+)	117	0.08%

Other interesting facts:

There were only about 3K non-latin searches (lots of Greek, Cyrillic, Hebrew, Arabic, Devanagari, Thai, Japanese, Chinese, emoji, and more).

A quick skim of the zero results show lots of names.. but not a lot else stuck out.

The top 12 wikis are responsible for 145,325 of the zero results queries (95.2%) (with Wikidata on top and most of the rest coming from itwiki...):

•   20699 wikidatawiki_content (13.5%)
•   19798 itwiktionary_content (13.0%)
•   18753 itwikiversity_content
•   18080 itwikinews_content
•   16587 itwikivoyage_content
•   15504 itwikibooks_content
•   13460 itwikisource_content
•   12148 itwikiquote_content
•    3773 eswiki
•    3764 nlwiktionary_content
•    1419 sewiki_content
•    1340 thwiki_content

Change in Zero Results Rate by Wiki (July to August)

The table below shows the change in zero results rate from a week's sample (1-in-750 ) each from July (7/22-7/28) and August (8/19-8/25). The wikis shown are those with at least 1000 queries in either week sampled. In total, they account for just under 98% of queries each week.

• zero rate is the zero results rate.
• % of all is the percentage of all queries that come from this wiki.
• % of zero is the percentage of zero results that come from this wiki.
• zero/all is the ratio of the previous two percentages. Values > 1 indicate more than average zero results, < 1indicates fewer than average zero results.
• query vol Δ is the change in query volume from July to August.
• zero rate Δ is the change in zero results rate from July to August.

N.B.: This analysis does not properly account for itwiki's interwiki search (which searches several Italian wikis at once) because this is a 1-in-750 sample, so will likely only see one element of the itwiki interwiki search, and itwik* is over represented. The Discovery Dashboards use unsampled data and so can account for this.

	July					August
wiki	zero rate	% of all	% of zero	zero/all	___	zero rate	% of all	% of zero	zero/all	___	query vol Δ	zero rate Δ
TOTAL	25.18%	100.00%	100.00%	1.00		28.75%	100.00%	100.00%	1.00		6.66%	3.56%
enwiki	14.18%	47.13%	26.54%	0.56		13.79%	38.03%	18.25%	0.48		-13.94%	-0.39%
dewiki	14.78%	8.19%	4.81%	0.59		17.51%	8.28%	5.04%	0.61		7.75%	2.73%
eswiki	21.87%	2.83%	2.46%	0.87		18.83%	3.97%	2.60%	0.65		49.57%	-3.04%
frwiki	11.58%	3.30%	1.52%	0.46		12.67%	3.13%	1.38%	0.44		1.06%	1.10%
itwiktionary	66.60%	2.27%	6.01%	2.64		67.87%	2.94%	6.95%	2.36		38.25%	1.27%
itwiki	18.52%	2.80%	2.06%	0.74		19.87%	2.93%	2.02%	0.69		11.38%	1.35%
itwikinews	61.16%	2.35%	5.71%	2.43		62.69%	2.90%	6.31%	2.18		31.48%	1.53%
itwikisource	47.03%	2.33%	4.35%	1.87		46.19%	2.90%	4.65%	1.61		32.56%	-0.83%
itwikiquote	43.33%	2.29%	3.94%	1.72		42.85%	2.89%	4.31%	1.49		34.83%	-0.48%
itwikiversity	64.41%	2.31%	5.90%	2.56		65.77%	2.89%	6.61%	2.29		33.63%	1.36%
itwikibooks	53.88%	2.29%	4.90%	2.14		53.43%	2.88%	5.36%	1.86		34.22%	-0.44%
itwikivoyage	60.09%	2.35%	5.60%	2.39		58.49%	2.86%	5.82%	2.03		29.95%	-1.60%
wikidatawiki	88.28%	1.23%	4.31%	3.51		72.08%	2.83%	7.11%	2.51		145.62%	-16.20%
ruwiki	16.14%	2.79%	1.79%	0.64		17.19%	2.52%	1.51%	0.60		-3.60%	1.06%
jawiki	23.74%	2.30%	2.17%	0.94		26.11%	2.25%	2.05%	0.91		4.40%	2.37%
ptwiki	23.41%	2.15%	2.00%	0.93		22.33%	2.13%	1.65%	0.78		5.28%	-1.08%
eswiki	25.25%	1.03%	1.03%	1.00		25.77%	1.61%	1.44%	0.90		66.91%	0.52%
nlwiki	23.76%	0.82%	0.78%	0.94		34.47%	1.33%	1.60%	1.20		72.53%	10.71%
zhwiki	27.77%	1.05%	1.16%	1.10		41.58%	1.06%	1.53%	1.45		7.19%	13.81%
plwiki	33.45%	0.48%	0.63%	1.33		44.24%	0.88%	1.35%	1.54		96.25%	10.79%
trwiki	19.99%	0.47%	0.37%	0.79		34.40%	0.79%	0.95%	1.20		78.98%	14.41%
viwiki	53.43%	0.25%	0.53%	2.12		42.57%	0.77%	1.14%	1.48		231.56%	-10.86%
arwiki	31.50%	0.71%	0.88%	1.25		27.86%	0.61%	0.59%	0.97		-7.98%	-3.65%
idwiki	36.12%	0.41%	0.58%	1.43		31.42%	0.65%	0.71%	1.09		68.86%	-4.70%
svwiki	23.31%	0.65%	0.60%	0.93		23.39%	0.50%	0.40%	0.81		-18.21%	0.08%
fawiki	32.52%	0.51%	0.66%	1.29		28.80%	0.42%	0.42%	1.00		-11.78%	-3.72%
nlwiktionary	99.59%	0.46%	1.82%	3.95		99.26%	0.41%	1.42%	3.45		-4.59%	-0.33%
commonswiki	8.94%	0.32%	0.11%	0.35		9.17%	0.38%	0.12%	0.32		25.04%	0.23%
kowiki	37.46%	0.36%	0.53%	1.49		37.18%	0.35%	0.45%	1.29		5.24%	-0.28%
cswiki	42.39%	0.17%	0.29%	1.68		37.32%	0.25%	0.32%	1.30		50.62%	-5.07%
fiwiki	38.62%	0.23%	0.35%	1.53		36.15%	0.24%	0.31%	1.26		14.42%	-2.47%
hewiki	25.61%	0.25%	0.26%	1.02		27.37%	0.18%	0.17%	0.95		-23.59%	1.76%
thwiki	42.47%	0.08%	0.14%	1.69		59.33%	0.20%	0.41%	2.06		159.87%	16.86%
commonswiki_file	44.95%	0.20%	0.35%	1.78		40.25%	0.18%	0.25%	1.40		-2.07%	-4.70%
dawiki	22.48%	0.07%	0.06%	0.89		35.95%	0.17%	0.21%	1.25		156.59%	13.47%
huwiki	51.06%	0.15%	0.31%	2.03		44.28%	0.12%	0.18%	1.54		-16.90%	-6.78%
nowiki	58.41%	0.12%	0.28%	2.32		47.78%	0.14%	0.23%	1.66		19.47%	-10.63%
hiwiki	33.09%	0.11%	0.15%	1.31		37.28%	0.13%	0.17%	1.30		23.50%	4.19%
ukwiki	61.76%	0.14%	0.34%	2.45		58.64%	0.10%	0.20%	2.04		-25.10%	-3.13%

Notes

• eswiki is listed twice; the first instance is searches against "eswiki_content", the second against "eswiki". (I'm not sure what that means.)
• enwiki had a significant drop in traffic from July to August
• wikidata, eswiki, and many of the itwikis had significant increases in traffic from July to August
• wikidata had a significant decrease in zero rate, though it is still incredibly high.
• nlwiktionary has a phenomenal zero results rate, over 99%. A quick check shows that it is getting spammed with URLs for German (.de) websites.

Full manual review of a 1K enwiki sample

Introduction

Looking for large recurring patterns of searches will not reveal the frequency of idiosyncratic erorrs (like typos, gibberish, and queries in foreign languages) that can't be easily be recognized automatically. So I undertook a manual review of 1047 randomly sampled queries from from one day's worth of logs of enwiki full-text searches, starting on 2015-07-29. This was a random sample (or as random as pseudorandom can be) rather than an every-N sample.

Caveats

All categories are at least somewhat subjective, and depend in part on my ability to recognize (or uncover) user intent. Many items could have been put in multiple categories, but I chose just one each time (some comments about this are included below). I know others would disagree on some categories, and I know that there are errors in categorization (i.e., I'd disagree with myself), but the overall trends are still illustrative.

Requestors

Overall, 67.2% were requested via API, 32.8% via web.

Typos

I broke typos up into two categories: apparent mistakes, and incomplete words or phrases. Note that the vast majority in the incomplete category come via API, hinting that some app may be sending incomplete queries.

153      14.6%  TYPO,    54.2% web,  45.8% api
93        8.9%  INCOMP,  97.8% api,   2.2% web
Total: 23.5%

Typo autosuggestions

I took a random sample of 40 of the queries in the TYPO category and searched for them in enwiki via the web. The results were oddly evenly distributed:

• 10/40 = 25.0% zero results
• 10/40 = 25.0% some results
• 20/40 = 50.0% correct results

Half had clearly correct suggestions, a quarter had no results, and a quarter had some non-zero results that were either wrong, or not clearly correct.

Typo reverse index

By manual inspection, 20 of the 153 TYPOs (13.1%) had an error in the first two characters of one of the search terms, and thus might benefit from a reverse index.

Previously seen categories

These are the previously discussed large categories of zero-return queries. The distributions are different from previous samples, sometimes drastically different. This could be attributed to the current small sample size, the time skew in the earlier sample, day-of-the-week effects, and random vagaries of millions of people searching—though that last can account for almost any variance!

125      11.9%  AND,   100.0% api	"Article_title" AND "title of link taken from article"
91        8.7%  UNIX,  100.0% api	Unix Timestamps
29        2.8%  FILM,  100.0% web	TV Episodes / Movies—"..." film
28        2.7%  QUOT,  100.0% api	quot
20        1.9%  DOI,   100.0% api	DOI
3         0.3%  TERM+, 100.0% api	term+term+term / term+term+term country
Total: 28.3%

Foreign languages

Though enwiki has many pages with titles in other languages, these searches didn't get any results. I didn't dig too deeply into most of them. However, numerous entries filed under MOVIES, MUSIC, and YOUTUBE are in Spanish, Portuguese, Turkish, or other languages.

13        1.2%  CHINESE,    61.5% api,  38.5% web
12        1.1%  ARABIC,     58.3% web,  41.7% api
7         0.7%  CYRILLIC,   57.1% web,  42.9% api
7         0.7%  TAGALOG,    85.7% web,  14.3% api
6         0.6%  SPANISH,    83.3% api,  16.7% web
5         0.5%  MALAY,      80.0% web,  20.0% api
3         0.3%  GERMAN,     66.7% api,  33.3% web
3         0.3%  DEVTRANS,  100.0% api
2         0.2%  NORWEGIAN, 100.0% api
2         0.2%  SWAHILI,   100.0% web
1 each (0.1%, 100% api) for GREEK, THAI, TAMIL
1 each (0.1%, 100% web) for PORTUGUESE, DUTCH, FRENCH, LATIN, SWEDISH, ITALIAN, 
                            CROATIAN, HINDI, ESTONIAN, HMONG, KANNADA, FINNISH
Total: 7.2%

One or Two Word Queries

How many of these foreign language queries are one or two words, and thus, for example, more likely to get a match in Wikitionary or Wikivoyage?

• Arabic: 0/12, 3/12
• Chinese: hard to judge, but apparently 0/13; not sure on "two words"
• Cyrillic: 4/7, 1/7
• Dev-Trans: 3/3
• German: 1/3, 1/3
• Malay: 0/5, 1/5
• Norwegian: 0/2, 2/2
• Spanish: 1/6, 2/6
• Swahili: 0/2, 0/2
• Tagalog: 0/7
• Croatian, Estonian, Swedish, Tamil: 1/1
• Dutch, Greek, Hindi, Hmong, Italian, Portuguese, Thai: 0/1, 0/1
• Finnish, French, Kannada, Latin: 0/1, 1/1

Totals:

• One word: 13/75
• Two words: 14/75

Mystery queries

These are ones that I just couldn't figure out. They weren't clearly junk. They could be typos, but often they are are too ambiguous to be sure.

66        6.3%  ??,  71.2% api,  28.8%

Not encyclopedic

These categories are potentially problematic, since they may depend in part on what you think should or should not be in Wikipedia.

Total: 28.0%

Wrong website

• PROD: These are queries that appear to be about general or specific products, including video games, clothing brands, drugs, decorations, laptop replacement parts, etc.
• QUESTION: These are queries that seem to be asking for non-encyclopedic information, including advice on romance, study habits, and home furnishing, job searches, making travel arrangements, celebrity facts, scholarly research, etc.
• URL: These look like they are or tried to be URLs.
• NEWS: Questions about current events.
• TWITTER: Actual tweets or parts of tweets.

57        5.4%  PROD,      66.7% api,  33.3% web
45        4.3%  QUESTION,  57.8% web,  42.2% api
32        3.1%  URL,       62.5% web,  37.5% api
3         0.3%  NEWS,      66.7% web,  33.3% api
2         0.2%  TWITTER,  100.0% api
Total: 13.3%

Content

These appear to be searches for particular content, including particular songs, albums, music by a particular artist, movies, TV episodes, books, or scholarly articles. The YOUTUBE category are queries that exactly match the titles of individual YouTube videos. ISBNs are specific, plain ISBN numbers.

42        4.0%  MUSIC,    78.6% api,  21.4% web
25        2.4%  YOUTUBE,  92.0% api,   8.0% web
10        1.0%  MOVIE,    50.0% api,  50.0% web
7         0.7%  ARTICLE,  85.7% web,  14.3% api
7         0.7%  ISBN,     71.4% web,  28.6% api
Total: 8.8%

People and places

These are queries for particular people or places that are not in Wikipedia, including named individuals or parts of names (PERSON), online usernames (USER), addresses of business (ADDRESS—note that all are in Las Vegas) or email addresses.

The LINKED category are searches in this form:

• "SURNAME" "FIRST MIDDLE" "COMPANY" LINKEDIN

Both LINKEDIN and VIADEO (professional social networking sites) were used.

26        2.5%  PERSON,   53.8% web,  46.2% api
16        1.5%  USER,     56.2% api,  43.8% web
10        1.0%  ADDRESS, 100.0% api
4         0.4%  LINKED,  100.0% api
2         0.2%  EMAIL,   100.0% web
Total: 5.6%

Misc

Stuff I could at least partly identify, but couldn't categorize elsewhere. One was just a number, one a slightly mangled Wikimedia Commons file name, the other a Wikimedia Commons category name.

2         0.2%  COMMONS,  50.0% api,  50.0% web
1         0.1%  NUMBER,  100.0% web
Total: 0.3%

Junk

• JUNK includes snippets of larger texts, multiply repeated letters, keyboard banging, and the like.
• OCR are "words" that seem to appear primarily as OCR errors in Google Books.
• ERRORs include "search_suggest_query" and the like.
• EMOJI are strings of emoji characters.
• SPAM includes an actual advertisement and what looks like a hacking probe attempt.
• NODE is a pattern that's come up before: Iamlookingfor[...]node[...]

34        3.2%  JUNK,   85.3% web,  14.7% api
9         0.9%  OCR,    55.6% api,  44.4% web
3         0.3%  ERROR,  66.7% api,  33.3% web
3         0.3%  EMOJI, 100.0% web
2         0.2%  SPAM,   50.0% api,  50.0% web
1         0.1%  NODE,  100.0% web
Total: 5.0%

Misses

Actual contentful queries that there are no entries for.

• DICT: mostly obscure words, but they are in Wiktionary.
• SPECIES: Latin species names.
• MISS: other misses that seem to describe reasonable things that could be in Wikipedia, but either aren't or weren't found.

12        1.1%  MISS,     50.0% api,  50.0% web
4         0.4%  SPECIES,  75.0% web,  25.0% api
5         0.5%  DICT,     80.0% api,  20.0% web
Total: 2.0%