User:TJones (WMF)/Notes

This is an index for the reports I've written up on various search- & discovery-related topics.

So Many Search Options
So Many Search Options (December 2016): an initial proposal to encourage thinking about how to deal with all the different additional ways of searching when a query doesn't give great results ("Did you mean" suggestions, language detection, quote stripping, wrong keyboard detection, etc).

January 2017: Lots of updates and refinements, and the first draft of a proposal to update the API.

TextCat Improvements
November 2016—These are all on one wiki page if you want to browse them all, or jump to a specific section. December 2016 January 2017
 * Optimization Framework: I slapped together a grid-search optimizer around my existing tools.
 * Multiple Language Model Directories: Initial analysis of effects of allowing wiki-text models in addition to query-based models.
 * Maximum Returned Languages and Results Ratio: Initial analysis of the effects of optimizing what counts as "ambiguous"; also, turns out that model size is interesting and important.
 * Minimum Input Length: Initial analysis of the effects of introducing a minimum input length.
 * Max Proportion of Max Score: scores that are too close the worst possible score are likely junk!
 * Optimization Framework updates (Dec/Jan): now with coordinate descent!
 * Bucketing and Bonuses (Dec/Jan): give the most likely languages—esp. the "host" language—a boost so that ambiguity or near ambiguity comes out in their favor. Also, re-evaluate whether we've made enough progress to warrant putting back some languages we had to exclude (spoiler: we have!)
 * Unknown n-gram Penalty: Maybe an extra penalty for unknown n-grams will reduce ambiguity; or maybe the penalty is too high and we're throwing out the baby with the bathwater; or maybe it's just right.
 * Final Summary & Recommendations: stick a fork in it; it's done! A review of the overall findings, and the general improvement in F0.5 accuracy across the nine corpora we currently have.

TextCat Released into Production!
July 2016—There's a blog post on the Wikimedia blog that Deb and I worked on, announcing TextCat/language ID being in production for five wikis, and a PDF of I wrote on Commons. And while I'm here, I'll suggest the online demo if you want to play around with language identification directly.

TextCat and Confidence
TextCat and Confidence (July 2016) Quick summary of concerns and ideas for assigning a confidence score to TextCat's language identification.

Typing on the Wrong Keyboard / Russian and English
Typing on the Wrong Keyboard / Russian and English (June 2016) A quick attempt to identify and convert queries typed on the wrong keyboard on the English and Russian Wikipedias.

Favoring Recall in Language Identification
Favoring Recall in Language Identification (May 2016) Analysis of recall-favoring options for language detection (rather than precision-favoring), using the same data from frwiki, eswiki, itwiki, and dewiki as below.

Balanced Language Identification Evaluation Set for Queries
Balanced Language Identification Evaluation Set for Queries (February 2016) Creation of a 21-language balanced query corpus, and the evaluation of TextCat against that corpus.

TextCat with Additional Non-Word Characters
TextCat with Additional Non-Word Characters (January 2016) A follow up on an idea from Stas about modifying the non-word characters in TextCat. Ignoring parens helps a wee bit.

ElasticSearch Plugin—Limiting Languages & Retraining
ES Plugin, Limiting Language Options and Retraining on Query Data (December 2015) David retrained the ES Plugin models using the data from the TextCat evaluations, and figured out how to limit the plugin to the "useful" languages. The results are much improved and on-par with TextCat.

Language Detection Evaluation—TextCat
Language Detection with TextCat (December 2015)—An evaluation of TextCat (an n-gram–based language identifier) on the enwiki zero-results queries. Includes updates to TextCat, re-training on query data, and limiting language identification to "useful" languages. Offers an improvement over the ES Plugin.

Language Detection Evaluation—Update: Thresholds by Language
Language Detection Evaluation—Update: Thresholds by Language (October 2015)—Evaluated adding a language specific threshold (i.e., "it's never Romanian" on enwiki!) to the ElasticSearch language detection plugin. Results are overfitted because of small available data set, but are indicative of significant improvement to precision in language detection.

Language Detection Evaluation
Language Detection Evaluation (September 2015)—A test of language detection against a representative sample of hand-coded zero-results queries from enwiki.
 * ElasticSearch language detection plugin—A language detection plugin available for ElasticSearch;
 * also evaluated with initial and final spaces added (which gives better results, probably because of better recognition of letters at the edges of words)
 * Always "English" detector—Baseline against the current de facto default; also demonstrates that F-score is not necessarily the only relevant measure for search purposes.

On Generic ICU Folding
On Generic ICU Folding ( December 2016) Copy of quick discussion on Phab about the goals of generic ICU folding, and how to apply it to specific language wikis, hopefully with input from the wiki/language communities.

Upgrading ASCII Folding to ICU Folding for French and English
Upgrading ASCII Folding to ICU Folding for French and English (September 2016) A quick analysis of the effects of enabling ICU folding instead of simple ASCII folding for French and English.

Removing Stress Accents and Folding Ё to Е for Russian Wikis
Removing Stress Accents and Folding Ё to Е for Russian Wikis (September 2016) A quick-ish test on the effects of adding stress-accent-stripping and ё-folding to Russian wikis.

On Merging Apostrophes and Other Unicode Characters
On Merging Apostrophes and Other Unicode Characters (August 2016) Copied from a quick analysis in a Phab ticket on merging Unicode characters so I can easily find it later.

Adding Ascii-Folding to French Wikipedia
Adding Ascii-Folding to French Wikipedia (August 2016) A not-so-quick test on the effects of adding ascii-folding to French Wikipedia. Many unexpected twists and surprises!

Re-Ordering Stemming and Ascii-Folding on English Wikipedia
Re-Ordering Stemming and Ascii-Folding on English Wikipedia (August 2016) A quick test of the effects of moving ascii-folding before stemming on English Wikipedia.

TextCat Optimization for plwiki, arwiki, zhwiki, and nlwiki
TextCat Optimization for plwiki, arwiki, zhwiki, and nlwiki (September 2016) Analysis of low-performing queries (< 3 results) to optimize languages to be used for language detection.

TextCat Optimization for ptwiki, ruwiki, and jawiki
TextCat Optimization for ptwiki, ruwiki, jawiki (July 2016) Analysis of low-performing queries (< 3 results) to optimize languages to be used for language detection.

TextCat Re-optimization for enwiki
TextCat Re-optimization for enwiki (June 2016) Analysis of low-performing queries (< 3 results) to optimize languages to be used for language detection; plus comparison to similar previous ZRR-based enwiki corpus from 2015.

TextCat Optimization for frwiki, eswiki, itwiki, and dewiki
TextCat Optimization for frwiki, eswiki, itwiki, and dewiki (April 2016) Analysis of low-performing queries (< 3 results) to optimize languages to be used for language detection.

Spaceless Writing Systems and Wiki-Projects
Spaceless Writing Systems and Wiki-Projects (November 2016) A quick review of languages/projects that don't use spaces between most words in their writing systems.

Fallback Langauges
Fallback Langauges (October 2016) A list of languages that are potentially used as fallbacks for other languages in language analysis.

Top Unsuccessful Search Queries
Top Unsuccessful Search Queries (July 2016) Analysis of the top 100 most frequent zero-results queries for enwiki for the month of May, 2016, to help determine whether mining such queries is worthwhile.

Dropping Final Question Marks in the Top 10 Wikipedias
Dropping Final Question Marks in the Top 10 Wikipedias (June 2016) More detailed look at the effects on search results (especially Zero Results Rate and Poorly Performing Queries) of dropping final question marks from queries on the top 10 Wikipedias.

Quotes and Questions
Quotes and Questions (May 2016) Quick write up of effects of removing quotation marks and question marks from poorly performing queries.

How Wrong Would Using Out of Date Page View Data Be?
How Wrong Would Using Out of Date Page View Data Be? (January 2016) We want to integrate page view information into the scoring algorithms we use for both the completion suggester and our regular search results. Our initial idea is we only update this page view information when doing normal document updates after a page edit (for technical reasons, page view data is available/provided when a page is edited). We need to analyze if this page view data will be "good enough" or if we need to do something more.

Relevance Lab!
Relevance Lab (October 2015)—High level description and design of a Relevance Lab for Discovery, which would allow us (and others!) to experiment with proposed modifications to our search process and gauge their effectiveness and impact before deploying them.

Why People Use Search Engines
Why People Use Search Engines (September 2015)—An overview of how well English Wikipedia Search performs on a sample of ~4K queries that came from Google, with analysis of categories of unsuccessful queries and lots of ideas (not all necessarily practical) for Wikimedia search improvements.

Cross Language Wiki Searching
Cross Language Wiki Searching (September 2015)—An attempt to estimate the impact on enwiki's zero-results rate given "perfect" (or at least human-level) language identification.

Phrase Slop Pre-Test
Phrase Slop Pre-Test (August 2015)—An in vitro test of the ElasticSearch phrase slop parameter against ptwiki and dewiki before the in vivo A/B test. The final report, prepared by Mikhail, is here.

Survey of Zero-Results Queries
Survey of Zero-Results Queries (July 2015)—A survey of the readily identifiable patterns in full-text zero-results queries. Lots of potential bots and bugs identified.
 * One Month Followup (August 2015)—Overview of day-by-day changes in full-text traffic for known bots and bugs one month later, and monthly changes in zero-results rate for top wikis by volume.
 * Full manual review of a 1K enwiki sample (August 2015)—Hand coding and categorization of a 1K sample of full-text zero-results queries.