User:TJones (WMF)/Notes/TextCat Optimization for plwiki arwiki zhwiki and nlwiki

From MediaWiki.org
Jump to navigation Jump to search

September 2016 — See TJones_(WMF)/Notes for other projects. (Phabricator ticket: T142140)

TextCat Optimization for plwiki, arwiki, zhwiki, and nlwiki[edit]

Summary of Results[edit]

Using the default 3K models, the best options for each wiki are presented below:

nlwiki

  • languages: Dutch, English, Chinese, Arabic, Korean, Greek, Hebrew, Japanese, Russian
  • lang codes: nl, en, zh, ar, ko, el, he, ja, ru
  • relevant poor-performing queries: 36%
  • f0.5: 82.3%

Background[edit]

See the earlier report on frwiki, eswiki, itwiki, and dewiki for information on how the corpora were created.

Dutch Results[edit]

About 16.8% of the original 10K corpus was removed in the initial filtering. A 1200-query random sample was taken, and 57.1% of those queries were discarded, leaving a 515-query corpus. Thus only about 35.7% of low-performing queries are in an identifiable language.

Other languages searched on nlwiki[edit]

Based on the sample of 515 poor-performing queries on nlwiki that are in some language, about 63% are in Dutch, 25% in English, 2-3% in French and German, less than 2% each are in a handful of other languages.

Below are the results for nlwiki, with raw counts, percentage, and 95% margin of error.

count lg % +/-
326 nl 63.30% 4.16%
128 en 24.85% 3.73%
16 fr 3.11% 1.50%
11 de 2.14% 1.25%
6 es 1.17% 0.93%
5 it 0.97% 0.85%
4 la 0.78% 0.76%
3 zh 0.58% 0.66%
2 tr 0.39% 0.54%
2 pl 0.39% 0.54%
2 fi 0.39% 0.54%
2 ar 0.39% 0.54%
1 vi 0.19% 0.38%
1 pt 0.19% 0.38%
1 my 0.19% 0.38%
1 ko 0.19% 0.38%
1 hr 0.19% 0.38%
1 da 0.19% 0.38%
1 cs 0.19% 0.38%
1 af 0.19% 0.38%

In order, those are Dutch, English, French, German, Spanish, Italian, Latin, Chinese, Turkish, Polish, Finnish, Arabic, Vietnamese, Portuguese, Burmese, Korean, Croatian, Danish, Czech, Afrikaans.

We don’t have query-trained language models for all of the languages represented here, such as Afrikaans, Danish, Finnish, Croatian, Latin, and Burmese (af, da, fi, hr, la, my). Since these each represent very small slices of our corpus (< 5 queries each), we aren’t going to worry about them, and accept that they will not be detected correctly.

Looking at the larger corpus of 8,323 remaining queries after the initial filtering, focusing on queries in other writing systems, there are also a small number of Russian, Hebrew, Greek, and Japanese queries, and Amharic (for which we do not have models).

Analysis and Optimization[edit]

Using all of the language models available, I ran tests on various model sizes, in increments of 500 up to 5,000 and increments of 1,000 up to 10,000. Results for the 3K models, and some of the models that did better are here:

               3000    3500    4000    4500    5000    6000    9000    
      TOTAL    74.5%   74.5%   75.5%   76.1%   77.0%   78.3%   78.7%   
      Dutch    88.2%   88.6%   89.2%   88.8%   89.3%   89.9%   89.9%   
    English    71.3%   69.8%   71.0%   73.6%   75.1%   75.9%   78.2%   
     French    57.1%   58.3%   59.6%   66.7%   64.0%   71.1%   71.1%   
     German    32.1%   32.7%   34.0%   32.7%   34.6%   39.2%   37.7%   
    Spanish    46.2%   44.4%   44.4%   46.2%   46.2%   46.2%   50.0%   
    Italian    20.7%   21.4%   14.8%   14.8%   15.4%   13.8%   13.3%   
      Latin     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
    Chinese    80.0%   80.0%   80.0%   80.0%   80.0%  100.0%  100.0%  
     Arabic   100.0%  100.0%  100.0%  100.0%  100.0%  100.0%  100.0%  
    Finnish     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
     Polish    50.0%   50.0%   66.7%   66.7%   66.7%   66.7%   66.7%   
    Turkish    57.1%   50.0%   50.0%   40.0%   44.4%   44.4%   44.4%   
  Afrikaans     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
    Burmese     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
   Croatian     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
      Czech    22.2%   25.0%   28.6%   40.0%   50.0%   50.0%   50.0%   
     Danish     0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    
     Korean   100.0%  100.0%  100.0%  100.0%  100.0%  100.0%  100.0%  
 Portuguese    25.0%   20.0%   22.2%   25.0%   22.2%   28.6%   28.6%   
 Vietnamese    40.0%   40.0%   40.0%   40.0%   50.0%   66.7%   66.7%   

Performance details for the 3K model are here (details for larger models are similar in terms of which language models perform the most poorly):

               f0.5    f1      f2      recall  prec    total   hits    misses
      TOTAL     74.6%   74.5%   74.4%   74.4%   74.7%  515     383     130
      Dutch     93.7%   88.2%   83.3%   80.4%   97.8%  326     262     6
    English     80.2%   71.3%   64.2%   60.2%   87.5%  128     77      11
     French     47.3%   57.1%   72.2%   87.5%   42.4%  16      14      19
     German     23.6%   32.1%   50.6%   81.8%   20.0%  11      9       36
    Spanish     34.9%   46.2%   68.2%  100.0%   30.0%  6       6       14
    Italian     14.9%   20.7%   34.1%   60.0%   12.5%  5       3       21
      Latin      0.0%    0.0%    0.0%    0.0%    0.0%  4       0       0
    Chinese     90.9%   80.0%   71.4%   66.7%  100.0%  3       2       0
     Arabic    100.0%  100.0%  100.0%  100.0%  100.0%  2       2       0
    Finnish      0.0%    0.0%    0.0%    0.0%    0.0%  2       0       0
     Polish     38.5%   50.0%   71.4%  100.0%   33.3%  2       2       4
    Turkish     45.5%   57.1%   76.9%  100.0%   40.0%  2       2       3
  Afrikaans      0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
    Burmese      0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
   Croatian      0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
      Czech     15.2%   22.2%   41.7%  100.0%   12.5%  1       1       7
     Danish      0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
     Korean    100.0%  100.0%  100.0%  100.0%  100.0%  1       1       0
 Portuguese     17.2%   25.0%   45.5%  100.0%   14.3%  1       1       6
 Vietnamese     29.4%   40.0%   62.5%  100.0%   25.0%  1       1       3
               f0.5    f1      f2      recall  prec    total   hits    misses

French, German, Spanish, and Italian all do very poorly, with too many false positives. (When Spanish and Italian are disabled, French does even worse). Polish, Turkish, Czech, Portuguese and Vietnamese aren’t terrible in terms of raw false positives, but aren’t great, either.

As noted above, Greek, Hebrew, Japanese, and Russian are present in the larger sample, and as our models for these languages are very high accuracy, I’ve included them.

The final language set is Dutch, English, Chinese, Arabic, Korean, Greek, Hebrew, Japanese, and Russian. As above, 3K is not the optimal model size, but it is within 1.5%. The 3K results are shown below along with the best performing model sizes:

               3000    3500    6000    7000    9000   
      TOTAL    82.3%   82.5%   82.9%   83.3%   83.7%   
      Dutch    92.1%   92.4%   92.8%   92.9%   92.7%   
    English    76.0%   76.2%   76.5%   77.6%   79.1%   
     French     0.0%    0.0%    0.0%    0.0%    0.0%    
     German     0.0%    0.0%    0.0%    0.0%    0.0%    
    Spanish     0.0%    0.0%    0.0%    0.0%    0.0%    
    Italian     0.0%    0.0%    0.0%    0.0%    0.0%    
      Latin     0.0%    0.0%    0.0%    0.0%    0.0%    
    Chinese   100.0%  100.0%  100.0%  100.0%  100.0%  
     Arabic    80.0%   80.0%   80.0%   80.0%   80.0%   
    Finnish     0.0%    0.0%    0.0%    0.0%    0.0%    
     Polish     0.0%    0.0%    0.0%    0.0%    0.0%    
    Turkish     0.0%    0.0%    0.0%    0.0%    0.0%    
  Afrikaans     0.0%    0.0%    0.0%    0.0%    0.0%    
    Burmese     0.0%    0.0%    0.0%    0.0%    0.0%    
   Croatian     0.0%    0.0%    0.0%    0.0%    0.0%    
      Czech     0.0%    0.0%    0.0%    0.0%    0.0%    
     Danish     0.0%    0.0%    0.0%    0.0%    0.0%    
     Korean   100.0%  100.0%  100.0%  100.0%  100.0%  
 Portuguese     0.0%    0.0%    0.0%    0.0%    0.0%    
 Vietnamese     0.0%    0.0%    0.0%    0.0%    0.0%    


The accuracy is very high, and the differences are <2%, so it makes sense to stick with the default 3K models for now, but keep an eye out for significant performance improvements with other model sizes.

The detailed report for the 3K model is here:

              f0.5    f1      f2      recall  prec    total   hits    misses
      TOTAL    82.3%   82.3%   82.3%   82.3%   82.3%  515     424     91
      Dutch    92.4%   92.1%   91.9%   91.7%   92.6%  326     299     24
    English    68.5%   76.0%   85.4%   93.0%   64.3%  128     119     66
     French     0.0%    0.0%    0.0%    0.0%    0.0%  16      0       0
     German     0.0%    0.0%    0.0%    0.0%    0.0%  11      0       0
    Spanish     0.0%    0.0%    0.0%    0.0%    0.0%  6       0       0
    Italian     0.0%    0.0%    0.0%    0.0%    0.0%  5       0       0
      Latin     0.0%    0.0%    0.0%    0.0%    0.0%  4       0       0
    Chinese   100.0%  100.0%  100.0%  100.0%  100.0%  3       3       0
     Arabic    71.4%   80.0%   90.9%  100.0%   66.7%  2       2       1
    Finnish     0.0%    0.0%    0.0%    0.0%    0.0%  2       0       0
     Polish     0.0%    0.0%    0.0%    0.0%    0.0%  2       0       0
    Turkish     0.0%    0.0%    0.0%    0.0%    0.0%  2       0       0
  Afrikaans     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
    Burmese     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
   Croatian     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
      Czech     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
     Danish     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
     Korean   100.0%  100.0%  100.0%  100.0%  100.0%  1       1       0
 Portuguese     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
 Vietnamese     0.0%    0.0%    0.0%    0.0%    0.0%  1       0       0
              f0.5    f1      f2      recall  prec    total   hits    misses


Recall went up and precision went down for Dutch and English, but overall performance improved. Queries in unrepresented languages were almost all identified as either Dutch or English (decreasing precision for both), but those now unused models are no longer generating lots of false positives and bringing down precision overall. (The one query in Burmese was identified as Arabic, probably because it scored the same in all languages—with the max “unknown” score—and Arabic is alphabetically first among the contenders.)

nlwiki: Best Options[edit]

The barely sub-optimal settings (though consistent with others using 3K models) for nlwiki, based on these experiments, would be to use models for Dutch, English, Chinese, Arabic, Korean, Greek, Hebrew, Japanese, and Russian (nl, en, zh, ar, ko, el, he, ja, ru), using the default 3000-ngram models.