User:OrenBochman/Search/NLP Tools

=Introduction=

NLP (Natural Language Programming) Tools are tools geared at broadening the scope and increasing the power of search by modeling the languges in which articles are written in. They require more sophisticated algorithems greater processing power. THier capabilites can be disruptive and could revolutionize how people use/edit a wiki. For example it could be possible to use such a tool to
 * search names phoneticaly or using transliteration
 * spell check an article (using a search engine spell checker nor a lexeme spell checker)
 * search for entities (people,institutions,deaseas)
 * search & visualise memes in a wiki or an article.
 * identify articles most in need of expansion.
 * identify an article's degree of asherence to a project's writing style.
 * identify uninteresting articles
 * search terms across projects and languages (cross-language search)
 * translate an article from another language

= Wiktionary Analyzer = Goal Produce a Lucene analyzer which based on data extracted from Provide a better, faster, smarter search across the Wikimedia projet. (Suggestions, spellings, corrections, etc)
 * Wiktionary projects
 * Wikipedia projects
 * a few other selected projects

The analysis would be CPU intensive. To be done in sensible time development would require
 * an integration server (Hudson scales nicely)
 * Wikimedia project dumps (openzim or xml) source and HTML
 * Hadoop cluster running advanced Mahut algorithms(SVM,LDA and others)
 * Iterative production of more poweful Lucene analyzers. Bootsraped via simple scripts followed by unsupervised learning cycles (to complete the picture)
 * Since these jobs could easily become intractable (though bugs, bad algorithms)
 * running dev job on wiki subsets
 * job length and current progress/cost estimation are design goals.

Corpus Linguistics
Since R now provides powerful tools for doing corpus linguistics - these should be levreaged. A suitable tools for converting Wikipedia dump to a corpus should be found or developed. From this three immediate deliverables can be produced


 * frequency lists - words, shingels, phrases.
 * colocations - phrases that are more than the sum of thier words.
 * concordences - KWIC Databases.

Lexical Data

 * 1) Lemma Extraction
 * 2) Scan English wiktionary at the POS section level.
 * 3) Extract
 * 4) Lang,
 * 5) POS,
 * 6) Lemma information,
 * 7) Glosses
 * 8) co-location
 * 9) proper names
 * 10) silver bullets
 * 11) Entropy reducing heuristic -> to induct missing lemmas from free text.
 * 12) Inducting word-sense based on Topic Map 'Contexts'.
 * 13) Introducing a disambiguation procedure to Semantic/Lexical/Entities.

Semantic Data

 * 1) Word Sense enumeration - Via Grep
 * 2) Word Sense context collection Via Mahut SVD
 * 3) Word Sense context description(req an algorithm)
 * 4) Word nets - synonyms,antonyms,
 * 5) Entity type
 * 6) Categories

Entity Data
The idea is to generate a database of entities referenced in Wikipedia. Entities are

Bootstrapping via article headers and categories. Once the most obvious entities are listed one proceeds to train classifiers to find the remaining entities via NLP.


 * 1) category Based Classification of:
 * 2) People, organizations, companies, bands, nations, imagined, dead etc
 * 3) Animals, Species, etc
 * 4) Places, counties, cities,
 * 5) Dates, Time Lines, Duration
 * 6) Events, Wars, Treaties, films, awards,
 * 7) Chemicals, medicine, drugs
 * 8) commodities, Stocks etc
 * 9) Publications, Journals, periodicals, Citations etc
 * 10) External Web locations.
 * 11) Unsupervised acquisition of More entities
 * 12) Train a SVM classifiers Mahut via Hadup using tagging/parsing low ambiguity snippets referencing wide choice of terms.
 * 13) Accuire more entities.
 * 14) Crosswikify Top enteties.
 * 15) Cross wiki links.
 * 16) Run Mahut LDA via Haddop on Articles/Sections with correlated entities.

Etymology
If we trust etymologies in one language we could suggest them for others.
 * 1) would require a model (loan, analogy, language change).
 * 2) requires/implies a phonological distance, semantic distance, word sense.
 * 3) requires/implies a graph of time, language, location.
 * 4) historical linguistics rules could be used to refine such a model.

MT translation Data
This is not high priority deliverable since its utility is doubtful.
 * Offline bilingual dictionaries may be of interest. c.f. wikt:en:User:Matthias Buchmeier/trans-en-es.awk
 * As wiktionaries improve they could become a significant contribution where statistical methods fall short.
 * Wikipedias clearly contain large volume of text for generating statistical language models.

Filing in the gaps
During analysis it may be able to do some extra tasks.
 * 1) Multilingual context sensitive spell-checking the wikis. Both offline and online.
 * 2) foreign Language Learning spelling dictionary
 * 3) Identify "missing templates" requires lemma to template mapping data struction and a generalization algorithm
 * 4) Identify "missing pages/section" in the wiktionary.

Language Instruction
Language Instruction would benefit from a database of about language pairs or groups: [i.e. it could help chart an optimal curriculum for teaching a nth language to a speaker of n-1 languages by producing a graph of list least resistance.


 * Top Frequency Lexical Charts
 * Topical Word Lists
 * Lexical problem areas
 * Word Order (requires lemma-n-gram frequencies)
 * Verbal Phrase/ Verbal Complement Misalignment.

Compression

 * 1) frequency information together with lexical data can be used to make a text compressor optimized for a specific wiki.
 * 2) this type of compressed text would be faster to search.

Tagger/Parser
the lexical data + frequency + n-gram frequency could be used to make a parametric translingual parser.

Machine Translation
see also Wikipedia_Machine_Translation_Project


 * 1) export data into apertium format. cf http://wiki.apertium.org
 * a morphological dictionary for language xx called apertium-sh-en.sh.dix, apertium-sh-en.en.dix etc. which contains the rules of how words in language xx are inflected.
 * Bilingual dictionaries which contain correspondences between words and symbols in the two languages. called: apertium-sh-en.sh-en.dix
 * language xx to language yy transfer rules: this file has rules for how language xx will be changed into language yy. In our example this will be: apertium-sh-en.sh-en.t1x
 * language yy to xx language transfer rules: this file has rules for how language yy will be changed into language xx. In our example this will be: apertium-sh-en.en-sh.t1x


 * apertium likes [w:FSM]s so it could be possible to adapt its morphological data into an efficient spell-cheking dictionary.
 * it format may able to support collocation.
 * it does not seem to have a word-sense notion.

Crowdsourcing Wiktionary
Using existing categories, templates, the semantic wiki extension, and some scripts one could
 * automate generation of morphological information.
 * again it is possible to automate generation of bi-lingual information from wiktionary pages

Topological Translation Algorithm

 * Statistical translation works by matching parts of sentences. (This has many problems)
 * Requires a large parallel corpus of translated texts. (not available)
 * Assumes that words in a N-gram of words operate. Some languages have free word order.
 * In reality statistical lexical data is sparse.

My idea is to develop a:
 * topological algorithm for wikis
 * based on related documents and their revisions.
 * can use non-paralel categorised sets of translation sets of documents.
 * generates seme lattices I.E. cross lingual semantic algorithm
 * (semes N-nets)
 * with (morphological-state)
 * morpho-syntactic Lagrangian for MT.


 * the lattice should converge due to product theorem


 * 1) Translation matrix
 * 2) maps source word-sense to a target word-sense.
 * 3) translate cross language.
 * 4) simplify a single language text.
 * 5) make text clearer via a disambiguate operation
 * 6) Numbered list item

Unsupervised Morphology Induction
Parametric version unsupervised induction of morphology based on [GoldSmith]

Parameters
comma seperetes equivelents, dor seperetes entries
 * 1) ALPHA_FULL: [a,A.b,B. ... σ,sz,SZ. ... z,Z]
 * 2) ALPHA_VOWL: [a.e.i.o.u]
 * 3) Derive:
 * 4) * ALPHA_SIZE
 * 5) * CONS_COUNT
 * 6) * VOWL_COUNT
 * 7) BOOTSTRP_MIN_STEM_LEN = 5
 * 8) BOOTSTRP_SUCCESOR_SIGNICICANCE = 25
 * 9) BOOTSTRP_SIGN_SEED = (2,2) two suffix of length two
 * 10) BOOTSTRP_NULL_LEN = 2
 * 11) INDUCT_PREFIX
 * 12) SPLIT_COMPOUNDS
 * 13) assigns stems n-grams signature
 * 14) use the signature to detect compunds

Algorithm

 * 1) Preprocess recode corpus. Cann be done automaticaly via minentropy binary codes like in arithmatic coding, but these rules can override for debugability. e.g.
 * 2) * [sz<>σ] hungarian encode bi-gram as foreign unigram
 * 3) * [ssz<>σσ] hungarian long consonant as two foreign unigrams
 * 4) Store words in a pat-trie
 * 5) bootstrap morphology using huristics.
 * 6) huristic 1: split at Li where successorFrequency(i)>= 25 and successorFrequency(i+1) == successorFrequency(i-1) == 1
 * 7) huristic 2: seed signitures with stems having 2 suffixes of length 2
 * 8) repeat;
 * 9) evaluate signiture entopy, and morphology entropy recode signatures for min entorpy and max reboustnesss (samething).
 * 10) generalize for unknown words.
 * 11) generalize for unknown stems.
 * 12) generalize for unknown suffixes.

Data

 * 1) trie word list (or AWG)
 * 2) atoms sorted by frequency
 * 3) stems
 * 4) suffixes
 * 5) prefixes
 * 6) signatures listed by robustness

Improvments

 * 1) solve aliphony using wiktionary
 * 2) solve aliphony using LPSA or TOPIC map variant
 * 3) collect lemmas database from templates categories. (Template extraction)
 * 4) Using hand-built table miner, SED (template) structured text (wiktionary) and free-text miner (Wikipedia) collect
 * 5) map templates to morphological-state via a "model".
 * 6) maprduce via mahut

Lemma Mining
A couple of themes in searching for lemmas. 1. geometry. (Hamming for equal length) 2. N-gram.


 * 1) A bootstrap known knowledge.
 * 2) Lemma base.
 * 3) unknown base.
 * 4) parallel multi-pattern matches
 * 5) affix mode

other iteration modes:
 * 1) Levinstien clustering using spheres round/between existing lemma and lemma members. lets say R=d(L1,M1) is the max distance between any two known lemma representatives. Words equidistant from L or M within R/2 are lemma candidates.
 * 2) generalizing the Levinstien distance to complex distance (we could map strings to sort be predicitive.

Generalized Morphological Analyzer
Based on the above morphology datastructure analyze text storing a full merphological analysis in a payload for downstream ana;ysis tasks.

An enumeration of all possible morphological states (in all languages). Each language has a sparse subset depending on it's morphological parameters.

e.g. the hungarian has:

Uses

 * 1) Enumeration Develop a language independent view of morphology with chief application in MT. In this sense morphology is viewed as a semi group generated as a Cartesian product of its feature subsets.
 * 2) Compression. Since in reality feature availability varies across languages matrix is not only sparse but also within a language feature availability is co-dependent. (a verb has time but a noun does not) Therefore the minimal sparse matrix can be extracted and used to compress morphological state. But to create such a compression scheme it is necessary to collect statistics showing lemma frequency, and feature dependency.
 * 3) IR. By supplying a lemma-id and a morphological-state one provide superior search capabilities in certain languages.