User:OrenBochman/Search/Porting

=Search 2 Code Review=

Why the long review
I've had to wait for months to get access to a search environment which is still offline. I could not run tests so I've written down my comments in this reviewing instead of trying to resolve them since this has not been possible.

Location of sorce code

 * project is http://svn.wikimedia.org/svnroot/mediawiki/trunk/lucene-search-2/src/
 * source is located at http://svn.wikimedia.org/svnroot/mediawiki/trunk/lucene-search-2/src/
 * test is http://svn.wikimedia.org/svnroot/mediawiki/trunk/lucene-search-2/test/

Package org.apache.lucene.search/ArticleInfo.Java
note: the only implementation wraps methods of ArticleMetaSource so it could be refactored away
 * (limited) interface for metadata on article
 * isSubpage - if it is a subpage
 * daysOld - age in index
 * namespace - articles nameSpace
 * interface implementation is in org.wikimedia.lsearch.search/ArticleInfoImpl.java

org.apache.lucene.search/ArticleNamespaceScaling.Java

 * boosts article using its namespace.
 * is used in:
 * ArticleQueryWrap.customExplain,
 * ArticleQueryWrap.customScore
 * SearchEngine.PrefixMatch
 * tested in
 * testComplex
 * testDefalut

org.apache.lucene.search/ConstMinScore

 * provides a boost queary with a minumum score
 * used by
 * CustomScorer

org.apache.lucene.search/CustomBoostQuery

 * Query that sets document score as a programmatic function of (up to) two (sub) scores.

Filters
Some missing filters.
 * WikiArticleFilter indexes all wikipedia articles titles and redirects referenced in article texts using DB dump of title tables.
 * NamedEntity filter - uses a database to index named enteties mined from various wikis (semanticMW could help here)
 * WikiParserAnalyser - uses the latest parser implimentation to parse and index wiki source.

Others

 * WikiQueryParser
 * Aggregate  - Aggregate bean that captures information about one item going into the some index aggregate field.

Problems in Aggregate
 * 1) Token.termText - undefined
 * 2) Analyzer.tokenStream(String,String) - undefined
 * 3) TokenStream.next - undefined

Porting Filters
Filters should be ported from the 2.4.x to Lucene 2.9.x api. This involves: private CharTermAttribute termAtt; //Copies the contents of buffer, at an offset for length characters, to the termBuffer array. private TypeAttribute typeAttr; which should be intiated in the constructor via: public filterXtor {  super(input); termAttr = (CharTermAttribute) addAttribute(CharTermAttribute .class); typeAttr = addAttribute(TypeAttribute.class); }
 * 1) writing unit tests for the old filter and seeing that they still work with new input....
 * 2) Token next and Token next(Token) have been deprected.
 * 3) incrementToken needs to be called on the input token string, not on the filter which will cause a stack overflow).
 * 4) to process the token add to the filter properties:

boolean incrementToken {  if (!input.incrementToken) return false;
 * 1) boolean incrementToken is now required.
 * 2) it moves the token stream one step forward.
 * 3) it returns true is there are more tokens, false otherwise.

// process token via termAttr.term // next update buffers termAttr.setTermBuffer(modifiedToken); termAttr.setTermLength(this.parseBuffer(termAtt.termBuffer, termAtt.termLength)); typeAttr.setType(TOKEN_TYPE_NAME); return true; }


 * Porting SOLR Token Filter from Lucene 2.4.x to Lucene 2.9.x
 * Porting to Lucene 4.0.x

Package org.apache.xmlrpc.webserver

 * no issues

Package org.wikimedia.lsearch.beans

 * no issues

Package org.wikimedia.lsearch.benchmark

 * no issues

Package org.wikimedia.lsearch.config

 * no issues

Package org.wikimedia.lsearch.frontend

 * no issues

Package org.wikimedia.lsearch.highlight

 * review
 * migrate to

Broking Api Changes
IndexWriter writer = new IndexWriter(String path,null,boolean newIndex) Has been deprecated from the api. Indexing has advanced considerably since since 2.3 and these changes should be intergrated into the indexer code.
 * it is possible to use an index while it is being updated.
 * it is possible to update documents in the index.

Non Breaking Api Changes
writer.setSimilarity(new WikiSimilarity) IndexWriterConfig.setMaxBufferedDocs(int) writer.setMergeFactor(mergeFactor) writer.setUseCompoundFile(true); writer.setMaxBufferedDocs(maxBufDocs) writer.setMaxFieldLength(WikiIndexModifier.MAX_FIELD_LENGTH); Have been deprecated from the api and should be replaced with: IndexWriterConfig.setSimilarity(Similarity)} IndexWriterConfig.setMaxBufferedDocs(int)} LogMergePolicy.setMergeFactor(int) LogMergePolicy.setUseCompoundFile(boolean) LimitTokenCountAnalyzer Porting notes: needs to be created and then passed to the constructor. IndexWriterConfig conf = new IndexWriterConfig(analyzer); conf.setSimilarity(new WikiSimilarity).setMaxBufferedDocs(maxBufDocs); //allows chaining writer=new IndexWriter(Directory d, conf)     // Constructs a new IndexWriter per the settings given in conf. All but the last are pretty trivial changes. However using  instead of   means that another step needs to be added to analysis also the  fact that some of the anlysis uses mutivalued where?? fields means that LimitTokenCountAnalyzer will operate differently from before. The analyzer limits the number of tokens per token stream created, while this setting limits the total number of tokens to index. So that more tokens would get indexed

Package org.wikimedia.lsearch.index

 * Indexer and indexing related classes.

org.wikimedia.lsearch.indexWikiSimilarity
public float lengthNorm(String fieldName, int numTokens) has been deprecated and needs to be replaced by public float computeNorm(String field, FieldInvertState state)

Package org.wikimedia.lsearch.index

 * three files don't compile
 * review

Package org.wikimedia.lsearch.interoperability

 * no issues

Package org.wikimedia.lsearch.oai

 * no open issues - may be obsolete
 * review later

Package org.wikimedia.lsearch.prefix

 * one file needs review
 * generates title prefixes for search. is this still needed if title field is indexed as term position vectors

Package org.wikimedia.lsearch.ranks

 * another indexer which chounts links for page rank purposes
 * the analyser will need to be ported along the lines of the main anlayser in the impoerter packacge.

Package org.wikimedia.lsearch.related

 * no open issues

Package org.wikimedia.lsearch.search

 * many issues
 * needs serious review.
 * losts of filter and filter wrappers - may be generaly useful code.
 * has distributed search related classes which sould be made obsolete by moving to solr

Package org.wikimedia.lsearch.spell

 * does the did you mean feature
 * two broken files

Package org.wikimedia.lsearch.spell.api

 * more of the did you mean feature
 * 3 indexers are broken.

Package org.wikimedia.lsearch.spell.dist
no issues

Package org.wikimedia.lsearch.statistics
no issues

Package org.wikimedia.lsearch.storage

 * Storage of data, mainly in database. E.g. page ranks, text for highlighting...
 * three files broken
 * problems are indexing ENUM
 * String path -> FSDirectory Dir

Package org.wikimedia.lsearch.util
=Search 2 Code Review=

General

 * uses junit3
 * test methods are poorly written and badly named
 * tests are not atomic
 * tests do not seperate behaviour
 * tests may have side effect (are thy safe to run in parallel)
 * tests should use testing patterns
 * no mocks of interfaces for complex behavoiur (may not be an issue)
 * most if not all extends WikiTestCase

WikiTestCase

 * WikiTestCase is OS dependent.
 * WikiTestCase hardcodes file locations.
 * WikiTestCase reads a global configuration
 * WikiTestCase sets WikiQueryParser
 * TITLE_BOOST
 * ALT_TITLE_BOOST
 * CONTENTS_BOOST
 * When WikiTestCase has issues one cannot run test.
 * WikiTestCase is in import org.wikimedia.lsearch.config.GlobalConfiguration;
 * It is not possible to run test either  or   can't compile. This is bad since they include most parts of the projects.

Package org.wikimedia.lsearch.util
=Search 2 Code Porting Notes=

Questions

 * is it possible to use the old filter code with new lucene libraries.
 * is it possible to make a non invasive modification which would allow using old filter code almost as is.
 * what do various filters do.
 * which filters are in use for various setups.

Contributing To Solr
It would be strtegicly wise to contributed back to Apache Lucne and/or Apache Solr. This is beacause:
 * Once it is integrated into SOLR they have to keeping it up to date with changes in lucene's API which invariably occuer over time
 * They have extensive tests
 * Their user base will find and fix bug much faster than can be done in the Wikipedia ecosystem.

Additional Porting notes

 * 1) The method tokenStream(String, Reader) in the type Analyzer is not applicable for the arguments (String, String)
 * 2) The constructor HitQueue(int) is undefined
 * 1) The constructor HitQueue(int) is undefined
 * 1) The constructor HitQueue(int) is undefined

=References=