Search 2 Code ReviewEdit

Why the long reviewEdit

I've had to wait for months to get access to a search environment which is still offline. I could not run tests so I've written down my comments in this reviewing instead of trying to resolve them since this has not been possible.

Location of sorce codeEdit


  • (limited) interface for metadata on article
    • isSubpage() - if it is a subpage
    • daysOld() - age in index
    • namespace() - articles nameSpace
  • interface implementation is in

note: the only implementation wraps methods of ArticleMetaSource so it could be refactored away

  • boosts article using its namespace.
  • is used in:
    • ArticleQueryWrap.customExplain(),
    • ArticleQueryWrap.customScore()
    • SearchEngine.PrefixMatch()
  • tested in
    • testComplex
    • testDefalut

  • provides a boost queary with a minumum score
  • used by
    • CustomScorer()

  • Query that sets document score as a programmatic function of (up to) two (sub) scores.

Package org.apache.lucene.analysisEdit

Package org.apache.lucene.searchEdit


Type Purpose Tests Lucene version Comments
AcronymFilter N.S.A. → NSA none 4.0.x
CJKFilter C1C2C3C4 → C1C2 C2C3 C3C4. none 2.4.x alternative[1]
EnglishSingularFilter Add English singular forms of words as aliases of type "singular" none 2.4.x
EsperantoStemFilter Stemming of esperanto none 2.4.x
HyphenFilter CAT-\nFISH → CATFISH none 2.4.x
LimitTokenFilter Return tokens whose positional incremental less than some limit none 2.4.x
PhraseFilter Adds word bigrams to token stream.
StopWords can force trigrams
none 2.4.x implimentation [2]


RussianStemFilter Wrapper for Lucene's implementation of Russian stem filter. none 2.4.x
SerbianFilter Filter for Serbian dual-script text. none 2.4.x
VietnameseFilter Vietnamese standard transliterations to ascii none 2.4.x

Some missing filters.

  • WikiArticleFilter indexes all wikipedia articles titles and redirects referenced in article texts using DB dump of title tables.
  • NamedEntity filter - uses a database to index named enteties mined from various wikis (semanticMW could help here)
  • WikiParserAnalyser - uses the latest parser implimentation to parse and index wiki source.


Type Purpose Tests Lucene version
AcronymFilter N.A.S.A. -> N.A.S.A , NASA I've added some 4.0.x


  • WikiQueryParser
  • Aggregate - Aggregate bean that captures information about one item going into the some index aggregate field.

Problems in Aggregate

  1. Token.termText() - undefined
  2. Analyzer.tokenStream(String,String) - undefined
  3. - undefined

Porting FiltersEdit

Filters should be ported from the 2.4.x to Lucene 2.9.x api. This involves:

  1. writing unit tests for the old filter and seeing that they still work with new input....
  2. Token next() and Token next(Token) have been deprected.
    1. incrementToken needs to be called on the input token string, not on the filter which will cause a stack overflow).
    2. to process the token add to the filter properties:
private CharTermAttribute termAtt; //Copies the contents of buffer, at an offset for length characters, to the termBuffer array.
private TypeAttribute typeAttr;

which should be intiated in the constructor via:

public filterXtor()
   termAttr = (CharTermAttribute) addAttribute(CharTermAttribute .class);
   typeAttr = addAttribute(TypeAttribute.class);
  1. boolean incrementToken() is now required.
    1. it moves the token stream one step forward.
    2. it returns true is there are more tokens, false otherwise.
boolean incrementToken()
   if (!input.incrementToken()) 
     return false;

 // process token via termAttr.term()
 // next update buffers
  termAttr.setTermLength(this.parseBuffer(termAtt.termBuffer(), termAtt.termLength()));
  return true;

  • Porting SOLR Token Filter from Lucene 2.4.x to Lucene 2.9.x [4]
  • Porting to Lucene 4.0.x [5]

Package org.apache.lucene.searchEdit

Package org.apache.xmlrpc.webserverEdit

  • no issues

Package org.wikimedia.lsearch.analyzersEdit

Package org.wikimedia.lsearch.beansEdit

  • no issues

Package org.wikimedia.lsearch.benchmarkEdit

  • no issues

Package org.wikimedia.lsearch.configEdit

  • no issues

Package org.wikimedia.lsearch.frontendEdit

  • no issues

Package org.wikimedia.lsearch.highlightEdit

  • review
  • migrate to FastVectorHighlighter

Package org.wikimedia.lsearch.importerEdit


Broking Api ChangesEdit

IndexWriter writer = new IndexWriter(String path,null,boolean newIndex)

Has been deprecated from the api. Indexing has advanced considerably since since 2.3 and these changes should be intergrated into the indexer code.

  • it is possible to use an index while it is being updated.
  • it is possible to update documents in the index.

Non Breaking Api ChangesEdit

writer.setSimilarity(new WikiSimilarity())

Have been deprecated from the api and should be replaced with:


Porting notes: IndexWriterConfig needs to be created and then passed to the constructor.

IndexWriterConfig conf = new IndexWriterConfig(analyzer);
conf.setSimilarity(new WikiSimilarity()).setMaxBufferedDocs(maxBufDocs); //allows chaining
writer=new IndexWriter(Directory d, conf)      // Constructs a new IndexWriter per the settings given in conf.

All but the last are pretty trivial changes. However using LimitTokenCountAnalyzer instead of writer.setMaxFieldLength() means that another step needs to be added to analysis also the fact that some of the anlysis uses mutivalued<reference>where??</reference> fields means that LimitTokenCountAnalyzer will operate differently from before. The analyzer limits the number of tokens per token stream created, while this setting limits the total number of tokens to index. So that more tokens would get indexed

Package org.wikimedia.lsearch.indexEdit

  • Indexer and indexing related classes.


public float lengthNorm(String fieldName, int numTokens)

has been deprecated and needs to be replaced by

public float computeNorm(String field, FieldInvertState state)

Package org.wikimedia.lsearch.indexEdit

  • three files don't compile
  • review

Package org.wikimedia.lsearch.interoperabilityEdit

  • no issues

Package org.wikimedia.lsearch.oaiEdit

  • no open issues - may be obsolete
  • review later

Package org.wikimedia.lsearch.prefixEdit

  • one file needs review
  • generates title prefixes for search. is this still needed if title field is indexed as term position vectors

Package org.wikimedia.lsearch.ranksEdit

  • another indexer which chounts links for page rank purposes
  • the analyser will need to be ported along the lines of the main anlayser in the impoerter packacge.

Package org.wikimedia.lsearch.relatedEdit

  • no open issues

Package org.wikimedia.lsearch.searchEdit

  • many issues
  • needs serious review.
  • losts of filter and filter wrappers - may be generaly useful code.
  • has distributed search related classes which sould be made obsolete by moving to solr

Package org.wikimedia.lsearch.spellEdit

  • does the did you mean feature
  • two broken files

Package org.wikimedia.lsearch.spell.apiEdit

  • more of the did you mean feature
  • 3 indexers are broken.

Package org.wikimedia.lsearch.spell.distEdit

no issues

Package org.wikimedia.lsearch.statisticsEdit

no issues

Package org.wikimedia.lsearch.storageEdit

  • Storage of data, mainly in database. E.g. page ranks, text for highlighting...
  • three files broken
  • problems are indexing ENUM
  • String path -> FSDirectory Dir

Package org.wikimedia.lsearch.utilEdit

Search 2 Code ReviewEdit


  • uses junit3
  • test methods are poorly written and badly named
    • tests are not atomic
    • tests do not seperate behaviour
    • tests may have side effect (are thy safe to run in parallel)
    • tests should use testing patterns
    • no mocks of interfaces for complex behavoiur (may not be an issue)
  • most if not all extends WikiTestCase


    • WikiTestCase is OS dependent.
    • WikiTestCase hardcodes file locations.
    • WikiTestCase reads a global configuration
    • WikiTestCase sets WikiQueryParser
    • When WikiTestCase has issues one cannot run test.
    • WikiTestCase is in import org.wikimedia.lsearch.config.GlobalConfiguration;
    • It is not possible to run test either WikiQueryParser or GlobalConfiguration can't compile. This is bad since they include most parts of the projects.

Package org.wikimedia.lsearch.analyzersEdit

Package org.wikimedia.lsearch.beansEdit

Package org.wikimedia.lsearch.configEdit

Package org.wikimedia.lsearch.highlightEdit

Package org.wikimedia.lsearch.indexEdit

Package org.wikimedia.lsearch.ranksEdit

Package org.wikimedia.lsearch.searchEdit

Package org.wikimedia.lsearch.spellEdit

Package org.wikimedia.lsearch.spell.apiEdit

Package org.wikimedia.lsearch.storageEdit

Package org.wikimedia.lsearch.testEdit

Package org.wikimedia.lsearch.utilEdit

Search 2 Code Porting NotesEdit


  • is it possible to use the old filter code with new lucene libraries.
  • is it possible to make a non invasive modification which would allow using old filter code almost as is.
  • what do various filters do.
  • which filters are in use for various setups.

Contributing To SolrEdit

It would be strtegicly wise to contributed back to Apache Lucne and/or Apache Solr. This is beacause:

  • Once it is integrated into SOLR they have to keeping it up to date with changes in lucene's API which invariably occuer over time
  • They have extensive tests
  • Their user base will find and fix bug much faster than can be done in the Wikipedia ecosystem.

Additional Porting notesEdit

  1. The method tokenStream(String, Reader) in the type Analyzer is not applicable for the arguments (String, String)
    1. analyzer.tokenStream(field,text) → analyzer.tokenStream(field,new StringReader(text))
    2. analyzer.tokenStream(field,"") → analyzer.tokenStream(field,(Reader)null)
  2. The constructor HitQueue(int) is undefined
    1. HitQueue hq = new HitQueue(nDocs,true) → #HitQueue hq = new HitQueue(nDocs,true)


  1. A Japanese stemmers now exsits and may do a better job than this filter
  2. implimentation is problematic because it could conflict with Ngram tokens in text. It would be better to prefix ngrams with a standard prefix or better yet to store them in payloads
  3. this filter may be obolete since indexing using term position vectors will allow to query for these token sequences using a standard index and would also be great for highlighting as well