User:Gautham shankar/Gsoc

Identity

 * Name : Gautham Shankar G
 * Email : gautham.shankar@hiveusers.com
 * Project Title: Lucene Automatic Query Expansion From Wikipedia Text

Contact / Working Info

 * Timezone: UTC/GMT +5:30 hours
 * Typical working hours: 9 AM to 9 PM (3:30 AM to 4:30 PM UTC) (flexible)
 * IRC handle: gauthamshankar
 * Skype: gautham.shankar3
 * Phone: +919884837794

Project Summary
Query expansion is a method used to improve the performance of information retrieval systems. The following problems may exist when an user gives a search query,


 * Users typically formulate very short queries and are not likely to take the trouble of constructing long and carefully stated queries.

Hence the search results that are obtained are not satisfactory and might not contain the relevant information that the user is looking for. This project aims to solve the problem in two stages
 * The words used to describe a concept in the query are different from the words authors use to describe the same concept in their documents.


 * 1) Creation of a multilingual wordnet
 * 2) *Wordnet is a lexical database that has become a standard resource in NLP research.
 * 3) *In wordnet nouns, verbs,adjectives and adverbs are organized in sets of synonyms, called synsets which convey a concept.
 * 4) *These synsets are connected to other synsets by semantic relations (hiponymy,antonomy etc.)
 * 5) *The wordnet can be built using the vast multilingual data in Wiktionary.
 * 6) Query expansion
 * 7) *The input query is expanded with relevant terms in order to encompass a wider range of context for a successful search.
 * 8) *The search query is mapped with wordnet to obtain relevant expansion terms.
 * 9) *Integrating wordnet and search will provide data on the effectiveness of the wordnet.
 * 10) *The query expansion is added as a Lucene filter.

Both the stages are very big and most of the time during Gsoc will go into building the wordnet. If time permits the lucene filter can be added. Completing the wordnet would provide a vast lexical database in machine readable format for future NLP projects. The completion of the entire project would greatly enhance the quality of the search results obtained by the user.

Deliverables
Required deliverables


 * 1) A framework to mine Wiktionary to create a word net
 * 2) Parse through the word list and obtain the Synonyms,Hyponyms, Derived terms and other available data for a particular word.
 * 3) organize them into a hierarchical dataset with a IS A relationship.

Once the word net is completed, we can obtain the synonyms of a word, or obtain the lemma of a word etc.


 * 1) Search Engine filter (adhoc feedback system)
 * 2) First Search (obtain feedback)
 * 3) A search needs to be run with the query given by the user.The pages obtained are used to perform Word-sense Disambiguation [WSD].
 * 4) A re-ranking of search results can be performed in case the query does not yield relevant results.(optional)
 * 5) Second Search (use feedback to improve search)
 * 6) Using the results from WSD the required expansion queries are obtained from the word net.
 * 7) The expansion terms are appended to the search query.
 * 8) The query terms are given a rank based on their relevance to the search context.
 * 9) The expanded query is then given as input to obtain the final results.

Research has proved that directly expanding the query has at times degraded the results due to lack of relevant terms being appended. Hence the feedback system is used to improve the quality of the terms.

Community bonding Period

 * Interact with the mentors and the community.
 * Discuss the deliverables with the mentor and finalize the approach to be taken to solve the problem.
 * familiarize myself with the required algorithms and compare performance of various algorithms to choose the best.

Coding Period
I have my university exams until 31st of may and will start coding from 1st of June.

Schedule for the first leg :


 * 1) 1st June to 16th June (Milestone 1, 2.2 weeks)
 * 2) *Download the required Wiktionary dumps
 * 3) *Decide on the data structure to store the word net
 * 4) *Write a script to Prune through the data and organize it into the required data structure
 * 5) 17th June to 30th June (Milestone 2, 2 weeks)
 * 6) *Run the first search to obtain the feedback
 * 7) *Perform a re-ranking for the results
 * 8) 1st July to 7th July (Milestone 3, 1 week)
 * 9) * Completion of coding for first leg of Gsoc
 * 10) * Prepare documentation for Mid-term Evaluation

Schedule for the second leg :
 * 1) 8th July to 14th July (Milestone 4, 1 week)
 * 2) * feedback on performance from mentors
 * 3) *Parse the documents from the search results for Word-sense Disambiguation
 * 4) 15th July to 4th August (Milestone 5, 3 weeks)
 * 5) *implement different WSD algorithms to find the optimal algorithm
 * 6) 5th August to 11th August (Milestone 6, 1 week)
 * 7) * Use the WSD results to get relevant expansion terms from word net
 * 8) 12th August to 18th August (Milestone 7, 1 week)
 * 9) * Obtain the final search results
 * 10) * Complete coding for second leg
 * 11) * Prepare documentation for Evaluation
 * 12) 19th August to 29th August (Milestone 8, 1.5 weeks)
 * 13) * Make final changes if any to make it presentable

About Me
I'm Gautham Shankar, final year engineering student pursuing by B.E in computer science and engineering. I have a great passion for programming and problem solving.I used to primarily code in c/c++ and have created a basic MS paint replica in c++. moving into college i got interested in the web and especially search.I have developed websites using php, mysql, javascript and built a recommendation framework for search query suggestion.I have extensively used open source technologies and I would like to contribute back to this community by being a part of gsoc2012.

Participation
Coding has never felt like work.Would love to spend the summer doing what i truly love.I'm willing to work from 9am to 10pm. I'm usually online on skype or gtalk and respond to mails without much delay. I work very hard in solving problems and use the forums and blogs when i am stuck with a problem.

Past open source experience
I have not contributed to any open source software, but would like to this as an opportunity to do so.I have used the a lot of open source softwares in my projects like the Lucene search engine and phpbb.