docusearch

Inactive

Project Summary

Analyzed 1 day ago based on code collected 1 day ago.

Getting StartedTo get started download the source using:

svn checkout http://docusearch.googlecode.com/svn/trunk/ docusearch-read-only You will need to install Java 1.6, Maven 2.0+ and CouchDB before start using the services. On Mac, you can install CouchDB via:

sudo port install couchdbThen manually start the CouchDB using

sudo /opt/local/bin/couchdbYou can verify if CouchDB is running using http://localhost:5984/_utils/index.html.

BuildingType "mvn" to build the project. Maven will download a bunch of files that may take a few minutes and will cache those locally and will then proceed to compile, test and build war file.

Populating DatabaseYou are free to choose your favorite way to add or import data into CouchDB, though the DocuSearch includes some ETL programs to add comma or tab delimited data into CouchDB. For example, let say you want to find authorized e-file providers for IRS, so you download some data from http://www.irs.gov/taxpros/providers/article/0,,id=98168,00.html that has following format:

business_name,street_address_1,street_address_2,city,state,zip,zip_4,contact_first_name,contact_middle_name,contact_last_name,phone,flag1,flag2,flag3,flag4You can import it to the couchdb using

mvn exec:java -Dexec.mainClass="com.plexobject.docusearch.etl.DocumentLoader" \
-Dexec.args="efile_providers data/wa.txt none business_name,street_address_1,street_address_2,city,state,zip,zip_4,contact_first_name,contact_middle_name,contact_last_name,phone"Which takes following arguments:

name-of-database, e.g. efile_providers name of comma delimited file, e.g. wa.txt id-column or none if database ids will automatically be generated comma-delimited list of fields to be imported Once the data is loaded, you can create Lucene index, but before that you will have to specify the index policy, which is just another CouchDB document. The index policy specifies fields to be indexed, whether they should be stored in index, score and boost values. These policy configurations are stored in the_config database and you can add the policy using:

curl -X PUT http://127.0.0.1:5984/the_config/index_policy_for_efile_providers -d \
'{"_id":"index_policy_for_efile_providers","dbname":"the_config","score":0,"boost":0,"fields":[{"name":"business_name", "storeInIndex":"true"},{"name":"street_address_1"},{"name":"city"},{"name":"zip"},{"name":"contact_first_name"},{"name":"contact_last_name"}]}'It will return
{"ok":true,"id":"index_policy_for_efile_providers","rev":"1-0fd2f5b2e2012f898df677c68daf4592"}Note that you will need to pass the "rev" parameter if you need to update the index policy. Later, you can retrieve the policy using:

curl http://localhost:5984/the_config/index_policy_for_efile_providersNow you are ready to build the index but let's first start the Jetty with the REST based services via

mvn jetty:run-warNow hop on to browser and point to

http://localhost:8080Finally, you can use curl to build the index via:

curl -vX POST http://localhost:8080/api/index/primary/efile_providersBefore you can query, you will have to specify query policy that is also stored in CouchDB and specifies list of fields that are searched, e.g.

curl -X PUT http://127.0.0.1:5984/the_config/query_policy_for_efile_providers -d \ '{"_id":"query_policy_for_efile_providers","dynamo":"the_config","fields":[{"name":"efile_providers.business_name", "boost":2},{"name":"efile_providers.street_address_1"},{"name":"efile_providers.city"},{"name":"efile_providers.zip"},{"name":"efile_providers.contact_first_name"},{"name":"efile_providers.contact_last_name"}]}'Which will return

{"ok":true,"id":"query_policy_for_efile_providers","rev":"1-618703c1fd66996f23b89c4414dd0842"}Again, you will need to pass "rev" parameter when updating the query policy. Next you can search contents of the index via:

curl "http://localhost:8080/api/search/efile_providers?keywords=mike"Which will return

{"suggestions":[],"keywords":"mike","start":0,"limit":0,"totalHits":7,"docs":[{"_id":"0352d18145532a05714bfec2e1e649dd","dbname":"efile_providers","indexDate":"20091121","doc":"53","score":"0.0","owner":"*","efile_providers.business_name":"Mr Tax Man"},{"_id":"062d548eb394db3534782c5b6ded0529","dbname":"efile_providers","indexDate":"20091121","doc":"96","score":"0.0","owner":"*","efile_providers.business_name":"Liberty Tax Service"},{"_id":"1ddc6006a2315dd0b0119c0dbc22c1a7","dbname":"efile_providers","indexDate":"20091121","doc":"450","score":"0.0","owner":"*","efile_providers.business_name":"1040 PLUS INC"},{"_id":"3621cc7edde5f191bcc5f3a41160f61e","dbname":"efile_providers","indexDate":"20091121","doc":"793","score":"0.0","owner":"*","efile_providers.business_name":"MIKE A PASSECK CPA"},{"_id":"37a2a152ff120ac293ea67daac1a11aa","dbname":"efile_providers","indexDate":"20091121","doc":"811","score":"0.0","owner":"*","efile_providers.business_name":"Liberty Tax Service"},{"_id":"be0fd60800b9eed6d418601f8cba06f3","dbname":"efile_providers","indexDate":"20091121","doc":"2856","score":"0.0","owner":"*","efile_providers.business_name":"Liberty Tax Service"},{"_id":"dfa948236e87d0c6ba90c612cb166635","dbname":"efile_providers","indexDate":"20091121","doc":"3395","score":"0.0","owner":"*","efile_providers.business_name":"MIKE FOLEYS TAX SERVICE"}]}This query functionality can also be tested through a simple html based interface by just pointing your browser to http://localhost:8080/, e.g.

The index stores id of the document that is indexed so you can also retrieve details of each link using

http://localhost:8080/api/storage/efile_providers/0352d18145532a05714bfec2e1e649ddThis feature can be tested from HTML interface by clicking on details link, e.g.

You can also debug why certain results are showing up using following API

http://localhost:8080/api/search/explain/efile_providers?keywords=mikeThis feature can be tested from HTML interface by clicking on explain button, e.g.

Next, you can also find top terms used in the index using:

http://localhost:8080/api/search/rank/efile_providers?limit=1000Again, this feature can be tested from HTML interface by clicking on top terms button, e.g.

You can also find similar searches for a particular search using

http://localhost:8080/api/search/similar/efile_providers?externalId=37a2a152ff120ac293ea67daac1a11aa&luceneId=811&detailedResults=trueWhich will return

ConclusionDocuSearch makes it easy to query documents on CouchDB, however I have also started adding support for Berkley DB if you choose to use it. I found CouchDB wastes a lot of space and is a bit slow so that may be alternative option for some. I also plan to add ngrams and stem based analyzers to create better search experience. I also welcome you to join the project. You can add yourself to http://code.google.com/p/docusearch/ project and start contributing.