StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow do I index all the revisions of a couchdb doc using elasticsearch river plugin
text
Body
copied!<p>I know how to set up the river plugin and search across it. The problem is if the same document is edited multiple times (multiple revisions), the data from oldest revision is retained and older data is lost. I intend to be able keep an index all revisions for my entire couchdb, so I don'thave to keep the history on my couchdb and retrieve history on a doc using elasticsearch and not have to go to the futon. I know the issue will be to uniqely determine a key for a couchdb doc while indexing, but we can append the "revision" number to the key and every key will be unique.</p> <p>I couldn't find for a way to do that in any documentation. Does anyone have an idea as to how to do it.</p> <p>Any suggestions/thoughts are welcome.</p> <p>EDIT 1 : To be more explicit, at the moment elasticsearch saves couchdb docs like this:</p> <pre><code>"_index": "foo", "_type": "foo", "_id": "27fd33f3f51e16c0262e333f2002580a", "_score": 1.0310782, "_source": { "barVal": "bar", "_rev": "3-d10004227969c8073bc573c33e7e5cfd", "_id": "27fd33f3f51e16c0262e333f2002580a", </code></pre> <p>here the _id from couchdb is same as _id for search index. I want the search index to be concat("_id","_rev") from couchdb.</p> <p>EDIT 2: (after trying out @DaveS solution) So I tried the following, but It didn't work - the search still indexes it based on the couchdb's _id</p> <p>What I did:</p> <pre><code>curl -XDELETE 127.0.0.1:9200/_all curl -XPUT 'localhost:9200/foo_test' -d '{ "mappings": { "foo_test": { "_id": { "path": "newId", "index": "not_analyzed", "store": "yes" } } } }' curl -XPUT 'localhost: 9200/_river/foo_test/_meta' -d '{ "type": "couchdb", "couchdb": { "host": "127.0.0.1", "port": 5984, "db": "foo_test", "script": "ctx.doc.newId = ctx.doc._id + ctx.doc._rev", "filter": null }, "index": { "index": "foo_test", "type": "foo_test", "bulk_size": "100", "bulk_timeout": "10ms" } }' </code></pre> <p>And after this, when I search for a doc I added, I get:</p> <pre><code>_index: foo_test _type: foo_test _id: 53fa6fcf981a01b05387e680ac4a2efa _score: 8.238497 _source: { _rev: 4-8f8808f84eebd0984d269318ad21de93 content: { foo: bar foo3: bar3 foo2: bar2 } _id: 53fa6fcf981a01b05387e680ac4a2efa newId: 53fa6fcf981a01b05387e680ac4a2efa4-8f8808f84eebd0984d269318ad21de93 </code></pre> <p>@DaveS - Hope this helps in explaining that elasticsearch is not not using the new path to define its "_id" field.</p> <p>EDIT 3 - for @dadoonet. Hope this helps</p> <p>This is how you get all older rev info for a couchdb. Then you can iterate through the ones available and get their data and index them:</p> <ol> <li><p>Get a list of all revisions on a doc id:</p> <p>curl http://:5984/testdb/cde07b966fa7f32433d33b8d16000ecd?revs_info=true {"_id":"cde07b966fa7f32433d33b8d16000ecd", "_rev":"2-16e89e657d637c67749c8dd9375e662f", "foo":"bar", "foo2":"bar2", "_revs_info":[ {"rev":"2-16e89e657d637c67749c8dd9375e662f", "status":"available"}, {"rev":"1-4c6114c65e295552ab1019e2b046b10e", "status":"available"}]}</p></li> </ol> <p>And then you can retrieve each version by (if the status is available):</p> <pre><code>curl http://<foo>:5984/testdb/cde07b966fa7f32433d33b8d16000ecd?rev=1-4c6114c65e295552ab1019e2b046b10e {"_id":"cde07b966fa7f32433d33b8d16000ecd", "_rev":"1-4c6114c65e295552ab1019e2b046b10e", "foo":"bar"} curl http://<foo>:5984/testdb/cde07b966fa7f32433d33b8d16000ecd?rev=2-16e89e657d637c67749c8dd9375e662f {"_id":"cde07b966fa7f32433d33b8d16000ecd", "_rev":"2-16e89e657d637c67749c8dd9375e662f", "foo":"bar", "foo2":"bar2"} </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload