Databases
StackOverflow2013
Postsdbo
POBetter way of calculating document Similarity using Lucene
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POBetter way of calculating document Similarity using Lucene
    primarykey
    data
    text
    <p>I’m indexing a collection of documents using Lucene by specifying TermVector at indexing time. Then I retrieve terms and their frequencies by reading the index and calculating TF-IDF score vectors for each document. Then, using the TF-IDF vectors, I calculate pairwise cosine similarity between documents using <a href="http://en.wikipedia.org/wiki/Cosine_similarity" rel="nofollow">Wikipedia's cosine similarity equation</a>.</p> <p>This is my problem: Say I have two identical documents “A” and “B” in this collection (A and B have more than 200 sentences). If I calculate pairwise cosine similarity between A and B it gives me cosine value=1 which is perfectly OK. But if I remove a single sentence from Doc “B”, it gives me cosine similarity value around 0.85 between these two documents. The documents are almost similar but cosine values are not. I understand the problem is with the equation that I’m using. </p> <p>Is there better way / equation that I can use for calculating cosine similarity between documents? </p> <p><strong>Edited</strong></p> <p>This is how I calculate Cosine Similarity, <code>doc1[]</code> and <code>doc2[]</code> are TF-IDF vectors for corresponding document. the vector contains only the <code>scores</code> but not the <code>words</code> </p> <pre><code>private double cosineSimBetweenTwoDocs(float doc1[], float doc2[]) { double temp; int doc1Len = doc1.length; int doc2Len = doc2.length; float numerator = 0; float temSumDoc1 = 0; float temSumDoc2 = 0; double equlideanNormOfDoc1 = 0; double equlideanNormOfDoc2 = 0; if (doc1Len &gt; doc2Len) { for (int i = 0; i &lt; doc2Len; i++) { numerator += doc1[i] * doc2[i]; temSumDoc1 += doc1[i] * doc1[i]; temSumDoc2 += doc2[i] * doc2[i]; } equlideanNormOfDoc1=Math.sqrt(temSumDoc1); equlideanNormOfDoc2=Math.sqrt(temSumDoc2); } else { for (int i = 0; i &lt; doc1Len; i++) { numerator += doc1[i] * doc2[i]; temSumDoc1 += doc1[i] * doc1[i]; temSumDoc2 += doc2[i] * doc2[i]; } equlideanNormOfDoc1=Math.sqrt(temSumDoc1); equlideanNormOfDoc2=Math.sqrt(temSumDoc2); } temp = numerator / (equlideanNormOfDoc1 * equlideanNormOfDoc2); return temp; } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USDavid J.
    1. USKasun
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. USHelium
      1. VTFavorite
    2. VO
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI guess something is wrong about your code. Removing one sentence from 200 sentences should give you a number > 0.98. To verify it, you can generate a random vector, make a modification to the vector and compute the cosine similarity for it to see what you get. For a vector of size 1000, and random numbers in the range [10,100], if I subtract a random number in the range [10,20] from all the numbers in the vector, the resulting similarity measure is always > 0.98 for me.
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. USHelium
    2. COI used Mathematica to verify the case. Here is my code: a = RandomInteger[{10, 100}, 1000]; b = a - RandomInteger[{10, 20}, 1000]; {Total[a], Total[b], Total[a - b], N[(a.b)/(Norm[a] Norm[b])]}, and here is the output: {55419, 40271, 15148, 0.98811}
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. USHelium
    3. CO@Mohsen Removing One sentences from the Vector B will reduce the number of elements in that vector, if we get a vector of size 1000 after removing sentences the size of vector B will become say 995, and now vector A is size of 1000 but, two vectors are not aligned too. By removing a sentence, the vector elements are removed from middle but not from end of the vector. So if you can try by removing vector elements from middle, you can observe 0.85 value
      singulars
      1. POBetter way of calculating document Similarity using Lucene
      1. USKasun
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload