Note that there are some explanatory texts on larger screens.

plurals
  1. POEffective clustering of a similarity matrix
    primarykey
    data
    text
    <p>my topic is similarity and clustering of (a bunch of) text(s). In a nutshell: I want to cluster collected texts together and they should appear in meaningful clusters at the end. To do this, my approach up to now is as follows, my problem is in the clustering. The current software is written in php.</p> <p>1) Similarity: I treat every document as a "bag-of-words" and convert words into vectors. I use</p> <ul> <li>filtering (only "real" words)</li> <li>tokenization (split sentences into words)</li> <li>stemming (reduce words to their base form; Porter's stemmer)</li> <li>pruning (cut of words with too high &amp; low frequency)</li> </ul> <p>as methods for dimensionality reduction. After that, I'm using cosine similarity (as suggested / described on various sites on the web <a href="https://stackoverflow.com/a/1290286/519270">and here</a>.</p> <p>The result then is a similarity matrix like this:</p> <pre><code> A B C D E A 0 30 51 75 80 B X 0 21 55 70 C X X 0 25 10 D X X X 0 15 E X X X X 0 </code></pre> <p>A…E are my texts and the number is the similarity in percent; the higher, the more similar the texts are. Because sim(A,B) == sim(B,A) only half of the matrix is filled in. So the similarity of Text A to Text D is 71%.</p> <p>I want to generate a a priori unknown(!) number of clusters out of this matrix now. The clusters should represent the similar items (up to a certain stopp criterion) together.</p> <p>I tried a basic implementation myself, which was basically like this (60% as a fixed similarity threshold)</p> <pre><code> foreach article get similar entries where sim &gt; 60 foreach similar entry check if one of the entries already has a cluster number if no: assign new cluster number to all similar entries if yes: use that number </code></pre> <p>It worked (somehow), but wasn't good at all and the results were often monster-clusters. So, I want to redo this and already had a look into all kinds of clustering algorithms, but I'm still not sure which one will work best. I think it should be an agglomerative algoritm, because every pair of texts can be seen as a cluster in the beginning. But still the questions are what the stopp criterion is and if the algorithm should divide and / or merge existing clusters together.</p> <p>Sorry if some of the stuff seems basic, but I am relatively new in this field. Thanks for the help.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload