Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>There are several tricks to make <em>k</em>-means work for text:</p> <ol> <li>Get rid of the terms that occur in only a few documents (that have low df). These artificially blow up the distances in vector space.</li> <li>Normalize vectors. That helps a bit, since it gets rid of length differences between documents. It also makes document vectors have similar variances, as pointed out by @Anony-Mousse.</li> <li>Perform dimensionality reduction using <a href="http://nlp.stanford.edu/IR-book/pdf/18lsi.pdf" rel="noreferrer">LSA</a>, aka truncated <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition" rel="noreferrer">SVD</a>, before doing the actual clustering. That helps a lot. (Be sure to normalize the LSA results as well.)</li> </ol> <p>Short explanation of why normalization works: suppose you have three documents {d₁, d₂, d₃}, and the tiny vocabulary {cat, dog, tax}. The term-document matrix (raw counts or tf-idf, doesn't matter) looks like</p> <pre><code> | cat | dog | tax d₁ | 100 | 100 | 0 d₂ | 10 | 10 | 0 d₃ | 0 | 0 | 100 </code></pre> <p>Now we're going to do 2-means. We can reasonably expect to find a pets cluster {d₁, d₂} and a finance singleton cluster {d₃}. However, the distances between the pairs are</p> <pre><code>D(d₁, d₂) = 127.28 D(d₁, d₃) = 173.21 D(d₂, d₃) = 101.00 </code></pre> <p>so a density-based method like <em>k</em>-means will tend to group d₂ with d₃. By normalizing the vectors, you effectively map both d₁ and d₂ to the same vector [0.71, 0.71, 0] so D(d₁, d₂) = 0 and they will always be in the same cluster.</p> <p>(<em>k</em>-means applied to normalized vectors is sometimes called "spherical" <em>k</em>-means because unit vectors lie on a hypersphere centered at the origin.)</p>
    singulars
    1. This table or related slice is empty.
    1. PODocument Clustering in Java using Weka
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USFred Foo
    1. USFred Foo
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PODocument Clustering in Java using Weka
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COReally appreciate the info. I'm already normalizing, the tfidf values for my keywords are floats between 0 and 1. My feature selection is pretty dumb though, as I mentioned in my explanation my dumb initial implementation is to consider any term that is in the top 30 for any article a feature and use it in clustering. I will experiment with LSA tonight, examples are pretty sparse so any pointers to using LSA in Java code would be super helpful. Per @Anony-Mousse's question, are you confident that K-Means will be a good fit with good feature selection or should I try other clusterers?
      singulars
      1. PO
      1. USTyson
    2. COI played with #1/3 (trim low frequency, use LSA to reduce dimentionality) and the results are slightly better. 44% of the articles still fall into one cluster, but there are 7 clusters with 5+ articles which seems to fit the data better. How do I map the post LSA instances to my original ones? Do I depend on index and assume that order is preserved?
      singulars
      1. PO
      1. USTyson
    3. CO@Tyson: there's no reason why the order should not be preserved (unless you have a really buggy implementation).
      singulars
      1. PO
      1. USFred Foo
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload