Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>If I understand correctly, an <code>lapply</code> solution is probably the way to answer your question. This is the same approach as the answer that you link to, but here's a self-contained example that might be closer to your use case:</p> <p>Load libraries and reproducible data (please include these in your future questions here)</p> <pre><code>library(tm) library(RWeka) data(crude) </code></pre> <p>Your bigram tokenizer...</p> <pre><code>#Tokenizer for n-grams and passed on to the term-document matrix constructor BigramTokenizer &lt;- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2)) txtTdmBi &lt;- TermDocumentMatrix(crude, control = list(tokenize = BigramTokenizer)) </code></pre> <p>Check that it worked by inspecting a random sample...</p> <pre><code>inspect(txtTdmBi[1000:1005, 10:15]) A term-document matrix (6 terms, 6 documents) Non-/sparse entries: 1/35 Sparsity : 97% Maximal term length: 18 Weighting : term frequency (tf) Docs Terms 248 273 349 352 353 368 for their 0 0 0 0 0 0 for west 0 0 0 0 0 0 forced it 0 0 0 0 0 0 forced to 0 0 0 0 0 0 forces trying 1 0 0 0 0 0 foreign investment 0 0 0 0 0 0 </code></pre> <p><strong>Here is the answer to your question:</strong></p> <p>Now use a <code>lapply</code> function to calculate the associated words for every item in the vector of terms in the term-document matrix. The vector of terms is most simply accessed with <code>txtTdmBi$dimnames$Terms</code>. For example <code>txtTdmBi$dimnames$Terms[[1005]]</code> is "foreign investment".</p> <p>Here I've used <code>llply</code> from the <code>plyr</code> package so we can have a progress bar (comforting for big jobs), but it's basically the same as the base <code>lapply</code> function. </p> <pre><code>library(plyr) dat &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocs(txtTdmBi, i, 0.5), .progress = "text" ) </code></pre> <p>The output is a list where each item in the list is a vector of named numbers where the name is the term and the number is the correlation value. For example, to see the terms associated with "foreign investment", we can access the list like so:</p> <pre><code>dat[[1005]] </code></pre> <p>and here are the terms associated with that term (I've just pasted in the top few)</p> <pre><code>168 million 1986 was 1987 early 300 mln 31 pct 1.00 1.00 1.00 1.00 1.00 a bit a crossroads a leading a political a population 1.00 1.00 1.00 1.00 1.00 a reduced a series a slightly about zero activity continues 1.00 1.00 1.00 1.00 1.00 advisers are agricultural sector agriculture the all such also reviews 1.00 1.00 1.00 1.00 1.00 and advisers and attract and imports and liberalised and steel 1.00 1.00 1.00 1.00 1.00 and trade and virtual announced since appears to are equally 1.00 1.00 1.00 1.00 1.00 are recommending areas for areas of as it as steps 1.00 1.00 1.00 1.00 1.00 asia with asian member assesses indonesia attract new balance of 1.00 1.00 1.00 1.00 1.00 </code></pre> <p>Is that what you want to do?</p> <p>Incidentally, if your term-document matrix is very large, you may want to try this version of <code>findAssocs</code>:</p> <pre><code># u is a term document matrix # term is your term # corlimit is a value -1 to 1 findAssocsBig &lt;- function(u, term, corlimit){ suppressWarnings(x.cor &lt;- gamlr::corr(t(u[ !u$dimnames$Terms == term, ]), as.matrix(t(u[ u$dimnames$Terms == term, ])) )) x &lt;- sort(round(x.cor[(x.cor[, term] &gt; corlimit), ], 2), decreasing = TRUE) return(x) } </code></pre> <p>This can be used like so: </p> <pre><code>dat1 &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocsBig(txtTdmBi, i, 0.5), .progress = "text" ) </code></pre> <p>The advantage of this is that it uses a different method of converting the TDM to a matrix <code>tm:findAssocs</code>. This different method uses memory more efficiently and so prevents this kind of message: <code>Error: cannot allocate vector of size 1.9 Gb</code> from occurring. </p> <p>Quick benchmarking shows that both <code>findAssocs</code> functions are about the same speed, so the main difference is in the use of memory:</p> <pre><code>library(microbenchmark) microbenchmark( dat1 &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocsBig(txtTdmBi, i, 0.5)), dat &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocs(txtTdmBi, i, 0.5)), times = 10) Unit: seconds expr min lq median dat1 &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocsBig(txtTdmBi, i, 0.5)) 10.82369 11.03968 11.25492 dat &lt;- llply(txtTdmBi$dimnames$Terms, function(i) findAssocs(txtTdmBi, i, 0.5)) 10.70980 10.85640 11.14156 uq max neval 11.39326 11.89754 10 11.18877 11.97978 10 </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POfindAssocs for multiple terms in R
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USBen
    1. USBen
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POfindAssocs for multiple terms in R
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks Ben! I managed to find the associations for all bigrams in my term-document matrix. However, in the variable `dat` I replaced `txtTdmBi$dimnames$Terms` with `txtTdmBi` since I kept receiving an error: `0%Error in UseMethod("findAssocs", x): no applicable method for 'findAssocs' applied to an object of class "character"`
      singulars
      1. PO
      1. USGrote
    2. COIn addition to this topic; the function `findAssocsBig` in your example is a nifty solution for memory preservation. How can one apply for multiple terms. Also the `lapply` approach?
      singulars
      1. PO
      1. USGrote
    3. COThanks for spotting that mistake, I've corrected my answer accordingly, and added in how you can use the other function.
      singulars
      1. PO
      1. USBen
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload