StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POGenerating a very large matrix of string combinations using combn() and bigmemory package
text
Body
copied!<p>I have a vector x of 1,344 unique strings. I want to generate a matrix that gives me all possible groups of three values, regardless of order, and export that to a csv. </p> <p>I'm running R on EC2 on a m1.large instance w 64bit Ubuntu. When using combn(x, 3) I get an out of memory error: </p> <pre><code>Error: cannot allocate vector of size 9.0 Gb </code></pre> <p>The size of the resulting matrix is C1344,3 = 403,716,544 rows and three columns - which is the transpose of the result of combn() function.</p> <p>I thought of using the bigmemory package to create a file backed big.matrix so I can then assign the results of the combn() function. I can create a preallocated big matrix:</p> <pre><code>library(bigmemory) x <- as.character(1:1344) combos <- 403716544 test <- filebacked.big.matrix(nrow = combos, ncol = 3, init = 0, backingfile = "test.matrix") </code></pre> <p>But when I try to allocate the values <code>test <- combn(x, 3)</code> I still get the same: <code>Error: cannot allocate vector of size 9.0 Gb</code></p> <p>I even tried coercing the result of <code>combn(x,3)</code> but I think that because the combn() function is returning an error, the big.matrix function doesn't work either.</p> <pre><code>test <- as.big.matrix(matrix(combn(x, 3)), backingfile = "abc") Error: cannot allocate vector of size 9.0 Gb Error in as.big.matrix(matrix(combn(x, 3)), backingfile = "abc") : error in evaluating the argument 'x' in selecting a method for function 'as.big.matrix' </code></pre> <p>Is there a way to combine these two functions together to get what I need? Are there any other ways of achieving this? Thanks.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload