Databases
StackOverflow2013
Postsdbo
POLarge-scale pseudoinverse
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLarge-scale pseudoinverse
    primarykey
    data
    text
    <p>I would like to compute the <a href="http://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse" rel="nofollow noreferrer">Moore–Penrose pseudoinverse</a> of an enormous matrix. Ideally, I would like to do it on a matrix that has 23 million rows and 1000 columns, but if necessary I can reduce the number of rows to 4 million by only running on one part of my experiment.</p> <p>Obviously, loading the matrix in to memory and running SVD on it is not going to work. <a href="http://en.wikipedia.org/wiki/Block_matrix_pseudoinverse" rel="nofollow noreferrer">Wikipedia</a> points to <a href="http://en.wikipedia.org/wiki/Krylov_subspace" rel="nofollow noreferrer">Krylov subspace</a> methods and mention the <a href="http://en.wikipedia.org/wiki/Arnoldi_iteration" rel="nofollow noreferrer">Arnoldi</a>, <a href="http://en.wikipedia.org/wiki/Lanczos_iteration" rel="nofollow noreferrer">Lanczos</a>, <a href="http://en.wikipedia.org/wiki/Conjugate_gradient" rel="nofollow noreferrer">Conjugate gradient</a>, <a href="http://en.wikipedia.org/wiki/GMRES" rel="nofollow noreferrer">GMRES</a> (generalized minimum residual), BiCGSTAB (biconjugate gradient stabilized), QMR (quasi minimal residual), TFQMR (transpose-free QMR), and MINRES (minimal residual) methods as being among the best Krylov subspace methods. But I don't know where to go from here. Is computing the pseudoinverse of such a huge matrix even feasible? If so, using which algorithms or software libraries? I have a large computing cluster available, so parallel approaches are welcome.</p> <p><a href="https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix#3172815">This answer</a> points to the R package <a href="http://cran.r-project.org/web/packages/biglm/index.html" rel="nofollow noreferrer">biglm</a>. Would that work? Has anyone used it? I normally work in Python, but don't mind using other languages and tools for this particular task.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USVebjorn Ljosa
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POLarge-scale pseudoinverse
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POLarge-scale pseudoinverse
      1. USgatoatigrado
      1. VTFavorite
    3. VO
      singulars
      1. POLarge-scale pseudoinverse
      1. This table or related slice is empty.
      1. VTUpMod
    1. CODoes the matrix have any special block structure?
      singulars
      1. POLarge-scale pseudoinverse
      1. USJoel
    2. CO@Joel: No, almost all elements are nonzero.
      singulars
      1. POLarge-scale pseudoinverse
      1. USVebjorn Ljosa
    3. COIs the pseudoinverse the end product or are you computing something with it?
      singulars
      1. POLarge-scale pseudoinverse
      1. USuser382751
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload