StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POComputing sparse pairwise distance matrix in R
primarykey
Id
5560218
data
AcceptedAnswerId
5569719
AnswerCount
3
ClosedDate
CommentCount
4
CommunityOwnedDate
CreationDate
2011-04-06T00:34:51.637
FavoriteCount
7
LastActivityDate
2011-04-07T15:54:59.147
LastEditDate
2017-05-23T10:29:58.373
LastEditorUserId
-1
OwnerUserId
84458
ParentId
0
PostTypeId
1
Score
21
ViewCount
6323
LastEditorDisplayName
text
Body
I have a <code>NxM</code> matrix and I want to compute the <code>NxN</code> matrix of Euclidean distances between the <code>M</code> points. In my problem, <code>N</code> is about 100,000. As I plan to use this matrix for a k-nearest neighbor algorithm, I only need to keep the <code>k</code> smallest distances, so the resulting <code>NxN</code> matrix is very sparse. This is in contrast to what comes out of <code>dist()</code>, for example, which would result in a dense matrix (and probably storage problems for my size <code>N</code>). The packages for kNN that I've found so far (<code>knnflex</code>, <code>kknn</code>, etc) all appear to use dense matrices. Also, the <code>Matrix</code> package does not offer a pairwise distance function. Closer to my goal, I see that the <code>spam</code> package has a <code>nearest.dist()</code> function that allows one to only consider distances less than some threshold, <code>delta</code>. In my case, however, a particular value of <code>delta</code> may produce too many distances (so that I have to store the <code>NxN</code> matrix densely) or too few distances (so that I can't use kNN). I have seen previous discussion on trying to perform <a href="https://stackoverflow.com/questions/3039646/k-means-clustering-in-r-on-very-large-sparse-matrix">k-means clustering</a> using the <code>bigmemory/biganalytics</code> packages, but it doesn't seem like I can leverage these methods in this case. Does anybody know a function/implementation that will compute a distance matrix in a sparse fashion in R? My (dreaded) backup plan is to have two <code>for</code> loops and save results in a <code>Matrix</code> object.
Tags
<r><distance><sparse-matrix><knn>
Title
Computing sparse pairwise distance matrix in R
singulars
PostAcceptedAnswerId
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. USCommunity
UserOwnerUserId
1. USChristopher DuBois
plurals
PostLinksPostIdRelatedPostId
1. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
2. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
PostLinksRelatedPostIdPostId
1. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
2. PO
 singulars
 PostTypePostTypeId
 PTAnswer
3. PO
 singulars
 PostTypePostTypeId
 PTAnswer
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 POComputing sparse pairwise distance matrix in R
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
2. VO
 singulars
 PostPostId
 POComputing sparse pairwise distance matrix in R
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
3. VO
 singulars
 PostPostId
 POComputing sparse pairwise distance matrix in R
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.