Databases
StackOverflow2013
Postsdbo
POUsing snow (and snowfall) with AWS for parallel processing in R
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUsing snow (and snowfall) with AWS for parallel processing in R
    primarykey
    data
    text
    <p>In relation to my earlier <a href="https://stackoverflow.com/questions/7241244/using-aws-for-parallel-processing-with-r">similar SO question </a>, I tried using snow/snowfall on AWS for parallel computing.</p> <p>What I did was:</p> <ul> <li>In the <code>sfInit()</code> function, I provided the public DNS to <code>socketHosts</code> parameter like so <code>sfInit(parallel=TRUE,socketHosts =list("ec2-00-00-00-000.compute-1.amazonaws.com"))</code></li> <li>The error returned was <code>Permission denied (publickey)</code></li> <li>I then followed the instructions (I presume correctly!) on <a href="http://www.imbi.uni-freiburg.de/parallel/" rel="nofollow noreferrer">http://www.imbi.uni-freiburg.de/parallel/</a> in the 'Passwordless Secure Shell (SSH) login' section</li> <li>I just cat the contents of the .pem file that I created on AWS into the ~/.ssh/authorized_keys of the AWS instance I want to connect to from my master AWS instance and for the master AWS instance as well</li> </ul> <p>Is there anything I am missing out ? I would be very grateful if users can share their experiences in the use of snow on AWS.</p> <p>Thank you very much for your suggestions.</p> <p>UPDATE: I just wanted to update the solution I found to my specific problem:</p> <ul> <li>I used StarCluster to setup my AWS cluster : <a href="http://web.mit.edu/stardev/cluster/docs/0.92rc2/index.html" rel="nofollow noreferrer">StarCluster</a></li> <li>Installed package <code>snowfall</code> on all the nodes of the cluster</li> <li>From the master node issued the following commands</li> <li><code>hostslist &lt;- list("ec2-xxx-xx-xxx-xxx.compute-1.amazonaws.com","ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com")</code></li> <li><code>sfInit(parallel=TRUE, cpus=2, type="SOCK",socketHosts=hostslist)</code></li> <li><code>l &lt;- sfLapply(1:2,function(x)system("ifconfig",intern=T))</code></li> <li><code>lapply(l,function(x)x[2])</code></li> <li><code>sfStop()</code></li> <li>The ip information confirmed that the AWS nodes were being utilized</li> </ul>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USharshsinghal
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. USIterator
      1. VTFavorite
    2. VO
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI believe .pem file to be a X509 certificate, not a RSA public key. You should generate the key pair on the master node, as described in the section, and copy the public key to the authorized_keys of the slave node(s).
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. USAnatoliy
    2. COI believe @Anatoliy is correct; he should make an answer of that, rather than just a comment, so we can upvote. :)
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. USIterator
    3. COPerhaps disregard my answer on your other question regarding "use StarCluster" as I now see you have, but give a shot running the whole cluster within the private IP range, I had no need to fuss with keys or certificates once I started doing that.
      singulars
      1. POUsing snow (and snowfall) with AWS for parallel processing in R
      1. USThell
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload