Databases
StackOverflow2013
Postsdbo
POSender and receiver to transfer files over ssh on request?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSender and receiver to transfer files over ssh on request?
    primarykey
    data
    text
    <p>I created a program that iterates over a bunch of files and invokes for some of them:</p> <pre><code>scp &lt;file&gt; user@host:&lt;remotefile&gt; </code></pre> <p>However, in my case, there may be thousands of small files that need to transferred, and scp is opening a new ssh connection for each of them, which has quite some overhead.</p> <p>I was wondering if there is no solution where I keep one process running that maintains the connection and I can send it "requests" to copy over single files.</p> <p>Ideally, I'm looking for a combination of some sender and receiver program, such that I can start a single process (1) at the beginning: </p> <pre><code>ssh user@host receiverprogram </code></pre> <p>And for each file, I invoke a command (2):</p> <pre><code>senderprogram &lt;file&gt; &lt;remotefile&gt; </code></pre> <p>and pipe the output of (2) to the input of (1), and this would cause the file to be transferred. In the end, I can just send process (1) some signal to terminate.</p> <p>Preferably the sender and receiver programs are open source C programs for Unix. They may communicate using a socket instead of a pipe, or any other creative solution.</p> <p>However, it is an <strong>important constraint</strong> that each file gets transferred at the moment I iterate over it: it is not acceptable to collect a list of files and then invoke one instance of <code>scp</code> to transfer all the files at once at the end. Also, I have only simple shell access to the receiving host.</p> <p><strong>Update:</strong> I found a solution for the problem of the connection overhead using the multiplexing features of ssh, see my own answer below. Yet, I'm starting a bounty because I'm curious to find if there exists a sender/receiver program as I describe here. It seems there should exist something that can be used, e.g. xmodem/ymodem/zmodem?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBruno De Fraine
    1. USBruno De Fraine
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSender and receiver to transfer files over ssh on request?
      1. USBruno De Fraine
      1. VTBountyStart
    2. VO
      singulars
      1. POSender and receiver to transfer files over ssh on request?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POSender and receiver to transfer files over ssh on request?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COOne minor note - if you end up using scp/ssh as Bruno mentioned, one relative "freebie" for picking up performance is to add the argument "-c arcfour" - this makes the encryption step much more lightweight, and can roughly double througput.
      singulars
      1. POSender and receiver to transfer files over ssh on request?
      1. USjasedit
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload