Databases
StackOverflow2013
Postsdbo
POHow do I queue FTP commands in Twisted?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow do I queue FTP commands in Twisted?
    primarykey
    data
    text
    <p>I'm writing an FTP client using Twisted that downloads a lot of files and I'm trying to do it pretty intelligently. However, I've been having the problem that I'll download several files very quickly (sometimes ~20 per batch, sometimes ~250) and then the downloading will hang, only to eventually have connections time out and then the download and hang start all over again. I'm using a DeferredSemaphore to only download 3 files at a time, but I now suspect that this is probably not the right way to avoid throttling the server.</p> <p>Here is the code in question:</p> <pre><code>def downloadFiles(self, result, directory): # make download directory if it doesn't already exist if not os.path.exists(directory['filename']): os.makedirs(directory['filename']) log.msg("Downloading files in %r..." % directory['filename']) files = filterFiles(None, self.fileListProtocol) # from http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker/2862440#2862440 # use a DeferredSemaphore to limit the number of files downloaded simultaneously from the directory to 3 sem = DeferredSemaphore(3) jobs = [sem.run(self.downloadFile, f, directory) for f in files] d = gatherResults(jobs) return d def downloadFile(self, f, directory): filename = os.path.join(directory['filename'], f['filename']).encode('ascii') log.msg('Downloading %r...' % filename) d = self.ftpClient.retrieveFile(filename, FTPFile(filename)) return d </code></pre> <p>You'll noticed that I'm reusing an FTP connection (active, by the way) and using my own FTPFile instance to make sure the local file object gets closed when the file download connection is 'lost' (ie completed). Looking at <a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.protocols.ftp.FTPClient.html" rel="nofollow">FTPClient</a> I wonder if I should be using <a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.protocols.ftp.FTPClientBasic.html#queueCommand" rel="nofollow">queueCommand</a> directly. To be honest, I got lost following the <a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.protocols.ftp.FTPClient.html#retrieveFile" rel="nofollow">retrieveFile</a> command to <a href="http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.0.0//twisted/protocols/ftp.py#L2342" rel="nofollow">_openDataConnection</a> and beyond, so maybe it's already being used.</p> <p>Any suggestions? Thanks!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USpr1001
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow do I queue FTP commands in Twisted?
      1. USquick_dry
      1. VTFavorite
    2. VO
      singulars
      1. POHow do I queue FTP commands in Twisted?
      1. USunutbu
      1. VTFavorite
    3. VO
      singulars
      1. POHow do I queue FTP commands in Twisted?
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload