Databases
StackOverflow2013
Postsdbo
POIs it possible to pass multiple args to multiprocessing.pool?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs it possible to pass multiple args to multiprocessing.pool?
    primarykey
    data
    text
    <blockquote> <p><strong>Possible Duplicate:</strong><br> <a href="https://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments">Python multiprocessing pool.map for multiple arguments</a> </p> </blockquote> <p>I'd like to feed two arguments to subprocesses in a multiprocessing.Pool? I feel like I'm pushing uphill. Is it possible as 2 args, or a tuple, or... at all? It seems to work fine for some, correctly passing out the two filenames (in and out), but then barfs unexpectedly at a variable point. Sadly it's not really working because the output files are all empty - which is not what happens if I call it directly, or single processedly. There is another complicating factor, the called routine is in another, imported, module. Localising that as a 'foo' stub module does fix things but its only printing the args, not trying to do any actual work.</p> <p>Which is perhaps a long way to stubbornly resist learning how to use a Queue, but I'd just like to confirm I'm not going to get anywhere pushing down the path I'm on.</p> <pre><code>fixtures/txt_data/AAD.txt obj/txt_data/AAD.txt fixtures/txt_data/ANZSMW.txt obj/txt_data/ANZSMW.txt fixtures/txt_data/BENPA.txt obj/txt_data/BENPA.txt fixtures/txt_data/CBAIZQ.txt obj/txt_data/CBAIZQ.txt Traceback (most recent call last): File "./jobflow.py", line 60, in &lt;module&gt; main() File "./jobflow.py", line 57, in main args.func(args) File "./jobflow.py", line 40, in market pool.map(foo, market_files()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 225, in map return self.map_async(func, iterable, chunksize).get() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 522, in get raise self._value TypeError: function takes exactly 1 argument (2 given) fixtures/txt_data/CSDO.txt obj/txt_data/CSDO.txt fixtures/txt_data/EMB.txt obj/txt_data/EMB.txt fixtures/txt_data/GGG.txt obj/txt_data/GGG.txt fixtures/txt_data/IDL.txt obj/txt_data/IDL.txt </code></pre> <p>That's an error sample. It halts before the files are exhausted. It alternatively moans that it wants 2 args but only got 1 when I change it to try and pass two args:</p> <pre><code>def foo(c): a, b, = c print a, b market2.file_main((a, b)) # does comment/uncommenting this break it only because it's in another python file? def market(args): """ read raw ticker data files and output nice, clean, more valid ticker data files """ pool = multiprocessing.Pool() class market_files(object): for infile in args.infiles: outfile = os.path.join(args.outdir, os.path.basename(infile)) yield (infile, outfile) pool.map(foo, market_files()) </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USJohn Mee
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTDuplicate
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs it possible to pass multiple args to multiprocessing.pool?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COoh. nice find. thx.
      singulars
      1. POIs it possible to pass multiple args to multiprocessing.pool?
      1. USJohn Mee
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload