Databases
StackOverflow2013
Postsdbo
PORun parallel Invoke-WebRequest jobs in PowerShell v3
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORun parallel Invoke-WebRequest jobs in PowerShell v3
    primarykey
    data
    text
    <p>Running simultaneous background jobs in PowerShell in pretty straightforward, but I cannot seem to get it working with the new (in v3) cmdlet Invoke-WebRequest.</p> <p>I have several thousand files I am downloading scriptomagically via PowerShell. Works great, but doing them in series is taking days:</p> <pre><code>for($f=0;$f -lt $urlList.Count;$f++) { $remote = $urlList[$f] + $fileList[$f] $local = 'C:\folder\' + $fileList[$f] Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox } </code></pre> <p>I've made several attempts at leveraging the the 'AsJob' method, but either they error-out or complete fine yet no local files get saved. Here's an example of the latter:</p> <pre><code>for($f=0;$f -lt $urlList.Count;$f++) { $remote = $urlList[$f] + $fileList[$f] $local = 'C:\folder\' + $fileList[$f] $command = "Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox" Start-Job {Invoke-Expression -Command $command} } Get-Job|Wait-Job </code></pre> <p>Output examples:</p> <pre><code>Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 339 Job339 BackgroundJob Running True localhost Invoke-Expression -Com... 341 Job341 BackgroundJob Running True localhost Invoke-Expression -Com... 343 Job343 BackgroundJob Running True localhost Invoke-Expression -Com... 339 Job355 BackgroundJob Completed True localhost Invoke-Expression -Com... 341 Job357 BackgroundJob Completed True localhost Invoke-Expression -Com... 343 Job359 BackgroundJob Completed True localhost Invoke-Expression -Com... </code></pre> <p>The weird thing is, the jobs above complete roughly as fast as it would take to download the associated file...so it seems that data is going somewhere -- just not to my save location. I assume it's going to memory but being dumped without being saved. Tried adding and removing the 'PassThru' option, but get the same results either way. Also tried piping it to Out-File, still no joy. Not sure why this particular cmdlet is being such a bugger.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USTrevor Sullivan
    1. USLocal Needs
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORun parallel Invoke-WebRequest jobs in PowerShell v3
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORun parallel Invoke-WebRequest jobs in PowerShell v3
      1. USRob Kielty
      1. VTFavorite
    3. VO
      singulars
      1. PORun parallel Invoke-WebRequest jobs in PowerShell v3
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload