StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>You can try to use <a href="http://search.cpan.org/perldoc/LWP%3a%3aParallel" rel="nofollow">LWP::Parallel</a>. </p> <h3>Update:</h3> <p>I just tried to build it on Windows XP with ActiveState's 5.10.1 and encountered a bunch of test failures some which are due to the <code>TEST</code> script blindly prepending <code>..</code> to all entries in <code>@INC</code> and others seem to be due to a version mismatch with <code>LWP::Protocol::*</code> classes.</p> <p>This is a concern. I might go with <a href="http://search.cpan.org/perldoc/Parallel%3a%3aForkManager" rel="nofollow">Parallel::ForkManager</a> in conjunction with <a href="http://search.cpan.org/perldoc/LWP" rel="nofollow">LWP</a>.</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use Config::Std { def_sep => '=' }; use File::Slurp; use HTTP::Request::Common qw(POST); use LWP::UserAgent; use Parallel::ForkManager; die "No config file specified\n" unless @ARGV; my ($ini) = @ARGV; read_config $ini, my %config; my $pm = Parallel::ForkManager->new(10); my @urls = @{ $config{''}{url} }; for my $url ( @urls ) { $pm->start and next; my $param = [ %{ $config{$url} } ]; my $request = POST $url, $param; my $ua = LWP::UserAgent->new; my $fn = sprintf '%s-%s-%s.xml', map $request->$_, qw( method uri content); $fn =~ s/\W+/_/g; my $response = $ua->request( $request ); if ( $response->code == 200 ) { write_file $fn, \ $response->as_string; } else { warn $response->message, "\n"; } $pm->finish; } $pm->wait_all_children; </code></pre> <p>Here is a sample config file:</p> <pre>url = http://one.example.com/search url = http://two.example.com/query url = http://three.example.com/question [http://one.example.com/search] keyword = Perl limit = 20 [http://two.example.com/query] type = Who is limit = 10 [http://three.example.com/question] use = Perl result = profit</pre> <h3>Update:</h3> <p>If you need to convince yourself that execution is not serial, try the following short script:</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use Parallel::ForkManager; my $pm = Parallel::ForkManager->new(2); for my $sub (1 .. 4) { $pm->start and next; for my $i ('a' .. 'd') { sleep rand 3; print "[$sub]: $i\n"; } $pm->finish; } $pm->wait_all_children; </code></pre> <p>Output:</p> <pre>[1]: a [1]: b [2]: a [1]: c [1]: d [2]: b [3]: a [3]: b [3]: c [2]: c [3]: d [2]: d [4]: a [4]: b [4]: c [4]: d</pre> <p>Regarding your comment about "reliability", I believe it's misguided. What you are doing is simulated by the following script:</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use Parallel::ForkManager; use YAML; my @responses = parallel_run(); print Dump \@responses; sub parallel_run { my $pm = Parallel::ForkManager->new(2); my @responses; for my $sub (1 .. 4) { $pm->start and next; for my $i ('a' .. 'd') { sleep rand 3; push @responses, "[$sub]: $i"; } $pm->finish; } $pm->wait_all_children; return @responses; } </code></pre> <p>The output you get from that will be:</p> <pre>--- []</pre> <p>It is up to you to figure out why. That's why <code>Parallel::ForkManager</code> allows you to register callbacks. Just like the ones you are using with <code>AnyEvent::HTTP</code>.</p> <p>What module you use is your own business. Just don't keep making blatantly false statements.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload