StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>There are two mechanisms for including other jars to your job's classpath:</p> <p>If you haven't already stored them in HDFS, you can use the GenericOptionsParser's <code>-libjars</code> argument. This will cause the JobClient to upload the jars to a temp directory in HDFS for you job, and include them in the distributed cache for your job. For this to work, you'll need to run your job via the ToolRunner.run interface:</p> <pre><code>public class MyJob extends COnfigured implements Tool { public int run(String args[]) { Job job = new Job(getConf()); // configure your job // .. return job.waitForCompletion() ? 0 : 1; } public static void main(String args[]) throws Exception { ToolRunner.run(new MyJob(), args)); } } </code></pre> <p>Then you'd run your job as follows (adding jars 1-3 to the job classpath):</p> <pre><code>#> hadoop jar myjob.jar MyJob -libjars jar1.jar,jar2.jar,jar3.jar [other args] </code></pre> <p>If your Jars are already in HDFS, then you just need to add the jars the distributed cache:</p> <pre><code>public int run(String args[]) { Job job = new Job(getConf()); // configure your job // .. // acquire job configuration Configuration conf = job.getConf(); // create a FileSystem FileSystem fs = FileSystem.get(fs); DistributedCache.addFileToClassPath(new Path("/myapp/jar1.jar"), conf, fs); DistributedCache.addFileToClassPath(new Path("/myapp/jar2.jar"), conf, fs); DistributedCache.addFileToClassPath(new Path("/myapp/jar3.jar"), conf, fs); return job.waitForCompletion() ? 0 : 1; } </code></pre> <p>The only downside of this second method is that you cannot reference any class in theses jars in your job configuration (unless you have copies client-side too, and you configure the <code>HADOOP_CLASSPATH</code> env variable).</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload