StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>The docs say to use <a href="http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html" rel="nofollow"><code>org.apache.hadoop.mapreduce.lib.output.MultipleOutputs</code></a> instead.</p> <p>Below is a snippet of code that uses MultipleOutputs. Unfortunately I didn't write it and haven't spent much time with it... So I don't know exactly why things are where. I share with the hopes it helps. :)</p> <p>Job Setup</p> <pre><code>job.setJobName("Job Name"); job.setJarByClass(ETLManager.class); job.setMapOutputKeyClass(Text.class); job.setOutputKeyClass(NullWritable.class); job.setMapOutputValueClass(MyThing.class); job.setMapperClass(MyThingMapper.class); job.setReducerClass(MyThingReducer.class); MultipleOutputs.addNamedOutput(job, Constants.MyThing_NAMED_OUTPUT, TextOutputFormat.class, NullWritable.class, Text.class); job.setInputFormatClass(MyInputFormat.class); FileInputFormat.addInputPath(job, new Path(conf.get("input"))); FileOutputFormat.setOutputPath(job, new Path(String.format("%s/%s", conf.get("output"), Constants.MyThing_NAMED_OUTPUT))); </code></pre> <p>Reducer Setup</p> <pre><code>public class MyThingReducer extends Reducer<Text, MyThing, NullWritable, NullWritable> { private MultipleOutputs m_multipleOutputs; @Override public void setup(Context context) { m_multipleOutputs = new MultipleOutputs(context); } @Override public void cleanup(Context context) throws IOException, InterruptedException { if (m_multipleOutputs != null) { m_multipleOutputs.close(); } } @Override public void reduce(Text key, Iterable<MyThing> values, Context context)throws IOException, InterruptedException { for (MyThing myThing : values) { m_multipleOutputs.write(Constants.MyThing_NAMED_OUTPUT, EMPTY_KEY, generateData(context, myThing), generateFileName(context, myThing)); context.progress(); } } } </code></pre> <p>EDIT: Added link to MultipleOutputs.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload