Databases
StackOverflow2013
Postsdbo
POMultipleOutputs not writing to files, what am I doing wrong?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMultipleOutputs not writing to files, what am I doing wrong?
    text
    copied!<p>I'm basically trying to write out my own lines of Tab separated values to 3 different output files. Despite defining and sriting different named outputs, all files are still named "part-r-*"</p> <p>All code has been anonymized and condensed</p> <p>Driver class looks like this:</p> <pre><code>// Set up job Job job = new Job(conf, MyDriver.class.getSimpleName()); job.setJarByClass(MyJar.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); // Set mappers and reducers job.setMapperClass(MyMapper.class); job.setReducerClass(MyReducer.class); job.setInputFormatClass(TextInputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1]); // Set the 3 different outputs MultipleOutputs.addNamedOutput(job, "out1", TextOutputFormat.class, Text.class, Text.class); MultipleOutputs.addNamedOutput(job, "out2", TextOutputFormat.class, Text.class, Text.class); MultipleOutputs.addNamedOutput(job, "errors", TextOutputFormat.class, Text.class, Text.class); // Run System.exit(job.waitForCompletion(true) ? 0 : 1); </code></pre> <p>The reducer sets up the MultipleOutput in the setup() method:</p> <pre><code>public void setup(Context context) { // Set up multiple output files multiOut = new MultipleOutputs&lt;Text,Text&gt;(context); } </code></pre> <p>And called later from 3 separate methods each pertaining to the different outputs, for example, errors is simplest:</p> <pre><code>private void writeError(String error) { System.out.println("Writing: " + error); multiOut.write("errors", new Text(error), new Text()); } </code></pre> <p>The issue is that I can see that I have proper output from that System.out.println in my logs, but nothing actually gets written to file. I even call multiOut.close() in the cleanup.</p> <p>I imagine I'm just missing something in the driver class? Let me know if I need to provide any additional data.</p> <p>If its worth noting, this is being run on Amazon EMR/S3, but I dont think that's the issue.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload