StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POMapReduce: result is not complete
text
Body
copied!<p>The contents of wcin_file:</p> <pre><code>Run 1 access 1 default 2 out 2 project 1 task 1 windows 1 your 1 </code></pre> <p>I want to use MapReduce to sort these data in the file <strong>wcin_file</strong> descendingly by the second fild, just as follows:</p> <pre><code>default 2 out 2 access 1 ... </code></pre> <p>But I found the output files only contain two rows:</p> <pre><code>default 2 Run 1 </code></pre> <p>Why? Here is some of the source code:</p> <p><em>SortLogsMapper</em></p> <pre><code>public static class SortLogsMapper extends Mapper<Object, Text, Text, IntWritable> { public void map(Object key, Text value, Context context) throws IOException, InterruptedException { context.write(value, new IntWritable(0)); //the content of value is just every line, just as `Run 1`, `access 1` etc. } } </code></pre> <p><em>SortLogsReducer</em></p> <pre><code>public static class SortLogsReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private Text k = new Text(); private IntWritable v = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { k.set(key.toString().split(" ")[0]); //split to get the first filed v.set(Integer.parseInt(key.toString().split(" ")[1])); //second filed context.write(k, v); } } </code></pre> <p><em>LogDescComparator</em></p> <pre><code>public static class LogDescComparator extends WritableComparator { protected LogDescComparator() { super(Text.class, true); } @Override public int compare(WritableComparable w1, WritableComparable w2) { Text t1 = (Text) w1; Text t2 = (Text) w2; String[] t1Items = t1.toString().split("\t| "); String[] t2Items = t2.toString().split("\t| "); Integer t1Value = Integer.parseInt(t1Items[1]); Integer t2Value = Integer.parseInt(t2Items[1]); int comp = t2Value.compareTo(t1Value); return comp; </code></pre> <p><em>Then I started the job in the main function:</em></p> <pre><code>Job job2 = new Job(conf2, "sort"); job2.setNumReduceTasks(1); job2.setJarByClass(WordCount.class); job2.setMapperClass(SortLogsMapper.class); job2.setReducerClass(SortLogsReducer.class); job2.setSortComparatorClass(LogDescComparator.class); job2.setOutputKeyClass(Text.class); job2.setOutputValueClass(IntWritable.class); FileInputFormat.setInputPaths(job2, new Path("wcin_file")); FileOutputFormat.setOutputPath(job2, new Path("wcout")); System.exit(job2.waitForCompletion(true) ? 0 : 1); </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload