StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POIn Hadoop, if you want to keep the values from each key-value pair into an Array, why are all the elements that you add the same?
text
Body
copied!<p>I'm trying to store the values from the key-value pairs that the Map function gets and use them further. Given the following input:</p> <pre><code>Hello hadoop goodbye hadoop Hello world goodbye world Hello thinker goodbye thinker </code></pre> <p>An the following code:</p> <p><strong>Note</strong> - the map is the simple WordCount example</p> <pre><code>public class Inception extends Configured implements Tool{ public Path workingPath; public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); // initialising the arrays that contain the values and the keys public ArrayList<LongWritable> keyBuff = new ArrayList<LongWritable>(); public ArrayList<Text> valueBuff = new ArrayList<Text>(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); System.out.println(word + " / " + one); } } public void innerMap(LongWritable key, Text value, Context context) throws IOException, InterruptedException { // adding the value to the bufferr valueBuff.add(value); System.out.println("ArrayList addValue -> " + value); for (Text v : valueBuff){ System.out.println("ArrayList containedValue -> " + value); } keyBuff.add(key); } public void run(Context context) throws IOException, InterruptedException { setup(context); // going over the key-value pairs and storing them into the arrays while(context.nextKeyValue()){ innerMap(context.getCurrentKey(), context.getCurrentValue(), context); } Iterator itrv = valueBuff.iterator(); Iterator itrk = keyBuff.iterator(); while(itrv.hasNext()){ LongWritable nextk = (LongWritable) itrk.next(); Text nextv = (Text) itrv.next(); System.out.println("Value iterator -> " + nextv); System.out.println("Key iterator -> " + nextk); // iterating over the values and running the map on them. map(nextk, nextv, context); } cleanup(context); } } public int run(String[] args) throws Exception { ... } public static void main (..) { ... } </code></pre> <p>Ok, now the log output:</p> <p><strong>stdout logs</strong></p> <pre><code>ArrayList addValue -> Hello hadoop goodbye hadoop ArrayList containedValue -> Hello hadoop goodbye hadoop ArrayList addValue -> Hello world goodbye world ArrayList containedValue -> Hello world goodbye world ArrayList containedValue -> Hello world goodbye world ArrayList addValue -> Hello thinker goodbye thinker ArrayList containedValue -> Hello thinker goodbye thinker ArrayList containedValue -> Hello thinker goodbye thinker ArrayList containedValue -> Hello thinker goodbye thinker Value iterator -> Hello thinker goodbye thinker Key iterator -> 84 Hello / 1 thinker / 1 goodbye / 1 thinker / 1 Value iterator -> Hello thinker goodbye thinker Key iterator -> 84 Hello / 1 thinker / 1 goodbye / 1 thinker / 1 Value iterator -> Hello thinker goodbye thinker Key iterator -> 84 Hello / 1 thinker / 1 goodbye / 1 thinker / 1 </code></pre> <p>So what you can notice is, that every time i add a new value to the ArrayList valueBuff, all the values in the list gets overwritten. Does anybody know why this is going on an why the values are not added properly in the Array?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload