Databases
StackOverflow2013
Postsdbo
POHadoop 2 IOException only when trying to open supposed cache files
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHadoop 2 IOException only when trying to open supposed cache files
    text
    copied!<p>I recent updated to hadoop 2.2 (using this tutorial <a href="http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1" rel="nofollow">here</a>).</p> <p>My main job class looks like so, and throws an IOException:</p> <pre><code>import java.io.*; import java.net.*; import java.util.*; import java.util.regex.*; import org.apache.hadoop.conf.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.chain.*; import org.apache.hadoop.mapreduce.lib.input.*; import org.apache.hadoop.mapreduce.lib.output.*; import org.apache.hadoop.mapreduce.lib.reduce.*; public class UFOLocation2 { public static class MapClass extends Mapper&lt;LongWritable, Text, Text, LongWritable&gt; { private final static LongWritable one = new LongWritable(1); private static Pattern locationPattern = Pattern.compile("[a-zA-Z]{2}[^a-zA-Z]*$"); private Map&lt;String, String&gt; stateNames; @Override public void setup(Context context) { try { URI[] cacheFiles = context.getCacheFiles(); setupStateMap(cacheFiles[0].toString()); } catch (IOException ioe) { System.err.println("Error reading state file."); System.exit(1); } } public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = line.split("\t"); String location = fields[2].trim(); if (location.length() &gt;= 2) { Matcher matcher = locationPattern.matcher(location); if (matcher.find()) { int start = matcher.start(); String state = location.substring(start, start + 2); context.write(new Text(lookupState(state.toUpperCase())), one); } } } private void setupStateMap(String filename) throws IOException { Map&lt;String, String&gt; states = new HashMap&lt;String, String&gt;(); BufferedReader reader = new BufferedReader(new FileReader(filename)); String line = reader.readLine(); while (line != null) { String[] split = line.split("\t"); states.put(split[0], split[1]); line = reader.readLine(); } stateNames = states; } private String lookupState(String state) { String fullName = stateNames.get(state); return fullName == null ? "Other" : fullName; } } public static void main(String[] args) throws Exception { Configuration config = new Configuration(); Job job = Job.getInstance(config, "UFO Location 2"); job.setJarByClass(UFOLocation2.class); job.addCacheFile(new URI("/user/kevin/data/states.txt")); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); Configuration mapconf1 = new Configuration(false); ChainMapper.addMapper(job, UFORecordValidationMapper.class, LongWritable.class, Text.class, LongWritable.class,Text.class, mapconf1); Configuration mapconf2 = new Configuration(false); ChainMapper.addMapper(job, MapClass.class, LongWritable.class, Text.class, Text.class, LongWritable.class, mapconf2); job.setMapperClass(ChainMapper.class); job.setCombinerClass(LongSumReducer.class); job.setReducerClass(LongSumReducer.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } </code></pre> <p>I get an IOException because it can't find the file "/user/kevin/data/states.txt" when it tries to instantiate the <code>BufferredReader</code> in the method <code>setupStateMap()</code></p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload