Databases
StackOverflow2013
Postsdbo
POHadoop giving SCDynamicStore on my JAR but not on hadoop-examples.jar
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHadoop giving SCDynamicStore on my JAR but not on hadoop-examples.jar
    text
    copied!<p>I'm very confused about building and executing my first job in Hadoop and would love help from anyone who can clarify the error I am seeing and provide guidance :)</p> <p>I have a JAR file that I've compiled. When I try to execute a M/R job using it in OSX, I get the SCDynamicStore error that is often associated with the HADOOP_OPTS environment variable. However, this does not happen when I run examples from the example JAR file. I have set the variable in hadoop-env.sh and it appears to be recognized in the cluster.</p> <p>Running a test from hadoop-examples.jar works:</p> <pre><code>$ hadoop jar /usr/local/Cellar/hadoop/1.1.2/libexec/hadoop-examples-1.1.2.jar wordcount /stock/data /stock/count.out 13/06/22 13:21:51 INFO input.FileInputFormat: Total input paths to process : 3 13/06/22 13:21:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/06/22 13:21:51 WARN snappy.LoadSnappy: Snappy native library not loaded 13/06/22 13:21:51 INFO mapred.JobClient: Running job: job_201306221315_0003 13/06/22 13:21:52 INFO mapred.JobClient: map 0% reduce 0% 13/06/22 13:21:56 INFO mapred.JobClient: map 66% reduce 0% 13/06/22 13:21:58 INFO mapred.JobClient: map 100% reduce 0% 13/06/22 13:22:04 INFO mapred.JobClient: map 100% reduce 33% 13/06/22 13:22:05 INFO mapred.JobClient: map 100% reduce 100% 13/06/22 13:22:05 INFO mapred.JobClient: Job complete: job_201306221315_0003 ... </code></pre> <p>Running a job using my own class does not work:</p> <pre><code>$ hadoop jar test.jar mapreduce.X /data /output 13/06/22 13:38:36 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 13/06/22 13:38:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/06/22 13:38:36 WARN snappy.LoadSnappy: Snappy native library not loaded 13/06/22 13:38:36 INFO mapred.FileInputFormat: Total input paths to process : 3 13/06/22 13:38:36 INFO mapred.JobClient: Running job: job_201306221328_0002 13/06/22 13:38:37 INFO mapred.JobClient: map 0% reduce 0% 13/06/22 13:38:44 INFO mapred.JobClient: Task Id : attempt_201306221328_0002_m_000000_0, Status : FAILED java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 9 more Caused by: java.lang.NoClassDefFoundError: com/google/gson/TypeAdapterFactory at mapreduce.VerifyMarket$Map.&lt;clinit&gt;(VerifyMarket.java:26) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:249) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:873) at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:947) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) ... 14 more Caused by: java.lang.ClassNotFoundException: com.google.gson.TypeAdapterFactory at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 22 more attempt_201306221328_0002_m_000000_0: 2013-06-22 13:38:39.314 java[60367:1203] Unable to load realm info from SCDynamicStore 13/06/22 13:38:44 INFO mapred.JobClient: Task Id : attempt_201306221328_0002_m_000001_0, Status : FAILED ... (This repeats a few times, but hopefully this is enough to see what I mean.) </code></pre> <p>Initially, I thought this was related to the aforementioned environment variable, but now I'm not so sure. Maybe I'm packaging my JAR incorrectly?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload