Databases
StackOverflow2013
Postsdbo
POhadoop -libjars and ClassNotFoundException
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POhadoop -libjars and ClassNotFoundException
    text
    copied!<p>please help, I'm stuck. Here is my code to run job.</p> <pre><code>hadoop jar mrjob.jar ru.package.Main -files hdfs://0.0.0.0:8020/MyCatalog/jars/metadata.csv -libjars hdfs://0.0.0.0:8020/MyCatalog/jars/opencsv.jar,hdfs://0.0.0.0:8020/MyCatalog/jars/gson.jar,hdfs://0.0.0.0:8020/MyCatalog/jars/my-utils.jar /MyCatalog/http_requests.seq-r-00000 /MyCatalog/output/result_file </code></pre> <p>I do get these WARNs:</p> <pre><code>12/10/26 18:35:50 WARN util.GenericOptionsParser: The libjars file hdfs://0.0.0.0:8020/MyCatalog/jars/opencsv.jar is not on the local filesystem. Ignoring. 12/10/26 18:35:50 WARN util.GenericOptionsParser: The libjars file hdfs://0.0.0.0:8020/MyCatalog/jars/gson.jar is not on the local filesystem. Ignoring. 12/10/26 18:35:50 WARN util.GenericOptionsParser: The libjars file hdfs://0.0.0.0:8020/MyCatalog/jars/my-utils.jar is not on the local filesystem. Ignoring. </code></pre> <p>Then: <strong>Exception in thread "main" java.lang.NoClassDefFoundError:</strong> on line in Main class where I try to instantiate class from jar named my-utils.jar</p> <ol> <li>All these jars are in hfds (I see them through file browser)</li> <li>my-utils.jar does contain class which is a reason for <strong>NoClassDefFoundError</strong></li> </ol> <p>What do I do wrong?</p> <p><strong>UPD:</strong> I'm inspecting sourcecode of <strong>GenericOptionsParser</strong>:</p> <pre><code>/** * If libjars are set in the conf, parse the libjars. * @param conf * @return libjar urls * @throws IOException */ public static URL[] getLibJars(Configuration conf) throws IOException { String jars = conf.get("tmpjars"); if(jars==null) { return null; } String[] files = jars.split(","); List&lt;URL&gt; cp = new ArrayList&lt;URL&gt;(); for (String file : files) { Path tmp = new Path(file); if (tmp.getFileSystem(conf).equals(FileSystem.getLocal(conf))) { cp.add(FileSystem.getLocal(conf).pathToFile(tmp).toURI().toURL()); } else { LOG.warn("The libjars file " + tmp + " is not on the local " + "filesystem. Ignoring."); } } return cp.toArray(new URL[0]); } </code></pre> <p>So: 1. no spaces between comma 2. still don't get it... I've tried to point to: local file system, hdfs file system, result is the same. Seems like class is not added...</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload