Databases
StackOverflow2013
Postsdbo
POHow do I build/run this simple Mahout program without getting exceptions?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow do I build/run this simple Mahout program without getting exceptions?
    text
    copied!<p>I would like to run this code which I found in Mahout In Action:</p> <pre><code>package org.help; import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.SequenceFile; import org.apache.hadoop.io.Text; import org.apache.mahout.math.DenseVector; import org.apache.mahout.math.NamedVector; import org.apache.mahout.math.VectorWritable; public class SeqPrep { public static void main(String args[]) throws IOException{ List&lt;NamedVector&gt; apples = new ArrayList&lt;NamedVector&gt;(); NamedVector apple; apple = new NamedVector(new DenseVector(new double[]{0.11, 510, 1}), "small round green apple"); apples.add(apple); Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); Path path = new Path("appledata/apples"); SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path, Text.class, VectorWritable.class); VectorWritable vec = new VectorWritable(); for(NamedVector vector : apples){ vec.set(vector); writer.append(new Text(vector.getName()), vec); } writer.close(); SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path("appledata/apples"), conf); Text key = new Text(); VectorWritable value = new VectorWritable(); while(reader.next(key, value)){ System.out.println(key.toString() + " , " + value.get().asFormatString()); } reader.close(); } } </code></pre> <p>I compile it with:</p> <pre><code>$ javac -classpath :/usr/local/hadoop-1.0.3/hadoop-core-1.0.3.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-sources.jar -d myjavac/ SeqPrep.java </code></pre> <p>I jar it:</p> <pre><code>$ jar -cvf SeqPrep.jar -C myjavac/ . </code></pre> <p>Now I'd like to run it on my local hadoop node. I've tried:</p> <pre><code> hadoop jar SeqPrep.jar org.help.SeqPrep </code></pre> <p>But I get:</p> <pre><code>Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/Vector at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:149) </code></pre> <p>So I tried using the libjars parameter: </p> <pre><code>$ hadoop jar SeqPrep.jar org.help.SeqPrep -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT.jar -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-sources.jar -libjars /home/hduser/mahout/trunk/math/target/mahout-math-0.8-SNAPSHOT.jar -libjars /home/hduser/mahout/trunk/math/target/mahout-math-0.8-SNAPSHOT-sources.jar </code></pre> <p>and got the same problem. I don't know what else to try.</p> <p>My eventual goal is to be able to read a .csv file on the hadoop fs into a sparse matrix and then multiply it by a random vector.</p> <p><strong>edit:</strong> Looks like Razvan got it (note: see below for another way to do this that does not mess with your hadoop installation). For reference:</p> <pre><code>$ find /usr/local/hadoop-1.0.3/. |grep mah /usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-tests.jar /usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT.jar /usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-job.jar /usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-sources.jar /usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT-sources.jar /usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT-tests.jar /usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT.jar </code></pre> <p>and then:</p> <pre><code>$hadoop jar SeqPrep.jar org.help.SeqPrep small round green apple , small round green apple:{0:0.11,1:510.0,2:1.0} </code></pre> <p><strong>edit:</strong> I'm trying to do this without copying the mahout jars into the hadoop lib/</p> <pre><code>$ rm /usr/local/hadoop-1.0.3/lib/mahout-* </code></pre> <p>and then of course:</p> <pre><code>hadoop jar SeqPrep.jar org.help.SeqPrep Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/Vector at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:149) Caused by: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) </code></pre> <p>and when I try the mahout job file:</p> <pre><code>$hadoop jar ~/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep Exception in thread "main" java.lang.ClassNotFoundException: org.help.SeqPrep at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:149) </code></pre> <p>If I try to include the .jar file I made:</p> <pre><code>$ hadoop jar ~/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar SeqPrep.jar org.help.SeqPrep Exception in thread "main" java.lang.ClassNotFoundException: SeqPrep.jar </code></pre> <p><strong>edit:</strong> Apparently I can only send one jar at a time to hadoop. This means I need to add the class I made into the mahout core job file:</p> <pre><code>~/mahout/trunk/core/target$ cp mahout-core-0.8-SNAPSHOT-job.jar mahout-core-0.8-SNAPSHOT-job.jar_backup ~/mahout/trunk/core/target$ cp ~/workspace/seqprep/bin/org/help/SeqPrep.class . ~/mahout/trunk/core/target$ jar uf mahout-core-0.8-SNAPSHOT-job.jar SeqPrep.class </code></pre> <p>And then:</p> <pre><code>~/mahout/trunk/core/target$ hadoop jar mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep Exception in thread "main" java.lang.ClassNotFoundException: org.help.SeqPrep </code></pre> <p><strong>edit:</strong> Ok, now I can do it without messing with my hadoop installation. I was updating the .jar wrong in that previous edit. It should be:</p> <pre><code>~/mahout/trunk/core/target$ jar uf mahout-core-0.8-SNAPSHOT-job.jar org/help/SeqPrep.class </code></pre> <p>then:</p> <pre><code>~/mahout/trunk/core/target$ hadoop jar mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep small round green apple , small round green apple:{0:0.11,1:510.0,2:1.0} </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload