Databases
StackOverflow2013
Postsdbo
POLaunch a mapreduce job from eclipse
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLaunch a mapreduce job from eclipse
    primarykey
    data
    text
    <p>I've written a mapreduce program in Java, which I can submit to a remote cluster running in distributed mode. Currently, I submit the job using the following steps:</p> <ol> <li>export the mapreuce job as a jar (e.g. <code>myMRjob.jar</code>)</li> <li>submit the job to the remote cluster using the following shell command: <code>hadoop jar myMRjob.jar</code></li> </ol> <p>I would like to submit the job directly from Eclipse when I try to run the program. How can I do this?</p> <p>I am currently using CDH3, and an abridged version of my conf is:</p> <pre class="lang-java prettyprint-override"><code>conf.set("hbase.zookeeper.quorum", getZookeeperServers()); conf.set("fs.default.name","hdfs://namenode/"); conf.set("mapred.job.tracker", "jobtracker:jtPort"); Job job = new Job(conf, "COUNT ROWS"); job.setJarByClass(CountRows.class); // Set up Mapper TableMapReduceUtil.initTableMapperJob(inputTable, scan, CountRows.MyMapper.class, ImmutableBytesWritable.class, ImmutableBytesWritable.class, job); // Set up Reducer job.setReducerClass(CountRows.MyReducer.class); job.setNumReduceTasks(16); // Setup Overall Output job.setOutputFormatClass(MultiTableOutputFormat.class); job.submit(); </code></pre> <p>When I run this directly from Eclipse, the job is launched but Hadoop cannot find the mappers/reducers. I get the following errors:</p> <pre><code>12/06/27 23:23:29 INFO mapred.JobClient: map 0% reduce 0% 12/06/27 23:23:37 INFO mapred.JobClient: Task Id : attempt_201206152147_0645_m_000000_0, Status : FAILED java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mypkg.mapreduce.CountRows$MyMapper at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996) at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:212) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) ... </code></pre> <p>Does anyone know how to get past these errors? If I can fix this, I can integrate more MR jobs into my scripts which would be awesome!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAlexey Grigorev
    1. USTucker
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. USAlex
      1. VTFavorite
    3. VO
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. This table or related slice is empty.
      1. VTUpMod
    1. COTucker - I was able to run the Hadoop job in stand alone, but not other modes from Eclipse. I posted the query in the Hadoop forums some time back and there was no +ve response. BTW, Hadoop runs in the stand alone mode without any configuration files (default parameters).
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. USPraveen Sripati
    2. COWhen you submit the job from within Eclipse, are the mapper / reducer classes in the same project, or is the jar containing them on the classpath, and the classes themselves nowhere else on the cp?
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. USChris White
    3. CO@ChrisWhite The class containing everything is called CountRows. This class contains a 'main' method which sets the job configurations. the CountRows class also contains the class for the mapper and reducer called MyMapper and MyReducer respectively. The job works fine as I said when I launch the job from the comandline by writing 'hadoop jar CountRows.jar'
      singulars
      1. POLaunch a mapreduce job from eclipse
      1. USTucker
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload