Databases
StackOverflow2013
Postsdbo
POHadoop: File Copy with Cascading 2.5.1 and Hadoop 2.2.0
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHadoop: File Copy with Cascading 2.5.1 and Hadoop 2.2.0
    primarykey
    data
    text
    <p>I have recently set up a pseudo-distributed hadoop 2.2.0 cluster on my Mac OSX following <a href="http://ac31004.blogspot.com/2013/10/installing-hadoop-2-on-mac_29.html" rel="nofollow">this guide</a>. Then, I tried the basic Cascading file copy with Cascading 2.5.1 However when I compiled the project using maven, I got the following error:</p> <pre><code>[ERROR] /Users/david/IdeaProjects//CascadingIntro/src/main/java/com/example/CascadingIntro.java:[24,24] cannot access org.apache.hadoop.mapred.JobConf class file for org.apache.hadoop.mapred.JobConf not found </code></pre> <p>What am I doing wrong and how do I fix this? I believe that Cascading 2.5.1 is compatible with Hadoop 2.2.0 from <a href="http://www.cascading.org/support/compatibility/" rel="nofollow">this page</a> on Cascading.org.</p> <p>My pom.xml is as follows:</p> <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt; &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt; &lt;groupId&gt;CascadingIntro&lt;/groupId&gt; &lt;artifactId&gt;CascadingIntro&lt;/artifactId&gt; &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt; &lt;properties&gt; &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt; &lt;/properties&gt; &lt;repositories&gt; &lt;repository&gt; &lt;id&gt;conjars.org&lt;/id&gt; &lt;url&gt;http://conjars.org/repo&lt;/url&gt; &lt;/repository&gt; &lt;/repositories&gt; &lt;dependencies&gt; &lt;dependency&gt; &lt;groupId&gt;cascading&lt;/groupId&gt; &lt;artifactId&gt;cascading-core&lt;/artifactId&gt; &lt;version&gt;2.5.1&lt;/version&gt; &lt;/dependency&gt; &lt;dependency&gt; &lt;groupId&gt;cascading&lt;/groupId&gt; &lt;artifactId&gt;cascading-hadoop&lt;/artifactId&gt; &lt;version&gt;2.5.1&lt;/version&gt; &lt;/dependency&gt; &lt;/dependencies&gt; &lt;build&gt; &lt;finalName&gt;CascadingIntro&lt;/finalName&gt; &lt;plugins&gt; &lt;plugin&gt; &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt; &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt; &lt;version&gt;3.0&lt;/version&gt; &lt;configuration&gt; &lt;source&gt;1.7&lt;/source&gt; &lt;target&gt;1.7&lt;/target&gt; &lt;/configuration&gt; &lt;/plugin&gt; &lt;plugin&gt; &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt; &lt;artifactId&gt;maven-jar-plugin&lt;/artifactId&gt; &lt;version&gt;2.4&lt;/version&gt; &lt;configuration&gt; &lt;archive&gt; &lt;manifest&gt; &lt;mainClass&gt;com.example.CascadingIntro&lt;/mainClass&gt; &lt;/manifest&gt; &lt;/archive&gt; &lt;/configuration&gt; &lt;/plugin&gt; &lt;/plugins&gt; &lt;/build&gt; &lt;/project&gt; </code></pre> <p>And in my CascadingIntro class:</p> <pre><code>package com.example; import cascading.flow.FlowDef; import cascading.flow.hadoop.HadoopFlowConnector; import cascading.pipe.Pipe; import cascading.property.AppProps; import cascading.scheme.hadoop.TextDelimited; import cascading.tap.Tap; import cascading.tap.hadoop.Hfs; import java.util.Properties; public class CascadingIntro { public static void main(String[] args) { Properties properties = new Properties(); AppProps.setApplicationJarClass( properties, CascadingIntro.class ); HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties ); String inputPath = args[0]; Tap inputTap = new Hfs(new TextDelimited(true,"\t"), inputPath); String outputPath = args[1]; Tap outputTap = new Hfs(new TextDelimited(true,"\t"),outputPath); Pipe copyPipe = new Pipe("copy"); FlowDef flowDef = FlowDef .flowDef() .addSource(copyPipe,inputTap) .addTailSink(copyPipe,outputTap); flowConnector.connect(flowDef).complete(); } } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USDavid Williams
    1. USDavid Williams
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHadoop: File Copy with Cascading 2.5.1 and Hadoop 2.2.0
      1. USDavid Williams
      1. VTBountyStart
    2. VO
      singulars
      1. POHadoop: File Copy with Cascading 2.5.1 and Hadoop 2.2.0
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload