Databases
StackOverflow2013
Postsdbo
POUnable to use flume to extract files into hdfs
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUnable to use flume to extract files into hdfs
    text
    copied!<p>I am using flume-ng-1.2.0 with cdh3u5. I am simply trying to extract data from a text file and putting it into hdfs. Here is the Configuration I am using:</p> <pre><code>agent1.sources = tail1 agent1.channels = Channel-2 agent1.sinks = HDFS agent1.sources.tail1.type = exec agent1.sources.tail1.command = tail -F /usr/games/sample1.txt agent1.sources.tail1.channels = Channel-2 agent1.sinks.HDFS.channel = Channel-2 agent1.sinks.HDFS.type = hdfs agent1.sinks.HDFS.hdfs.path = hdfs://10.12.1.2:8020/user/hdfs/flume agent1.sinks.HDFS.hdfs.fileType = DataStream agent1.channels.Channel-2.type = memory agent1.channels.Channel-2.capacity = 1000 </code></pre> <p>and I am running agent by <code>bin/flume-ng agent -n agent1 -c ./conf/ -f conf/flume.conf</code></p> <p>and the logs which I am getting is</p> <pre><code>2012-10-11 12:10:36,626 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1 2012-10-11 12:10:36,631 INFO node.FlumeNode: Flume node starting - agent1 2012-10-11 12:10:36,639 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting 2012-10-11 12:10:36,639 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 12 2012-10-11 12:10:36,641 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting 2012-10-11 12:10:36,646 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume.conf 2012-10-11 12:10:36,657 INFO conf.FlumeConfiguration: Processing:HDFS 2012-10-11 12:10:36,670 INFO conf.FlumeConfiguration: Processing:HDFS 2012-10-11 12:10:36,670 INFO conf.FlumeConfiguration: Processing:HDFS 2012-10-11 12:10:36,670 INFO conf.FlumeConfiguration: Processing:HDFS 2012-10-11 12:10:36,671 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: agent1 2012-10-11 12:10:36,758 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 2012-10-11 12:10:36,758 INFO properties.PropertiesFileConfigurationProvider: Creating channels 2012-10-11 12:10:36,800 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: Channel-2, registered successfully. 2012-10-11 12:10:36,800 INFO properties.PropertiesFileConfigurationProvider: created channel Channel-2 2012-10-11 12:10:36,835 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 2012-10-11 12:10:37,753 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 2012-10-11 12:10:37,896 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully. 2012-10-11 12:10:37,899 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{tail1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource@362f0d54 }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4b142196 counterGroup:{ name:null counters:{} } }} channels:{Channel-2=org.apache.flume.channel.MemoryChannel@16a9255c} } 2012-10-11 12:10:37,900 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel Channel-2 2012-10-11 12:10:37,901 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: Channel-2 started 2012-10-11 12:10:37,901 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink HDFS 2012-10-11 12:10:37,905 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 2012-10-11 12:10:37,910 INFO nodemanager.DefaultLogicalNodeManager: Starting Source tail1 2012-10-11 12:10:37,912 INFO source.ExecSource: Exec source starting with command:tail -F /usr/games/sample1.txt </code></pre> <p>I don't know where am I doing mistake. As I am a beginner I am not getting anything in hdfs and the flume-agent is kept on running. any suggestion and correction will be very helpful to me ,thanks.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload