Databases
StackOverflow2013
Postsdbo
POHow to tail a named pipe to Flume Windows node (Flume Node Service)
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to tail a named pipe to Flume Windows node (Flume Node Service)
    text
    copied!<p>Not sure if this is a SO, SuperUser or ServerFault question, but I just installed the FlumeNode service on my windows machine and I wanted to send logging infromation via a named pipe to the FlumeNode.</p> <p>The <a href="http://archive.cloudera.com/cdh/3/flume/Cookbook/index.html" rel="nofollow">Flume Cookbook</a> states that we can tail the named pipe to Flume, but it seems that this requires cygwin and it's unlikely that we're going to install cygwin on all the Flume nodes. Here are the instructions from the cookbook:</p> <blockquote> <p>Console sources are useful because we can pipe data into Flume directly. The next example pipes data from a program into Flume which then delivers it.</p> </blockquote> <pre><code>$ &lt;external process&gt; | flume node_nowatch -1 -s -n foo -c 'foo:console|agentBESink("collector");' </code></pre> <blockquote> <p>Ideally, you could write data to a named pipe and just have Flume read data from a named pipe using text or tail. Unfortunately, this version of Flume’s text and tail are not currently compatible with named pipes in a Linux environment. However, you could pipe data to a Flume node listening on the stdin console:</p> </blockquote> <pre><code>$ tail -f namedpipe | flume node_nowatch -1 -s -n foo -c 'foo:console|agentBESink;' </code></pre> <blockquote> <p>Or you can use the exec source to get its output data:</p> </blockquote> <pre><code>$ flume node_nowatch -1 -s -n bar -c 'bar:exec("cat pipe")|agentBESink;' </code></pre> <p>I also found a discussion on <a href="https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/5a75bf6fee5cae52?pli=1" rel="nofollow">google groups regarding this idea</a> and it seems that the Claudera people say it's possible, but there is nothing specific explaining how Flume would connect to a pipe and if it acts as the server or a client in that case.</p> <p>Ideally I should be able to create a <code>NamedPipeClientSteram</code> or a <code>NamedPipeServerStream</code> in my application. The client requires a named pipe server to be running and the clientlooks for clients which connect to it. I assume that the Flume will be acting as a server, but I can't confirm this. E.g.</p> <pre><code>NamedPipeClientStream pipeClient = new NamedPipeClientStream(".", "FlumeNamedPipe", PipeDirection.Out, pIpeOptions.None); pipeClient.Connect(); </code></pre> <p>Is there something that can be added to the command line arguments, when starting the Flume service in Windows, in order to tail a pipe?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload