StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PODeserializing Flume Events in C# via Avro
text
Body
copied!<p>I've set up a Flume service that can monitor Netcat or tail a log with Exec as a Source, that sort of thing. Im using Memory as a channel, and Avro as the sink (Thrift is specified in the docs, but doesn't seem to work in Flume 1.3 or 1.4)</p> <p>I've set up a C# socket server to receive the messages, and I get a stream of bytes. If I use read them using Encoding.UTF8.GetString(buffer), then I can see something like:</p> <pre><code>"\0\0\0\0\0\0\0\0\00��Tt��5\ne\0��Tt��5\ne\0\0appendBatch\0\0�\0�127.0.0.1 - - [12/Nov/2013:22:42:50 +0000] \"GET /docs/appdev/index.html HTTP/1.1\" 200 7645\0�127.0.0.1 - - [12/Nov/2013:22:44:07 +0000] \"GET /docs/appdev/introduction.html HTTP/1.1\" 200 8619\0�127.0.0.1 - - [12/Nov/2013:22:44:09 +0000] \"GET /docs/appdev/installation.html HTTP/1.1\" 200 9045\0�127.0.0.1 - - [12/Nov/2013:22:44:12 +0000] \"GET /docs/appdev/deployment.html HTTP/1.1\" 200 18800\0�127.0.0.1 - - [12/Nov/2013:22:49:07 +0000] \"GET /docs/appdev/source.html HTTP/1.1\" 200 24554\0�127.0.0.1 - - [12/Nov/2013:22:50:38 +0000] \"GET /docs/appdev/processes.html HTTP/1.1\" 200 30743\0�127.0.0.1 - - [12/Nov/2013:22:51:39 +0000] \"GET /docs/appdev/sample/ HTTP/1.1\" 200 1852\0�0:0:0:0:0:0:0:1 - - [12/Nov/2013:22:51:48 +0000] \"GET /sample HTTP/1.1\" 404 963\0�0:0:0:0:0:0:0:1 - - [12/Nov/2013:22:51:48 +0000] \"GET /favicon.ico HTTP/1.1\" 200 21630\0�0:0:0:0:0:0:0:1 - - [12/Nov/2013:23:02:13 +0000] \"GET /sample HTTP/1.1\" 404 963\0" </code></pre> <p>So obviously I'm getting the data across, but I'd like to deserialize it properly rather than doing some kind of regex extractions. I can see there's an official Avro C# library, and there's a Microsoft Hadoop library that has deserialization libraries. I created a local object to deserialize to:</p> <pre><code>[DataContract] public class AvroEvent { [DataMember] public byte[] Body { get; set; } } </code></pre> <p>and try deserializing with this:</p> <pre><code> client = serverSocket.EndAccept(result); var myNetworkStream = new NetworkStream(client); myNetworkStream.Read(buffer, 0, size); var avro = new AvroSerializer(typeof(AvroEvent)); var deser = avro.Deserialize(myNetworkStream); </code></pre> <p>then I get this error:</p> <pre><code> System.InvalidOperationException was unhandled HResult=-2146233079 Message=Unexpected number of bytes. Source=Microsoft.Hadoop.Avro </code></pre> <p>I'm almost certainly going about all this in the wrong way, and I'm sure people are going to be telling me not to use C#, but I've pretty much run out of sources on Google, so if anyone else has actually done this and point me in the right direction, I'd be very grateful</p> <p>Toby</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload