Databases
StackOverflow2013
Postsdbo
POLazy, stream driven object serialization with protobuf-net
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLazy, stream driven object serialization with protobuf-net
    primarykey
    data
    text
    <p>We are developing a WCF service for streaming a large amount of data, therefore we have chosen to use <a href="http://msdn.microsoft.com/en-us/library/ms733742.aspx">WCF Streaming</a> functionality combined with a <a href="http://code.google.com/p/protobuf-net/">protobuf-net</a> serialization.</p> <p><b>Context:</b></p> <p>Generally an idea is to serialize objects in the service, write them into a stream and send. On the other end the caller will receive a Stream object and it can read all data.</p> <p>So currently the service method code looks somewhat like this:</p> <pre><code>public Result TestMethod(Parameter parameter) { // Create response var responseObject = new BusinessResponse { Value = "some very large data"}; // The resposne have to be serialized in advance to intermediate MemoryStream var stream = new MemoryStream(); serializer.Serialize(stream, responseObject); stream.Position = 0; // ResultBody is a stream, Result is a MessageContract return new Result {ResultBody = stream}; } </code></pre> <p>The BusinessResponse object is serialized to a MemoryStream and that is returned from a method. On the client side the calling code looks like that:</p> <pre><code>var parameter = new Parameter(); // Call the service method var methodResult = channel.TestMethod(parameter); // protobuf-net deserializer reads from a stream received from a service. // while reading is performed by protobuf-net, // on the service side WCF is actually reading from a // memory stream where serialized message is stored var result = serializer.Deserialize&lt;BusinessResponse&gt;(methodResult.ResultBody); return result; </code></pre> <p>So when <code>serializer.Deserialize()</code> is called it reads from a stream <code>methodResult.ResultBody</code>, on the same time on the service side WCF is reading a MemoryStream, that has been returned from a <code>TestMethod</code>.</p> <p><b>Problem:</b></p> <p>What we would like to achieve is to get rid of a <code>MemoryStream</code> and initial serialization of the whole object on the service side at once. Since we use streaming we would like to avoid keeping a serialized object in memory before sending.</p> <p><b>Idea:</b></p> <p>The perfect solution would be to return an empty, custom-made Stream object (from <code>TestMethod()</code>) with a reference to an object that is to be serialized ('BusinessResponse' object in my example). So when WCF calls a <code>Read()</code> method of my stream, I internally serialize a piece of an object using protobuf-net and return it to the caller without storing it in the memory.</p> <p>And now there is a problem, because what we actually need is a possibility to serialize an object piece by piece in the moment when stream is read. I understand that this is totally different way of serialization - instead of pushing an object to a serializer, I'd like to request a serialized content piece by piece.</p> <p>Is that kind of serialization is somehow possible using protobuf-net?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USHubert R. Skrzypek
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs this one object? Or a series of objects (a collection)? Whether it is worth looking at this much actually depends on your WCF config - in most configurations it will always buffer the entire message in memory *anyway* - so it may be as easy to not change anything.
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. USMarc Gravell
    2. COHi Marc, WCF is configured to not use buffering at all - that is the point of streaming - I want to decrease memory footprint on server side. Additionally if I would like to serialize collection of objects I would use `SerializeWithLengthPrefix()` each time the Client calls `Read()` and my underlying buffer is smaller than requested data amount. The issue here is that I'd like to be able to split single object serialization.
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. USHubert R. Skrzypek
    3. COinteresting question. I *think* this can be generalised, essentially to a spoof Stream that makes the Read and Write work as co-routines. If you don't mind having an extra Thread, it could be done with a simple gate, however iirc Jon had some interesting ideas. I'll have to take a look and get back to you. However, I can say without doubt that I don't intend hacking the core of protobuf-net for the purpose :)
      singulars
      1. POLazy, stream driven object serialization with protobuf-net
      1. USMarc Gravell
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload