Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I'm going to read between some lines here... because <code>List&lt;T&gt;</code> (mapped as <code>repeated</code> in protobuf parlance) doesn't have an overall length-prefix, and <code>byte[]</code> (mapped as <code>bytes</code>) has a trivial length-prefix that shouldn't cause additional buffering. So I'm guessing what you <em>actually</em> have is more like:</p> <pre><code>[ProtoContract] public class A { [ProtoMember(1)] public B Foo {get;set;} } [ProtoContract] public class B { [ProtoMember(1)] public List&lt;byte[]&gt; Bar {get;set;} } </code></pre> <p>Here, the need to buffer for a length-prefix is actually when writing <code>A.Foo</code>, basically to <em>declare</em> "the following complex data is the value for <code>A.Foo</code>"). Fortunately there is a simple fix:</p> <pre><code>[ProtoMember(1, DataFormat=DataFormat.Group)] public B Foo {get;set;} </code></pre> <p>This changes between 2 packing techniques in protobuf:</p> <ul> <li>the default (google's stated preference) is length-prefixed, meaning you get a marker indicating the length of the message to follow, then the sub-message payload</li> <li>but there is also an option to use a start-marker, the sub-message payload, and an end-marker</li> </ul> <p>When using the second technique <em>it doesn't need to buffer</em>, so: it doesn't. This does mean it will be writing slightly different bytes for the same data, but protobuf-net is very forgiving, and will happily deserialize data from <em>either</em> format here. Meaning: if you make this change, you can still read your existing data, but new data will use the start/end-marker technique.</p> <p>This demands the question: why do google prefer the length-prefix approach? <em>Probably</em> this is because it is more efficient <strong>when reading</strong> to skip through fields (either via a raw reader API, or as unwanted/unexpected data) when using the length-prefix approach, as you can just read the length-prefix, and then just progress the stream [n] bytes; by contrast, to skip data with a start/end-marker you still need to crawl through the payload, skipping the sub-fields individually. Of course, this theoretical difference in read performance doesn't apply if you <em>expect</em> that data and want to read it into your object, which you almost certainly do. Also, in the google protobuf implementation, because it isn't working with a regular POCO model, the size of the payloads are already known, so they don't really see the same issue when writing.</p>
    singulars
    1. This table or related slice is empty.
    1. POMemory usage serializing chunked byte arrays with Protobuf-net
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USMarc Gravell
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POMemory usage serializing chunked byte arrays with Protobuf-net
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks for the quick reply. Your guess about our data structure was correct. Would I be right in saying that we need to change the DataFormat to Group for any properties containing a reference to A as well, and so on up to the root of the object graph? And this change would also need to be on relevant ProtoInclude attributes too?
      singulars
      1. PO
      1. USJames Thurley
    2. CO@James essentially, yes. Hmmm... Maybe I should add a model-level default for that!
      singulars
      1. PO
      1. USMarc Gravell
    3. COI've updated my question with my attempt at using DataFormat.Group to solve the problem but I'm still having problems getting the buffer to flush. Apologies if I'm being an idiot..
      singulars
      1. PO
      1. USJames Thurley
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload