StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow should I serialize domain model snapshots for event sourcing
text
Body
copied!<p>We are building an application using the <a href="http://lmax-exchange.github.io/disruptor/" rel="noreferrer">LMAX Disruptor</a>. When using <a href="http://martinfowler.com/eaaDev/EventSourcing.html" rel="noreferrer">Event Sourcing</a>, you often want to persist periodic snapshots of your domain model (some people call this the <a href="http://martinfowler.com/bliki/MemoryImage.html" rel="noreferrer">Memory Image</a> pattern).</p> <p><strong>I need a better solution than what we are currently using to serialize our domain model when taking a snapshot. I want to be able to "pretty-print" this snapshot in a readable format for debugging, and I want to simplify snapshot schema migration.</strong></p> <p>Currently, we are using <a href="https://code.google.com/p/protobuf/" rel="noreferrer">Googles' Protocol Buffers</a> to serialize our domain model to a file. We chose this solution, because protocol buffers are more compact than XML / JSON, and using a compact binary format seemed like a good idea to serialize a big Java domain model.</p> <p>The problem is, Protocol Buffers were designed for relatively small messages, and our domain model is quite big. So the domain model does not fit in one big hierarchical protobuf message, and we end up serializing various protobuf messages to a file, like this:</p> <pre><code>for each account { write simple account fields (id, name, description) as one protobuf message write number of user groups for each user group { convert user group to protobuf message, and serialize it } for each user { convert user to protobuf message, and serialize it } for each sensor { convert sensor to protobuf message, and serialize it } ... } </code></pre> <p>This is annoying, because manipulating a stream of heterogenous protobuf messages is complicated. It would be a lot easier if we had one big protobuf message that contained all of our domain model, like this:</p> <pre><code>public class AggregateRoot { List<Account> accounts; } --> convert to big hierarchical protobuf message using some mapping code: message AggregateRootMessage { repeated AccountMessage accounts = 1; } --> persist this big message to a file </code></pre> <p>If we do this, it's easy to prettyprint a snapshot: simply read the big protobuf message, then prettyprint it using protobuf's <a href="https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/TextFormat" rel="noreferrer">TextFormat</a>. With our current approach, we need to read the various protobuf messages one by one, and pretty-print them, which is harder, since the order of the protobuf messages in the stream depends on the current snapshot schema, so our pretty-printing tool needs to be aware of that.</p> <p>I also need a tool to migrate snapshots to the new snapshot schema when our domain model evolves. I'm still working on this tool, but it's hard, because I have to deal with a stream of various protobuf messages, instead of dealing with just one big message. If it were just one big message, I could: - take the snapshot file - parse the file as a big Java protobuf message, using the .proto schema for the previous snapshot version - convert this big protobuf message into a big protobuf message for the new version, using Dozer and some mapping code - write this new protobuf message in a new file, using the .proto schema for the new version</p> <p>But since I am dealing with a stream of protobuf messages of various types, my tool needs to handle this stream in the correct order.</p> <p>So, yeah... I guess my questions are:</p> <ul> <li><p>Do you know any serialization tool that can serialize a big domain model into a file, without protobuf's limitations, possibly using streaming to avoid OutOfMemorryErrors?</p></li> <li><p>If you use event sourcing or memory images, what do you use to serialize your domain model? JSON? XML? Protobuf? Something else?</p></li> <li><p>Are we doing it wrong? Do you have any suggestions?</p></li> </ul>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload