Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Hibernate (or any JPA solution) is the wrong tool for this job.</p> <p>JPA/Hibernate isn't a lightweight solution. In high-volume applications, the overhead is not only significant but prohibitive. You really need to look into <a href="https://stackoverflow.com/questions/383920/what-is-the-best-library-for-java-to-grid-cluster-enable-your-application#383929">grid and cluster solutions</a>. I won't repeat the overview of the various technologies here.</p> <p>I've got a lot of experience in financial market information systems. A few of the things you said stuck out to me:</p> <ul> <li>You have a lot of raw data;</li> <li>You want to apply various aggregations to that data (eg open/high/low/close daily summaries);</li> <li>High availability is probably an issue (it always is in these kinds of systems); and</li> <li>Low latency is probably an issue (ditto).</li> </ul> <p>Now for grid/cluster type solutions I divide them loosely into two categories:</p> <ol> <li>Map-based solutions like Coherence or Terracotta; and</li> <li>Javaspaces-based solutions like GigaSpaces.</li> </ol> <p>I've used Coherence a lot and the Map solution can be nice but it can be problematic too. Coherence maps can have listeners on them and you can use this sort of thing to do things like:</p> <ul> <li>Market price alerts (users may want a notification when a price reaches a certain level);</li> <li>Derivative pricing (eg an exchange-traded option pricing system will want to reprice when an underlying security changes last traded price);</li> <li>A trade-matching/booking system may want to match received trade notifications for reconciliation purposes;</li> <li>etc.</li> </ul> <p>All of these can be done with listeners but in Coherence for example listeners have to be cheap, which leads to things like a Map having a listener than writes something to another Map and this can chain on for awhile. Also, modifying the cache entry can be problematic (although there are mechanisms for dealing with that kind of problem too; I'm talking about situations like turning off a market price alert so it doesn't trigger a second time).</p> <p>I found GigaSpaces type grid solutions to be far more compelling for this kind of application. The read (or destructive read) operation is a highly elegant and scalable solution and you can get transactional grid updates with sub-millisecond performance.</p> <p>Consider the two classic queueing architectures:</p> <ul> <li>Request/Response: a bad message can block the queue and while you can many senders and receivers (for scalability) scaling up the number of pipes isn't always straightforward; and</li> <li>Publish/Subscribe: this decouples the sender and receiver but lacks scalability in that if you have multiple subscribers they'll each receive the message (not necessarily what you want with say a booking system).</li> </ul> <p>In GigaSpaces, a destructive read is like a scalable publish-subscribe system and a read operation is like the traditional publish-subscribe model. There is a Map and JMS implementation built on top of the grid and it can do FIFO ordering.</p> <p>Now whaqt about persistence I hear you ask? Persistence is a consequence of deciding all the other stuff. For this kind of application, I like the <a href="http://natishalom.typepad.com/nati_shaloms_blog/2007/09/paas-persistenc.html" rel="nofollow noreferrer">Persistence as a Service</a> model (ironically written about Hibernate but it applies to anything).</p> <p>Basically this means your date store hits are asynchronous and it works nicely with doing summary data. Like you can have a service listening for trade notifications and persist just the ones it's interested in (aggregating in memory if required). You can do open/high/low/close prices this way.</p> <p>For high volume data you don't really want to write it all to the database. Not synchronously anyway. A persistent store plus a data warehouse is probably more the route you want to go but again this depends on requirements, volumes, etc.</p> <p>It's a complicated topic and I've only really touche don it. Hope that helps you.</p>
    singulars
    1. This table or related slice is empty.
    1. POWhat are my options to store and query huge amounts of data where a lot of it is repeating?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USCommunity
    1. UScletus
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload