Databases
StackOverflow2013
Postsdbo
PORefactoring large data object
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORefactoring large data object
    primarykey
    data
    text
    <p><strong>What are some common strategies for refactoring large "state-only" objects?</strong></p> <p>I am working on a specific soft-real-time decision support system which does online modeling/simulation of the national airspace. This piece of software consumes a number of live data feeds, and produces a once-per-minute estimate of the "state" of a large number of entities in the airspace. The problem breaks down neatly until we hit what is currently the lowest-level entity.</p> <p>Our mathematical model estimates/predicts upwards of 50 parameters for a timeline of several hours into the past and future for each of these entities, roughly once per minute. Currently, these records are encoded as a single Java class with a lot of fields (some get collapsed into an <code>ArrayList</code>). Our model is evolving, and the dependencies among the fields are not yet set in stone, so each instance wanders through a convoluted model, accumulating settings as it goes along. </p> <p>Currently we have something like the following, which uses a builder pattern approach to build up the contents of the record, and enforce what the known dependencies are (as a check against programmer error as evolve the mode.) Once the estimate is done, we convert the below into an immutable form using a <code>.build()</code> type method.</p> <pre><code>final class OneMinuteEstimate { enum EstimateState { INFANT, HEADER, INDEPENDENT, ... }; EstimateState state = EstimateState.INFANT; // "header" stuff DateTime estimatedAtTime = null; DateTime stamp = null; EntityId id = null; // independent fields int status1 = -1; ... // dependent/complex fields... ... goes on for 40+ more fields... void setHeaderFields(...) { if (!EstimateState.INFANT.equals(state)) { throw new IllegalStateException("Must be in INFANT state to set header"); } ... } } </code></pre> <p>Once a very large number of these estimates are complete, they are assembled into timelines where aggregate patterns/trends are analyzed. We have looked at using an embedded database but have struggled with performance issues; we'd rather get this sorted out in terms of data modeling and then incrementally move portions of the soft-real-time code into an embedded data store.</p> <p>Once the "time sensitive" pieces of this are done, the products are flushed to flat files and a database.</p> <p><strong>Problems:</strong></p> <ul> <li>It's a giant class, with way too many fields. </li> <li>There is very little behavior encoded in the class; it's mostly a holder for data fields.</li> <li>Maintaining the <code>build()</code> method is extremely cumbersome.</li> <li>It feels clumsy to manually maintain a "state machine" abstraction merely for the purpose of ensuring that a large number of dependent modeling components are properly populating a data object, but it has saved us a lot of frustration as the model evolves.</li> <li>There is a lot of duplication, particularly when the records described above are aggregated into very similar "rollups" which amount to rolling sums/averages or other statistical products of the above structure in time series.</li> <li>While some of the fields could be clumped together, they are all logically "peers" of one another, and any breakdown we've tried has resulted in having behavior/logic artificially split and needing to reach two levels deep in indirection.</li> </ul> <p>Out of the box ideas entertained, but this is something we need to evolve incrementally. Before anyone else says it, I'll note that one could suggest that our mathematical model is insufficiently crisp if the data representation for that model is this hard to get ahold of. Fair point, and we're working that, but I think that's a side-effect of an R&amp;D environment with a lot of contributors, and a lot of concurrent hypotheses in play.</p> <p>(Not that it matters, but this is implemented in Java. We use HSQLDB or Postgres for output products. We don't use any persistence framework, partly out of a lack of familiarity, partly because we have enough performance trouble with just the database alone and hand-coded storage routines... we're skeptical of moving towards additional abstraction.)</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USMartin Vidner
    1. USandersoj
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORefactoring large data object
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORefactoring large data object
      1. USPéter Török
      1. VTFavorite
    3. VO
      singulars
      1. PORefactoring large data object
      1. This table or related slice is empty.
      1. VTUpMod
    1. CO+1 and starred. Great question and what looks like a wonderfully fun problem. I'm eager to see the answers.
      singulars
      1. PORefactoring large data object
      1. USCarl Manaster
    2. COLikely relevant http://martinfowler.com/bliki/AnemicDomainModel.html or http://stackoverflow.com/questions/258534/anemic-domain-model-pros-cons
      singulars
      1. PORefactoring large data object
      1. USandersoj
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload