Databases
StackOverflow2013
Postsdbo
POHow do I cache a large dataset that takes several minutes to return?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow do I cache a large dataset that takes several minutes to return?
    text
    copied!<p>Here's the background, I have a webservice from a vendor/partner that returns a large XML document that I use to display data. Simple enough. The problem is that the service takes several minutes to execute. The vendor is returning an entire normalized database as an XML document in one method call and it's slow. Stupid vendor but I can't control that.</p> <p>I have two questions:</p> <ol> <li><p>Where do I cache it? Disk, database, memory (I'm leaning to putting it on disk. It's too big and not accessed enough to store in memory. Putting it in a DB is probably <em>right</em> but I don't want to write ETL and have a job running somewhere.)</p></li> <li><p>How do I cache it? I can't just let it expire and have the next request refresh the cache because it takes so long. I need to serve stale data while the cache is rebuilding. I can think of a couple of ways of doing it but nothing that seems simple or very elegant. I'm looking to do something simple; I'm not looking for the best engineered solution ever.</p></li> </ol> <p>Here's my plan...please tell me that I'm an idiot and suggest something easier that I'm overlooking.</p> <ol> <ul>Write the XML to disk</ul> <ul>At the same time write the filename and some type of expiry bomb into cache (obviously the filename would not expire for a very long time)</ul> <ul>In my worker thread, try to read my bomb from cache</ul> <ul>If the cache is expired (my bomb went off) then spawn a thread to get and save the XML</ul> <ul>meanwhile, take the filename from cache and read in the old XML while the new one builds</ul> <ul>when the new one is finished, expire the old filename from cache and write the filename to cache</ul> </ol> <p>This sounds pretty much ridiculous to me and there has to be a better way.</p> <p>I'm using .Net and IIS6 and I can stick the XML in SQL Server if needed.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload