Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>You might need to experiment with the optimal configuration of SQL Server (not an expert for that)... but the basic scheme goes like this:</p> <ul> <li><p>Have 5 columns to manage job processing<br> One called DONEUNTIL that acts as a failsafe in case your processing node dies... it is basically an "estimated processing completion point in time"...<br> Second called NODEID that contains the ID of the NODE processing that job.<br> Third called DONE that gets set 1 once the job is finished.<br> Fourth called CREATED which contains the timestamp when the job was put into the table.<br> Fifth called JOBID which is the primary key.</p></li> <li><p>Have every node frequently cleanup all jobs where DONEUNTIL has passed and DONE != 1 by setting DONEUNTIL and NODEID to NULL</p></li> <li><p>When a Node is ready to take the next job it just selects the JOBID with the oldest CREATED having NULL in NODEID and DONE != 1<br> It then just updates DONEUNTIL and NODEID appropriately before it starts processing.<br> After finishing processing it updates DONE = 1.</p></li> </ul> <p>You could use variations of the above - like having job-priorities, retry-counters etc. or putting the "job-management-information" into a separate table etc.</p> <p>This scheme works really fine... if your have lots of jobs it might make sense to move jobs with DONE=1 into an archive table... this way your job table only contains active jobs (either waiting to be processed or being processed) which should keep it running smoothly...</p>
    singulars
    1. This table or related slice is empty.
    1. POArchitecture: Distributed Data Processing w/ Node Failure Tolerance
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USYahia
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks, Yahia! This is pretty much in line with what I had in mind (per my question), except for the DONEUNTIL idea which I like. I was trying to see if there's a completely different approach to avoid the frequent status updating (again, there are tens of thousands of jobs, and many are executed in a recurring fashion at a high frequency) but I guess in the end that information exchange has to happen somewhere. Thanks!
      singulars
      1. PO
      1. USAlex
    2. CO@Alex you are welcome... depending on how important persistence is you might be able to use some in-memory-DB and/or create a WebService (potentially with a built-in in-memory-cache) which in turn gets called by the different nodes...
      singulars
      1. PO
      1. USYahia
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload