Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>If you don't want a database, you could store just the next run timestamp and task id in memory. You could store the properties for each task in a file named [task_id].txt. You would need a data structure to store all the tasks, sorted by timestamp in memory, an AVL tree seems like it would work, here's a simple one for python: <a href="http://bjourne.blogspot.com/2006/11/avl-tree-in-python.html" rel="nofollow noreferrer">http://bjourne.blogspot.com/2006/11/avl-tree-in-python.html</a>. Hopefully Linux (I assume that's what you are running on) could handle millions of files in a directory, otherwise you might need to hash on the task id to get a sub folder).</p> <p>Your master server would just need to run a loop, popping off tasks out of the AVL tree until the next task's timestamp is in the future. Then you could sleep for a few seconds and start checking again. Whenever a task runs, you would update the next run timestamp in the task file and re-insert it into the AVL tree.</p> <p>When the master server reboots, there would be the overhead of reloading all tasks id and next run timestamp back into memory, so that might be painful with millions of files. Maybe you just have one giant file and give each task 1K space in the file for properties and next run timestamp and then use [task_id] * 1K to get to the right offset for the task properties.</p> <p>If you are willing to use a database, I am confident MySQL could handle whatever you throw at it given the conditions you describe, assuming you have 4GB+ RAM and several hard drives in RAID 0+1 on your master server.</p> <p>Finally, if you really want to get complicated, Hadoop might work too: <a href="http://hadoop.apache.org/" rel="nofollow noreferrer">http://hadoop.apache.org/</a></p>
    singulars
    1. This table or related slice is empty.
    1. POHow to schedule hundreds of thousands of tasks?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USjjxtra
    1. USjjxtra
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COJeff, appreciate your response. We're at about 200 writes per second now and are planning for 10x the number of jobs. It may be that a database solution is the the only way to go, but I am (perhaps vainly) looking for an approach that doesn't require constant indexing of timestamps.
      singulars
      1. PO
      1. USwehriam
    2. COI updated my answer with some new ideas
      singulars
      1. PO
      1. USjjxtra
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload