StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POArchitecture: Distributed Data Processing w/ Node Failure Tolerance
primarykey
Id
13905665
data
AcceptedAnswerId
0
AnswerCount
2
ClosedDate
CommentCount
3
CommunityOwnedDate
CreationDate
2012-12-16T21:30:24.813
FavoriteCount
2
LastActivityDate
2012-12-16T22:15:24.470
LastEditDate
2012-12-16T21:37:42.493
LastEditorUserId
114916
OwnerUserId
114916
ParentId
0
PostTypeId
1
Score
0
ViewCount
177
LastEditorDisplayName
text
Body
I have lots of pieces (tens of thousands) of data/jobs that need frequent recurring processing. The jobs to be processed are stored in a SQL Server 2012 (Web Ed) database that gets updated with new jobs frequently and/or may have jobs deleted from it. Sample: <pre><code>Id | WorkItem 1 | Copy X to Y 2 | Ping stackoverflow.com 3 | Verify backupset 4 | Send an email </code></pre> My goal is to distribute the job processing across multiple nodes, both for performance reasons and to ensure jobs get processed even if a node fails. Two considerations: <ul> <li>I need to be sure that all jobs eventually get executed by some node, and that this continually happens as jobs are added to the database driven queue.</li> <li>I'd like all nodes to some work, and prevent a single node from fetching most of the work, hence implementing some kind of round-robin seems to make sense.</li> </ul> Because the entire system needs to be resistant to node failure, I can't really "assign" the work to nodes as I don't know what's up and what isn't. My initial thoughts to work around this were to have each node "fetch" one or more jobs from the database, process them, and then return data to the SQL database. However, there are some issues then: <ul> <li>Nodes need to be aware what other nodes are already processing, so jobs don't get incorrectly processed twice. This means jobs would need to be marked as being processed.</li> <li>Many of these jobs are incredibly small, but need to be executed very frequently (maybe every 10-30 seconds). Constantly updating who is processing jobs and then releasing the job, and then processing again seems like major database overhead.</li> <li>What happens if a node dies mid-processing? Is there a way to recover from this?</li> </ul> What is the most efficient way to solve this problem? Thank you!
Tags
<architecture><queue><distributed-computing><system-design>
Title
Architecture: Distributed Data Processing w/ Node Failure Tolerance
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. USAlex
UserOwnerUserId
1. USAlex
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
2. PO
 singulars
 PostTypePostTypeId
 PTAnswer
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 POArchitecture: Distributed Data Processing w/ Node Failure Tolerance
 UserUserId
 USAlex
 VoteTypeVoteTypeId
 VTFavorite
2. VO
 singulars
 PostPostId
 POArchitecture: Distributed Data Processing w/ Node Failure Tolerance
 UserUserId
 UScsg
 VoteTypeVoteTypeId
 VTFavorite
CommentsPostId

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.