Databases
StackOverflow2013
Postsdbo
PODual-queue producer-consumer in .NET (forcing member variable flush)
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PODual-queue producer-consumer in .NET (forcing member variable flush)
    text
    copied!<p>I have a thread which produces data in the form of simple object (record). The thread may produce a thousand records for each one that successfully passes a filter and is actually enqueued. Once the object is enqueued it is read-only. </p> <p>I have one lock, which I acquire once the record has passed the filter, and I add the item to the back of the producer_queue. </p> <p>On the consumer thread, I acquire the lock, confirm that the producer_queue is not empty, set consumer_queue to equal producer_queue, create a new (empty) queue, and set it on producer_queue. Without any further locking I process consumer_queue until it's empty and repeat. </p> <p>Everything works beautifully on most machines, but on one particular dual-quad server I see in ~1/500k iterations an object that is not fully initialized when I read it out of consumer_queue. The condition is so fleeting that when I dump the object after detecting the condition the fields are correct 90% of the time. </p> <p>So my question is this: how can I assure that the writes to the object are flushed to main memory when the queue is swapped? </p> <p>Edit: </p> <p>On the producer thread: (producer_queue above is m_fillingQueue; consumer_queue above is m_drainingQueue)</p> <pre><code>private void FillRecordQueue() { while (!m_done) { int count; lock (m_swapLock) { count = m_fillingQueue.Count; } if (count &gt; 5000) { Thread.Sleep(60); } else { DataRecord rec = GetNextRecord(); if (rec == null) break; lock (m_swapLock) { m_fillingQueue.AddLast(rec); } } } } </code></pre> <p>In the consumer thread:</p> <pre><code>private DataRecord Next(bool remove) { bool drained = false; while (!drained) { if (m_drainingQueue.Count &gt; 0) { DataRecord rec = m_drainingQueue.First.Value; if (remove) m_drainingQueue.RemoveFirst(); if (rec.Time &lt; FIRST_VALID_TIME) { throw new InvalidOperationException("Detected invalid timestamp in Next(): " + rec.Time + " from record " + rec); } return rec; } else { lock (m_swapLock) { m_drainingQueue = m_fillingQueue; m_fillingQueue = new LinkedList&lt;DataRecord&gt;(); if (m_drainingQueue.Count == 0) drained = true; } } } return null; } </code></pre> <p>The consumer is rate-limited, so it can't get ahead of the consumer. </p> <p>The behavior I see is that sometimes the Time field is reading as DateTime.MinValue; by the time I construct the string to throw the exception, however, it's perfectly fine. </p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload