Databases
StackOverflow2013
Postsdbo
POHow to remove duplicate items from a queue within a time frame?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to remove duplicate items from a queue within a time frame?
    primarykey
    data
    text
    <p>I would like to remove duplicate entries from a queue in an efficient way. The queue has a custom class with DateTime and FullPath and a few other things</p> <pre><code>private Queue&lt;MyCustomClass&gt; SharedQueue; </code></pre> <p>The DateTime in the class is the timestamp when inserted into the queue. The logic I would like to use is as following: Remove duplicates from the queue if the FullPath is identical within a 4 second window (i.e. if added to queue within 4 seconds of a duplicate fullpath). I have the events that I want to watch but a few duplicates will still arrive and that is OK.</p> <p>I am using c# 2.0 and the FileSystemWatcher class and a worker queue.</p> <p>There are a bunch of ways to do this: Trim the queue each time an item is added to it, or when I am working on the queue skip the processing of the current duplicate item.</p> <p>Or should I use a 'global private' variable Dictionary&lt; String, DateTime> ? So I can quickly search it? or a local copy of the queue ? Perhaps it is best to limit the local queue to 100 items in case of many file events? Though in my case it 'should be' only a relatively few files to monitor in a folder... but things always change...</p> <p>Thanks for any help.</p> <p>:Edit: Feb 10 8:54 EST: So I decided to implement a good simple solution as far as I can tell. I don't think I am holding on to the Dict keys too long...</p> <p>:Edit: Feb 10 9:53 EST: Updated as my Dictionary cannot contain duplicate values.</p> <pre><code> public void QueueInput(HotSynchUnit.RcdFSWFile rcd) // start the worker thread when program starts. // call Terminate.Set() in the programs exit routine or close handler etc. { // lock shared queue lock (SharedQueue) { if (!IsDuplicateQueueInput(rcd)) // only add unique values to queue { SharedQueue.Enqueue(rcd); SomethingToDo.Set(); } } } // public void QueueInput private bool IsDuplicateQueueInput(HotSynchUnit.RcdFSWFile rcd) /* Return true if the object is a duplicate object. * Pseudo Code: * * isDuplicate = false * Lock Dictionary * -If lastTimeStamp &gt; 4 seconds ago then // Optimization: save lastTimeStamp * if Dict.Count &gt; 0 then clear Dictionary * return isDuplicate * -If not Dict.TryGetValue(sPath, dtTimeStamp) then * Dict.AddKey() * -Else * Compare key timestamp to Currenttime * if key timestamp is &lt;= 4 seconds ago then * IsDuplicate = True * * Dict.RemoveKey() * Dict.AddKey() * * return isDuplicate */ { // put real code here } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USuser610064
    1. USuser610064
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COWhy 'global private' dictionary? Why not just put it at the same scope as your queue?
      singulars
      1. POHow to remove duplicate items from a queue within a time frame?
      1. USrecursive
    2. COThat is actually where it is. My bad.
      singulars
      1. POHow to remove duplicate items from a queue within a time frame?
      1. USuser610064
    3. COThank you @StevenJeuris for the idea of checking before I Enqueue - I forgot that I had that routine there. I did not want to get into overriding or extending classes...
      singulars
      1. POHow to remove duplicate items from a queue within a time frame?
      1. USuser610064
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload