Databases
StackOverflow2013
Postsdbo
PO.NET Heap fragmentation when using MSMQ async IO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO.NET Heap fragmentation when using MSMQ async IO
    primarykey
    data
    text
    <p>I have an application that reads from large amount of MSMQ queues(about 10000 at the moment). I use <code>queue.BeginPeek</code> with UInt32.MaxValue timeout to receive message from queue. When the message appears in queue, I process it and call <code>queue.BeginPeek</code> again. So I listen to all the queues, but message processing is done on Thread Pool. </p> <p>I noticed that memory usage slowly grows(two weeks of work cause growth from 200 MB to 800 MB). After investigating the dump file I see typical heap fragmentation picture with many free objects (some of them have size about several megabytes). And there are pinned objects between the holes.</p> <p>This seems to be common situation when working with calls to unmanaged code, that create pinned objects. But I did not find any solution in the internet. </p> <p>So is memory management in .NET so pure, that it does not allow to complete even such simple scenarios, or I miss something?</p> <p><strong>Edit :</strong> I've made some investigations in sample applications. The holes(free memory zones, so called free objects) between pinned objects are reused by GC when allocating memory for new objects. But in my production application, pinned objects are long living, and they finally appear in 2nd generation with the holes between them(because GC just shifts the border which separates generations). As I have really little normal long-living objects, I see this holes in 2nd generation in dump file. </p> <p>So memory consumption of my application can grow to 10000*(average size of the hole). (10000 is the number of queues that can also increase in future). I have no ideas how to fix this at the moment. The only way is to restart application from time to time.</p> <p>Once again I can only ask, why .NET does not have separate heap for pinned objects? (maybe this is newbie question). At the moment I see that calling async operations that work with unmanaged code can cause memory issues.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USstruhtanov
    1. USstruhtanov
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. USDmitry Borovsky
      1. VTFavorite
    3. VO
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COYes, lots of pinning going on when you've got 10,000 BeginPeek() calls going. Don't expect miracles. Growth doesn't mean much, you've only got a problem if that ultimately never stabilizes and crashes your app with OOM.
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. USHans Passant
    2. COI've edited my post, I got linear growth (from 200MB to 800MB in two weeks). I'm not sure when I'll get OOM, if this will happen, but this is still huge amount of memory. I can only repeat -- this scenario is typical in my opinion. I'm newbie in memory management tools, but why they can't implement separate heap for pinned objects? Another option is to create cache with OverlappedData structures, but as System.Messaging is not open source, it is hard to implement such behavior. With current GC implementation pinned objects can be used only in simple scenarios
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. USstruhtanov
    3. COHow many concurrent calls to BeginPeek do you have. And you always call EndPeek, right? In my estimation nothing should be left pinned one you quiesce the system (no outstanding calls).
      singulars
      1. PO.NET Heap fragmentation when using MSMQ async IO
      1. USusr
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload