Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I hope this is okay. It's my first answer on StackOverflow.</p> <p>Yes absolutely if your chunksize is over 85000 bytes then the array will get allocated on the large object heap. You will probably not run out of memory very quickly as you are allocating and deallocating contiguous areas of memory that are all the same size so when memory fills up the runtime can fit a new chunk into an old, reclaimed memory area.</p> <p>I would be a little worried about the Array.Resize call as that will create another array (see <a href="http://msdn.microsoft.com/en-us/library/1ffy6686(VS.80).aspx" rel="noreferrer">http://msdn.microsoft.com/en-us/library/1ffy6686(VS.80).aspx</a>). This is an unecessary step if actualLength==Chunksize as it will be for all but the last chunk. So I would as a minimum suggest:</p> <pre><code>if (actualLength != chunkSize) Array.Resize(ref buffer, actualLength); </code></pre> <p>This should remove a lot of allocations. If the actualSize is not the same as the chunkSize but is still > 85000 then the new array will also be allocated on the Large object heap potentially causing it to fragment and possibly causing apparent memory leaks. It would I believe still take a long time to actually run out of memory as the leak would be quite slow.</p> <p>I think a better implementation would be to use some kind of Buffer Pool to provide the arrays. You could roll your own (it would be too complicated) but WCF does provide one for you. I have rewritten your code slightly to take advatage of that:</p> <pre><code>BufferManager bm = BufferManager.CreateBufferManager(chunkSize * 10, chunkSize); for (int i = resumeChunk; i &lt; chunks; i++) { byte[] buffer = bm.TakeBuffer(chunkSize); try { fileStream.Position = i * chunkSize; int actualLength = fileStream.Read(buffer, 0, (int)chunkSize); if (actualLength == 0) break; //Array.Resize(ref buffer, actualLength); using (MemoryStream stream = new MemoryStream(buffer)) { UploadFile(stream, actualLength); } } finally { bm.ReturnBuffer(buffer); } } </code></pre> <p>this assumes that the implementation of UploadFile Can be rewritten to take an int for the no. of bytes to write.</p> <p>I hope this helps</p> <p>joe</p>
    singulars
    1. This table or related slice is empty.
    1. POMemorystream and Large Object Heap
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USJoe Simmonds
    1. USJoe Simmonds
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POMemorystream and Large Object Heap
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks a lot! This is seriously a brillian answer. Thanks for pointing out the Array.Resize issue. I also never heard of the BufferManager, sounds like this will help me lot in other areas, too. This is a lot more than I expected so I thought about starting a small bounty and giving it to you but I have to wait 23h after starting a bounty... So you have to wait, too :)
      singulars
      1. PO
      1. USflayn
    2. COThanks for that. I'm really glad I can be of help. Let me know if there's anything else. Looking back at it, it might be worth pointing out that the optimum implementation would share a single instance of the BufferManager across the whole service. I don't know how practical that would be for you.
      singulars
      1. PO
      1. USJoe Simmonds
    3. CO+1 Just stumbled across this one looking for an answer to a similar issue. Never heard of BufferManager before - awesome! Guess this will be something to remember for the future.
      singulars
      1. PO
      1. USalexander.biskop
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload