Databases
StackOverflow2013
Postsdbo
POHow to make a .NET process run out of memory without exhausting all system memory
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to make a .NET process run out of memory without exhausting all system memory
    primarykey
    data
    text
    <p>The problem is simple I have a process, that does ETL on some xml files. We started getting really large xml files and I started getting OutOfMemoryExceptions.</p> <p>Fixing the process is relatively simple. However, I'd like to make a unit test for my NUnit suite to make sure the process will continue to be able to handle really large files. However, actually running out of memory on my development workstation slows down my machine, and is time consuming. Storing a humongous test file in version control is also a bad idea. If I could artificially limit a process, thread or appdomain to only use a fixed amount of ram, lets say 128 megs, I could make a smaller unit test that would not bring my workstation, to its knees.</p> <p>Any suggestions? Is their some unmanaged API I can P/Invoke?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USbendewey
    1. USJustin Dearing
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. USGWLlosa
      1. VTFavorite
    3. VO
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI didn't want to put this in my main answer, because it's not exactly answering your question, but this just sounds like a weird thing to unit test. On a high level, either your algorithm for reading the files will handle very large files of arbitrary size (because it streams or chunks the file) or it won't. That isn't going to change very often, if ever, and I don't see what you gain from testing it every time with your test suite, especially given that (if it fails) it may not fail reliably every time on a given file.
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. USmquander
    2. COI open and scan the file several times. SOmetimes I do it as a stream, sometimes I do not. 99.9% of the files I scan are small enough that I can load the whole file into memory. The other .01% need to process. Therefore, if I make all the file operations stream based (fixing the issue), someone might add a new step that processes the file by loading it all into memory. Without a unit test that performs the entire ETL operation on a file big enough to eat all the processing ram, this will make it to production, because QA might not test the system with a really large file.
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. USJustin Dearing
    3. COI think that the best answer here is to encapsulate the operations you need to do very well, so that it's difficult for anyone else to accidentally go down the route of getting the file handle and opening the whole thing to try and fetch something. Of course, someone could still go and hack through your jungle to open the whole file if they tried hard enough, but I think you could prevent anyone from doing it out of sheer ignorance. (I know this is ancillary to the discussion of how to test it, but prevention is the best cure.)
      singulars
      1. POHow to make a .NET process run out of memory without exhausting all system memory
      1. USmquander
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload