Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>What sort of problem are you running into, and what does "large" mean to you? I have friends you need to load 200 Gb files into memory, so their idea of good tips is a lot different than the budget shopper for minimal VM slices suffering with 250 Mb of RAM (really? My phone has more than that).</p> <p>In general, Perl holds on to any memory you use, even if it's not using it. Realize that optimizing in one direction, e.g. memory, might negatively impact another, such as speed.</p> <p>This is not a comprehensive list (and there's more in <a href="http://www.programmingperl.org" rel="nofollow noreferrer">Programming Perl</a>):</p> <p>☹ Use Perl memory profiling tools to help you find problem areas. See <a href="https://stackoverflow.com/q/6645073/8817">Profiling heap memory usage on perl programs</a> and <a href="https://stackoverflow.com/questions/106555/how-to-find-the-amount-of-physical-memory-occupied-by-a-hash-in-perl">How to find the amount of physical memory occupied by a hash in Perl?</a></p> <p>☹ Use lexical variables with the smallest scope possible to allow Perl to re-use that memory when you don't need it.</p> <p>☹ Avoid creating big temporary structures. For instance, reading a file with a <code>foreach</code> reads all the input at once. If you only need it line-by-line, use <code>while</code>.</p> <pre><code> foreach ( &lt;FILE&gt; ) { ... } # list context, all at once while( &lt;FILE&gt; ) { ... } # scalar context, line by line </code></pre> <p>☹ You might not even need to have the file in memory. <a href="http://www.effectiveperlprogramming.com/2010/01/memory-map-files-instead-of-slurping-them/" rel="nofollow noreferrer">Memory-map files instead of slurping them</a></p> <p>☹ If you need to create big data structures, consider something like <a href="https://www.metacpan.org/module/DBM::Deep" rel="nofollow noreferrer">DBM::Deep</a> or other storage engines to keep most of it out of RAM and on disk until you need it.</p> <p>☹ Don't let people use your program. Whenever I've done that, I've reduced the memory footprint by about 100%. It also cuts down on support requests.</p> <p>☹ Pass large chunks of text and large aggregates by reference so you don't make a copy, thus storing the same information twice. If you have to copy it because you want to change something, you might be stuck. This goes both ways as subroutine arguments and subroutine return values:</p> <pre><code> call_some_sub( \$big_text, \@long_array ); sub call_some_sub { my( $text_ref, $array_ref ) = @_; ... return \%hash; } </code></pre> <p>☹ Track down memory leaks in modules. I had big problems with an application until I realized that <a href="http://blogs.perl.org/users/brian_d_foy/2011/03/fixing-yamlxss-memory-leak.html" rel="nofollow noreferrer">a module wasn't releasing memory</a>. I found a patch in the module's RT queue, applied it, and solved the problem.</p> <p>☹ If you need to handle a big chunk of data once but don't want the persistent memory footprint, offload the work to a child process. The child process only has the memory footprint while it's working. When you get the answer, the child process shuts down and releases it memory. Similarly, work distribution systems, such as <a href="https://metacpan.org/release/Gearman" rel="nofollow noreferrer">Gearman</a>, can spread work out among machines.</p> <p>☹ Turn recursive solutions into iterative ones. Perl doesn't have tail recursion optimization, so every new call adds to the call stack. You can optimize the tail problem yourself with tricks with <a href="http://perldoc.perl.org/functions/goto.html" rel="nofollow noreferrer">goto</a> or a module, but that's a lot of work to hang onto a technique that you probably don't need.</p> <p>☹ Did he use 6 Gb or only five? Well, to tell you the truth, in all this excitement I kind of lost track myself. But being as this is Perl, the most powerful language in the world, and would blow your memory clean off, you've got to ask yourself one question: Do I feel lucky? Well, do ya, punk? </p> <p>There are many more, but it's too early in the morning to figure out what those are. I cover some in <a href="http://www.masteringperl.org" rel="nofollow noreferrer">Mastering Perl</a> and <a href="http://www.effectiveperlprogramming.com" rel="nofollow noreferrer">Effective Perl Programming</a>.</p>
    singulars
    1. This table or related slice is empty.
    1. POTips for keeping Perl memory usage low
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USCommunity
    1. USbrian d foy
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POTips for keeping Perl memory usage low
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs it correct that when you create a hash Perl will never release the memory it used? Can you re-use the same hash for something else to save memory?
      singulars
      1. PO
      1. USØyvind Skaar
    2. COIIRC, memory is not release back out of the program, but the program itself will reuse that which has already been claimed. Therefore if your hash goes out of scope, and you then make a new hash it will likely use the same memory that the old one occupied (many caveats, but close enough). Also IIRC this is not a Perl-specific problem but basically the way the OSes distribute memory to programs.
      singulars
      1. PO
      1. USJoel Berger
    3. COAlso I think Reini Urban is probably the guy to ask for deeper looks on reduced-memory-Perling. Checkout his [technical blog](http://blogs.perl.org/users/rurban/) if you dare.
      singulars
      1. PO
      1. USJoel Berger
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload