StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POphp script outpacing garbage collection?
text
Body
copied!<p>I am writing a script to create sitemaps for an ecommerce site with ~700k products. The ecommerce package already had a built-in sitemap generator but it doesn't format properly for large product databases (master with links to gzipped 50k url files).</p> <p>I am running into a memory management problem when running the script and it seems as though the script is outpacing php's ability to garbage collect. Here is what I have:</p> <pre><code>function buildSitemapFile() { // prep sitemap string $content = "sitemap header info"; $max_entries_per_sitemap = 50000; $current_batch = 0; while (buildSitemapZip($current_batch, $content, $max_entries_per_sitemap)) { $current_batch++; } // finish sitemap string, open/create file, write to it, close file } function buildSitemapZip($batch, &$main_content, $max_urls = 50000) { $file_name = *create file path string* // add this file to the index file's content string $main_content .= *this sitemap info* // open and prep this batch's site map file $handle = fopen($file_name, "w"); $content = *sitemap header info* require_once('path_to_Product.class.php'); $product = new Product(); $group_size = 1000; // on first time called, $group_start = 0 (0 * 50000), group_start loops till >= 50000 ((0+1) * 50000), and group_start increments by $group_size on each pass for ($group_start = $batch * $max_urls; $group_start < ($batch + 1) * $max_urls; $group_start += $group_size) { // request the next group of products. the query ends with "order by id limit $group_start, $group_size" $prods = $product->query_db_for_products($group_start, $group_size); // loop product set adding urls to the content string foreach($prods as $key => $prod) { $content .= *get info from product object to fill out the sitemap content string* } if (count($prods) != $group_size) { // end of list $return_val = *stop looping* break; } } // complete sitemap string, write it to file, close file // create and open gzip file, write to it, close it, erase temp sitemap file return $return_val; } </code></pre> <p>When I run the script with smaller $group_size values it doesn't seem to be such an issue, but it is very slow and starts to slow down access to the database (through repeated queries).</p> <p>The eccomerce package has no __construct of __destruct methods on it's objects.</p> <p>The last time I ran the test was with a $group_size of 10k (was supposed to be 1k...) and the mem usage got out of control. I stopped the script but the mem usage (as per the top shell command) kept increasing and several hours later hasn't been released.</p> <p>Any thoughts as to what might be going on with the script to cause the problem?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload