StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POFastest way to manage large ammount of data with PHP? (data upto 100mb altogether per client request)
text
Body
copied!<p><strong>The situation:</strong> </p> <ul> <li>I have multiple arrays of containing multiple complex objects, each storing different data, but in same format.</li> <li>Now, these arrays (containing objects) are too complex to be stored in a sql table, so i serialize them, and store each array in separate files.</li> <li>I use PHP function <code>file_get_contents()</code> read the data, and then i use <code>unserialize()</code> on the data. </li> <li>I have to load one file (max 100mb) per client request and 'unserialize()' it and process it.</li> <li>This data is not the same for every client</li> <li>All data in total is around 3GB. </li> <li>This data is updated every 24 hours, and the size of data is increased per update. </li> <li>Maximum data per file is 100mb.</li> </ul> <p><strong>The problem:</strong> </p> <ul> <li>The method i currently am using works fine for small file sizes (upto 5mb).</li> <li>But when it comes to larger files of sizes, its taking too much time.</li> <li>The function <code>unserialize()</code> is taking about 33 seconds to execute if i try to load a file with size around 40mb.</li> <li>So the main problem with my current method is with <code>unserialize()</code>.</li> </ul> <p><strong>The main question:</strong> </p> <ul> <li>How can i store my very complex objects without serializing them, or how can i make my unserialization faster?</li> </ul>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload