Databases
StackOverflow2013
Postsdbo
POSegmentation Fault while using Pandas in uWSGI
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSegmentation Fault while using Pandas in uWSGI
    primarykey
    data
    text
    <p>while doing a select from a HDF5 Store in Pandas running as a uWSGI process under Flask, I get a segmentation fault. It does several chunks fine but then runs into a problem after a while. The <em>*</em> are my own logging, basically we get to round 16 of the process, which compares a chunk of 14531 rows to a base table and then saves the results. It goes wrong while retrieving the data from HDF5:</p> <pre><code>*** Processing Chunk: 16 *** Saving # Rows: 14531 *** Measure list: [-3, -2] *** Retrieve Data *** No Cache In Memory! !!! uWSGI process 14786 got Segmentation Fault !!! *** backtrace of 14786 *** uwsgi(uwsgi_backtrace+0x29) [0x45f029] uwsgi(uwsgi_segfault+0x21) [0x45f1b1] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fb50c92c4a0] /srv/www/li/venv/local/lib/python2.7/site-packages/pandas/hashtable.so(+0x10ae9) [0x7fb506cc9ae9] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /srv/www/li/venv/local/lib/python2.7/site-packages/pandas/index.so(+0xda6c) [0x7fb505f90a6c] /srv/www/li/venv/local/lib/python2.7/site-packages/pandas/index.so(+0xac77) [0x7fb505f8dc77] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3bff) [0x7fb50cf88e2f] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(+0x5c86d) [0x7fb50cf4a86d] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(+0x12539f) [0x7fb50d01339f] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(+0xc7676) [0x7fb50cfb5676] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3bff) [0x7fb50cf88e2f] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(+0x5c970) [0x7fb50cf4a970] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2b2a) [0x7fb50cf87d5a] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fb50cf8b37b] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fb50cf8a650] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fb50cf8b37b] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(+0x5c970) [0x7fb50cf4a970] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2b2a) [0x7fb50cf87d5a] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(+0x5c970) [0x7fb50cf4a970] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2b2a) [0x7fb50cf87d5a] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fb50cf8b37b] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fb50cf8b37b] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fb50cf8b37b] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fb50cf4a6b5] /usr/lib/libpython2.7.so.1.0(+0x5c86d) [0x7fb50cf4a86d] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(+0x12539f) [0x7fb50d01339f] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(+0x13153c) [0x7fb50d01f53c] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fb50d02f053] /usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb50d02f9a7] uwsgi(python_call+0x1f) [0x470daf] uwsgi(uwsgi_request_wsgi+0x11e) [0x472ffe] *** end of backtrace *** DAMN ! worker 1 (pid: 14786) died, killed by signal 11 :( trying respawn ... Respawned uWSGI worker 1 (new pid: 15975) mapping worker 1 to CPUs: 0 </code></pre> <p>Does anyone have any idea what is causing this? If i run the process using the Flask development :5000 port, it runs fine from start to end. It's also not a lack of memory issue (each uWSGI has sufficient memory assigned and it doesn't run out of it during the run). I'm aware i should decouple the process from uWSGI to Celery or something in the near future ;) but it would be nice if it just would work for now! I don't think it's uWSGI harakiri either (which would give a different log anyway)</p> <p>The setup:</p> <ul> <li>Flask: 0.10.0</li> <li>uWSGI: 1.9.18</li> <li>Pandas: 0.12.0</li> <li>Python: 2.7.3</li> <li>Ubuntu 12.04LTS Server, 64bit</li> <li>32gb ram, 4 8gb uwsgi workers</li> </ul> <p>the uwsgi config:</p> <pre><code>&lt;uwsgi&gt; &lt;uid&gt;www-data&lt;/uid&gt; &lt;gid&gt;www-data&lt;/gid&gt; &lt;socket&gt;/tmp/li.socket&lt;/socket&gt; &lt;chmod-socket&gt;666&lt;/chmod-socket&gt; &lt;chdir&gt;/srv/www/li&lt;/chdir&gt; &lt;pythonpath&gt;/srv/www/li&lt;/pythonpath&gt; &lt;virtualenv&gt;/srv/www/li/venv&lt;/virtualenv&gt; &lt;module&gt;li&lt;/module&gt; &lt;wsgi-file&gt;/srv/www/li/li.py&lt;/wsgi-file&gt; &lt;callable&gt;app&lt;/callable&gt; &lt;master/&gt; &lt;processes&gt;4&lt;/processes&gt; &lt;pidfile&gt;/tmp/li.pid&lt;/pidfile&gt; &lt;harakiri&gt;12000&lt;/harakiri&gt; &lt;reload-mercy&gt;8&lt;/reload-mercy&gt; &lt;cpu-affinity&gt;1&lt;/cpu-affinity&gt; &lt;stats&gt;/tmp/stats.socket&lt;/stats&gt; &lt;max-requests&gt;2000&lt;/max-requests&gt; &lt;limit-as&gt;8192&lt;/limit-as&gt; &lt;reload-on-as&gt;8192&lt;/reload-on-as&gt; &lt;reload-on-rss&gt;8192&lt;/reload-on-rss&gt; &lt;no-orphans/&gt; &lt;vacuum/&gt; &lt;/uwsgi&gt; </code></pre> <p>Any help will be highly appreciated!</p> <p>Edit: after trying a series of configs, 3 workers with 10gb do not run into segmentation faults. it's still really weird as the workers do not use more than 4.5gb at most and 1.8gb normally...</p> <p>Edit 2: For anyone finding this by google, in the end i created a pre-fork caching solution meaning that I could have many workers but still be very memory efficient. it does mean you have to restart a uwsgi process after data is altered though</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCarst
    1. USCarst
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POSegmentation Fault while using Pandas in uWSGI
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHDFStore is not thread or process safe for writing. you CAN read multiple processes, but writing to a single store can only be from a single process at a time, see (here)[http://pandas.pydata.org/pandas-docs/dev/io.html#notes-caveats] ; HDF5 is itself not safe for synchronized writing
      singulars
      1. POSegmentation Fault while using Pandas in uWSGI
      1. USJeff
    2. COyes i confirm the previous comment, you can use uwsgi.lock() and uwsgi.unlock() to serialize writes (independently by processes or threads)
      singulars
      1. POSegmentation Fault while using Pandas in uWSGI
      1. USroberto
    3. COHi, thanks for your comments Jeff, Roberto. The only thing is: there are no concurrent writes by multiple uwsgi processes (I've built in a check that guarantees a unique write process per HDF5 file). Also, the error happens during a read, not a write... I'll look into the lock() but it shouldn't cause it...
      singulars
      1. POSegmentation Fault while using Pandas in uWSGI
      1. USCarst
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload