StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POWhy is my C module leaking memory?
text
Body
copied!<p>I'm reading lists from a large file, which I eventually want to store as <code>array.array</code>s. Because</p> <pre><code>map(int, line.split()) </code></pre> <p>is very slow, I wrote a small C module which does strtok and a faster version of atoi:</p> <pre><code>inline long minhashTables_myatoi(const char* s) { int r; for (r = 0; *s; r = r * 10 + *s++ - '0'); return r; } static PyObject* minhashTables_ints(PyObject *self, PyObject *args) { char* s; Py_ssize_t slen; if(!PyArg_ParseTuple(args, "s#", &s, &slen)) return NULL; long* buf = malloc(sizeof(long) * (slen+1)/2); const char* tok = strtok(s, " "); buf[0] = minhashTables_myatoi(tok); Py_ssize_t i; for(i = 1; (tok = strtok(NULL, " ")) != NULL; i++) buf[i] = minhashTables_myatoi(tok); Py_ssize_t buflen = i; PyObject* list = PyList_New(buflen); PyObject *o; for(i = 0; i < buflen; i++) { o = PyInt_FromLong(buf[i]); PyList_SET_ITEM(list, i, o); } free(buf); return list; } </code></pre> <p>So my python script calls <code>ints()</code> with a string and passes it to the <code>array.array</code> constructor and saves the resulting array in a <code>list</code>.</p> <p>My problem is, that now the script leaks memory, which it did not with the map instead of the <code>ints()</code> function, of course.</p> <p>Also using my own version of Pythons <code>int()</code> using a C module does not leak memory.</p> <p>Thanks for your help!</p> <p><strong>Edit:</strong> To valgrind the module I used this script:</p> <pre><code>import minhashTables data = ' '.join(map(str, range(10))) print 'start' foo = minhashTables.ints(data) del data del foo print 'stop' </code></pre> <p>And I run <code>valgrind --tool=memcheck --leak-check=full --show-reachable=yes python test.py</code>, but there so no output from valgrind between <code>start</code> and <code>stop</code>, through there are tons before and afterwards.</p> <p><strong>Edit:</strong> Code for confirming it's leaking: import minhashTables</p> <pre><code>for i in xrange(1000000000): data = ' '.join(map(str, range(10, 10000))) foo = minhashTables.ints(data) </code></pre> <p>I have to recreate the string, because strtok changes it. By the way, copying the string into another memory location doesn't change the behavior.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload