StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POPyODBC Cursor.fetchall() causes python to crash (segfault)
text
Body
copied!<p>I am using Python 2.7 on Windows XP.</p> <p>I have a simple python script on a schedule that uses pyodbc to grab data from an AR database which has worked perfectly until today. I get a segfault once the cursor reaches a particular row. I have similar code in C++ which has no problem retrieving the results, so I figure this is an issue with pyodbc. Either way, I'd like to "catch" this error. I've tried to use the subprocess module, but it doesn't seem to work since once the script hits a segfault it just hangs on the "python.exe has encountered a problem and needs to close." message. I guess I could set some arbitrary time frame for it to complete in and, if it doesn't, force close the process, but that seems kind of lame.</p> <p>I have reported the issue here as well - <a href="http://code.google.com/p/pyodbc/issues/detail?id=278" rel="nofollow">http://code.google.com/p/pyodbc/issues/detail?id=278</a></p> <p>@paulsm4 - I have answered your questions below, thanks!</p> <blockquote> <p>Q: You're on Windows/XP (32-bit, I imagine), Python 2.7, and BMC Remedy AR. Correct?</p> </blockquote> <p>A: Yes, it fails on Win XP 32 bit and Win Server 2008 R2 64 bit.</p> <blockquote> <p>Q: Is there any chance you (or perhaps your client, if they purchased Remedy AR) can open a support call with BMC?</p> </blockquote> <p>A: Probably not...</p> <blockquote> <p>Q: Can you isolate which column causes the segfault? "What's different" when the segfault occurs?</p> </blockquote> <p>A: Just this particular row...but I have now isolated the issue with your suggestions below. I used a loop to fetch each field until a segfault occurred.</p> <pre><code>cursor.columns(table="mytable") result = cursor.fetchall() columns = [x[3] for x in result] for x in columns: print x cursor.execute("""select "{0}" from "mytable" where id = 'abc123'""".format(x)) cursor.fetchall() </code></pre> <p>Once I identified the column that causes the segfault I tried a query for all columns EXCEPT that one and sure enough it worked no problem.</p> <p>The column's data type was CHAR(1024). I used C++ to grab the data and noticed that the column for that row had the most characters in it out of any other row...1023! Thinking that maybe there is a buffer in the C code for PyODBC that is getting written to beyond its boundaries.</p> <blockquote> <p>2) Enable ODBC tracing: <a href="http://support.microsoft.com/kb/274551" rel="nofollow">http://support.microsoft.com/kb/274551</a></p> <p>3) Post back the results (including the log trace of the failure)</p> </blockquote> <p>Ok, I have created a pastebin with the results of the ODBC trace - <a href="http://pastebin.com/6gt95rB8" rel="nofollow">http://pastebin.com/6gt95rB8</a>. To protect the innocent, I have masked some string values.</p> <p>Looks like it may have been due to data truncation.</p> <p>Does this give us enough info as to how to fix the issue? I'm thinking it's a bug within PyODBC since using the C ODBC API directly works fine.</p> <p><b>Update</b></p> <p>So I compiled PyODBC for debugging and I got an interesting message -</p> <pre><code>Run-Time Check Failure #2 - Stack around the variable 'tempBuffer' was corrupted. </code></pre> <p>While I don't currently understand it, the call stack is as follows -</p> <pre><code>pyodbc.pyd!GetDataString(Cursor * cur=0x00e47100, int iCol=0) Line 410 + 0xf bytes C++ pyodbc.pyd!GetData(Cursor * cur=0x00e47100, int iCol=0) Line 697 + 0xd bytes C++ pyodbc.pyd!Cursor_fetch(Cursor * cur=0x00e47100) Line 1032 + 0xd bytes C++ pyodbc.pyd!Cursor_fetchlist(Cursor * cur=0x00e47100, int max=-1) Line 1063 + 0x9 bytes C++ pyodbc.pyd!Cursor_fetchall(_object * self=0x00e47100, _object * args=0x00000000) Line 1142 + 0xb bytes C++ </code></pre> <p><b>Resolved!</b></p> <p>Problem was solved by ensuring that the buffer had enough space.</p> <p>In <code>getdata.cpp</code> on line 330</p> <pre><code>char tempBuffer[1024]; </code></pre> <p>Was changed to</p> <pre><code>char tempBuffer[1025]; </code></pre> <p>Compiled and replaced the old pyodbc.pyd file in site-packages and we're all good!</p> <p>Thanks for your help!</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload