Databases
StackOverflow2013
Postsdbo
PONumpy reshape yields a different size error
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONumpy reshape yields a different size error
    primarykey
    data
    text
    <p>I have a piece of Python code which has been used for about a year without any issues (it reads, uncompresses/unpacks data, selects a window and plots it using Numpy/Matplotplib). </p> <p>We recently got a new machine which stores the data in a 64-bit encoded format instead of a 32-bit encoding which should not be a real issue. I rewrote a few lines of the code to deal with this (both functions of interest are included below) but I keep getting an error that I do not quite understand.</p> <p><strong>The Error:</strong></p> <pre><code>Traceback (most recent call last): File "./3D_Visualizer.py", line 238, in &lt;module&gt; File "./3D_Visualizer.py", line 232, in main main() File "./3D_Visualizer.py", line 93, in plot_data ax.set_ylabel('Time (s)',color='r') File "/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 170, in reshape return _wrapit(a, 'reshape', newshape, order=order) File "/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) ValueError: total size of new array must be unchanged </code></pre> <p>I would love if someone is able to explain to me why this error occurs as to my understanding the numpy reshape function takes the size of the input array for the actual resize. The code of the 2 functions which lead to this error are listed below:</p> <p><strong>Reading function:</strong></p> <pre><code>def numpy_array(data, peaks): """Fills the NumPy array 'data' with m/z-intensity values acquired from b64 decoding and unpacking the binary string read from the mzXML file, which is stored in the list 'peaks'. The m/z values are assumed to be ordered without validating this assumption. Note: This function is the performance bottleneck """ rt_counter=0 for x in peaks: if rt_counter %(len(peaks)/20) == 0: update_progress() peak_counter=0 data_buff=base64.b64decode(x) endian = '!' precision = 'd' buff_size = len(data_buff) / struct.calcsize(endian + precision) index=0 for y in struct.unpack(endian + precision * buff_size, data_buff[0:len(data_buff)]): if (index % 2 == 0): data[rt_counter][1][peak_counter][0]= y else: data[rt_counter][1][peak_counter][1]= y peak_counter+=1 index+=1 rt_counter+=1 </code></pre> <p><strong>The plotting function:</strong></p> <pre><code>def plot_data(X,Y,Z): """Plots a 3D wireframe based on the x, y and z datapoints passed to this function in the python lists 'X', 'Y' and 'Z'. Custom labels are created for the Y (m/z) axis since matplotlib creates 'ugly' labels by default (TODO: labelling goes wrong). """ fig=plt.figure() x=sorted(set(X)) y=sorted(set(Y)) labels=['%.2f'%k for k in y] XX,YY=np.meshgrid(y,x) ZZ=np.reshape(Z,XX.shape) ax=fig.add_subplot(111,projection='3d') ax.plot_wireframe(XX,YY,ZZ) ax.set_title('3D projection of LC-MS region',size='large',color='r') ax.set_xlabel('m/z',color='r',style='italic') ax.set_xticklabels(labels) ax.set_ylabel('Time (s)',color='r') ax.set_zlabel('Intensity',color='r') plt.show() </code></pre> <p><strong>-- 30/07/13 2:10 --</strong></p> <p>the X, Y and Z all have the same length in the test case that throws this error (namely 184). The length for x and y is 18 and 20 respectively after <code>set</code> lines in the error test case while they were 18 and 11 after <code>set</code> lines in the working test case.</p> <p>example contents of <code>y</code> in the error case (64-bit encoded):</p> <pre><code>[1398.51513671875, 1398.5152587890625, 1398.5225830078125, 1398.522705078125, 1398.530029296875, 1398.5301513671875, 1398.5374755859375, 1398.53759765625, 1398.5447998046875, 1398.544921875, 1398.55224609375, 1398.5523681640625, 1398.5596923828125, 1398.559814453125, 1398.567138671875, 1398.5672607421875, 1398.5745849609375, 1398.5819091796875, 1398.58203125, 1398.58935546875] </code></pre> <p>example contents of 'y' in the working case (32-bit encoded):</p> <pre><code>[1398.51171875, 1398.5191650390625, 1398.526611328125, 1398.533935546875, 1398.5413818359375, 1398.548828125, 1398.5562744140625, 1398.5635986328125, 1398.571044921875, 1398.5784912109375, 1398.5859375] </code></pre> <p>This shows that in the error case the decoding has issues regarding fitting the value, I think?</p> <p><strong>-- 31/07/13 10:20 --</strong></p> <p>I dove back into the raw data acquired from the actual machine and it showed that the measurement coordinates are not the same for all time points (something which was the case for all previous versions of the machine + controlling software). This has the effect that certain coordinates are ever so slightly shifted a litle bit (ie 1398.5152... vs 1398.5151) causing the reshape to fail. </p> <p>I am currently in the process of just assigning each coordinate to an integer value (1, 2 ...) to 'hot fix' this for now.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBas Jansen
    1. USBas Jansen
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONumpy reshape yields a different size error
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI'm actually surprised the lengths of 18 and 11 (x and y) work, because 18 * 11 != 184, and the reshape() call should fail. I have to assume that the length of Z is different in the working case (198 perhaps)? The origin of those different sizes will indeed be in the unpacking.
      singulars
      1. PONumpy reshape yields a different size error
      1. This table or related slice is empty.
    2. COHave you compared data_buff (or part of it) for the working and the failing cases? The decoded data string should be the same, right? So that will simply tell you whether that part causes the failure.
      singulars
      1. PONumpy reshape yields a different size error
      1. This table or related slice is empty.
    3. COCan you cut this down to the _minimum_ needed to generate the error?
      singulars
      1. PONumpy reshape yields a different size error
      1. UStacaswell
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload