Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I had the same question and was disappointed when I read Sven's reply. Seems as though numpy would be missing out on some key functionality if you couldn't have a huge array on file and work on little pieces of it at a time. Your case seems to be close to one of the use cases in the origional rational for making the .npy format (see: <a href="http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt" rel="noreferrer">http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt</a>).</p> <p>I then ran into numpy.lib.format, which seems to be full useful goodies. I have no idea why this functionality is not available from the numpy root package. The key advantage over HDF5 is that this ships with numpy.</p> <pre><code>&gt;&gt;&gt; print numpy.lib.format.open_memmap.__doc__ """ Open a .npy file as a memory-mapped array. This may be used to read an existing file or create a new one. Parameters ---------- filename : str The name of the file on disk. This may not be a filelike object. mode : str, optional The mode to open the file with. In addition to the standard file modes, 'c' is also accepted to mean "copy on write". See `numpy.memmap` for the available mode strings. dtype : dtype, optional The data type of the array if we are creating a new file in "write" mode. shape : tuple of int, optional The shape of the array if we are creating a new file in "write" mode. fortran_order : bool, optional Whether the array should be Fortran-contiguous (True) or C-contiguous (False) if we are creating a new file in "write" mode. version : tuple of int (major, minor) If the mode is a "write" mode, then this is the version of the file format used to create the file. Returns ------- marray : numpy.memmap The memory-mapped array. Raises ------ ValueError If the data or the mode is invalid. IOError If the file is not found or cannot be opened correctly. See Also -------- numpy.memmap """ </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POHow can I create a numpy .npy file in place on disk?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USkiyo
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHow can I create a numpy .npy file in place on disk?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI've just got back to this to try it out. It works - thanks a lot. A word of caution: if you use the wrong mode string it might be silently accepted and a file will be created, but the header won't be written properly. The only mode string for writing a new file is 'w+'.
      singulars
      1. PO
      1. UShamish
    2. COThat seems worrisome. Maybe file a bug report?
      singulars
      1. PO
      1. USkiyo
    3. COI just used this method to write a 38G file and read (at least a few rows) out of it successfully. It is my understanding that this is not supposed to work bc there is a 2GB limit on this files. Any ideas if this limit is still true or how to observe whatever problem might happen when dealing with large files?
      singulars
      1. PO
      1. USmathtick
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload