Databases
StackOverflow2013
Postsdbo
POIs it possible to map a discontiuous data on disk to an array with python?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs it possible to map a discontiuous data on disk to an array with python?
    primarykey
    data
    text
    <p>I want to map a big fortran record (12G) on hard disk to a numpy array. (Mapping instead of loading for saving memory.)</p> <p>The data stored in fortran record is not continuous as it is divided by record markers. The record structure is as "marker, data, marker, data,..., data, marker". The length of data regions and markers are known. </p> <p>The length of data between markers is not multiple of 4 bytes, otherwise I can map each data region to an array.</p> <p>The first marker can be skipped by setting offset in memmap, is it possible to skip other markers and map the data to an array?</p> <p>Apology for possible ambiguous expression and thanks for any solution or suggestion.</p> <hr> <p>Edited May 15</p> <p>These are fortran unformatted files. The data stored in record is a (1024^3)*3 float32 array (12Gb). </p> <p>The record layout of variable-length records that are greater than 2 gigabytes is shown below:</p> <p><img src="https://i.stack.imgur.com/WNqIJ.jpg" alt="data structure"></p> <p>(For details see <a href="http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-win/index.htm#GUID-64D43E4C-68E7-4C48-8B50-B49F1F7DA46C.htm" rel="nofollow noreferrer">here</a> -> the section [Record Types] -> [Variable-Length Records].)</p> <p>In my case, except the last one, each subrecord has a length of 2147483639 bytes and separated by 8 bytes (as you see in the figure above, a end marker of the previous subrecord and a begin marker of the following one, 8 bytes in total ) .</p> <p>We can see the first subrecord ends with the first 3 bytes of certain float number and the second subrecord begins with the rest 1 byte as 2147483639 mod 4 =3.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USSyrtis Major
    1. USSyrtis Major
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    3. VO
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COCan you give us a bit more details about the data structure? Based on what I think you're saying, you have variable-length arrays between your markers? How are they packed (e.g. float, int8, int16, whatever)?
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. USJoe Kington
    2. COThanks for attention and sorry for lack of details. More imformation is added. I'm trying h5py as suggested by Castro.
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. USSyrtis Major
    3. COSorry, I fogot to notify you @JoeKington.
      singulars
      1. POIs it possible to map a discontiuous data on disk to an array with python?
      1. USSyrtis Major
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload