Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>As requested, I'm showing an example of <a href="http://www.mathworks.com/help/matlab/memory-mapping.html" rel="nofollow noreferrer">memory-mapped</a> files using <a href="http://www.mathworks.com/help/matlab/ref/memmapfile.html" rel="nofollow noreferrer"><code>memmapfile</code></a> class.</p> <p>Since you didn't provide the exact format of the data file, I will create my own. The data I am creating is a table of <code>N</code> rows, each consisting of 4 columns:</p> <ul> <li>first is a <code>double</code> scalar value</li> <li>second is a <code>single</code> value</li> <li>third is a fixed-length string representing a <code>uint32</code> in HEX notation (e.g: <code>D091BB44</code>)</li> <li>fourth column is a <code>uint8</code> value</li> </ul> <p>The code to generate the random data, and write it to binary file structured as described above:</p> <pre><code>% random data N = 10; data = [... num2cell(rand(N,1)), ... num2cell(rand(N,1,'single')), ... cellstr(dec2hex(randi(intmax('uint32'), [N,1]),8)), ... num2cell(randi([0 255], [N,1], 'uint8')) ... ]; % write to binary file fid = fopen('file.bin', 'wb'); for i=1:N fwrite(fid, data{i,1}, 'double'); fwrite(fid, data{i,2}, 'single'); fwrite(fid, data{i,3}, 'char'); fwrite(fid, data{i,4}, 'uint8'); end fclose(fid); </code></pre> <p>Here is the resulting file viewed in a HEX editor:</p> <p><img src="https://i.stack.imgur.com/vmXjU.png" alt="binary file viewed in a hex editor"></p> <p>we can confirm the first record (note that my system uses <a href="https://en.wikipedia.org/wiki/Endianness" rel="nofollow noreferrer">Little-endian</a> byte ordering):</p> <pre><code>&gt;&gt; num2hex(data{1,1}) ans = 3fd4d780d56f2ca6 &gt;&gt; num2hex(data{1,2}) ans = 3ddd473e &gt;&gt; arrayfun(@dec2hex, double(data{1,3}), 'UniformOutput',false) ans = '46' '35' '36' '32' '37' '35' '32' '46' &gt;&gt; dec2hex(data{1,4}) ans = C0 </code></pre> <hr> <p>Next we open the file using memory-mapping:</p> <pre><code>m = memmapfile('file.bin', 'Offset',0, 'Repeat',Inf, 'Writable',false, ... 'Format',{ 'double', [1 1], 'd'; 'single', [1 1], 's'; 'uint8' , [1 8], 'h'; % since it doesnt directly support char 'uint8' , [1 1], 'i'}); </code></pre> <p>Now we can <a href="http://www.mathworks.com/help/matlab/import_export/constructing-a-memmapfile-object.html#braidws-28" rel="nofollow noreferrer">access the records</a> as an ordinary <a href="http://www.mathworks.com/help/matlab/examples/create-a-structure-array.html" rel="nofollow noreferrer">structure array</a>:</p> <pre><code>&gt;&gt; rec = m.Data; % 10x1 struct array &gt;&gt; rec(1) % same as: data(1,:) ans = d: 0.3257 s: 0.1080 h: [70 53 54 50 55 53 50 70] i: 192 &gt;&gt; rec(4).d % same as: data{4,1} ans = 0.5799 &gt;&gt; char(rec(10).h) % same as: data{10,3} ans = 2B2F493F </code></pre> <p>The benefit is that for large data files, is that you can restrict the mapping "viewing window" to a small subset of the records, and move this view along the file:</p> <pre><code>% read the records two at-a-time numRec = 10; % total number of records lenRec = 8*1 + 4*1 + 1*8 + 1*1; % length of each record in bytes numRecPerView = 2; % how many records in a viewing window m.Repeat = numRecPerView; for i=1:(numRec/numRecPerView) % move the window along the file m.Offset = (i-1) * numRecPerView*lenRec; % read the two records in this window: %for j=1:numRecPerView, m.Data(j), end m.Data(1) m.Data(2) end </code></pre> <p><img src="https://i.stack.imgur.com/veXDd.png" alt="access a portion of a file using memory-mapping"></p>
    singulars
    1. This table or related slice is empty.
    1. POCan I read a gigantic text file with Parallel Computing?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USAmro
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POCan I read a gigantic text file with Parallel Computing?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COAwesome, thanks! I will have to adapt this for my needs but this is definitely a great start.
      singulars
      1. PO
      1. USDang Khoa
    2. COJust want to remind you that the above works for binary files with fixed-length fields and records (we know in advance how many bytes and the precision of each field). It wont work for space-delimited text files, not unless you use some kind of padding to get all fields aligned.
      singulars
      1. PO
      1. USAmro
    3. CORight. I might go back to the source of this file and ask him to make it a binary format for me.
      singulars
      1. PO
      1. USDang Khoa
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload