Databases
StackOverflow2013
Postsdbo
POFastest way of reading bytes in D2
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFastest way of reading bytes in D2
    primarykey
    data
    text
    <p>I want to read single bytes as fast as possible from a file into a D2 application. The application need byte per byte, so reading larger blocks of data is not an option for the interface to the reader.</p> <p>For this I created some trivial implementations in C++, Java, D2 at: <a href="https://github.com/gizmomogwai/performance">https://github.com/gizmomogwai/performance</a>.</p> <p>As you can see I tried plain reads, buffers in the application code and memory mapped files. For my usecase the memory mapped solution worked best, but the strange thing is that D2 is slower than java. I would have hoped for D2 to land between C++ and Java (C++ code is compiled with -O3 -g, D2 code is compiled with -O -release).</p> <p>So please tell me what I am doing wrong here and how to speed up the D2 implementation.</p> <p>To give you an idea of the use case here is a C++ implementation:</p> <pre><code>class StdioFileReader { private: FILE* fFile; static const size_t BUFFER_SIZE = 1024; unsigned char fBuffer[BUFFER_SIZE]; unsigned char* fBufferPtr; unsigned char* fBufferEnd; public: StdioFileReader(std::string s) : fFile(fopen(s.c_str(), "rb")), fBufferPtr(fBuffer), fBufferEnd(fBuffer) { assert(fFile); } ~StdioFileReader() { fclose(fFile); } int read() { bool finished = fBufferPtr == fBufferEnd; if (finished) { finished = fillBuffer(); if (finished) { return -1; } } return *fBufferPtr++; } private: bool fillBuffer() { size_t l = fread(fBuffer, 1, BUFFER_SIZE, fFile); fBufferPtr = fBuffer; fBufferEnd = fBufferPtr+l; return l == 0; } }; size_t readBytes() { size_t res = 0; for (int i=0; i&lt;10; i++) { StdioFileReader r("/tmp/shop_with_ids.pb"); int read = r.read(); while (read != -1) { ++res; read = r.read(); } } return res; } </code></pre> <p>which is much faster compared to the "same" solution in D:</p> <pre><code>struct FileReader { private FILE* fFile; private static const BUFFER_SIZE = 8192; private ubyte fBuffer[BUFFER_SIZE]; private ubyte* fBufferPtr; private ubyte* fBufferEnd; public this(string fn) { fFile = std.c.stdio.fopen("/tmp/shop_with_ids.pb", "rb"); fBufferPtr = fBuffer.ptr; fBufferEnd = fBuffer.ptr; } public int read(ubyte* targetBuffer) { auto finished = fBufferPtr == fBufferEnd; if (finished) { finished = fillBuffer(); if (finished) { return 0; } } *targetBuffer = *fBufferPtr++; return 1; } private bool fillBuffer() { fBufferPtr = fBuffer.ptr; auto l = std.c.stdio.fread(fBufferPtr, 1, BUFFER_SIZE, fFile); fBufferEnd = fBufferPtr + l; return l == 0; } } size_t readBytes() { size_t count = 0; for (int i=0; i&lt;10; i++) { auto reader = FileReader("/tmp/shop_with_ids.pb"); ubyte buffer[1]; ubyte* p = buffer.ptr; auto c = reader.read(p); while (1 == c) { ++count; c = reader.read(p); } } return count; } </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USGizmomogwai
    1. USGizmomogwai
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POFastest way of reading bytes in D2
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POFastest way of reading bytes in D2
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    3. VO
      singulars
      1. POFastest way of reading bytes in D2
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI have done some other non-related coding in D and Java, (math intensive computations), and turns out Java is marginally faster in my tests. I guess that you should not expect java to be that much slower nowadays, the JIT compiler is VERY good at optimization.
      singulars
      1. POFastest way of reading bytes in D2
      1. USPer Alexandersson
    2. COYeah ... you are right ... I dont expect java to be much slower than cpp (which it still is in my demo example using the default jit), but my point is that d is even slower. I hoped d to be on par with cpp.
      singulars
      1. POFastest way of reading bytes in D2
      1. USGizmomogwai
    3. COYes, I did that as well, when I converted a Java algorithm to D a few months ago. I think they have some quirks to fix in the code optimization. or maybe the GC is just really bad, and slow, so try turning that of?
      singulars
      1. POFastest way of reading bytes in D2
      1. USPer Alexandersson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload