Databases
StackOverflow2013
Postsdbo
POLinux & Windows: Using Large Files to save physical memory
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLinux & Windows: Using Large Files to save physical memory
    primarykey
    data
    text
    <p>Good afternoon, We have implemented a C++ cKeyArray class to test whether we can use the Large File API to save physical memory. During Centos Linux testing, we found that the Linux File API was just as fast as using the heap for random access processing. Here are the numbers: for a 2,700,000 row SQL database where the KeySize for each row is 62 bytes,</p> <p>cKeyArray class using LINUX File API BruteForceComparisons = 197275 BruteForceTimeElapsed = 1,763,504,445 microsecs Each BruteForce Comparisons requires two random access, there the mean time required for each random access = 1,763,504,445 microsecs / (2 * 197275) = 4470 microsecs</p> <p>Heap , no cKeyArray class </p> <p>BruteForceComparisons = 197275 BruteForceTimeElapsed = 1,708,442,690microsecs the mean time required for each random access = 4300 microsecs.</p> <p>On 32 bit Windows,the numbers are,</p> <p>cKeyArray class using Windows File API BruteForceComparisons = 197275 BruteForceTimeElapsed = 9243787 millisecs the mean time for each random access is 23.4 millisec</p> <p>Heap, no cKeyArray class BruteForceComparisons = 197275 BruteForceTimeElapsed = 2,141,941 millisecs the mean time requires for each random access is 5.4 millisec</p> <p>We are wondering why the Linux cKeyArray numbers are just as good the Linux heap numbers while on 32 bit Windows the mean heap random access time is 4 times as fast the cKeyArray Windows File API. Is there some way we can speed up the Windows cKeyArray File API? </p> <p>Previouly, we received a lot of good suggestions from Stack Overflow on using the Windows Memory Mapped File API. Based on these Stack Overflow suggestions we have implemented a Memory Mapped File MRU caching class which functions properly.</p> <p>Because we want to devlop a cross-platform solution, we want to do due diligence to see why the Linux File API is so fast? Thank you. We are trying to post a portion of the cKeyArray class implementation below.</p> <pre><code>#define KEYARRAY_THRESHOLD 100000000 // Use file instead of memory if requirement is above this number cKeyArray::cKeyArray(long RecCount_,int KeySize_,int MatchCodeSize_, char* TmpFileName_) { RecCount=RecCount_; KeySize=KeySize_; MatchCodeSize=MatchCodeSize_; MemBuffer=0; KeyBuffer=0; MemFile=0; MemFileName[0]='\x0'; ReturnBuffer=new char[MatchCodeSize + 1]; if (RecCount*KeySize&lt;=KEYARRAY_THRESHOLD) { InMemory=true; MemBuffer=new char[RecCount*KeySize]; memset(MemBuffer,0,RecCount*KeySize); } else { InMemory=false; strcpy(MemFileName,TmpFileName_); try { MemFile= new cFile(MemFileName,cFile::CreateAlways,cFile::ReadWrite); } catch (cException e) { throw e; } try { MemFile-&gt;SetFilePointer( (int64_t)(RecCount*KeySize),cFile::FileBegin); } catch (cException e) { throw e; } if (!(MemFile-&gt;SetEndOfFile())) throw cException(ERR_FILEOPEN,MemFileName); KeyBuffer=new char[KeySize]; } } char *cKeyArray::GetKey(long Record_) { memset(ReturnBuffer,0,MatchCodeSize + 1); if (InMemory) { memcpy(ReturnBuffer,MemBuffer+Record_*KeySize,MatchCodeSize); } else { MemFile-&gt;SetFilePointer((int64_t)(Record_*KeySize),cFile::FileBegin); MemFile-&gt;ReadFile(KeyBuffer,KeySize); memcpy(ReturnBuffer,KeyBuffer,MatchCodeSize); } return ReturnBuffer; } uint32_t cKeyArray::GetDupeGroup(long Record_) { uint32_t DupeGroup(0); if (InMemory) { memcpy((char*)&amp;DupeGroup, MemBuffer+Record_*KeySize + MatchCodeSize,sizeof(uint32_t)); } else { MemFile-&gt;SetFilePointer( (int64_t)(Record_*KeySize + MatchCodeSize) ,cFile::FileBegin); MemFile-&gt;ReadFile((char*)&amp;DupeGroup,sizeof(uint32_t)); } return DupeGroup; } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USErik
    1. USFrank
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POLinux & Windows: Using Large Files to save physical memory
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI wouldn't bother with a cross-platform solution; just refactor the platform-specific stuff into separate files.
      singulars
      1. POLinux & Windows: Using Large Files to save physical memory
      1. USIgnacio Vazquez-Abrams
    2. CO@Ignacio Vazquez-Abrams, Are you suggesting that we refactor the Linux file API into one file and the Windows memory mapped file implementation into another file? We are puzzled why the Linux File API is faster than the the Windows File API? Do you have clues about this ? Thank you for your help.
      singulars
      1. POLinux & Windows: Using Large Files to save physical memory
      1. USFrank
    3. CONo. Use memory-mapped in both cases with the same interface, then fill in the implementation for each.
      singulars
      1. POLinux & Windows: Using Large Files to save physical memory
      1. USIgnacio Vazquez-Abrams
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload