Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <h1>edit</h1> <ul> <li>now has an option to store in 16bit floats. <ul> <li>you lose some precision(depends on data range,vary the rnd distribution to see different error rates).</li> <li>slower(20-30ms)(see the link below for probably faster tricks if it matters)</li> <li>2mb</li> </ul></li> <li>if you know the nature of your data(range,distribution surface(maybe variable length encoding)) more can be done probably <ul> <li>see here(<a href="https://stackoverflow.com/questions/1659440/32-bit-to-16-bit-floating-point-conversion">32-bit to 16-bit Floating Point Conversion</a>).</li> </ul></li> </ul> <h1>code</h1> <pre><code>//using code lifted from http://www.mathworks.com/matlabcentral/fileexchange/23173 #include "opencv2/opencv.hpp" #include &lt;string.h&gt; #include &lt;math.h&gt; using namespace cv; #define INT16_TYPE short #define UINT16_TYPE unsigned short #define INT32_TYPE long #define UINT32_TYPE unsigned long int doubles2halfp(void *target, void *source, int numel); int halfp2doubles(void *target, void *source, int numel); void writemat(char* fpath,Mat&amp; data,bool isf16) { FILE* fp = fopen(fpath,"wb"); if (!fp)perror("fopen"); double dbuf[1024]; for(int i=0;i&lt;1024;++i) { for(int j=0;j&lt;1024;++j) dbuf[j]=data.at&lt;double&gt;(i,j); if(isf16) { UINT16_TYPE hbuf[1024]; doubles2halfp(&amp;hbuf,&amp;dbuf,1024); fwrite(&amp;hbuf,sizeof(UINT16_TYPE),1024,fp); }else { fwrite(&amp;dbuf,sizeof(double),1024,fp); } } fclose(fp); } void readmat(char* fpath,Mat&amp; data,bool isf16) { FILE* fp = fopen(fpath,"rb"); if (!fp)perror("fopen"); double dbuf[1024]; for(int i=0;i&lt;1024;++i) { if(isf16) { UINT16_TYPE hbuf[1024]; fread(&amp;hbuf,sizeof(UINT16_TYPE),1024,fp); halfp2doubles(&amp;dbuf,&amp;hbuf,1024); }else{ fread(&amp;dbuf,sizeof(double),1024,fp); } for(int j=0;j&lt;1024;++j) { data.at&lt;double&gt;(i,j)=dbuf[j]; } } fclose(fp); } int main() { RNG rng= theRNG(); Mat data = Mat::zeros(Size(1024,1024),CV_64FC1); for(int i=0;i&lt;1024;++i) for(int j=0;j&lt;1024;++j) data.at&lt;double&gt;(i,j)=rng.uniform(-1.,1.); writemat("img.bin",data,true); Mat res = Mat::zeros(Size(1024,1024),CV_64FC1); readmat("img.bin",res,true); double error=0; for(int i=0;i&lt;1024;++i) for(int j=0;j&lt;1024;++j) { //printf("%f %f\n",data.at&lt;double&gt;(i,j),res.at&lt;double&gt;(i,j)); error+=abs(data.at&lt;double&gt;(i,j)-res.at&lt;double&gt;(i,j)); } printf("err=%f avgerr=%f\n",error,error/1024/1024); getchar(); return 0; } //////////////////////////////////////// //////////////////////////////////////// //////////////////////////////////////// //////////////////////////////////////// int doubles2halfp(void *target, void *source, int numel) { UINT16_TYPE *hp = (UINT16_TYPE *) target; // Type pun output as an unsigned 16-bit int UINT32_TYPE *xp = (UINT32_TYPE *) source; // Type pun input as an unsigned 32-bit int UINT16_TYPE hs, he, hm; UINT32_TYPE x, xs, xe, xm; int hes; static int next; // Little Endian adjustment static int checkieee = 1; // Flag to check for IEEE754, Endian, and word size double one = 1.0; // Used for checking IEEE754 floating point format UINT32_TYPE *ip; // Used for checking IEEE754 floating point format if( checkieee ) { // 1st call, so check for IEEE754, Endian, and word size ip = (UINT32_TYPE *) &amp;one; if( *ip ) { // If Big Endian, then no adjustment next = 0; } else { // If Little Endian, then adjustment will be necessary next = 1; ip++; } if( *ip != 0x3FF00000u ) { // Check for exact IEEE 754 bit pattern of 1.0 return 1; // Floating point bit pattern is not IEEE 754 } if( sizeof(INT16_TYPE) != 2 || sizeof(INT32_TYPE) != 4 ) { return 1; // short is not 16-bits, or long is not 32-bits. } checkieee = 0; // Everything checks out OK } xp += next; // Little Endian adjustment if necessary if( source == NULL || target == NULL ) { // Nothing to convert (e.g., imag part of pure real) return 0; } while( numel-- ) { x = *xp++; xp++; // The extra xp++ is to skip over the remaining 32 bits of the mantissa if( (x &amp; 0x7FFFFFFFu) == 0 ) { // Signed zero *hp++ = (UINT16_TYPE) (x &gt;&gt; 16); // Return the signed zero } else { // Not zero xs = x &amp; 0x80000000u; // Pick off sign bit xe = x &amp; 0x7FF00000u; // Pick off exponent bits xm = x &amp; 0x000FFFFFu; // Pick off mantissa bits if( xe == 0 ) { // Denormal will underflow, return a signed zero *hp++ = (UINT16_TYPE) (xs &gt;&gt; 16); } else if( xe == 0x7FF00000u ) { // Inf or NaN (all the exponent bits are set) if( xm == 0 ) { // If mantissa is zero ... *hp++ = (UINT16_TYPE) ((xs &gt;&gt; 16) | 0x7C00u); // Signed Inf } else { *hp++ = (UINT16_TYPE) 0xFE00u; // NaN, only 1st mantissa bit set } } else { // Normalized number hs = (UINT16_TYPE) (xs &gt;&gt; 16); // Sign bit hes = ((int)(xe &gt;&gt; 20)) - 1023 + 15; // Exponent unbias the double, then bias the halfp if( hes &gt;= 0x1F ) { // Overflow *hp++ = (UINT16_TYPE) ((xs &gt;&gt; 16) | 0x7C00u); // Signed Inf } else if( hes &lt;= 0 ) { // Underflow if( (10 - hes) &gt; 21 ) { // Mantissa shifted all the way off &amp; no rounding possibility hm = (UINT16_TYPE) 0u; // Set mantissa to zero } else { xm |= 0x00100000u; // Add the hidden leading bit hm = (UINT16_TYPE) (xm &gt;&gt; (11 - hes)); // Mantissa if( (xm &gt;&gt; (10 - hes)) &amp; 0x00000001u ) // Check for rounding hm += (UINT16_TYPE) 1u; // Round, might overflow into exp bit, but this is OK } *hp++ = (hs | hm); // Combine sign bit and mantissa bits, biased exponent is zero } else { he = (UINT16_TYPE) (hes &lt;&lt; 10); // Exponent hm = (UINT16_TYPE) (xm &gt;&gt; 10); // Mantissa if( xm &amp; 0x00000200u ) // Check for rounding *hp++ = (hs | he | hm) + (UINT16_TYPE) 1u; // Round, might overflow to inf, this is OK else *hp++ = (hs | he | hm); // No rounding } } } } return 0; } int halfp2doubles(void *target, void *source, int numel) { UINT16_TYPE *hp = (UINT16_TYPE *) source; // Type pun input as an unsigned 16-bit int UINT32_TYPE *xp = (UINT32_TYPE *) target; // Type pun output as an unsigned 32-bit int UINT16_TYPE h, hs, he, hm; UINT32_TYPE xs, xe, xm; INT32_TYPE xes; int e; static int next; // Little Endian adjustment static int checkieee = 1; // Flag to check for IEEE754, Endian, and word size double one = 1.0; // Used for checking IEEE754 floating point format UINT32_TYPE *ip; // Used for checking IEEE754 floating point format if( checkieee ) { // 1st call, so check for IEEE754, Endian, and word size ip = (UINT32_TYPE *) &amp;one; if( *ip ) { // If Big Endian, then no adjustment next = 0; } else { // If Little Endian, then adjustment will be necessary next = 1; ip++; } if( *ip != 0x3FF00000u ) { // Check for exact IEEE 754 bit pattern of 1.0 return 1; // Floating point bit pattern is not IEEE 754 } if( sizeof(INT16_TYPE) != 2 || sizeof(INT32_TYPE) != 4 ) { return 1; // short is not 16-bits, or long is not 32-bits. } checkieee = 0; // Everything checks out OK } xp += next; // Little Endian adjustment if necessary if( source == NULL || target == NULL ) // Nothing to convert (e.g., imag part of pure real) return 0; while( numel-- ) { h = *hp++; if( (h &amp; 0x7FFFu) == 0 ) { // Signed zero *xp++ = ((UINT32_TYPE) h) &lt;&lt; 16; // Return the signed zero } else { // Not zero hs = h &amp; 0x8000u; // Pick off sign bit he = h &amp; 0x7C00u; // Pick off exponent bits hm = h &amp; 0x03FFu; // Pick off mantissa bits if( he == 0 ) { // Denormal will convert to normalized e = -1; // The following loop figures out how much extra to adjust the exponent do { e++; hm &lt;&lt;= 1; } while( (hm &amp; 0x0400u) == 0 ); // Shift until leading bit overflows into exponent bit xs = ((UINT32_TYPE) hs) &lt;&lt; 16; // Sign bit xes = ((INT32_TYPE) (he &gt;&gt; 10)) - 15 + 1023 - e; // Exponent unbias the halfp, then bias the double xe = (UINT32_TYPE) (xes &lt;&lt; 20); // Exponent xm = ((UINT32_TYPE) (hm &amp; 0x03FFu)) &lt;&lt; 10; // Mantissa *xp++ = (xs | xe | xm); // Combine sign bit, exponent bits, and mantissa bits } else if( he == 0x7C00u ) { // Inf or NaN (all the exponent bits are set) if( hm == 0 ) { // If mantissa is zero ... *xp++ = (((UINT32_TYPE) hs) &lt;&lt; 16) | ((UINT32_TYPE) 0x7FF00000u); // Signed Inf } else { *xp++ = (UINT32_TYPE) 0xFFF80000u; // NaN, only the 1st mantissa bit set } } else { xs = ((UINT32_TYPE) hs) &lt;&lt; 16; // Sign bit xes = ((INT32_TYPE) (he &gt;&gt; 10)) - 15 + 1023; // Exponent unbias the halfp, then bias the double xe = (UINT32_TYPE) (xes &lt;&lt; 20); // Exponent xm = ((UINT32_TYPE) hm) &lt;&lt; 10; // Mantissa *xp++ = (xs | xe | xm); // Combine sign bit, exponent bits, and mantissa bits } } xp++; // Skip over the remaining 32 bits of the mantissa } return 0; } </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POhow to save a double matrix quickly and compactly if some loss of precision is acceptable?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USCommunity
    1. USZaw Lin
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POhow to save a double matrix quickly and compactly if some loss of precision is acceptable?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks for solution. It works fine most of the times, but when I run write function 3-4 times, it gives exception. when there is no output file it creates and writes, for the second one it overwrites, third on gives exception, namely, `Access violation`. Also when I run it over and over, sometimes, it is not filling the files(file size = 0). Is this something expected or do I miss something?
      singulars
      1. PO
      1. USsmttsp
    2. COIn term of time, it is really good.
      singulars
      1. PO
      1. USsmttsp
    3. COno it wasnt expected, i have run it in a loop a thousand times and it finishes fine.. are you perhaps using threads? can you post a simple example that reproduce the problem?
      singulars
      1. PO
      1. USZaw Lin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload