Note that there are some explanatory texts on larger screens.

plurals
  1. POConverting number stored in a VARCHAR field in SQL Server
    primarykey
    data
    text
    <p>I have a table with payments stored in a column that is of type <code>VARCHAR</code></p> <p>This is a sample data from the field:</p> <pre><code>10.01 55.11 </code></pre> <p>Lately, as part of a new request, we had to remove the decimal point from the number we store in the <code>VARCHAR</code> column.</p> <p>So, I had this:</p> <pre><code>CONVERT(BIGINT, CONVERT(REAL, RTRIM(LTRIM(@amt))) * 100) as PaymentAmount </code></pre> <p>to convert to <code>REAL</code> (cannot convert to <code>BIGINT</code> from <code>VARCHAR</code>), multiply by 100 and then convert to <code>BIGINT</code></p> <p>Now the issue is, sometimes, during conversion, the end result is 1 cent less.</p> <p>Example: <code>'10.11'</code> gets converted to <code>1010</code>, <code>10.15</code> to <code>1014</code></p> <p>Easy way to test this:</p> <pre><code>SELECT CONVERT(BIGINT, CONVERT(REAL, RTRIM(LTRIM(10.15))) * 100) as PaymentAmount </code></pre> <p>The result of the query is <code>1014</code>. </p> <p>I fixed the issue by changing conversion from <code>REAL</code> to <code>FLOAT</code> and removed conversion to <code>BIGINT</code>. </p> <p>An alternative fix is to use <code>CAST</code> like this:</p> <pre><code>CAST(@amt as float)*100 as PaymentAmount </code></pre> <p>I ran conversions with numbers ranging from <code>10.00</code> to <code>10.99</code> and in that range of 100 values, the <strong>wrong</strong> numbers are:</p> <ul> <li><code>10.11</code> converts to <code>1010</code></li> <li><code>10.15</code> converts to <code>1014</code></li> <li><code>10.19</code> converts to <code>1018</code></li> <li><code>10.23</code> converts to <code>1022</code></li> </ul> <p>All other values from that range are fine. Curiously, <code>11.11</code> converts just fine to <code>1111</code>.</p> <p>So the question is, Why the heck is doing it and whats the pattern? From the <a href="http://msdn.microsoft.com/en-us/library/ms187752.aspx" rel="nofollow">DataType MSDN</a> REAL is not a precise number, but so is FLOAT and yet float works for the whole range.</p> <p>EDIT: I failed to mention the <code>varchar</code> is of size 1000.</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload