Databases
StackOverflow2013
Postsdbo
POHLSL tex2d sampler seemingly using inconsistent rounding; why?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHLSL tex2d sampler seemingly using inconsistent rounding; why?
    text
    copied!<p>I have code that needs to render regions of my object differently depending on their location. I am trying to use a colour map to define these regions.</p> <p>The problem is when I sample from my colour map, I get collisions. Ie, two regions with different colours in the colourmap get the same value returned from the sampler.</p> <p>I've tried various formats of my colour map. I set the colours for each region to be "5" apart in each case;</p> <ul> <li><b>Indexed colour</b></li> <li><b>RGB, RGBA</b>: region 1 will have RGB 5%,5%,5%. region 2 will have RGB 10%,10%,10% and so on. </li> <li><b>HSV Greyscale</b>: region 1 will have HSV 0,0,5%. region 2 will have HSV 0,0,10% and so on. </li> </ul> <p>(Values selected in The Gimp)</p> <p>The tex2D sampler returns a value [0..1].</p> <p>[ I then intend to derive an int array index from region. Code to do with that is unrelated, so has been removed from the question ]</p> <pre><code>float region = tex2D(gColourmapSampler,In.UV).x; </code></pre> <p>Sampling the "5%" colour gave a "region" of 0.05098 in hlsl. From this I assume the 5% represents 5/100*255, or 12.75, which is rounded to 13 when stored in the texture. (Reasoning: 0.05098 * 255 ~= 13)</p> <p>By this logic, the 50% should be stored as 127.5. <br> Sampled, I get 0.50196 which implies it was stored as 128.</p> <p>the 70% should be stored as 178.5. <br> Sampled, I get 0.698039, which implies it was stored as 178. </p> <p><b>What rounding is going on here</b>? <br> (127.5 becomes 128, 178.5 becomes 178 ?!)</p> <p>Edit: OK, <a href="http://en.wikipedia.org/wiki/Bankers_rounding#Round_half_to_even" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/Bankers_rounding#Round_half_to_even</a> Apparently this is "banker's rounding". I have no idea why this is being used, but it solves my problem. Apparently, it's a Gimp issue. </p> <hr> <p>I am using Shader Model 2 and FX Composer. This is my sampler declaration;</p> <pre><code>//Colour map texture gColourmapTexture &lt; string ResourceName = "Globe_Colourmap_Regions_Greyscale.png"; string ResourceType = "2D"; &gt;; sampler2D gColourmapSampler : register(s1) = sampler_state { Texture = &lt;gColourmapTexture&gt;; #if DIRECT3D_VERSION &gt;= 0xa00 Filter = MIN_MAG_MIP_LINEAR; #else /* DIRECT3D_VERSION &lt; 0xa00 */ MinFilter = Linear; MipFilter = Linear; MagFilter = Linear; #endif /* DIRECT3D_VERSION */ AddressU = Clamp; AddressV = Clamp; }; </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload