Rounding errors when subtracting

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Rounding errors when subtracting

Ian Wood-3
Help!

Rev Studio 2.9.0 on OS X 10.5.2 Intel

'put 51.905719161 - 51.905658591'

results in 0.000061 instead of 0.0000607

Is there some global setting I'm missing, or is this a bug (which I  
can't find on RevQC)?

I'm getting the same error if I divide by 1,000,000. :-(

Ian
_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Rounding errors when subtracting

masmit
Ian, take a look at the 'numberFormat' property in the docs.

set the numberformat to "0.#########"
put 51.905719161 - 51.905658591

produces 0.00006057 - which is correct, I think.

Best,

Mark

On 20 Jun 2008, at 14:38, Ian Wood wrote:

> Help!
>
> Rev Studio 2.9.0 on OS X 10.5.2 Intel
>
> 'put 51.905719161 - 51.905658591'
>
> results in 0.000061 instead of 0.0000607
>
> Is there some global setting I'm missing, or is this a bug (which I  
> can't find on RevQC)?
>
> I'm getting the same error if I divide by 1,000,000. :-(
>
> Ian
> _______________________________________________
> use-revolution mailing list
> [hidden email]
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Reply | Threaded
Open this post in threaded view
|

Re: Rounding errors when subtracting

Ian Wood-3
Thanks, that did the trick!

Ian

On 20 Jun 2008, at 15:18, Mark Smith wrote:

> Ian, take a look at the 'numberFormat' property in the docs.

_______________________________________________
use-revolution mailing list
[hidden email]
Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution