How to eliminate floating point errors

Part of the problem is that people math is base 10, computer math is base 2.

As all your results are expected to be whole numbers, the suggestion of using ceiling might work.

Another option is to take the result of your calculation, add 0.5 and then floor the result, this will always round to the nearest whole number.

I also see a problem with your test code sqrt(4)*2 = 4 not 2 first of all. Here’s an example of using floor, I only used a partial gamma value but it should provide an adequate example



const E = exp(1)
const GAMMA = 8.5772156649

Called when the node enters the scene tree for the first time.

func _ready() → void:
var y = PI1/PI
y += 0.5
var x = floor(y)
print(x)
var b = floor(E1/E + 0.5)
print(b)
var q = floor(sqrt(4)*2 + 0.5)
print(q)

Will print 1.0, 1.0 and 4.0

There’s also round() that does exactly that, no need to reinvent the wheel :slight_smile:

Rounding to the whole number might not be the best choice though as e.g. 2.0 and 2.4 are very different results. I’d still suggest rounding to some higher precision, but still before the floating point error starts showing up, as I suggested in my previous message:

There’s a snapped() method for that.

i found this, is that |x| the function abs(x)?

Yes, for a scalar x, it’s the equivalent of abs(x).

For a vector v, |v| is magnitude of v, e.g length(v)

Cheers!

To be fully correct the general notation for vector length is ||v||, though both are used the double version is generally preferred to avoid confusion (note however that |AB| does generally represent the distance from point A to point B in Euclidean space, or the length of the line segment defined by the points, which would be equivalent to ||B - A||)