Tag Info

Hot answers tagged


As background, Floating point precision is a way of storing numbers such that the precision is relative to the largest digit. For instance, the number $0.00123$ stored in fixed precision needs 6 digits of precision (3 zeros and the 3 non-zero numbers). However, this same number stored as floating point precision $1.23 \cdot 10^{-3}$ needs only 3 ...

Only top voted, non community-wiki answers of a minimum length are eligible