summaryrefslogtreecommitdiffstatshomepage
path: root/tests/float/float_format_ints_doubleprec.py
Commit message (Collapse)AuthorAge
* py/formatfloat: Use pow(10, e) instead of pos/neg_pow lookup tables.Dan Ellis2022-08-12
| | | | | | | | | | | | | Rework the conversion of floats to decimal strings so it aligns precisely with the conversion of strings to floats in parsenum.c. This is to avoid rendering 1eX as 9.99999eX-1 etc. This is achieved by removing the power- of-10 tables and using pow() to compute the exponent directly, and that's done efficiently by first estimating the power-of-10 exponent from the power-of-2 exponent in the floating-point representation. Code size is reduced by roughly 100 to 200 bytes by this commit. Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
* py/formatfloat: Format all whole-number floats exactly.Dan Ellis2022-07-26
Formerly, py/formatfloat would print whole numbers inaccurately with nonzero digits beyond the decimal place. This resulted from its strategy of successive scaling of the argument by 0.1 which cannot be exactly represented in floating point. The change in this commit avoids scaling until the value is smaller than 1, so all whole numbers print with zero fractional part. Fixes issue #4212. Signed-off-by: Dan Ellis dan.ellis@gmail.com