summaryrefslogtreecommitdiffstatshomepage
path: root/tests/float/float_parse.py
diff options
context:
space:
mode:
authorRomain Goyet <romain.goyet@numworks.com>2018-09-13 14:00:42 +0200
committerDamien George <damien.p.george@gmail.com>2018-09-20 22:06:41 +1000
commitb768cc6ca8ed6d7430b86dc7ccb9ee2391b3a251 (patch)
treea49a5ad758e3df4f7c96b035d80f43d2df365e87 /tests/float/float_parse.py
parent185716514f110560adaa35367aa6886023f29120 (diff)
downloadmicropython-b768cc6ca8ed6d7430b86dc7ccb9ee2391b3a251.tar.gz
micropython-b768cc6ca8ed6d7430b86dc7ccb9ee2391b3a251.zip
py/parsenum: Avoid rounding errors with negative powers-of-10.
This patches avoids multiplying with negative powers-of-10 when parsing floating-point values, when those powers-of-10 can be exactly represented as a positive power. When represented as a positive power and used to divide, the resulting float will not have any rounding errors. The issue is that mp_parse_num_decimal will sometimes not give the closest floating representation of the input string. Eg for "0.3", which can't be represented exactly in floating point, mp_parse_num_decimal gives a slightly high (by 1LSB) result. This is because it computes the answer as 3 * 0.1, and since 0.1 also can't be represented exactly, multiplying by 3 multiplies up the rounding error in the 0.1. Computing it as 3 / 10, as now done by the change in this commit, gives an answer which is as close to the true value of "0.3" as possible.
Diffstat (limited to 'tests/float/float_parse.py')
0 files changed, 0 insertions, 0 deletions