diff options
| author | Winfried Donkers <winfrieddonkers@libreoffice.org> | 2017-12-30 18:25:02 +0100 |
|---|---|---|
| committer | Eike Rathke <erack@redhat.com> | 2018-01-16 15:45:02 +0100 |
| commit | b97a0df0f3234b4c1140ba1418d4b96a592afa4a (patch) | |
| tree | fd03478f4e33b643fb635087b202364fb2174564 /sal | |
| parent | ada02c556531e9e5f28a159223fc7e2b36a7a84d (diff) | |
tdf#96821 fix corner cases for Calc function ROUND.
For very large integer numbers, e.g. 2^52+1, ROUND produced incorrect
results because rtl::math::round uses
floor( number + 0.5 + small_correction_value ), which reduces the maximum
possible mantissa resolution.a
Correction of several unit test documents (Calc functions) with rounding
errors that only became apparent with this fix.
Change-Id: I1769c9939a3d6118d3bfbfdf8e41dd4619997232
Reviewed-on: https://gerrit.libreoffice.org/47179
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Eike Rathke <erack@redhat.com>
Diffstat (limited to 'sal')
| -rw-r--r-- | sal/rtl/math.cxx | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/sal/rtl/math.cxx b/sal/rtl/math.cxx index 4b149b9f99dd..387ae3b2b4aa 100644 --- a/sal/rtl/math.cxx +++ b/sal/rtl/math.cxx @@ -1076,6 +1076,9 @@ double SAL_CALL rtl_math_round(double fValue, int nDecPlaces, if (fValue == 0.0) return fValue; + if ( nDecPlaces == 0 && eMode == rtl_math_RoundingMode_Corrected ) + return std::round( fValue ); + // sign adjustment bool bSign = rtl::math::isSignBitSet( fValue ); if (bSign) |