Related: tdf#138360 Use approxFloor() in rtl_math_round()

Ditch mostly but not always working correction value and use
approxFloor() instead which yields better results. With this we
now have one single place approxValue() in case more fine grained
tuning is needed.

Unfortunately all numeric spreadsheet function tests use ROUND()
in a manner such that they mostly blindly do a ROUND(...;12)
regardless of magnitude, sometimes effectively rounding to the
14th significant digit that may fail in cases like for

14.2040730851385
^
where the constant (rounded) value is stored as is but the
calculated value is
14.204073085138471
and the old round() yielded
14.204073085139 for both but the new round() more correctly
results in
14.204073085139 and
14.204073085138
so the spreadsheet test case sources had to be changed to
ROUND(...;11) in such places.

Change-Id: I211bb868a4f4dc9e68f4c7dcc2a187b5e175416f
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/107135
Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Jenkins

1 files changed, 5 insertions, 19 deletions

diff --git a/sal/rtl/math.cxx b/sal/rtl/math.cxx
index e6f09f18030e..a296927635bf 100644
--- a/sal/rtl/math.cxx
+++ b/sal/rtl/math.cxx
@@ -1133,6 +1133,9 @@ double SAL_CALL rtl_math_round(double fValue, int nDecPlaces,
 {
     OSL_ASSERT(nDecPlaces >= -20 && nDecPlaces <= 20);
 
+    if (!std::isfinite(fValue))
+        return fValue;
+
     if (fValue == 0.0)
         return fValue;
 
@@ -1190,24 +1193,7 @@ double SAL_CALL rtl_math_round(double fValue, int nDecPlaces,
     switch ( eMode )
     {
         case rtl_math_RoundingMode_Corrected :
-        {
-            int nExp;       // exponent for correction
-            if ( fValue > 0.0 )
-                nExp = static_cast<int>( floor( log10( fValue ) ) );
-            else
-                nExp = 0;
-
-            int nIndex;
-
-            if (nExp < 0)
-                nIndex = 15;
-            else if (nExp >= 14)
-                nIndex = 0;
-            else
-                nIndex = 15 - nExp;
-
-            fValue = floor(fValue + 0.5 + nCorrVal[nIndex]);
-        }
+            fValue = rtl::math::approxFloor(fValue + 0.5);
         break;
         case rtl_math_RoundingMode_Down:
             fValue = rtl::math::approxFloor(fValue);
@@ -1321,7 +1307,7 @@ double SAL_CALL rtl_math_approxValue( double fValue ) SAL_THROW_EXTERN_C()
     if (!std::isfinite(fValue))
         return fOrigValue;
 
-    fValue = rtl_math_round(fValue, 0, rtl_math_RoundingMode_Corrected);
+    fValue = std::round(fValue);
     fValue /= fExpValue;
 
     // If the original value was near DBL_MAX we got an overflow. Restore and