diff options
| author | Eike Rathke <erack@redhat.com> | 2016-09-13 15:26:09 +0200 |
|---|---|---|
| committer | Tor Lillqvist <tml@collabora.com> | 2016-09-16 14:51:48 +0000 |
| commit | 9c4fb18ba92265910d9c837d83c28926242b7915 (patch) | |
| tree | 0c562f2b762def9ff5057bef7e941ecc345912c3 | |
| parent | d17c9626f8997cef8ded6f4736003fcf694bc175 (diff) | |
recognize NaN with no bits set in lower word as error
Which can happen for example for -nan(0x8000000000000) as a result of
calculating with -inf. This was displayed as NaN instead of a proper error
value, now #NUM!
Example test case:
=FORECAST.ETS.ADD(50, {-1,-2,-3,-4}, {10,20,30,40})
Change-Id: I1e1d95e1f188e0036b72be37dd20039c9a9a13f6
(cherry picked from commit 4ef10fce39575ec0bd3793b5fdf731c0b9af25a3)
Reviewed-on: https://gerrit.libreoffice.org/28867
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
(cherry picked from commit ca19404c254c306d31a332958da5dde5d69a8d20)
Reviewed-on: https://gerrit.libreoffice.org/28876
Reviewed-by: Markus Mohrhard <markus.mohrhard@googlemail.com>
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Tested-by: Tor Lillqvist <tml@collabora.com>
| -rw-r--r-- | include/formula/errorcodes.hxx | 9 |
1 files changed, 6 insertions, 3 deletions
diff --git a/include/formula/errorcodes.hxx b/include/formula/errorcodes.hxx index 13597b9d4d53..173c7440bbad 100644 --- a/include/formula/errorcodes.hxx +++ b/include/formula/errorcodes.hxx @@ -103,11 +103,14 @@ inline sal_uInt16 GetDoubleErrorValue( double fVal ) return 0; if ( ::rtl::math::isInf( fVal ) ) return errIllegalFPOperation; // normal INF - sal_uInt32 nErr = reinterpret_cast< sal_math_Double * >( - &fVal)->nan_parts.fraction_lo; + sal_uInt32 nErr = reinterpret_cast< sal_math_Double * >( &fVal)->nan_parts.fraction_lo; if ( nErr & 0xffff0000 ) return errNoValue; // just a normal NAN - return (sal_uInt16)(nErr & 0x0000ffff); // any other error + if (!nErr) + // Another NAN, e.g. -nan(0x8000000000000) from calculating with -inf + return errIllegalFPOperation; + // Any other error known to us as error code. + return (sal_uInt16)(nErr & 0x0000ffff); } } // namespace formula |