Clamp equation values when exporting to binary MS format

CppunitTest_sw_ooxmlexport failed under
-fsanitize=implicit-signed-integer-truncation when writing 100000 (from the
triangle equation in oox/source/drawingml/customshapes/oox-drawingml-cs-presets)
as a 16-bit value (see below), and as discussed at FreeNode #libreoffice-dev:

Nov 12 09:12:32 <vmiklos> sberg: ah, that's custom shape equation, read from
ooxml but then written to ms binary format; i'm not sure if there is a better
option there other than just clamping the value.
Nov 12 09:14:33 <vmiklos> sberg: i would expect that information is 1) used in
general when writing drawingml+vml markup for a shape (where the vml fallback
is not read by LO nor by MSO) and 2) i don't expect that our poor vml export
actually reads those equations. so the vml export builds on top of the binary
export, but at the end probably that information (equations) is not read by
anyone

(At least for CppunitTest_sw_ooxmlexport, equation.nPara[1] never needed such
clamping, just nPara[0] and nPara[2].)

> filter/source/msfilter/escherex.cxx:2929:50: runtime error: implicit conversion from type 'sal_Int32' (aka 'int') of value 100000 (32-bit, signed) to type 'sal_Int16' (aka 'short') changed the value to -31072 (16-bit, signed)
>  #0 in EscherPropertyContainer::CreateCustomShapeProperties(MSO_SPT, com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) at filter/source/msfilter/escherex.cxx:2929:50 (instdir/program/libmsfilterlo.so +0x54f1e6)
>  #1 in ImplEESdrWriter::ImplWriteShape(ImplEESdrObject&, EscherSolverContainer&, bool) at filter/source/msfilter/eschesdo.cxx:283:26 (instdir/program/libmsfilterlo.so +0x67a775)
>  #2 in ImplEESdrWriter::ImplWriteTheShape(ImplEESdrObject&, bool) at filter/source/msfilter/eschesdo.cxx:932:12 (instdir/program/libmsfilterlo.so +0x69059d)
>  #3 in EscherEx::AddSdrObject(SdrObject const&, bool) at filter/source/msfilter/eschesdo.cxx:951:35 (instdir/program/libmsfilterlo.so +0x691064)
>  #4 in oox::vml::VMLExport::AddSdrObject(SdrObject const&, short, short, short, short, bool) at oox/source/export/vmlexport.cxx:1425:15 (instdir/program/libooxlo.so +0x2ab3157)
>  #5 in DocxSdrExport::writeVMLDrawing(SdrObject const*, SwFrameFormat const&) at sw/source/filter/ww8/docxsdrexport.cxx:772:38 (instdir/program/libmswordlo.so +0x168bfb2)
>  #6 in DocxSdrExport::writeDMLAndVMLDrawing(SdrObject const*, SwFrameFormat const&, int) at sw/source/filter/ww8/docxsdrexport.cxx:975:9 (instdir/program/libmswordlo.so +0x16938f5)
>  #7 in DocxAttributeOutput::OutputFlyFrame_Impl(ww8::Frame const&, Point const&) at sw/source/filter/ww8/docxattributeoutput.cxx:5357:57 (instdir/program/libmswordlo.so +0x1371a67)
>  #8 in AttributeOutputBase::OutputFlyFrame(ww8::Frame const&) at sw/source/filter/ww8/wrtw8nds.cxx:3173:5 (instdir/program/libmswordlo.so +0x1befe5a)
>  #9 in SwWW8AttrIter::OutFlys(int) at sw/source/filter/ww8/wrtw8nds.cxx:667:41 (instdir/program/libmswordlo.so +0x1beeb70)
>  #10 in MSWordExportBase::OutputTextNode(SwTextNode&) at sw/source/filter/ww8/wrtw8nds.cxx:2275:42 (instdir/program/libmswordlo.so +0x1c32e4d)
>  #11 in MSWordExportBase::OutputContentNode(SwContentNode&) at sw/source/filter/ww8/wrtw8nds.cxx:3237:13 (instdir/program/libmswordlo.so +0x1c5c7e4)
>  #12 in MSWordExportBase::WriteText() at sw/source/filter/ww8/wrtww8.cxx:2725:13 (instdir/program/libmswordlo.so +0x1efcdc2)
>  #13 in DocxExport::WriteMainText() at sw/source/filter/ww8/docxexport.cxx:1453:5 (instdir/program/libmswordlo.so +0x1607d1f)
>  #14 in DocxExport::ExportDocument_Impl() at sw/source/filter/ww8/docxexport.cxx:514:5 (instdir/program/libmswordlo.so +0x1604baf)
>  #15 in MSWordExportBase::ExportDocument(bool) at sw/source/filter/ww8/wrtww8.cxx:3206:19 (instdir/program/libmswordlo.so +0x1f3f022)
>  #16 in DocxExportFilter::exportDocument() at sw/source/filter/ww8/docxexportfilter.cxx:86:17 (instdir/program/libmswordlo.so +0x166b5e3)
>  #17 in oox::core::FilterBase::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at oox/source/core/filterbase.cxx:489:55 (instdir/program/libooxlo.so +0x1bc3880)
>  #18 in WriterFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at writerfilter/source/filter/WriterFilter.cxx:144:23 (instdir/program/libwriterfilterlo.so +0x1a250bf)
>  #19 in SfxObjectShell::ExportTo(SfxMedium&) at sfx2/source/doc/objstor.cxx:2441:25 (instdir/program/libsfxlo.so +0x38d1352)
>  #20 in SfxObjectShell::SaveTo_Impl(SfxMedium&, SfxItemSet const*) at sfx2/source/doc/objstor.cxx:1535:19 (instdir/program/libsfxlo.so +0x38c1818)
>  #21 in SfxObjectShell::PreDoSaveAs_Impl(rtl::OUString const&, rtl::OUString const&, SfxItemSet const&) at sfx2/source/doc/objstor.cxx:2848:39 (instdir/program/libsfxlo.so +0x38f0e90)
>  #22 in SfxObjectShell::CommonSaveAs_Impl(INetURLObject const&, rtl::OUString const&, SfxItemSet&) at sfx2/source/doc/objstor.cxx:2705:9 (instdir/program/libsfxlo.so +0x38ea470)
>  #23 in SfxObjectShell::APISaveAs_Impl(rtl::OUString const&, SfxItemSet&) at sfx2/source/doc/objserv.cxx:308:19 (instdir/program/libsfxlo.so +0x38667e3)
>  #24 in SfxBaseModel::impl_store(rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, bool) at sfx2/source/doc/sfxbasemodel.cxx:2969:46 (instdir/program/libsfxlo.so +0x3a3c48e)
>  #25 in SfxBaseModel::storeToURL(rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at sfx2/source/doc/sfxbasemodel.cxx:1639:13 (instdir/program/libsfxlo.so +0x3a425d5)
>  #26 in SwModelTestBase::reload(char const*, char const*, char const*) at sw/qa/extras/inc/swmodeltestbase.hxx:797:20 (workdir/LinkTarget/CppunitTest/libtest_sw_ooxmlexport.so +0x364711)
>  #27 in SwModelTestBase::executeImportExportImportTest(char const*, char const*) at sw/qa/extras/inc/swmodeltestbase.hxx:283:9 (workdir/LinkTarget/CppunitTest/libtest_sw_ooxmlexport.so +0x362d08)
>  #28 in testTextboxTable::Import_Export_Import() at sw/qa/extras/ooxmlexport/ooxmlexport.cxx:559:1 (workdir/LinkTarget/CppunitTest/libtest_sw_ooxmlexport.so +0x43c8dd)
[...]

Change-Id: I21d028af121691d51b053c1bf9e49c656be62b77
Reviewed-on: https://gerrit.libreoffice.org/63309
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>

1 files changed, 9 insertions, 2 deletions

diff --git a/filter/source/msfilter/escherex.cxx b/filter/source/msfilter/escherex.cxx
index ecffc34b4cb0..a41f5c1d9be9 100644
--- a/filter/source/msfilter/escherex.cxx
+++ b/filter/source/msfilter/escherex.cxx
@@ -19,6 +19,7 @@
 
 #include "eschesdo.hxx"
 #include <o3tl/any.hxx>
+#include <o3tl/clamp.hxx>
 #include <o3tl/make_unique.hxx>
 #include <svx/svdxcgv.hxx>
 #include <svx/svdomedia.hxx>
@@ -2926,9 +2927,15 @@ void EscherPropertyContainer::CreateCustomShapeProperties( const MSO_SPT eShapeT
                             for (auto const& equation : aEquations)
                             {
                                 aMemStrm.WriteUInt16( equation.nOperation )
-                                    .WriteInt16( equation.nPara[ 0 ] )
+                                    .WriteInt16(
+                                        o3tl::clamp(
+                                            equation.nPara[ 0 ], sal_Int32(SAL_MIN_INT16),
+                                            sal_Int32(SAL_MAX_INT16)) )
                                     .WriteInt16( equation.nPara[ 1 ] )
-                                    .WriteInt16( equation.nPara[ 2 ] );
+                                    .WriteInt16(
+                                        o3tl::clamp(
+                                            equation.nPara[ 2 ], sal_Int32(SAL_MIN_INT16),
+                                            sal_Int32(SAL_MAX_INT16)) );
                             }
 
                             AddOpt(DFF_Prop_pFormulas, true, 6, aMemStrm);