Use default density if rPPI values are too large

`instdir/program/soffice --convert-to pdf
caolan/id:001708,src:001481,op:havoc,rep:8.jpg` (with the file from
vm138.documentfoundation.org:/srv/crashtestdata/files/) failed with UBSan
float-cast-overflow

> vcl/source/filter/jpeg/jpegc.cxx:371:23: runtime error: 160000 is outside the range of representable values of type 'unsigned short'
>  #0 in WriteJPEG(JPEGWriter*, void*, long, long, basegfx::B2DVector const&, bool, long, long, com::sun::star::uno::Reference<com::sun::star::task::XStatusIndicator> const&) at vcl/source/filter/jpeg/jpegc.cxx:371:23
>  #1 in JPEGWriter::Write(Graphic const&) at vcl/source/filter/jpeg/JpegWriter.cxx:243:16
>  #2 in ExportJPEG(SvStream&, Graphic const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const*, bool*) at vcl/source/filter/jpeg/jpeg.cxx:58:32
>  #3 in GraphicFilter::ExportGraphic(Graphic const&, rtl::OUString const&, SvStream&, unsigned short, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const*) at vcl/source/filter/graphicfilter.cxx:1991:22
>  #4 in (anonymous namespace)::GraphicProvider::storeGraphic(com::sun::star::uno::Reference<com::sun::star::graphic::XGraphic> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at vcl/source/graphic/UnoGraphicProvider.cxx:818:25
>  #5 in vcl::PDFWriterImpl::implWriteBitmapEx(Point const&, Size const&, BitmapEx const&, Graphic const&, VirtualDevice const*, vcl::PDFWriter::PlayMetafileContext const&) at vcl/source/gdi/pdfwriter_impl2.cxx:217:35
>  #6 in vcl::PDFWriterImpl::playMetafile(GDIMetaFile const&, vcl::PDFExtOutDevData*, vcl::PDFWriter::PlayMetafileContext const&, VirtualDevice*) at vcl/source/gdi/pdfwriter_impl2.cxx:809:21
>  #7 in vcl::PDFWriter::PlayMetafile(GDIMetaFile const&, vcl::PDFWriter::PlayMetafileContext const&, vcl::PDFExtOutDevData*) at vcl/source/gdi/pdfwriter.cxx:464:22
>  #8 in PDFExport::ImplExportPage(vcl::PDFWriter&, vcl::PDFExtOutDevData&, GDIMetaFile const&) at filter/source/pdf/pdfexport.cxx:1094:13
>  #9 in PDFExport::ExportSelection(vcl::PDFWriter&, com::sun::star::uno::Reference<com::sun::star::view::XRenderable> const&, com::sun::star::uno::Any const&, StringRangeEnumerator const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue>&, int) at filter/source/pdf/pdfexport.cxx:242:25
>  #10 in PDFExport::Export(rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at filter/source/pdf/pdfexport.cxx:955:28
>  #11 in PDFFilter::implExport(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at filter/source/pdf/pdffilter.cxx:161:24
>  #12 in PDFFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at filter/source/pdf/pdffilter.cxx:224:23
>  #13 in SfxObjectShell::ExportTo(SfxMedium&) at sfx2/source/doc/objstor.cxx:2412:25 (instdir/program/libsfxlo.so +0x52260ad)

as the values read for that (artificial) jpg in
vcl/source/filter/png/pngread.cxx are

maOrigSize = 551 x 211

vs.

maPhysSize = 19435 x 7442

so Graphic::GetPPI returns 323.232 x 160000.

For jpeg_scan_info members density_unit, X_density, and Y_density,
workdir/UnpackedTarball/libjpeg-turbo/libjpeg.txt states:  "The default values
are 0,1,1 indicating square pixels of unknown size."  So lets stick to those if
the actual values would be too large.

(I only check for too large values, not for too small negative ones.  I do not
know if the code could legitimately be called with such---so should handle them
similarly---or not---in which case it should assert that.)

Change-Id: I4b93b278e14069c458c20cdd077ab4fa72ed1f2b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/92696
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>

1 files changed, 11 insertions, 3 deletions

diff --git a/vcl/source/filter/jpeg/jpegc.cxx b/vcl/source/filter/jpeg/jpegc.cxx
index 1ab261e3c8ce..ced814e2fccb 100644
--- a/vcl/source/filter/jpeg/jpegc.cxx
+++ b/vcl/source/filter/jpeg/jpegc.cxx
@@ -19,6 +19,7 @@
 
 #include <sal/config.h>
 #include <sal/log.hxx>
+#include <o3tl/float_int_conversion.hxx>
 #include <o3tl/safeint.hxx>
 
 #include <stdio.h>
@@ -366,9 +367,16 @@ bool WriteJPEG( JPEGWriter* pJPEGWriter, void* pOutputStream,
     jpeg_set_defaults( &cinfo );
     jpeg_set_quality( &cinfo, static_cast<int>(nQualityPercent), FALSE );
 
-    cinfo.density_unit = 1;
-    cinfo.X_density = rPPI.getX();
-    cinfo.Y_density = rPPI.getY();
+    if (o3tl::convertsToAtMost(rPPI.getX(), 65535) && o3tl::convertsToAtMost(rPPI.getY(), 65535))
+    {
+        cinfo.density_unit = 1;
+        cinfo.X_density = rPPI.getX();
+        cinfo.Y_density = rPPI.getY();
+    }
+    else
+    {
+        SAL_WARN("vcl.filter", "ignoring too large PPI " << rPPI);
+    }
 
     if ( ( nWidth > 128 ) || ( nHeight > 128 ) )
         jpeg_simple_progression( &cinfo );