Avoid gengal hanging in an --enable-dbgutil build on Windows

With a newer C++ debug runtime (in an --enable-dbgutil build), passing
an invalid locale name causes an attempt to display an error
dialog. Which does not even show up, at least for me, but instead the
process (gengal, at least) just hangs. Which is far from ideal.

Passing a POSIX-style locale name to the std::locale constructor on
Windows is a bit odd, but apparently in the normal C++ runtime it
"just" causes an exception to be thrown, that boost catches (see the
loadable(std::string name) in boost's libs\locale\src\std\std_backend.cpp),
and then instead uses the Windows style locale name it knows how to
construct. (Why it even tries the POSIX style name on Windows I can't
understand.)

Actually it isn't just the locale name part "en_US" of a locale like
"en_US.UTF-8" that is problematic, but also the encoding part,
"UTF-8". The Microsoft C/C++ library does not support UTF-8
locales. The error message that our own report hook catches says:
"f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c(89) : Assertion failed:
ploc->_Mbcurmax == 1 || ploc->_Mbcurmax == 2". Clearly in a UTF-8
locale (perhaps one that boost internally constructs?) the maximum
bytes per character will be more than 2.

With a debug C++ runtime, we need to avoid the error dialog, and just
ignore the error. So we install an own CRT error report hook that
ignores the error for the duration of the locale construcion.

Change-Id: Ia2ca994f03d1ce94ce7e9d7607f204c320ab8f2d
Reviewed-on: https://gerrit.libreoffice.org/54110
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
(cherry picked from commit 2ff121f298e64ff204621220622472fe697d599d)

1 files changed, 69 insertions, 1 deletions

diff --git a/unotools/source/i18n/resmgr.cxx b/unotools/source/i18n/resmgr.cxx
index a0ad625f7aff..4159cc670f81 100644
--- a/unotools/source/i18n/resmgr.cxx
+++ b/unotools/source/i18n/resmgr.cxx
@@ -1,4 +1,4 @@
-/* -*- Mode: C++; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */
+/* -*- Mode: C++; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4; fill-column: 100 -*- */
 /*
  * This file is part of the LibreOffice project.
  *
@@ -55,6 +55,13 @@
 #include <unordered_map>
 #include <memory>
 
+#if defined(_WIN32) && defined(DBG_UTIL)
+#include <o3tl/char16_t2wchar_t.hxx>
+#include <prewin.h>
+#include <crtdbg.h>
+#include <postwin.h>
+#endif
+
 namespace
 {
     OUString createFromUtf8(const char* data, size_t size)
@@ -87,6 +94,26 @@ namespace
     }
 }
 
+#if defined(_WIN32) && defined(DBG_UTIL)
+static int IgnoringCrtReportHook(int reportType, wchar_t *message, int * /* returnValue */)
+{
+    OUString sType;
+    if (reportType == _CRT_WARN)
+        sType = "WARN";
+    else if (reportType == _CRT_ERROR)
+        sType = "ERROR";
+    else if (reportType == _CRT_ASSERT)
+        sType = "ASSERT";
+    else
+        sType = "?(" + OUString::number(reportType) + ")";
+
+    SAL_WARN("unotools.i18n", "CRT Report Hook: " << sType << ": " << OUString(o3tl::toU(message)));
+
+    return TRUE;
+}
+#endif
+
+
 namespace Translate
 {
     std::locale Create(const sal_Char* pPrefixName, const LanguageTag& rLocale)
@@ -106,7 +133,48 @@ namespace Translate
         osl::File::getSystemPathFromFileURL(uri, path);
         gen.add_messages_path(OUStringToOString(path, osl_getThreadTextEncoding()).getStr());
         gen.add_messages_domain(pPrefixName);
+
+#if defined(_WIN32) && defined(DBG_UTIL)
+        // With a newer C++ debug runtime (in an --enable-dbgutil build), passing an invalid locale
+        // name causes an attempt to display an error dialog. Which does not even show up, at least
+        // for me, but instead the process (gengal, at least) just hangs. Which is far from ideal.
+
+        // Passing a POSIX-style locale name to the std::locale constructor on Windows is a bit odd,
+        // but apparently in the normal C++ runtime it "just" causes an exception to be thrown, that
+        // boost catches (see the loadable(std::string name) in boost's
+        // libs\locale\src\std\std_backend.cpp), and then instead uses the Windows style locale name
+        // it knows how to construct. (Why does it even try the POSIX style name I can't
+        // understand.)
+
+        // Actually it isn't just the locale name part "en_US" of a locale like "en_US.UTF-8" that
+        // is problematic, but also the encoding part, "UTF-8". The Microsoft C/C++ library does not
+        // support UTF-8 locales. The error message that our own report hook catches says:
+        // "f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c(89) : Assertion failed: ploc->_Mbcurmax == 1
+        // || ploc->_Mbcurmax == 2". Clearly in a UTF-8 locale (perhaps one that boost internally
+        // constructs?) the maximum bytes per character will be more than 2.
+
+        // With a debug C++ runtime, we need to avoid the error dialog, and just ignore the error.
+
+        struct CrtSetReportHook
+        {
+            int mnCrtSetReportHookSucceeded;
+
+            CrtSetReportHook()
+            {
+                mnCrtSetReportHookSucceeded = _CrtSetReportHookW2(_CRT_RPTHOOK_INSTALL, IgnoringCrtReportHook);
+            }
+
+            ~CrtSetReportHook()
+            {
+                if (mnCrtSetReportHookSucceeded >= 0)
+                    _CrtSetReportHookW2(_CRT_RPTHOOK_REMOVE, IgnoringCrtReportHook);
+            }
+        } aHook;
+
+#endif
+
         std::locale aRet(gen(sIdentifier.getStr()));
+
         aCache[sUnique] = aRet;
         return aRet;
     }