vcl: fix Win32 deadlocks from SolarMutexReleaser

To create and destroy thread-affine Win32 Windows and DCs, non-main
threads SendMessage() special messages like SAL_MSG_CREATEFRAME.

The main thread must handle these messages and return the result to
un-block the other thread.

This works fine as long as the main thread is in its message loop anyway
and blocked on GetMessage(); however if the main blocks trying to acquire
the SolarMutex that is held by the sending thread, deadlock results.

In order to work around this, there is some peculiar code in
ImplSalYieldMutexAcquireWithWait() to avoid blocking the main thread on
mpSalYieldMutex but instead block in GetMessage().

The crucial detail is that GetMessage() will immediately dispatch any
message sent via SendMessage(), which avoids the deadlock.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms644936.aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/ms644927.aspx

Most of the Win32 WndProc that acquire SolarMutex do so via
ImplSalYieldMutexAcquireWithWait(), but the main thread may also
temporarily drop SolarMutex and re-aquire it with the questionable
SolarMutexReleaser hack, which calls ImplSalAcquireYieldMutex()
instead, which blocks on mpSalYieldMutex.

Fix SolarMutexReleaser to call a new function
Application::ReAcquireSolarMutex() that does the right thing here:
acquire SolarMutex via ImplSalYieldMutexAcquireWithWait().

It turns out that this problem was already fixed before in commit
6a8fd4c76a969ac98d1aff91ff7442f43aee0006 but the problem was
insufficiently documented in the commit and it introduced the bug
that Application::Reschedule() was called without having the SolarMutex
locked, which caused timers to run without SolarMutex, so the commit
was reverted in 1ef1781390845d03b6e1518bbac81b818be62f3d.

Change-Id: I60aae555a9ee3c6390f584feddbc6b3cb7de0250

1 files changed, 30 insertions, 6 deletions

diff --git a/vcl/source/app/svapp.cxx b/vcl/source/app/svapp.cxx
index 536e81b7d229..42efa3b9c39c 100644
--- a/vcl/source/app/svapp.cxx
+++ b/vcl/source/app/svapp.cxx
@@ -338,19 +338,23 @@ void Application::Execute()
     pSVData->maAppData.mbInAppExecute = false;
 }
 
-inline void ImplYield( bool i_bWait, bool i_bAllEvents )
+inline void ImplYield(bool i_bWait, bool i_bAllEvents, sal_uLong const nReleased)
 {
     ImplSVData* pSVData = ImplGetSVData();
 
-    //Process all Tasks
-    Scheduler::ProcessTaskScheduling(false);
+    if (nReleased == 0) // else thread doesn't have SolarMutex so avoid race
+    {   // Process all Tasks
+        Scheduler::ProcessTaskScheduling(false);
+    }
 
     pSVData->maAppData.mnDispatchLevel++;
     // do not wait for events if application was already quit; in that
     // case only dispatch events already available
     // do not wait for events either if the app decided that it is too busy for timers
     // (feature added for the slideshow)
-    pSVData->mpDefInst->Yield( i_bWait && !pSVData->maAppData.mbAppQuit && !pSVData->maAppData.mbNoYield, i_bAllEvents );
+    pSVData->mpDefInst->DoYield(
+        i_bWait && !pSVData->maAppData.mbAppQuit && !pSVData->maAppData.mbNoYield,
+        i_bAllEvents, nReleased);
     pSVData->maAppData.mnDispatchLevel--;
 
     DBG_TESTSOLARMUTEX(); // must be locked on return from Yield
@@ -374,12 +378,32 @@ inline void ImplYield( bool i_bWait, bool i_bAllEvents )
 
 void Application::Reschedule( bool i_bAllEvents )
 {
-    ImplYield( false, i_bAllEvents );
+    ImplYield(false, i_bAllEvents, 0);
 }
 
 void Application::Yield()
 {
-    ImplYield( true, false );
+    ImplYield(true, false, 0);
+}
+
+void Application::ReAcquireSolarMutex(sal_uLong const nReleased)
+{
+    // 0 would mean that events/timers will be handled without locking
+    // SolarMutex (racy)
+    assert(nReleased != 0);
+#ifdef WNT
+    if (ImplGetSVData()->mbDeInit) // do not Yield in DeInitVCL
+        AcquireSolarMutex(nReleased);
+    else
+        ImplYield(false, false, nReleased);
+#else
+    // a) Yield is not needed on non-WNT platforms
+    // b) some Yield implementations for X11 (e.g. kde4) make it non-obvious
+    //    how to use nReleased
+    // c) would require a review of what all Yield implementations do
+    //    currently _before_ releasing SolarMutex that would run without lock
+    AcquireSolarMutex(nReleased);
+#endif
 }
 
 IMPL_STATIC_LINK_NOARG( ImplSVAppData, ImplQuitMsg )