tdf#159285 sw floattable: fix loop with inner table wrapped by inner table

Regression from 868140fcc1311259b9d5f666637b33d226511a53 (tdf#60558 sw
floattable: allow wrap of table on the right of a floattable,
2023-12-05), the bugdoc layout looped on load.

Somehow the big while() loop in SwTabFrame::MakeAll() never finishes: it
always tries again but can't reach a state where all of frame area
position, frame area size and frame print area is valid.

Fix the problem by going back to the old behavior (floating table is
wrapped by text frames, not by table frames) for the nested table case:
that keeps the old tdf#60558 use-case working and fixes the new
tdf#159285 use-case.

At some point it would be useful to support the combination of nested
floating tables and the "floating table wrapped by table" combination,
but that will be a new layout feature.

Change-Id: Ia3fdbd08de87e13ddef086ae1339e79a8833674d
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/163630
Reviewed-by: Miklos Vajna <vmiklos@collabora.com>
Tested-by: Jenkins

3 files changed, 23 insertions, 1 deletions

diff --git a/sw/qa/core/layout/data/floattable-wrapped-by-table-nested.docx b/sw/qa/core/layout/data/floattable-wrapped-by-table-nested.docx
new file mode 100644
index 000000000000..d6950a6c8e04
--- /dev/null
+++ b/sw/qa/core/layout/data/floattable-wrapped-by-table-nested.docx
diff --git a/sw/qa/core/layout/tabfrm.cxx b/sw/qa/core/layout/tabfrm.cxx
index 61b1a25109f5..e0d099c77102 100644
--- a/sw/qa/core/layout/tabfrm.cxx
+++ b/sw/qa/core/layout/tabfrm.cxx
@@ -221,6 +221,20 @@ CPPUNIT_TEST_FIXTURE(Test, testInlineTableThenSplitFly)
     // large positive one.
     CPPUNIT_ASSERT_LESS(static_cast<SwTwips>(0), nInlineLeft);
 }
+
+CPPUNIT_TEST_FIXTURE(Test, testSplitFlyWrappedByTableNested)
+{
+    // Given a document with 3 tables, one inline toplevel and two inner ones (one inline, one
+    // floating):
+    // When laying out that document:
+    // Without the accompanying fix in place, this test would have failed here with a layout loop.
+    createSwDoc("floattable-wrapped-by-table-nested.docx");
+
+    // Than make sure we have 3 tables, but only one of them is floating:
+    SwDoc* pDoc = getSwDoc();
+    CPPUNIT_ASSERT_EQUAL(static_cast<size_t>(3), pDoc->GetTableFrameFormats()->GetFormatCount());
+    CPPUNIT_ASSERT_EQUAL(static_cast<size_t>(1), pDoc->GetSpzFrameFormats()->GetFormatCount());
+}
 }
 
 /* vim:set shiftwidth=4 softtabstop=4 expandtab: */
diff --git a/sw/source/core/layout/tabfrm.cxx b/sw/source/core/layout/tabfrm.cxx
index c9b31e84599e..9642e85babc8 100644
--- a/sw/source/core/layout/tabfrm.cxx
+++ b/sw/source/core/layout/tabfrm.cxx
@@ -3373,7 +3373,15 @@ bool SwTabFrame::CalcFlyOffsets( SwTwips& rUpper,
         }
 
         bool bFlyHoriOrientLeft = text::HoriOrientation::LEFT == rHori.GetHoriOrient();
-        if (bSplitFly && !bFlyHoriOrientLeft)
+
+        bool bToplevelSplitFly = false;
+        if (bSplitFly)
+        {
+            // Floating table wrapped by table: avoid this in the nested case.
+            bToplevelSplitFly = !pFly->GetAnchorFrame()->IsInTab();
+        }
+
+        if (bToplevelSplitFly && !bFlyHoriOrientLeft)
         {
             // Only shift to the right if we don't have enough space on the left.
             SwTwips nTabWidth = getFramePrintArea().Width();