ITEM: Get away from classic 'poolable' Item flag

To understand this, some look back in history will be needed to see
why it is as it is today. In some (reworked) comments 'poolable' is
described as flag to hold Items in the ItemPool, also always having
only one incarnation of each possible Item.
This is not the original intention, but a side-effect. The reason is
what the binary format in the office did: To save a document, the
Objects & the Pool were saved, *not* individual Items *together*
with the objects. The Pool was completely (binary) saved (and loaded)
in one run.
Temporary IDs were used to represent at the objects in file which
Items were referenced. This *required* to have only one incarnation
per item to have a minimal binary file size, thus this high effort
was put into this. At doc load, the pool was loaded, all Items were
set to RefCount 5000, the references from the objects were restored
and then for each Item the RefCount was lowered by 5000 again
and - if being zero - deleted. Items for UI were marked 'non-poolable'
to *not* safe them with the document, so poolable was a flag to decide
if that Info/Item was to be saved with the document - or more direct:
if it is Model Data.
Items are small, so if we prefer runtime it is okay to no longer being
strict with this, anyways does not happen often and has only marginal
memory effects - compared to runtime effects/savings.
Other problems which this caused: One example is that objects in the
UNDO stack were still in the pool, so e.g. deleted pictures were saved
with the document despite no longer being used (!). That is the reason
we have an UndoItemPool and a method MigrateItemPool to move stuff to
that Pool when objects go to the UNDO stack - all of this is also no
longer needed.
Cleaning this up means to ideally have all items in the SfxItemSet,
no longer at the Pool. The Pool should be reduced to a 'Default-Item-
Holder' and a 'Slot-to-whichId-mapper'.

This needs thorough cleanups/removals, but will be worth it because
that massive simplification(s) will increase safety an runtime and make
migrating to the goal of completely type-based ItemSet stuff easier for
the future. Hopefully only view code in the office working with items
will have to be changed for this.

In this 1st step I already found that some 'compromizes' will be
needed:
- There are still Items that have to be at the pool to make the
Surrogate-stuff working. This gives back all Items in a Pool of a type
and is used in ca. 80 cases. Each one looks at these Items *without*
context (e.g. a SfxItemSet at an Object would be a context), so if e.g.
a dialog is open that temporarily uses Items of that type you would
also get these - without knowing about it...
To make that work there is still a mechanism to have Items at the Pool,
but now just *registering* (and un-reg) them without any sort/search/
remove needs. Also only for Items that need that, so I evaluated the
GetItemSurrogates calls and added some asserts when GetItemSurrogates
tries to access an unregistered item type which needs to be added.

- Another caveat is that there are about 250 places that directly put
Items to the Pool (not all remove these, that is done at pool deletion,
so some kind of silent 'garbage-collection' is in place). To have an
overview I renamed the accessing methods to separate them from the same
functionality at the SfxItemSet, which had the same names. An
implementation does still add these directly to the pool, there is no
way to cleanup those usages for now. In principle all these should be
changed to hold the data at an SfxItemSet.

I am still hunting problems. But you can build the office, all apps
work (including chart) and you can do speed comparisons already.

There are test throwing errors, so I hunt these now. It is hard to
give an estimation about how much more changes/corrections will be
needed.

Completed adaptions to new registered Items at Pool, that reduces the
failing tests. Still many that I need to hunt.

Added stuff to work around that 'compromize' in ScDocumentPool: It
overloads ::PutImpl of the pool to implement special handling for
a single Item in SC, the ScPatternAttr. In former code that method
was used from SfxItemSet and ::PutImpl at the pool directly, so it
was only used in one place. I am not sure if it was used from
the SfxItemSet functionality, but better offer it for now. To not
waste too much runtime the callbacks depend on the boolean
'NewItemCallback' at the SfxPoolItem, it gets set for that single
Item in SC and only then the callbacks trigger. I hope to get rid
of those again, e.g. newItem_UseDirect is only needed since we have
no 'real' StaticPoolDefaults currently - another thing that needs to
be cleaned up in a next step.

Since usages of impl(Create|Cleanup)ItemEntry and
Direct(Put|Remove)ItemInPoolImpl got more and more similar I decided to
unify that: move impl(Create|Cleanup)ItemEntry to tooling, make it
globally available in svl and use it also directly for
Direct(Put|Remove)ItemInPoolImpl. This slightly increases the failing
tests again, but only since in Direct(Put|Remove)ItemInPoolImpl that
fallback (e.g. tryToGetEqualItem) was used before, thus this is the
same class of errors (SfxPoolItem ptr-compare) as the others which I
will need to find anyways. Also fixed some missing stuff.

Have now idenified and redirected all SfxPoolItem ptr-compares
to be able to debug these - one cause for the remaining errors is
probably that before with bPoolable those often were sufficient, but
are no longer. Used the [loplugin:itemcompare] and a local clang
build to do so, see https://gerrit.libreoffice.org/c/core/+/157172

Stabilized Direct(Put|Remove)ItemInPoolImpl forwards, added parameter
to implCreateItemEntry to signal that it gets called from DirectPool
stuff - currently needed. Hopefully when getting rid of that DirectPool
stuff we can remove that again

Added two more debug functionalities:
- Added a SerialNumber to allow targeted debugging for deterministic
cases
- Added registering & listing of still-allocated SfxPoolItems at
office shutdown

Found PtrComp error in thints.cxx - POC, thanks to
areSfxPoolItemPtrsEqual. Will hopefully help more with other tests

Found some wrong asserts/warnings where I was too careful and not
finding something/succeeding is OK, fixes some UnitTests for SC

For SC I now just tried to replace all areSfxPoolItemPtrsEqual with
the full-ptr-content compare SfxPoolItem::areSame. I also needed to
experiment/adapt the newItem_Callback solution but got it working.

Did that replacement now for SW too, found some places where the
direct ptr compare is OK.

Continued for the rest of occurrences, now all 160 places evaluated.
Also done some cleanups.

Massive cleanups of stuff no longer needed with this paradigm change.
Also decided to keep tryToGetEqualItem/ITEM_CLASSIC_MODE for now.
It is used for *one* Item (ScPatternAttr/ATTR_PATTERN) in SC that
already needs many exceptions. Also useful for testing if errors
come up on this change to test if it is related to this.

Added forwarding of target Pool for ::Clone in SvxSetItem and
SvxSetItem, simplified SfxStateCache::SetState_Impl and returned
to simple ptr compares in SfxPoolItem::areSame to not do the test
in areSfxPoolItemPtrsEqual.

Debugged through UITest_calc_tests9 and found that in tdf133629
where BoxStyle is applied to fully selected empty calc the Item-
reuse fallback has to be used not only for ATTR_PATTERN, see
comment @implCreateItemEntry. Maybe more...

Problem with test_tdf156611_insert_hyperlink_like_excel. Found that
in ScEditShell::GetFirstURLFieldFromCell the correct SvxURLField
is found and returned as ptr, but it's usage crashes. That is due to
the SfxItemSet aEditSet used there gets destroyed at function return
what again deletes the SvxFieldItem that is holding the SvxURLField
that gets returned.
This shows a more general problem: There is no 'SfxPoolItemHolder'
that safely holds a single SfxPoolItem - like a SfxItemSet for a
single Item (if Items would be shared_ptrs, that would be a safe
return value).
That will be needed in the future, but for now use another solution:
Since I see no reason why EE_FEATURE_FIELD should not be shareable
I wil change this for ow in the SfxItemInfo for EditCharAttribField.
That way the Item returned will be shared (RefCnt > 1) and thus not
be deleted.

I changed the return value for GetURLField() and
GetFirstURLFieldFromCell() in ScEditShell: At least for
GetFirstURLFieldFromCell the return type/value was not safe: The
SvxFieldItem accessed there and held in the local temporary
SfxItemSet may be deleted with it, so return value can be
corrupted/deleted. To avoid that, return a Clone of SvxFieldData
as a unique_ptr.

With all that UnitTest debugging and hunting and to get the paradigm
change working to no longer rely on shared/pooled items I lost a
little bit focus on speed, so I made an optimization round for the
two central methods implCreateItemEntry/implCleanupItemEntry to
get back to the speed improvements that I detected when starting this
change. It was mainly lost due to that 'strange' chained pool stuff
we have, so I added to detect the target pool (the one at which the
WhichID is registered) directly and only once. Next thing to cleanup
will/should be the pool and it's concept, all this is not needed
and really costs runtime.
Since implCreateItemEntry/implCleanupItemEntry are executed millions
of times, each cycle counts here.

Had an error in the last changes: pool::*_Impl methods use index
instead of WhichID - most of them. Another bad trap, I really need
to cleanup pool stuff next.

Change-Id: I6295f332325b33268ec396ed46f8d0a1026e2d69
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/157559
Tested-by: Jenkins
Reviewed-by: Armin Le Grand <Armin.Le.Grand@me.com>

10 files changed, 133 insertions, 75 deletions

diff --git a/include/editeng/fontitem.hxx b/include/editeng/fontitem.hxx
index deabf3101f44..7cab5f462e4e 100644
--- a/include/editeng/fontitem.hxx
+++ b/include/editeng/fontitem.hxx
@@ -46,7 +46,6 @@ public:
 
     // "pure virtual Methods" from SfxPoolItem
     virtual bool operator==(const SfxPoolItem& rItem) const override;
-    bool operator<(const SfxPoolItem& rCmp) const override;
     virtual SvxFontItem* Clone(SfxItemPool *pPool = nullptr) const override;
     virtual bool QueryValue(css::uno::Any& rVal, sal_uInt8 nMemberId = 0) const override;
     virtual bool PutValue(const css::uno::Any& rVal, sal_uInt8 nMemberId) override;
diff --git a/include/editeng/wghtitem.hxx b/include/editeng/wghtitem.hxx
index 6f5f4b31ccc3..121cfcf318a6 100644
--- a/include/editeng/wghtitem.hxx
+++ b/include/editeng/wghtitem.hxx
@@ -59,14 +59,6 @@ public:
     FontWeight              GetWeight() const { return GetValue(); }
 
     void dumpAsXml(xmlTextWriterPtr pWriter) const override;
-
-    virtual bool IsSortable() const override { return true; }
-
-    virtual bool operator<(const SfxPoolItem& rCmp) const override
-    {
-        auto const & other = static_cast<const SvxWeightItem&>(rCmp);
-        return GetValue() < other.GetValue();
-    }
 };
 
 #endif // INCLUDED_EDITENG_WGHTITEM_HXX
diff --git a/include/sal/log-areas.dox b/include/sal/log-areas.dox
index 8e6d5d81f1ae..0b3907e08610 100644
--- a/include/sal/log-areas.dox
+++ b/include/sal/log-areas.dox
@@ -499,6 +499,7 @@ certain functionality.
 @li @c vcl.helper
 @li @c vcl.icontest
 @li @c vcl.ios.clipboard
+@li @c vcl.items
 @li @c vcl.kf5 - KF5
 @li @c vcl.layout - Widget layout
 @li @c vcl.lazydelete
diff --git a/include/svl/custritm.hxx b/include/svl/custritm.hxx
index 90215aff45d3..3159c41dfc96 100644
--- a/include/svl/custritm.hxx
+++ b/include/svl/custritm.hxx
@@ -39,8 +39,6 @@ public:
     {}
 
     virtual bool operator ==(const SfxPoolItem & rItem) const override;
-    virtual bool operator <(const SfxPoolItem & rItem) const override;
-    virtual bool IsSortable() const override { return true; }
 
     virtual bool GetPresentation(SfxItemPresentation,
                                  MapUnit, MapUnit,
diff --git a/include/svl/itempool.hxx b/include/svl/itempool.hxx
index e85451ecb738..8dfac4766ffb 100644
--- a/include/svl/itempool.hxx
+++ b/include/svl/itempool.hxx
@@ -26,6 +26,7 @@
 #include <svl/whichranges.hxx>
 #include <memory>
 #include <vector>
+#include <unordered_set>
 #include <o3tl/sorted_vector.hxx>
 #include <salhelper/simplereferenceobject.hxx>
 
@@ -34,11 +35,30 @@ struct SfxItemPool_Impl;
 
 struct SfxItemInfo
 {
+    // Defines a mapping between WhichID <-> SlotID
     sal_uInt16       _nSID;
-    bool            _bPoolable;
+
+    // Defines if this Item needs to be registered at the pool
+    // to make it accessible for the GetItemSurrogates call. It
+    // will not be included when this flag is not set, but also
+    // needs no registration. There are SAL_INFO calls in the
+    // GetItemSurrogates impl that will mention that
+    bool             _bNeedsPoolRegistration : 1;
+
+    // Defines if the Item can be shared/RefCounted else it will be cloned.
+    // Default is true - as it should be for all Items. It is needed by some
+    // SW items, so protected to let them set it in constructor. If this could
+    // be fixed at that Items we may remove this again.
+    bool             _bShareable : 1;
 };
 
 class SfxItemPool;
+typedef std::unordered_set<const SfxPoolItem*> registeredSfxPoolItems;
+
+#ifdef DBG_UTIL
+SVL_DLLPUBLIC size_t getAllDirectlyPooledSfxPoolItemCount();
+SVL_DLLPUBLIC size_t getRemainingDirectlyPooledSfxPoolItemCount();
+#endif
 
 /** Base class for providers of defaults of SfxPoolItems.
  *
@@ -53,14 +73,26 @@ class SVL_DLLPUBLIC SfxItemPool : public salhelper::SimpleReferenceObject
     friend class SfxItemSet;
     friend class SfxAllItemSet;
 
+    // allow ItemSetTooling to access
+    friend SfxPoolItem const* implCreateItemEntry(SfxItemPool&, SfxPoolItem const*, sal_uInt16, bool, bool);
+    friend void implCleanupItemEntry(SfxItemPool&, SfxPoolItem const*);
+
+    // unit testing
+    friend class PoolItemTest;
+
     const SfxItemInfo*              pItemInfos;
     std::unique_ptr<SfxItemPool_Impl>               pImpl;
 
+    registeredSfxPoolItems** ppRegisteredSfxPoolItems;
+
 private:
     sal_uInt16                      GetIndex_Impl(sal_uInt16 nWhich) const;
     sal_uInt16                      GetSize_Impl() const;
 
-    SVL_DLLPRIVATE bool             IsItemPoolable_Impl( sal_uInt16 nWhich ) const;
+    SVL_DLLPRIVATE bool             NeedsPoolRegistration_Impl(sal_uInt16 nPos) const
+    { return pItemInfos[nPos]._bNeedsPoolRegistration; }
+    SVL_DLLPRIVATE bool             Shareable_Impl(sal_uInt16 nPos) const
+    { return pItemInfos[nPos]._bShareable; }
 
 public:
     // for default SfxItemSet::CTOR, set default WhichRanges
@@ -135,18 +167,16 @@ public:
     virtual rtl::Reference<SfxItemPool> Clone() const;
     const OUString&                 GetName() const;
 
-    template<class T> const T&      Put( std::unique_ptr<T> xItem, sal_uInt16 nWhich = 0 )
-    { return static_cast<const T&>(PutImpl( *xItem.release(), nWhich, /*bPassingOwnership*/true)); }
-    template<class T> const T&      Put( const T& rItem, sal_uInt16 nWhich = 0 )
-    { return static_cast<const T&>(PutImpl( rItem, nWhich, /*bPassingOwnership*/false)); }
-    void                            Remove( const SfxPoolItem& );
+    template<class T> const T&      DirectPutItemInPool( std::unique_ptr<T> xItem, sal_uInt16 nWhich = 0 )
+    { return static_cast<const T&>(DirectPutItemInPoolImpl( *xItem.release(), nWhich, /*bPassingOwnership*/true)); }
+    template<class T> const T&      DirectPutItemInPool( const T& rItem, sal_uInt16 nWhich = 0 )
+    { return static_cast<const T&>(DirectPutItemInPoolImpl( rItem, nWhich, /*bPassingOwnership*/false)); }
+    void                            DirectRemoveItemFromPool( const SfxPoolItem& );
 
     const SfxPoolItem&              GetDefaultItem( sal_uInt16 nWhich ) const;
     template<class T> const T&      GetDefaultItem( TypedWhichId<T> nWhich ) const
     { return static_cast<const T&>(GetDefaultItem(sal_uInt16(nWhich))); }
 
-    bool                            CheckItemInPool(const SfxPoolItem *) const;
-
     struct Item2Range
     {
         o3tl::sorted_vector<SfxPoolItem*>::const_iterator m_begin;
@@ -158,8 +188,7 @@ public:
     template<class T> const T*      GetItem2Default( TypedWhichId<T> nWhich ) const
     { return static_cast<const T*>(GetItem2Default(sal_uInt16(nWhich))); }
 
-    sal_uInt32                      GetItemCount2(sal_uInt16 nWhich) const;
-    Item2Range                      GetItemSurrogates(sal_uInt16 nWhich) const;
+    const registeredSfxPoolItems&   GetItemSurrogates(sal_uInt16 nWhich) const;
     /*
         This is only valid for SfxPoolItem that override IsSortable and operator<.
         Returns a range of items defined by using operator<.
@@ -179,9 +208,14 @@ public:
 
     void                            Delete();
 
-    bool                            IsItemPoolable( sal_uInt16 nWhich ) const;
-    bool                            IsItemPoolable( const SfxPoolItem &rItem ) const
-                                    { return IsItemPoolable( rItem.Which() ); }
+    bool                            NeedsPoolRegistration(sal_uInt16 nWhich) const;
+    bool                            NeedsPoolRegistration(const SfxPoolItem &rItem) const
+                                    { return NeedsPoolRegistration(rItem.Which()); }
+
+    bool                            Shareable(sal_uInt16 nWhich) const;
+    bool                            Shareable(const SfxPoolItem &rItem) const
+                                    { return Shareable(rItem.Which()); }
+
     void                            SetItemInfos( const SfxItemInfo *pInfos );
     sal_uInt16                      GetWhich( sal_uInt16 nSlot, bool bDeep = true ) const;
     template<class T>
@@ -196,10 +230,34 @@ public:
     static bool                     IsSlot(sal_uInt16 nId) {
                                         return nId && nId > SFX_WHICH_MAX; }
 
+    // this method tries to register an Item at this Pool. If this
+    // is done depends on the SfxItemInfo-flag _bNeedsPoolRegistration
+    // which needs to be set for Items that are acessed using
+    // GetItemSurrogates, else the Item will not be returned/accessed
+    void tryRegisterSfxPoolItem(const SfxPoolItem& rItem, bool bPoolDirect);
+
+    // this method will register the Item at this Pool, no matter what.
+    // It is needed for all calls that directly register Items at the
+    // Pool, so the DirectPutItemInPool-methods
+    void doRegisterSfxPoolItem(const SfxPoolItem& rItem);
+
+    // this method will unregister an Item from this Pool
+    void unregisterSfxPoolItem(const SfxPoolItem& rItem);
+
+    // check if this Item is registered at this Pool, needed to detect
+    // if an Item is to be set at another Pool and needs to be cloned
+    bool isSfxPoolItemRegisteredAtThisPool(const SfxPoolItem& rItem) const;
+
+    // try to find an equal existing Item to given one in pool
+    const SfxPoolItem* tryToGetEqualItem(const SfxPoolItem& rItem, sal_uInt16 nWhich) const;
+
     void                            dumpAsXml(xmlTextWriterPtr pWriter) const;
 
 protected:
-    virtual const SfxPoolItem&      PutImpl( const SfxPoolItem&, sal_uInt16 nWhich = 0, bool bPassingOwnership = false );
+    const SfxPoolItem&      DirectPutItemInPoolImpl( const SfxPoolItem&, sal_uInt16 nWhich = 0, bool bPassingOwnership = false );
+    virtual void newItem_Callback(const SfxPoolItem& rItem) const;
+    virtual bool newItem_UseDirect(const SfxPoolItem& rItem) const;
+
 private:
     const SfxItemPool&              operator=(const SfxItemPool &) = delete;
 
diff --git a/include/svl/itemset.hxx b/include/svl/itemset.hxx
index 01d9d5058b14..06e82b87e054 100644
--- a/include/svl/itemset.hxx
+++ b/include/svl/itemset.hxx
@@ -36,11 +36,19 @@ SVL_DLLPUBLIC size_t getAllocatedSfxItemSetCount();
 SVL_DLLPUBLIC size_t getUsedSfxItemSetCount();
 #endif
 
+// ItemSet/ItemPool helpers
+SfxPoolItem const* implCreateItemEntry(SfxItemPool& rPool, SfxPoolItem const* pSource, sal_uInt16 nWhich, bool bPassingOwnership, bool bPoolDirect);
+void implCleanupItemEntry(SfxItemPool& rPool, SfxPoolItem const* pSource);
+
 class SAL_WARN_UNUSED SVL_DLLPUBLIC SfxItemSet
 {
     friend class SfxItemIter;
     friend class SfxWhichIter;
 
+    // allow ItemSetTooling to access
+    friend SfxPoolItem const* implCreateItemEntry(SfxItemPool&, SfxPoolItem const*, sal_uInt16, bool, bool);
+    friend void implCleanupItemEntry(SfxItemPool&, SfxPoolItem const*);
+
     SfxItemPool*      m_pPool;         ///< pool that stores the items
     const SfxItemSet* m_pParent;       ///< derivation
     sal_uInt16        m_nCount;        ///< number of items
@@ -88,7 +96,7 @@ private:
     const SfxItemSet&           operator=(const SfxItemSet &) = delete;
 
 protected:
-    virtual const SfxPoolItem*  PutImpl( const SfxPoolItem&, sal_uInt16 nWhich, bool bItemIsSetMember, bool bPassingOwnership );
+    virtual const SfxPoolItem*  PutImpl( const SfxPoolItem&, sal_uInt16 nWhich, bool bPassingOwnership );
 
     /** special constructor for SfxAllItemSet */
     enum class SfxAllItemSetFlag { Flag };
@@ -207,9 +215,9 @@ public:
     // add, delete items, work on items
 public:
     const SfxPoolItem*          Put( const SfxPoolItem& rItem, sal_uInt16 nWhich )
-    { return PutImpl(rItem, nWhich, /*bItemIsSetMember*/false, /*bPassingOwnership*/false); }
+    { return PutImpl(rItem, nWhich, /*bPassingOwnership*/false); }
     const SfxPoolItem*          Put( std::unique_ptr<SfxPoolItem> xItem, sal_uInt16 nWhich )
-    { return PutImpl(*xItem.release(), nWhich, /*bItemIsSetMember*/false, /*bPassingOwnership*/true); }
+    { return PutImpl(*xItem.release(), nWhich, /*bPassingOwnership*/true); }
     const SfxPoolItem*          Put( const SfxPoolItem& rItem )
                                 { return Put(rItem, rItem.Which()); }
     const SfxPoolItem*          Put( std::unique_ptr<SfxPoolItem> xItem )
@@ -261,7 +269,7 @@ private:
     sal_uInt16 ClearAllItemsImpl();
 
     // Merge two given Item(entries)
-    void MergeItem_Impl(const SfxPoolItem **ppFnd1, const SfxPoolItem *pFnd2, bool bItemIsSetMember, bool bIgnoreDefaults);
+    void MergeItem_Impl(const SfxPoolItem **ppFnd1, const SfxPoolItem *pFnd2, bool bIgnoreDefaults);
 
     // split version(s) of InvalidateItem for input types WhichID and Offset
     void InvalidateItem_ForWhichID(sal_uInt16 nWhich);
@@ -270,9 +278,6 @@ private:
     // split version(s) of GetItemStateImpl for input types WhichID and Offset
     SfxItemState GetItemState_ForWhichID( SfxItemState eState, sal_uInt16 nWhich, bool bSrchInParent, const SfxPoolItem **ppItem) const;
     SfxItemState GetItemState_ForOffset( sal_uInt16 nOffset, const SfxPoolItem **ppItem) const;
-
-    SfxPoolItem const* implCreateItemEntry(SfxPoolItem const* pSource, sal_uInt16 nWhich, bool bItemIsSetMember, bool bPassingOwnership);
-    void implCleanupItemEntry(SfxPoolItem const* pSource);
 };
 
 inline void SfxItemSet::SetParent( const SfxItemSet* pNew )
@@ -292,7 +297,7 @@ public:
 
     virtual std::unique_ptr<SfxItemSet> Clone( bool bItems = true, SfxItemPool *pToPool = nullptr ) const override;
 private:
-    virtual const SfxPoolItem*  PutImpl( const SfxPoolItem&, sal_uInt16 nWhich, bool bItemIsSetMember,bool bPassingOwnership ) override;
+    virtual const SfxPoolItem*  PutImpl( const SfxPoolItem&, sal_uInt16 nWhich, bool bPassingOwnership ) override;
 };
 
 
diff --git a/include/svl/poolitem.hxx b/include/svl/poolitem.hxx
index 4714e2f44be6..b434302a2f85 100644
--- a/include/svl/poolitem.hxx
+++ b/include/svl/poolitem.hxx
@@ -100,6 +100,7 @@ enum class SfxItemState {
 #ifdef DBG_UTIL
 SVL_DLLPUBLIC size_t getAllocatedSfxPoolItemCount();
 SVL_DLLPUBLIC size_t getUsedSfxPoolItemCount();
+SVL_DLLPUBLIC void listAllocatedSfxPoolItems();
 #endif
 
 class SfxItemPool;
@@ -108,31 +109,40 @@ typedef struct _xmlTextWriter* xmlTextWriterPtr;
 
 class SVL_DLLPUBLIC SfxPoolItem
 {
-friend class SfxItemPool;
-friend class SfxItemDisruptor_Impl;
-friend class SfxItemPoolCache;
-friend class SfxItemSet;
+    friend class SfxItemPool;
+    friend class SfxItemDisruptor_Impl;
+    friend class SfxItemPoolCache;
+    friend class SfxItemSet;
+
+    // allow ItemSetTooling to access
+    friend SfxPoolItem const* implCreateItemEntry(SfxItemPool&, SfxPoolItem const*, sal_uInt16, bool, bool);
+    friend void implCleanupItemEntry(SfxItemPool&, SfxPoolItem const*);
 
     mutable sal_uInt32 m_nRefCount;
     sal_uInt16  m_nWhich;
 
+#ifdef DBG_UTIL
+    // for debugging add a serial number, will be set in the constructor
+    // and count up from zero. If you have a deterministic error case and
+    // see the Item involved in the debugger you can  use that number in
+    // the next run to see where that Item gets constructed and how it is
+    // involved/ processed
+    sal_uInt32  m_nSerialNumber;
+#endif
+
     // bitfield for flags (instead of SfxItemKind)
-    bool        m_bIsVoidItem : 1;
-    bool        m_bDeleteOnIdle : 1;
-    bool        m_bStaticDefault : 1;
-    bool        m_bPoolDefault : 1;
+    bool        m_bIsVoidItem : 1;          // bit 0
+    bool        m_bDeleteOnIdle : 1;        // bit 1
+    bool        m_bStaticDefault : 1;       // bit 2
+    bool        m_bPoolDefault : 1;         // bit 3
+    bool        m_bRegisteredAtPool : 1;    // bit 4
+    bool        m_bNewItemCallback : 1;     // bit 5
+    bool        m_bIsSetItem : 1;           // bit 6
 
 protected:
-    // Defines if the Item can be shared/RefCounted else it will be cloned.
-    // Default is true - as it should be for all Items. It is needed by some
-    // SW items, so protected to let them set it in constructor. If this could
-    // be fixed at that Items we may remove this again.
-    bool        m_bShareable : 1;
-
-private:
 #ifdef DBG_UTIL
     // this flag will make debugging item stuff much simpler
-    bool        m_bDeleted : 1;
+    bool        m_bDeleted : 1;             // bit 7
 #endif
 
 private:
@@ -143,10 +153,13 @@ private:
     }
 
 protected:
-    void setVoidItem() { m_bIsVoidItem = true; }
+    void setIsVoidItem() { m_bIsVoidItem = true; }
     void setDeleteOnIdle() { m_bDeleteOnIdle = true; }
     void setStaticDefault() { m_bStaticDefault = true; }
     void setPoolDefault() { m_bPoolDefault = true; }
+    void setRegisteredAtPool(bool bNew) { m_bRegisteredAtPool = bNew; }
+    void setNewItemCallback() { m_bNewItemCallback = true; }
+    void setIsSetItem() { m_bIsSetItem = true; }
 
 public:
     inline void AddRef(sal_uInt32 n = 1) const
@@ -156,11 +169,23 @@ public:
         m_nRefCount += n;
     }
 
+#ifdef DBG_UTIL
+    sal_uInt32 getSerialNumber() const { return m_nSerialNumber; }
+#endif
+
     bool isVoidItem() const { return m_bIsVoidItem; }
     bool isDeleteOnIdle() const { return m_bDeleteOnIdle; }
     bool isStaticDefault() const { return m_bStaticDefault; }
     bool isPoolDefault() const { return m_bPoolDefault; }
-    bool isShareable() const { return m_bShareable; }
+    bool isRegisteredAtPool() const { return m_bRegisteredAtPool; }
+    bool isNewItemCallback() const { return m_bNewItemCallback; }
+    bool isSetItem() const { return m_bIsSetItem; }
+
+    // version that allows nullptrs
+    static bool areSame(const SfxPoolItem* pItem1, const SfxPoolItem* pItem2);
+
+    // if you have the items (not nullptrs) use this version
+    static bool areSame(const SfxPoolItem& rItem1, const SfxPoolItem& rItem2);
 
 private:
     inline sal_uInt32 ReleaseRef(sal_uInt32 n = 1) const
@@ -223,24 +248,6 @@ public:
     bool                     operator!=( const SfxPoolItem& rItem ) const
                              { return !(*this == rItem); }
 
-    // Sorting is only used for faster searching in a pool which contains large quantities
-    // of a single kind of pool item.
-    virtual bool             operator<( const SfxPoolItem& ) const { assert(false); return false; }
-    virtual bool             IsSortable() const { return false; }
-
-    // Some item types cannot be IsSortable() (such as because they are modified while stored
-    // in a pool, which would change the ordering position, see e.g. 585e0ac43b9bd8a2f714903034).
-    // To improve performance in such cases it is possible to reimplement Lookup() to do a linear
-    // lookup optimized for the specific class (avoiding virtual functions may allow the compiler
-    // to generate better code and class-specific optimizations such as hashing or caching may
-    // be used.)
-    // If reimplemented, the Lookup() function should search [begin,end) for an item matching
-    // this object and return an iterator pointing to the item or the end iterator.
-    virtual bool             HasLookup() const { return false; }
-    typedef std::vector<SfxPoolItem*>::const_iterator lookup_iterator;
-    virtual lookup_iterator  Lookup(lookup_iterator /*begin*/, lookup_iterator end ) const
-                             { assert( false ); return end; }
-
     /**  @return true if it has a valid string representation */
     virtual bool             GetPresentation( SfxItemPresentation ePresentation,
                                     MapUnit eCoreMetric,
@@ -299,6 +306,8 @@ inline bool IsInvalidItem(const SfxPoolItem *pItem)
     return pItem == INVALID_POOL_ITEM;
 }
 
+SVL_DLLPUBLIC bool areSfxPoolItemPtrsEqual(const SfxPoolItem* pItem1, const SfxPoolItem* pItem2);
+
 class SVL_DLLPUBLIC SfxPoolItemHint final : public SfxHint
 {
     SfxPoolItem* pObj;
diff --git a/include/svx/pageitem.hxx b/include/svx/pageitem.hxx
index 32702fe8089f..de93730731d2 100644
--- a/include/svx/pageitem.hxx
+++ b/include/svx/pageitem.hxx
@@ -98,7 +98,7 @@ class SVX_DLLPUBLIC SvxSetItem final : public SfxSetItem
 {
 public:
     SvxSetItem( const TypedWhichId<SvxSetItem> nId, const SfxItemSet& rSet );
-    SvxSetItem( const SvxSetItem& rItem );
+    SvxSetItem( const SvxSetItem& rItem, SfxItemPool* pPool = nullptr );
     SvxSetItem( const TypedWhichId<SvxSetItem> nId, SfxItemSet&& pSet );
 
     virtual SvxSetItem* Clone( SfxItemPool *pPool = nullptr ) const override;
diff --git a/include/svx/sdasitm.hxx b/include/svx/sdasitm.hxx
index e53e9c3604a2..cb7230afe703 100644
--- a/include/svx/sdasitm.hxx
+++ b/include/svx/sdasitm.hxx
@@ -70,8 +70,6 @@ private:
             SdrCustomShapeGeometryItem & operator =(SdrCustomShapeGeometryItem &&) = delete; // due to SfxPoolItem
 
             virtual bool                operator==( const SfxPoolItem& ) const override;
-            virtual bool                operator<( const SfxPoolItem& ) const override;
-            virtual bool                IsSortable() const override { return true; }
 
             virtual bool GetPresentation(SfxItemPresentation ePresentation,
                                          MapUnit eCoreMetric, MapUnit ePresentationMetric,
diff --git a/include/svx/xbtmpit.hxx b/include/svx/xbtmpit.hxx
index 0882081fb7ca..6d7d75f3de7c 100644
--- a/include/svx/xbtmpit.hxx
+++ b/include/svx/xbtmpit.hxx
@@ -41,8 +41,6 @@ public:
             XFillBitmapItem( const XFillBitmapItem& rItem );
 
     virtual bool            operator==( const SfxPoolItem& rItem ) const override;
-    // no idea why, but this item does not play nice with the sorting optimisation, get failures in sd_import_tests
-    virtual bool            IsSortable() const override { return false; }
     virtual XFillBitmapItem* Clone( SfxItemPool* pPool = nullptr ) const override;
 
     virtual bool            QueryValue( css::uno::Any& rVal, sal_uInt8 nMemberId = 0 ) const override;