xmlsecurity: some Distinguished Names are less equal than others

It turns out that the 2 backends NSS and MS CryptoAPI generate different
string representations of the same Distinguished Name in at least one
corner case, when a value contains a quote " U+0022.

The CryptoAPI function to generate the strings is:
CertNameToStr(..., CERT_X500_NAME_STR | CERT_NAME_STR_REVERSE_FLAG, ...)

This is documented on MSDN:
https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certnametostra#CERT_X500_NAME_STR

NSS appears to implement RFC 1485, at least that's what the internal
function is named after, or perhaps one of its several successor RFCs
(not clear currently if there's a relevant difference).

This is now causing trouble if a certificate with such a DN is used in a
signature, created on WNT but then verified on another platform, because
commit 5af5ea893bcb8a8eb472ac11133da10e5a604e66
introduced consistency checks that compare the DNs that occur as strings
in META-INF/documentsignatures.xml:

xmlsecurity/source/helper/xmlsignaturehelper.cxx:672: X509Data cannot be parsed

The reason is that in XSecController::setX509Data() the value read from
the X509IssuerSerial element (a string generated by CryptoAPI) doesn't
match the value generated by NSS from the certificate parsed from the
X509Certificate element, so these are erroneously interpreted as 2
distinct certificates.

Try to make the EqualDistinguishedNames() more flexible so that it can
try also a converted variant of the DN.

(libxmlsec's NSS backend also complains that it cannot parse the DN:
x509vfy.c:607: xmlSecNssX509NameRead() '' '' 12 'invalid data for 'char': actual=34 and expected comma ',''
but it manages to validate the signature despite this.)

Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124287
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
(cherry picked from commit e63611fabd38c757809b510fbb71c077880b1081)
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124196
Reviewed-by: Thorsten Behrens <thorsten.behrens@allotropia.de>
(cherry picked from commit 3dfe381032fc61ea31106f103dee9db8277d4d25)

Change-Id: I4f72900738d1f5313146bbda7320a8f44319ebc8
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124420
Tested-by: Michael Stahl <michael.stahl@allotropia.de>
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>

0 files changed, 0 insertions, 0 deletions