| Age | Commit message (Collapse) | Author |
|---|
|
084c64982ef6187292fd73e6deaa4448e0b6f3de "Create MAR updates from msi rather
than from archive" had patched make_add_instruction (in
workdir/UnpackedTarball/onlineupdate/tools/update-packagin/common.sh) to
unconditionally use add-if (with the file itself as test file) rather than plain
add (because we can't reuse Mozilla's extension logic there to determine
optional installation parts), and it noted in its commit message that "addition
of files will cause a MAR update to miss them".
Which turned out to be a problem with
<https://update-mar.libreoffice.org/24.2.2.2/LibreOffice_24.2.2.2_Windows_X86_64_d56cc158d8a96260b836f100ef4b4ef25d6f1a01_from_bf759d854b5ab45b6ef0bfd22e51c6dc4fb8b882_partial.mar>,
updating from 24.2.2.1 with instdir/program/python-core-3.8.18/ to 24.2.2.2 with
instdir/program/python-core-3.8.19/: All the files in python-core-3.8.19 were
recorded as add-if testing on themselves, so were never added.
So a better approach is, for newly added files to make them add (so they get
added unconditionally, even if they are part of some optional component that
happens to not be installed; this hopefully won't cause further issues) instead
of add-if. (Existing files that are added instead of patched still use add-if,
though.) This does not address the create-update-info target for full mar
updates (which is not used currently?), just the create-partial-info target.
(The existing calls to make_add_instruction in the upstream Mozilla code already
have a seemingly rotten mix of two-argument calls and calls with three arguments
where the third argument is ignored, and would check for a fourth "forced"
argument but which is never passed in. So avoid extending that mess further,
and just replace this never-use-add-if call to make_add_instruction directly
with that function's (accordingly tweaked) body.)
Change-Id: Ib596f3aa72c3ab9f896b20f7d623bc574eb15fd8
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/168285
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
Tested-by: Jenkins
|
|
On Windows, when a MAR update generated by create-partial-info contains very
many patches (as easily happens with LibreOffice), applying it would fail with
"failed: 7" (aka WRITE_ERROR, see
workdir/UnpackedTarball/onlineupdate/onlineupdate/source/update/common/updatererror.h)
because in
workdir/UnpackedTarball/onlineupdate/onlineupdate/source/update/updater/updater.cpp
PatchFile::mPatchStream holds open one FILE instance per patch from
PatchFile::Prepare to PatchFile::Execute (and which can't easily be reworked
because of the Lock/UnlockFile done on the underlying HANDLE "so it can't be
messed with [in] between"), so calling NS_tfopen in PatchFile::Prepare will
eventually start to fail with EMFILE.
To avoid that, try to raise the limit to its maximum (but don't fail immediately
if that fails, in case the given MAR update wouldn't run into the issue of too
many patches, anyway), and keep fingers crossed. (See
<https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170>
"_setmaxstdio" for details: "By default, up to 512 files can be open
simultaneously at the stream I/O level. This level includes files opened and
accessed using the fopen, fgetc, and fputc family of functions. The limit of
512 open files at the stream I/O level can be increased to a maximum of 8,192 by
use of the _setmaxstdio function.")
Change-Id: I6b3499f0f6c2060628418a15f5e36021bfe7dd18
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/162442
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
...as the former is more convenient for release engineering (see the discussion
in the comments at
<https://gerrit.libreoffice.org/c/core/+/162157/1#message-8d7ebbcc64a87ee8e4a073ae1a05e3b74f5a3d6a>
"Also enable --with-package-format=archive for LibreOfficeWin64.conf").
Instead of ONLINEUPDATE_MAR_OLDARCHIVE and ONLINEUPDATE_MAR_OLDMETADATA make
variables, the create-partial-info target now only needs an
ONLINEUPDATE_MAR_OLDMSI make variable.
TODO: There are two issues when comparing the content of msi files (extracted
with msiexec /a), which the old code comparing the content of archives had tried
to somewhat (but not fully) address with the metadata files that I had invented
(and now reverted): For one, msiexec /a also extracts content that would
normally be installed somewhere else in the system (e.g., it extracts Fonts,
System, and System64 directories). Differences in those directories will cause
a MAR update to create those directories in the installation directory, rather
than to update the corresponding files in their actual locations. For another,
optional components are not recognized as such, but their content must be added
to the MAR file as add/patch-if, not as plain add/patch. To work around that,
for now *all* files are added as add/patch-if, conditional on the files
themselves. Thus, addition of files will cause a MAR update to miss them.
(As they now exclusively operate on msi files, the create-update-info and
create-partial-info targets are no longer meaningful for non-Windows platforms,
so drop the non-Windows bin/update/create_full_mar_for_languages.py part.)
Change-Id: Ifb55b5e7d1a201b4f50a27cb449a634b96c2e29b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/162399
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
...along the lines of what had already been done for create-update-info, passing
files to the mar executable with an -f files.txt instead of on the command line
(which could grow too long for Windows), and taking the archive metadata into
account.
For the latter, `make create-partial-info` now expects a second make variable,
ONLINEUPDATE_MAR_OLDMETADATA, in addition to ONLINEUPDATE_MAR_OLDARCHIVE. And
there is a new TODO in update/create_partial_update.py that the old and new
metadata should be compared for problematic changes. (For now, any such changes
would just get lost by an update.)
Change-Id: Ide45adedb72958fc32c5b26aa7ebb01530ccf3d7
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161972
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
I got lost trying to figure out how the original
bin/update/create_partial_update.py code was meant to obtain old and new
installation trees to diff, so I simplified that down to the create-partial-info
make target now expecting an ONLINEUPDATE_MAR_OLDARCHIVE make variable that
points at the old archive install set. (And the
--with-online-update-mar-serverurl configure option is gone for good again.)
The remaining changes are similar to what was needed in
28bad382face10be75af3875e44dde89fbc78108 "Fix `make create-update-info` (for
Windows, at least)". (And the mbsdiff and mar tools expect Windows-style
pathnames, but mktemp returns a Unix-style pathname in cygwin shell scripts, so
this needed an additional Windows-only external/onlineupdate/cygpath.patch.)
Change-Id: I40690210d62e3f26fb2d574914a0dd4323e6cd62
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161924
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
That make target operates on an archive, but generates data to be applied to an
msi installation, so suffers from any mismatch between archive and msi install
sets. Two such mismatches, at least on Windows, are:
1 Files that msi will install outside the LibreOffice installation itself (but
which for an archive install set are included under LibreOffice/). This covers
(at least) .Net assemblies and associated files that are installed in the GAC
(scp2 styles ASSEMBLY and ASSIGNCOMPONENT) and fonts (scp2 style FONT).
2 Files that msi will only install conditionally. This covers optional
components (many of which in scp2 are assigned to gid_Module_Optional_...
modules) and user interface languages (which in scp2 are assigned to
gid_Module_Langpack_... modules).
The approach taken here is to create a
workdir/installation/LibreOffice/archive/install/metadata file while building an
archive install set, and to record any files matching 1 (as "skip" lines) or 2
(as "cond" lines). Then, the create-update-info target uses that metadata file
to act accordingly on those files:
1 Files from "skip" lines are simply removed for now from the extracted archive
that is passed to Mozilla's make_full_update.sh script. (TODO: That means that
changes to such files will not be updated with the MAR update mechanism. This
would apparently need some extra processing during the MAR update.)
2 Files from "cond" lines shall be recorded as add-if in the mar file manifest.
Mozilla's make_full_update.sh script already has support for emitting add-if vs.
plain add, but only for files under distribution/extensions/, which doesn't
match our needs. So we generate from the metadata file an ifs file that we pass
into the make_full_update.sh script, and patch that script to also take that ifs
file into account. (Each line in the ifs file is of the form
"testfile" "file"
which works as long as none of the pathnames contains double quote characters.
The Mozilla script code appears to be confused about the arguments to
make_add_instruction(), where this ifs file will be needed: There are calls to
make_add_instruction() with two or three arguments across make_full_update.sh
and make_incremental_update.sh, but make_add_instruction() checks $1, $2, and
$4 (but not $3), so leave that mess alone and pass the ifs file as a global
IFSFILE variable instead.)
The mar file manifest `add-if "testfile" "file"` adds "file" only if "testfile"
is already present, and those two can be different files. TODO: However, for
simplicity, for now I always use "file" also as the "testfile" (so that an
add-if file only gets updated if it was already present). That avoids having to
identify a specific "key file" for each optional component and for each user
interface language, where that key file would be used as the add-if testfile.
But on the other hand, it means that if an optional component or a user
interface language will bring along a completely new file in the future, we will
not install that file during a MAR update. What obviously remains to be done is
to properly assign each add-if file to a specific key file. (And the current
way of identifying add-if files by gid_Module_... names appears to be too
simplistic too. For example, there are some gid_Module_Optional_... that are
installed unconditionally for msi; but it should be harmless that those files
are recorded as add-if rather than as plain add.)
Change-Id: I2fdeed92604f3a2d8a0b500b9e3fa421cfb6a9cc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161917
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
* The shell scripts that had once been copied from Mozilla to bin/update/*.sh
are now included in
onlineupdate-c003be8b9727672e7d30972983b375f4c200233f-2.tar.xz (which is still
generated from the same
<https://github.com/mozilla/gecko-dev/commit/c003be8b9727672e7d30972983b375f4c200233f>
that was used for the original tarball in
3a445cb49795fabe0d8caaf12bfc38eb9e12d5fc "Turn onlineupdate into
external/onlineupdate").
* The additional modifications in external/onlineupdate/lo.patch are:
** Allowing to pass the list of files into the mar tool via -f instead of on the
command line, to avoid "command line too long" errors on Windows, inspired by
the modifications once made directly in our old downstream sources with
4165dd4e465a86ba6abe9afb3abfda5ef72493ea "add a way to create mar file from
file", 8e4d49340bd235a7db8fde1d24dd1d63ddc4d571 "use the new file based
approach for the mar creation", and fb13ed6955cd66017e5348b915af371a184ea633
"add the manifest file to the mar file". (To keep things simple for now, it
still uses a hard-coded maximum of 10000 lines in the file, marked with a
TODO.)
** Not failing when "precomplete file is missing!" (There is a comment
// Applications aren't required to have a precomplete manifest. The mar
// generation scripts enforce the presence of a precomplete manifest.
in
workdir/UnpackedTarball/onlineupdate/onlineupdate/source/update/updater/updater.cpp
and it appears to be OK that we don't have such a precomplete manifest file
and just skip that test.)
* In the Makefile.gbuild create-update-info, the create_full_mar.py script needs
to be called with a Unix pathname on Windows, or else the
#!/usr/bin/env python3
shebang in that script would get confused.
* The related Makefile.gbuild targets upload-update-info and create-partial-info
have not been addressed yet.
Change-Id: Iab4e083ddbe99e07d846e202f20c6031e2983e1b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161656
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
To get the MOZ_MAINTENANCE_SERVICE mode going at all, update.status needs to
contain a "pending-service" token. For Mozilla, code in its
toolkit/mozapps/update/UpdateService.sys.mjs takes care of writing that. For
us, lets always write that in update_checker() (even on Linux, where it's
apparently harmless).
Then, the MOZ_MAINTENANCE_SERVICE code is rather picky with its various sanity
checks: Among other things, it expects argv[0] to be a full path to the updater
executable, and it expects the update.mar (and its status and log files) to be
in a directory hierarchy named updates/0/ rather than patch/. So get all that
fixed in desktop/source/app/updater.cxx. And patch in
external/onlineupdate/lo.patch where it expects to find the updater executable
(just updater.exe vs. our program/updater.exe).
And we shouldn't interfere with the upstream Mozilla maintenance service, so
also rename that in external/onlineupdate/lo.patch.
And `update_service install` wants to read version resources from the
update_service.exe, so provide that (via gb_Executable_add_default_nativeres).
Also, `update_service install` wants to read a MozillaMaintenanceDescription
value from an updater.ini, so provide one (with contents of that value inspired
by Mozilla's browser/locales/en-US/updater/updater.ini).
As we now have an updater.ini anyway (and which apparently works fine with Unix
line ends on both Linux and Windows), also use it on Linux and drop the
onlineupdate/source/update/updater/progressui_gtk.cpp again from
external/onlineupdate/lo.patch. And update external/onlineupdate/README.md how
to manually execute that test against an updater.ini.
Change-Id: I0e3e5e5311be61e1224cda700af2e5d751113a99
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160996
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|
|
...and update to latest Mozilla sources.
Originally, this was a non-external onlineupdate module (plus correspsonding
top-level include/onlineupdate/ directory) that apparently contained sources
originally copied from Mozilla and subsequently modified in-place (plus, mixed
in, presumably some sources that were not copied from Mozilla but were our own
inventions). To clean up this mess, this has been turned into a proper
external/onlineupdate module with a tarball containing the pristine external
Mozilla sources.
The sources for the onlineupdate-c003be8b9727672e7d30972983b375f4c200233f.tar.xz
tarball are taken, somewhat arbitrarily, from a recent
<https://github.com/mozilla/gecko-dev/commit/c003be8b9727672e7d30972983b375f4c200233f>
("Bug 1867784 - Force reflow all kids in the last column balancing reflow.
r=layout-reviewers,dholbert") trunk state, by running
`external/onlineupdate/generate-sources.sh ~/github.com/mozilla/gecko-dev` on a
Fedora 39 machine.
The layout of the tarball still mostly follows the old onlineupdate/ layout,
even if that deviates heavily from the actual source layout at
<https://github.com/mozilla/gecko-dev/>. (And some files, which apparently are
not needed, anyway, lacked sources, see the "Missing source for" in
external/onlineupdate/generate-sources.sh. And win_dirent.h/.cpp has meanwhile
been superseded by updateutils_win.h/.cpp.) Merely newly included source files
are laid out in the tarball according to the actual source layout.
Any LO-specific modifications are made via patch files (rather than modifying
the sources inline, as was done in the past): external/onlineupdate/lo.patch
contains whatever modifications are needed to adapt the functionality, while
external/onlineupdate/gtk3deprecated.patch fixes
> workdir/UnpackedTarball/onlineupdate/onlineupdate/source/update/updater/progressui_gtk.cpp:97:21: error: use of undeclared identifier 'gtk_vbox_new'; did you mean 'gtk_box_new'?
> 97 | GtkWidget* vbox = gtk_vbox_new(TRUE, 6);
> | ^~~~~~~~~~~~
> | gtk_box_new
to not use the deprecated gtk_vbox_new, which is hidden because we include
-DGTK_DISABLE_DEPRECATED in our GTK3_CFLAGS as per our configure.ac.
On Windows, the definition of __BYTE_ORDER__ etc. is needed because
workdir/UnpackedTarball/onlineupdate/include/mozilla/ says "Our supported
compilers provide architecture-independent macros for this", but MSVC doesn't
actually, so define here what would implicitly be defined by GCC. Similarly, on
Windows -U_WIN32_WINNT is needed to undo -D_WIN32_WINNT=0x0601 in
solenv/gbuild/platform/windows.mk, which would cause
> workdir\UnpackedTarball\onlineupdate\include\mozilla/WinHeaderOnlyUtils.h(537): error C2065: 'FILE_ID_INFO': undeclared identifier
etc., despite the #include <windws.h> there.
Curiously, the original gb_CustomTarget_CustomTarget,onlineupdate/generated from
onlineupdate/CustomTarget_generated.mk had to be renamed to
gb_CustomTarget_CustomTarget,external/onlineupdate/generated when the file was
moved to external/onlineupdate/CustomTarget_generated.mk, as otherwise a
top-level `make CustomTarget_onlineupdate/generated` would have failed with "No
rule to make target..." Also, as there is no gb_CustomTarget_use_unpacked, its
effect has been poorly mimicked for now in
external/onlineupdate/CustomTarget_generated.mk.
Similarly, as there is no gb_WinResTarget_use_unpacked, its effect has been
poorly mimicked for now in external/onlineupdate/WinResTarget_updater.mk.
The original onlineupdate/workben/test_dialog.cxx, which is actually code
written by us, has been moved to external/onlineupdate/workben/test_dialog.cxx.
The original onlineupdate/qa/ sources (which were apparently not used during the
build) have been preserved for now as external/onlineupdate/qa/, for
documentation purposes.
The original onlineupdate/astyle.options (which was apparently not used during
the build) has been removed.
Change-Id: I5ea606202e7837269e7b128e45af2f0b8c277f9e
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160492
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@allotropia.de>
|