dnsdist: Generate our packages with `meson` #15184

rgacogne · 2025-02-20T14:50:17Z

Short description

This PR updates the build scripts to build our DNSdist >= 2.0 packages with meson instead of autotools. There are some rather big changes:

meson sub-directories that are shared with the authoritative server are no longer symbolic links because the meson directory at the root of the repository is not available when meson runs, causing issues. I think we could go for a different approach and make the meson available to the Docker environment instead
h2o support is removed from our packages
some features could not be properly disabled when building with meson
a recent enough version of meson is not available in most distributions, so I'm installing it from a release tarball. This is a bit messy to do without conflicting with the meson version packaged by the distribution itself
recent versions of meson require a recent enough version of Python, so I had to upgrade Python as well but fortunately there is a recent enough version in each distribution we support

I have a remaining issue that I don't know how to fix. Currently we set the correct version of DNSdist in configure.ac via the set-configure-ac-version.sh script. Unfortunately doing so creates an uncommitted change that meson refuses to pick up (Repository has uncommitted changes that will not be included in the dist tarball). We can use --allow-dirty so that meson keeps running but the resulting tarball does not have the correct version set in configure.ac, unsurprisingly. @omoerbeek @eli-schwartz any idea how to handle this properly?

DO NOT MERGE THIS!

Checklist

I have:

read the CONTRIBUTING.md document
compiled this code
tested this code
included documentation (including possible behaviour changes)
documented the code
added or modified regression test(s)
added or modified unit test(s)

eli-schwartz · 2025-02-20T15:17:55Z

I have a remaining issue that I don't know how to fix. Currently we set the correct version of DNSdist in configure.ac via the set-configure-ac-version.sh script. Unfortunately doing so creates an uncommitted change that meson refuses to pick up (Repository has uncommitted changes that will not be included in the dist tarball). We can use --allow-dirty so that meson keeps running but the resulting tarball does not have the correct version set in configure.ac, unsurprisingly. @omoerbeek @eli-schwartz any idea how to handle this properly?

Meson expects dist tarball changes to be applied to the "pure" sources in $MESON_DIST_ROOT when running meson.add_dist_script(), which is processed after snapshotting the current project tree into a temporary staging directory.

The snapshot is created via git archive which is why uncommitted changes aren't picked up (and also why it doesn't pick up builddir-debug/ or meson.build.bak / meson.build~ or __pycache__/ directories etc.). This is why meson is able to get away with not utilizing EXTRA_DIST = like Makefile.am requires.

coveralls · 2025-02-20T15:33:24Z

Pull Request Test Coverage Report for Build 13700093662

Details

1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
29 unchanged lines in 7 files lost coverage.
Overall coverage increased (+0.02%) to 64.49%

Files with Coverage Reduction	New Missed Lines	%
modules/gpgsqlbackend/gpgsqlbackend.cc	1	88.62%
pdns/recursordist/recursor_cache.cc	1	84.22%
pdns/misc.cc	2	62.24%
pdns/recursordist/test-syncres_cc2.cc	3	88.85%
pdns/signingpipe.cc	5	85.25%
pdns/recursordist/test-syncres_cc1.cc	8	89.68%
pdns/packethandler.cc	9	72.84%

Totals
Change from base Build 13694816685:	0.02%
Covered Lines:	127764
Relevant Lines:	167100

💛 - Coveralls

rgacogne · 2025-02-20T16:34:27Z

Thanks, Eli! I think I have something working now, test run in progress (will fail for el-7 and buster): https://github.com/rgacogne/pdns/actions/runs/13439966426

omoerbeek · 2025-02-25T13:16:50Z

Some thoughts:

For auth a specific version of meson is installed through pip, see build-and-test-all.yml. I have no particular preference, but I dislike having multiple install methods used by various scripts. So we have to decide what methods suits us best and make sure we are consistently using that method.
It is not clear to me why the scripts in the meson subdir should be copies instead of symlinks. meson dist makes sure symlinks end up into the dist tarball as files through meson-dist-script.sh . At which step of the package build process are the files needed but not available if you use symlinks in the meson subdir?
The change in the tarball name is unexpected and probably unneeded. You can use meson dist --formats bztar to get a .tar.bz2 dist file. Adding: I tried this but is seems the meson version in alpine (1.6.1) used to create the dist file has no bztar support. So we might want start to use our own installed meson for this step as well.

omoerbeek · 2025-02-25T13:17:32Z

builder-support/dockerfiles/Dockerfile.debbuild-prepare

 # create copy of source tarball with name that dpkg-source requires
-RUN cp /sdist/dnsdist-${BUILDER_VERSION}.tar.bz2 dnsdist_${BUILDER_VERSION}.orig.tar.bz2
+RUN cp /sdist/dnsdist-${BUILDER_VERSION}.tar.xz dnsdist_${BUILDER_VERSION}.orig.tar.zx


xz and zx ? but also see general comment wrt tarball name

Oops, that's indeed a typo. I'll fix for now, pending a decision on the actual tarball format.

eli-schwartz · 2025-02-25T14:30:08Z

The change in the tarball name is unexpected and probably unneeded. You can use meson dist --formats bztar to get a .tar.bz2 dist file. Adding: I tried this but is seems the meson version in alpine (1.6.1) used to create the dist file has no bztar support. So we might want start to use our own installed meson for this step as well.

Since meson 1.5.0, bztar has been a valid meson dist option: https://mesonbuild.com/Release-notes-for-1-5-0.html#support-for-bztar-in-meson-dist

CPython itself can be compiled without bzip2 support, although it seems odd to do so and most distributions should already have it in the base system...

zeha · 2025-02-25T14:31:03Z

Personally I'd welcome the tarballs changing to xz.

rgacogne · 2025-03-03T11:11:48Z

* For auth a specific version of meson is installed through `pip`, see `build-and-test-all.yml`. I have no particular preference, but I dislike having multiple install methods used by various scripts. So we have to decide what methods suits us best and make sure we are consistently using that method.

I agree we should unify the way we are installing meson. I went with the current approach of installing from a release tarball because it makes me feel more confident about reproducibility than using pip, which even with pinned versions seems to me to have more moving parts, but also because RPM-based distributions need the data/macros.meson file that did not seem to be installed via pip.

* It is not clear to me why the scripts in the meson subdir should be copies instead of symlinks. `meson dist` makes sure symlinks end up into the dist tarball as files through `meson-dist-script.sh `. At which step of the package build process are the files needed but not available if you use symlinks in the `meson` subdir?

It was initially needed because the Docker builder only included a partial view of the repository (pdns/dnsdistdist) so the symlinks were broken. Later in the process I realized that meson refuses to work on a partial view of the repository (rightfully claiming the repository is dirty) so I had to include the whole git repository in the Docker builder image anyway. I'll check whether I can revert the symlinks changes.

* The change in the tarball name is unexpected and probably unneeded. You can use `meson dist --formats bztar` to get a `.tar.bz2` dist file. Adding: I tried this but is seems the meson version in alpine (1.6.1) used to create the dist file has no bztar support. So we might want start to use our own installed meson for this step as well.

I personally would prefer to generate tar.xz archives rather than bz2 ones, and I would not mind keeping the alpine-provided meson for the dist step if possible, but I'll do what is needed if we prefer to keep tar.bz2 archives instead. @Habbie do you have a preference?

rgacogne · 2025-03-03T13:36:45Z

The symbolic links are back whenever possible.

builder-support/helpers/install_meson.sh

builder-support/dockerfiles/Dockerfile.dnsdist

omoerbeek · 2025-03-04T14:24:56Z

When building for debian-bookworm I see two packaging warnings:

dpkg-shlibdeps: warning: cannot find library /usr/lib/libdnsdist-quiche.so needed by debian/dnsdist/usr/bin/dnsdist (ELF format: 'elf64-x86-64' abi: '0201003e00000000'; RPATH: '')
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/dnsdist/usr/bin/dnsdist was not linked against libnetsnmpmibs.so.40 (it uses none of the library's symbols)

The first is a puzzling, the meson setup step does report:Run-time dependency quiche found: YES 0.23.2
I wanted to upload the build log, but now my test system's disk is dying.

rgacogne · 2025-03-04T14:41:20Z

I remember seeing the first warning and checking that the library was present in the deb (it was), the second does not ring a bell.

omoerbeek · 2025-03-04T14:42:53Z

I remember seeing the first warning and checking that the library was present in the deb (it was), the second does not ring a bell.

I suppose the 2nd is that the netsnmp pkconfig file lists that lib, but the code is not actually using it.

omoerbeek · 2025-03-05T12:37:06Z

I tried to build and install a debian-trixy package and that went fine:

$ dnsdist --version
dnsdist 9.9.100 (Lua 5.1.4 [LuaJIT 2.1.1737090214])
Enabled features: AF_XDP cdb dns-over-quic dns-over-http3 dns-over-tls(gnutls openssl) dns-over-https(nghttp2) dnscrypt ebpf fstrm ipcipher libedit libsodium lmdb protobuf re2 recvmmsg/sendmmsg snmp systemd

For rhel-9, the package built and installed without issues, but:

$ ls -l /usr/bin/dnsdist 
-rwxr-xr-x. 1 root root 13384208 Mar  5 12:24 /usr/bin/dnsdist
$ /usr/bin/dnsdist --version
/usr/bin/dnsdist: error while loading shared libraries: /usr/lib/libdnsdist-quiche.so: cannot open shared object file: No such file or directory
$ ldd /usr/bin/dnsdist 
	linux-vdso.so.1 (0x00007ffe28fe9000)
	libluajit-5.1.so.2 => /lib64/libluajit-5.1.so.2 (0x00007f4b61d6e000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f4b61c91000)
	libcdb.so.1 => /lib64/libcdb.so.1 (0x00007f4b62aff000)
	libfstrm.so.0 => /lib64/libfstrm.so.0 (0x00007f4b62af1000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f4b62ae7000)
	libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f4b61600000)
	libgnutls.so.30 => /lib64/libgnutls.so.30 (0x00007f4b61200000)
	libedit.so.0 => /lib64/libedit.so.0 (0x00007f4b62aa8000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f4b61c67000)
	/usr/lib/libdnsdist-quiche.so => not found
	libnetsnmpmibs.so.40 => /lib64/libnetsnmpmibs.so.40 (0x00007f4b60e00000)
	libnetsnmpagent.so.40 => /lib64/libnetsnmpagent.so.40 (0x00007f4b61bed000)
	libnetsnmp.so.40 => /lib64/libnetsnmp.so.40 (0x00007f4b61517000)
	libsodium.so.23 => /lib64/libsodium.so.23 (0x00007f4b61b94000)
	libssl.so.3 => /lib64/libssl.so.3 (0x00007f4b6111a000)
	liblmdb.so.0.0.0 => /lib64/liblmdb.so.0.0.0 (0x00007f4b61b7c000)
	libre2.so.9 => /lib64/libre2.so.9 (0x00007f4b614a3000)
	libbpf.so.1 => /lib64/libbpf.so.1 (0x00007f4b61b19000)
	libxdp.so.1 => /lib64/libxdp.so.1 (0x00007f4b6148b000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f4b60a00000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f4b60d25000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4b61470000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4b60600000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4b62b0d000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f4b608c4000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f4b61444000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f4b60c4e000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f4b610f6000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f4b610dc000)
	libp11-kit.so.0 => /lib64/libp11-kit.so.0 (0x00007f4b60469000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f4b60c2d000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f4b602e4000)
	libtasn1.so.6 => /lib64/libtasn1.so.6 (0x00007f4b608ac000)
	libnettle.so.8 => /lib64/libnettle.so.8 (0x00007f4b60855000)
	libhogweed.so.6 => /lib64/libhogweed.so.6 (0x00007f4b6024c000)
	libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f4b60825000)
	libsensors.so.4 => /lib64/libsensors.so.4 (0x00007f4b60814000)
	librpm.so.9 => /lib64/librpm.so.9 (0x00007f4b601c9000)
	librpmio.so.9 => /lib64/librpmio.so.9 (0x00007f4b6018f000)
	libperl.so.5.32 => /lib64/libperl.so.5.32 (0x00007f4b5fc00000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4b6017b000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f4b60141000)
	libelf.so.1 => /lib64/libelf.so.1 (0x00007f4b60126000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f4b60100000)
	libffi.so.8 => /lib64/libffi.so.8 (0x00007f4b61438000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f4b600ed000)
	liblua-5.4.so => /lib64/liblua-5.4.so (0x00007f4b600a7000)
	libpopt.so.0 => /lib64/libpopt.so.0 (0x00007f4b60098000)
	libacl.so.1 => /lib64/libacl.so.1 (0x00007f4b60809000)
	libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f4b5fac8000)
	libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f4b60064000)
	libattr.so.1 => /lib64/libattr.so.1 (0x00007f4b6005c000)
	libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f4b60053000)
$

zeha · 2025-03-05T17:21:38Z

When building for debian-bookworm I see two packaging warnings:

dpkg-shlibdeps: warning: cannot find library /usr/lib/libdnsdist-quiche.so needed by debian/dnsdist/usr/bin/dnsdist (ELF format: 'elf64-x86-64' abi: '0201003e00000000'; RPATH: '')

I think this is harmless.

It would be nicer to have the lib in a private directory, and not in the system dyld path. But meh?

> dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/dnsdist/usr/bin/dnsdist was not linked against libnetsnmpmibs.so.40 (it uses none of the library's symbols)

This is weird. In the last debian.org autoconf-based build I've found this weirdness: -Wl,--push-state -Wl,--as-needed -Wl,--pop-state ; maybe it causes the linker to drop -lnetsnmpmibs.

zeha · 2025-03-05T18:49:14Z

> dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/dnsdist/usr/bin/dnsdist was not linked against libnetsnmpmibs.so.40 (it uses none of the library's symbols)

I see now you've added -Wl,--no-as-needed to LDFLAGS. Probably that's why.

Otherwise our builder does not find it, because the root directory of the repository is not available to the Docker context.

…r` for `meson`

After all we are building right after.

rgacogne · 2025-03-06T09:40:53Z

I'm looking at the EL-9 issue right now. So far I see that the library is present in the package but perhaps in the wrong path: usr/lib64/libdnsdist-quiche.so

rgacogne · 2025-03-06T09:47:06Z

I see now you've added -Wl,--no-as-needed to LDFLAGS. Probably that's why.

It looks like it's not needed on the versions of Debian / Ubuntu we support, I'll get rid of it.

rgacogne · 2025-03-06T10:10:55Z

I'm looking at the EL-9 issue right now. So far I see that the library is present in the package but perhaps in the wrong path: usr/lib64/libdnsdist-quiche.so

We are installing in the right path, but LLD stores the full path to the shared library while linking, which is a problem because we initially install it in /usr/lib/. I guess I'll to install it to the correct path in install_quiche.sh, so perhaps it's the time to use a private directory as suggested by zeha.

rgacogne · 2025-03-06T11:00:10Z

I pushed new commits getting rid of -Wl,--no-as-needed and fixing the libdnsdist-quiche.so installation path on RHEL and friends.

rgacogne · 2025-03-06T11:06:11Z

Ah, except my idea of checking whether /usr/lib64 exists does not work, because it does exist on Debian even though it's not the correct libdir...

rgacogne · 2025-03-06T13:26:51Z

I'm testing a new, somewhat ugly, approach.

rgacogne added the dnsdist label Feb 20, 2025

rgacogne requested a review from omoerbeek February 21, 2025 11:05

omoerbeek reviewed Feb 25, 2025

View reviewed changes

rgacogne force-pushed the ddist-packages-with-meson branch from 1735e2a to a07052e Compare March 3, 2025 11:49

rgacogne marked this pull request as ready for review March 3, 2025 13:36

rgacogne requested a review from omoerbeek March 3, 2025 16:04

omoerbeek reviewed Mar 4, 2025

View reviewed changes

builder-support/helpers/install_meson.sh Outdated Show resolved Hide resolved

builder-support/dockerfiles/Dockerfile.dnsdist Outdated Show resolved Hide resolved

rgacogne force-pushed the ddist-packages-with-meson branch from a07052e to 73c8dfb Compare March 4, 2025 14:35

rgacogne added 8 commits March 6, 2025 10:24

dnsdist: Initial changes to build our DNSdist packages with meson

f5449b6

dnsdist: Fix detection of libcrypto's configured value in meson

08e1d64

dnsdist: Only compile lmdb-safe if LMDB has been found

6e44109

dnsdist: Only compile ipcipher if libcrypto has been found

88e7dad

dnsdist: Correct meson tarball sha256sum

db5342a

dnsdist: Use python3 when installing meson

a2c7fc7

dnsdist: Fix "unused parameter" warnings in some configurations

cbb071f

dnsdist: Copy the NOTICE file instead of using a link

0b5a03a

Otherwise our builder does not find it, because the root directory of the repository is not available to the Docker context.

rgacogne added 12 commits March 6, 2025 10:24

dnsdist: Fix the meson tarball hash

2d57960

dnsdist: Remove h2o from our packages

8850604

dnsdist: Remove time_t-related checks, we do not care

86a657c

dnsdist: Install already generated man pages

4eb9cb3

dnsdist: Fix package building on EL-9 and EL-8

eb228b1

dnsdist: Fix building Debian and Ubuntu packages with meson

ab5f007

dnsdist: Run set-configure-ac-version.sh in meson's dist script

dfccdb3

dnsdist: Provide a clean and complete git content available to `Docke…

1fd1610

…r` for `meson`

Remove files versioned in git from dockerignore of meson is unhappy

628932d

dnsdist: Fix .zx -> .xz typo

46d351f

dnsdist: Disable the built-in compile check of meson dist

78e17e6

After all we are building right after.

install_meson: Actually output two spaces for sha256sum

b648b14

rgacogne added 2 commits March 6, 2025 11:49

dnsdist: Test building packages without -Wl,--no-as-needed

9e344d8

dnsdist: Install the Quiche library in the correct library directory

660b53c

rgacogne force-pushed the ddist-packages-with-meson branch from 5c4a9bf to 660b53c Compare March 6, 2025 10:59

rgacogne added 3 commits March 6, 2025 14:15

dnsdist: Another attempt at finding the correct libdir

baa537f

dnsdist: Use the correct libdir for Quiche on Debian/Ubuntu

111b719

dnsdist: Update tasks.py for the new Quiche library installation path

6b7e7b2

dnsdist: Our pkgconfig file is also in the new library directory

345e062

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dnsdist: Generate our packages with `meson` #15184

dnsdist: Generate our packages with `meson` #15184

rgacogne commented Feb 20, 2025

eli-schwartz commented Feb 20, 2025

coveralls commented Feb 20, 2025 •

edited

Loading

rgacogne commented Feb 20, 2025

omoerbeek commented Feb 25, 2025 •

edited

Loading

omoerbeek Feb 25, 2025

rgacogne Mar 3, 2025

eli-schwartz commented Feb 25, 2025 •

edited

Loading

zeha commented Feb 25, 2025

rgacogne commented Mar 3, 2025

rgacogne commented Mar 3, 2025

omoerbeek commented Mar 4, 2025 •

edited

Loading

rgacogne commented Mar 4, 2025

omoerbeek commented Mar 4, 2025 •

edited

Loading

omoerbeek commented Mar 5, 2025 •

edited

Loading

zeha commented Mar 5, 2025

zeha commented Mar 5, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

dnsdist: Generate our packages with meson #15184

Are you sure you want to change the base?

dnsdist: Generate our packages with meson #15184

Conversation

rgacogne commented Feb 20, 2025

Short description

Checklist

eli-schwartz commented Feb 20, 2025

coveralls commented Feb 20, 2025 • edited Loading

Pull Request Test Coverage Report for Build 13700093662

Details

💛 - Coveralls

rgacogne commented Feb 20, 2025

omoerbeek commented Feb 25, 2025 • edited Loading

omoerbeek Feb 25, 2025

Choose a reason for hiding this comment

rgacogne Mar 3, 2025

Choose a reason for hiding this comment

eli-schwartz commented Feb 25, 2025 • edited Loading

zeha commented Feb 25, 2025

rgacogne commented Mar 3, 2025

rgacogne commented Mar 3, 2025

omoerbeek commented Mar 4, 2025 • edited Loading

rgacogne commented Mar 4, 2025

omoerbeek commented Mar 4, 2025 • edited Loading

omoerbeek commented Mar 5, 2025 • edited Loading

zeha commented Mar 5, 2025

zeha commented Mar 5, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

rgacogne commented Mar 6, 2025

dnsdist: Generate our packages with `meson` #15184

dnsdist: Generate our packages with `meson` #15184

coveralls commented Feb 20, 2025 •

edited

Loading

omoerbeek commented Feb 25, 2025 •

edited

Loading

eli-schwartz commented Feb 25, 2025 •

edited

Loading

omoerbeek commented Mar 4, 2025 •

edited

Loading

omoerbeek commented Mar 4, 2025 •

edited

Loading

omoerbeek commented Mar 5, 2025 •

edited

Loading