1 Building Debian Packages with buildah
Building packages in Debian seems to be a solved problem. But is it? At the bottom, installing the
The issue is that
The answer (in Debian and many other Linux distributions) is you do not at all. This is actually a somewhat surprising answer for software developers without a Linux distribution development background1. Instead, chroots "simulate" an environment that has all dependencies that we want to build against at the system locations, that is.
Chroots are basically full system installations in a subdirectory that includes system and application libraries. In order to use them, a package build needs to use the
What if there was an easy way to do all of the above without any process running as root? Enter rootless buildah.
Modern Linux container technologies allow unprivileged users to "untie" a process from the system (cf. the
dpkg-dev package provides all the basic tools needed. Assuming that you already succeeded with creating the necessary packaging metadata (i.e., debian/changelog, debian/control, debian/copyright, etc., and there are great helper tools for this such ash dh-make, dh-make-golang, etc.,) it should be as simple as invoking the dpkg-buildpackage tool. So what's the big deal here?The issue is that
dpkg-buildpackage expects to be called with an appropriately setup build context, that is, it needs to be called in an environment that satisfies all build dependencies on the system. Let's say you are building a package for Debian unstable on your Debian stable system (this is the common scenario for the official Debian build machines), you would need your build to link against libraries in unstable, not stable. So how to tell the package build process where to find its dependencies?The answer (in Debian and many other Linux distributions) is you do not at all. This is actually a somewhat surprising answer for software developers without a Linux distribution development background1. Instead, chroots "simulate" an environment that has all dependencies that we want to build against at the system locations, that is.
/usr/lib, etc.Chroots are basically full system installations in a subdirectory that includes system and application libraries. In order to use them, a package build needs to use the
chroot(2) system call, which is a privileged operation. Also creating these system installations is a somewhat finicky process. In Debian, we have tools that make this process easier, the most popular ones are probably pbuilder(1)2 and sbuild(1)3. Still, they are somewhat clumsy to use, add significant levels of abstraction in the sense that they do quite a bit of magic behind the scenes to work hide the fact that privileged operations (need root to run chroot(2), etc.) are required. They are also somewhat brittle, for instance, if a build process is aborted (SIGKILL, or system crash), you may end up with temporary directories and files under userids other than your own that again may require root-privileges to cleanup.What if there was an easy way to do all of the above without any process running as root? Enter rootless buildah.
Modern Linux container technologies allow unprivileged users to "untie" a process from the system (cf. the
unshare(2) system call). This means that a regular user may run a process (and its child processes) in an "environment" where system calls behave differently and provide a configurable amount of isolation levels. This article demonstrates a novel build tool buildah, which is:- easy to setup build environment from the command line
- secure, as no process needs to run as root
- simple in architecture, requires no running daemon like for example docker
- convenient to use: you can debug your debian/rules interactively
buildah is written in golang and compiled as a (mostly) statically linked executable. It builds on top a number of libraries written in golang, including github.com/containers/storage and github.com/containers/image. The overlay functionality is provided by the fuse-overlayfs(1) utility.1.1 Preparation:
The Kernel in Debian bullseye (and in buster, and recent Ubuntu kernels to the best of my knowledge) do support usernamespaces, but leave them disabled by default. Here is how to enable them:
The command on line 1 and 2 execute the installation of compilers and dpkg development tools such as
This output might look like this:
echo kernel.unprivileged_userns_clone = 1 | sudo tee -a /etc/sysctl.d/containers.conf systemctl -p /etc/sysctl.d/containers.confI have to admit that I'm not entirely sure why the Debian kernels don't enable usernamespaces by default. I've found a reference on stackexchange4 that claims this disables some "hardening" features in the Debian kernel. I also understand this step is not necessary if you chose to compile and run a vanilla upstream kernel. I'd appreciate a better reference and am happy to update this text.
$ c=$(buildah from docker.io/debian:sid) Getting image source signatures Copying blob 2bbc6b8c460d done Copying config 9b90abe801 done Writing manifest to image destination Storing signaturesThis command downloads the image from docker.io and stores it locally in your home directory. Let's install essential utilities for building Debian packages:
1: buildah run $c apt-get update -qq 2: buildah run $c apt-get install dpkg-dev -y 3: buildah config --workingdir /src $c 4: buildah commit $c dpkg-dev
dpkg-buildpackage, etc. The buildah config command in line 4 arranges that whenever you start a shell in the container, the current working directory in the container is in /src. Don't worry about this location not existing yet, we will make sources from your host system available there. The last command creates an OCI image with the name dpkg-dev. BTW, the name you use for the image in the commit command can be used in podman (but not the "containers"). See 5 and 6 for a comparison between podman and buildah.buildah images -a
| REPOSITORY | TAG | IMAGE | ID | CREATED | SIZE | | | | localhost/dpkg-dev | latest | b85c34f95d3e | 16 | seconds | ago | 406 | MB | | docker.io/library/debian | sid | 9b90abe801db | 11 | hours | ago | 124 | MB |
1.2 Running a package build
Now we have a working container with the reference in the variable
$c. To use it conveniently with source packages that I have stored in /srv/scratch/packages/containers, let's introduce a shell alias r like this:alias r='buildah run --tty -v /srv/scratch/packages/containers:/src $c 'This allows you to easily execute commands in that container:
r pwd r bashThe last command will give you an interactive shell that we'll be using for building packages!
siretart@x1:~/scratch/packages/containers/libpod$ r bash root@x1:/src# cd golang-github-openshift-imagebuilder root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps dpkg-checkbuilddeps: error: Unmet build dependencies: debhelper (>= 11) dh-golang golang-any golang-github-containers-storage-dev (>= 1.11) golang-github-docker-docker-dev (>= 18.09.3+dfsg1) golang-github-fsouza-go-dockerclient-dev golang-glog-dev root@x1:/src/golang-github-openshift-imagebuilder# apt-get build-dep -y . Note, using directory '.' to get the build dependencies Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: [...] root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps root@x1:/src/golang-github-openshift-imagebuilder#Now we have a environment that has all build-dependencies available and we are ready to build the package:
root@x1:/src/golang-github-openshift-imagebuilder# dpkg-buildpackage -us -uc -bAssuming the package builds, the package build results are placed in
/src inside the container, and are visible at ~/scratch/packages/containers on the host. There you can inspect, extract or even install them. The latter part allows you to interactively rebuild packages against updated dependencies, without the need of setting up an apt archive or similar.2 Availability
The
Keeping the package at the latest upstream version is challenging because of its fast development pace that picks up new dependencies on golang libraries and new versions of existing libraries with every new upstream release. In general, this requires updating several source packages, and in many cases also uploading new source packages to the Debian archive that need to be processed by the Debian ftp-masters.
As an exercise, I suggest to install the buildah package from bullseye, 'git clone' the packaging repository from https://salsa.debian.org/go-team/packages/golang-github-containers-buildah and build the latest version yourself. Note, I would expect the above to even work on a Fedora laptop.
The Debian packages have not swept into the Ubuntu distributions yet, but I expect them to be included in the Ubuntu 20.10 release. In the mean time, ubuntu users can install the package that are provided by the upstream maintainers in the "Project Atomic" PPA at https://launchpad.net/~projectatomic/+archive/ubuntu/ppa
buildah(1) tool is available in debian/bullseye since 2019-12-17. Since is a golang binary, it only links dynamically against system libraries such as libselinux and libseccomp, which are all available in buster-backports. I'd expect buildah to just work on a debian/buster system as well provided you install those system libraries and possibly the backported Linux kernel.Keeping the package at the latest upstream version is challenging because of its fast development pace that picks up new dependencies on golang libraries and new versions of existing libraries with every new upstream release. In general, this requires updating several source packages, and in many cases also uploading new source packages to the Debian archive that need to be processed by the Debian ftp-masters.
As an exercise, I suggest to install the buildah package from bullseye, 'git clone' the packaging repository from https://salsa.debian.org/go-team/packages/golang-github-containers-buildah and build the latest version yourself. Note, I would expect the above to even work on a Fedora laptop.
The Debian packages have not swept into the Ubuntu distributions yet, but I expect them to be included in the Ubuntu 20.10 release. In the mean time, ubuntu users can install the package that are provided by the upstream maintainers in the "Project Atomic" PPA at https://launchpad.net/~projectatomic/+archive/ubuntu/ppa
3 Related Tools
The buildah tool is accompanied with two "sister" tools:
The Skopeo package provides tooling to work with remote images registries. This allows you download, upload images to remote registries, convert container images between different formats. It has been available in Debian since 2020-04-20.
The podman tool is a 'docker' replacement. It provides a command-line interface that mimics the original
The Skopeo package provides tooling to work with remote images registries. This allows you download, upload images to remote registries, convert container images between different formats. It has been available in Debian since 2020-04-20.
The podman tool is a 'docker' replacement. It provides a command-line interface that mimics the original
docker command to an extend that a user familiar with docker might want to place this in their ~/.bashrc file:alias docker='podman'Unfortunately, at the time of writing podman is still being processed by the ftp-masters since 01-03-2020. At this point, I recommend building the package from our salsa repository at https://salsa.debian.org/debian/libpod
4 Conclusion
Building packages in the right build context is a fairly technical issue for which many tools have been written for. They come with different trade-off when it comes to usability. Containers promise a secure alternative to the tried and proven chroot-based approaches, and the buildah makes using this technology very easy to use.
I'd love to get into a conversation with you on how these tools work for you, and would like to encourage participation and assistance with keeping the complicated software stack up-to-date in Debian (and by extension, in derived distribution such as Ubuntu, etc.).
I'd love to get into a conversation with you on how these tools work for you, and would like to encourage participation and assistance with keeping the complicated software stack up-to-date in Debian (and by extension, in derived distribution such as Ubuntu, etc.).
Footnotes:
1
At my day-job, we build millions of lines of C++ code on Solaris10 and AIX6, where concepts such as "chroots" are restricted to the super user 'root' and is therefore not available to developers, not even through wrappers. Instead, libraries and headers are installed into "refroots", that is, subdirectories that mimic the structure of the "sysroot" directories that are used in the embedded Linux community for cross-compiling packages, and we use Makefiles that set include flags (
-I rules and -L flags) to tell the compiler and linker where to look.
7 comments:
Hi, thanks for your article! I'm still a bit confused. Why would I use buildah instead of `sbuild` with `--chroot-mode=unshare`? Right now I only see disadvantages: buildah sources from docker.io, so there is no trust path from the GPG keyrings I have on my system to the things that end up in my docker container. You also write "Still, they are somewhat clumsy to use" and you are right but by introducing a shell alias you also admit that this command is just too long to type manually all the time. I'm not saying that this is bad -- I just don't yet see an advantage over pbuilder or sbuild. Maybe you could expand more on the use-case that your tool is trying to solve?
On a technical level I'm interested in two things: How does your tool manage to not leave any temporary files behind even when aborted by a sigkill? Secondly, how do you make it such that files visible at `/src` inside the container are at the same time visible in `~/scratch/packages/containers` on the outside? The unshared process will give all files ownerships which make them useless on the outside or are you saying that they are only synchronized after a buildah session finished?
Thanks for the feedback, I'll try to incorporate some clarifications in an later revision of this article. In the mean time, let me address some of your questions directly:
Re: --chroot-mode=unshare: I have to admit that I haven't played with that option yet. According to sbuild(1), it requires a tarball which is unpacked at invocation time. I imagine this tarball takes time to unpack and to create. I claim buildah has significant performance benefits here: You can choose to re-use your build contexts (chroots) by the 'buildah commit' command, and save significant time for your next package build project. This pays off in particular in situations where the software you are building requires several layers of dependencies to build in a specific order (a situation I was running into when packaging buildah and skopeo).
Thanks for pointing out this novel mode to sbuild, I'll definitely try it out. This clearly addresses my security concern. I still see convenience and performance benefits with buildah. Is a similar unshare option available for pbuilder as well?
Re: your question of temporary files on power outage, sigkill, etc.: It doesn't TTBOMK. You can, however, use 'buildah rmi -a' to delete all of your images. See also https://stackoverflow.com/questions/56542220/how-to-reset-podman-and-buildah-after-experimenting-as-a-non-root-user
Re: your question about file visibility: This is what the option "--volume" (short -v) in buildah-run(1) does. Internally, the magic utilizes the "fuse-overlayfs" utility to provide the features of an file system overlays without requiring root privileges. As long as your containers run, files created by the "root" user will look like it is owned by "root". On the outside, the files are actually owned by the invoking user, which is exactly what I would want for building packages.
"I have to admit that I'm not entirely sure why the Debian kernels don't enable usernamespaces by default. "
I don't know if this is the kernel team's reasoning, but some folk disable it to close down some potential exploit vectors. There are some notes from someone who does this here
the URI was eaten by blogger. It's
https://utcc.utoronto.ca/~cks/space/blog/linux/UserNamespacesWhySecurityProblems
Thanks for the article! I'm really glad that podman and friends are getting into Debian. I've switched from schroot and mock (which is like Fedora's schroot) to a podman/toolbox workflow months ago, and don't want to go back. It's less privileged, much easier to maintain (no need to build base tarballs yourself, which is rather difficult with Fedora on the host), and I can do Debian/Ubuntu/Fedora/upstream development stuff with the same tool. I have not tried buildah yet, podman containers already seem to provide everything that's needed -- but I'll have a look at that.
Thanks, very informative.
Small issue:
systemctl -p /etc/sysctl.d/containers.conf
should be:
sudo sysctl -p /etc/sysctl.d/containers.conf
The rational behing disabling unpriviledged user namespaces four years ago is here
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1555321
Post a Comment