Podman

Podman - Auto Updates

Containers made it very easy to package and run applications on different Linux based platforms. Building and running a container can be done in seconds and is easy to reproduce. But what about updating the running containers on a regular basis?

Daniel Schier

Jul 1, 2021 — 7 min read

This article explains how you can run containers with Podman and also keep them updated without any manual step.

Why (not) Auto Update

Updating a Linux based system can be done with a few commands or even automatically. For example, if you want to update a CentOS based system, you can execute the below commands, and you are good to go.

# Update and reboot
$ sudo dnf update -y
$ sudo reboot

(Un)fortunately, running containers will not be updated with the above commands. Instead, you have to update each an every container separately. For example, updating a WordPress setup can look like this.

# Update and restart a WordPress container setup
$ podman image pull docker.io/library/wordpress:TAG
$ podman image pull docker.ui/library/mariadb:TAG

$ podman container rm -f myWordpressApp
$ podman container rm -f myWordpressDb

$ podman container run MANYOPTIONS docker.ui/library/mariadb:TAG
$ podman container run MANYOPTIONS docker.ui/library/wordpress:TAG

Docker Compose makes it a bit easier, but even then, you have to do it for each and every stack. For a production environment, where you really want to use specific image versions and full control or use some automation (ex. Ansible), this seems ok. For an environment, where you tested every setup anyway, or you don't care if something may be broken, and you are able to recover on your own, it's just tedious work.

Podman provides a very easy way to keep your container setups updated automatically or On-demand. Let's see how this works.

Hint

The guide is tested on Fedora 34 with Podman 3.2.1.

Podman and Systemd

The first thing, you need to know about the auto-update capabilities: It only works with systemd managed containers. I have already written an article about this here, so I will stick to the additions to make auto-updates working. Here is a fast-forward example to generate a (rootfull) container (httpd), which is managed via systemd.

# Run container and publish port 80
$ sudo podman container run --detach --tty --name web01 --publish 80:80/tcp docker.io/library/httpd:latest

# Test
$ curl localhost:80
<html><body><h1>It works!</h1></body></html>

# Generate systemd file and enable it
$ sudo podman generate systemd --name --new --files web01
/home/dschier/container-web01.service

$ sudo podman container rm -f web01

$ sudo cp /home/dschier/container-web01.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now container-web01

# Test again
$ curl localhost:80
<html><body><h1>It works!</h1></body></html>

The generated file container-web01.service will look something like the below example. I just added some newlines to make it better readable.

# container-web01.service
# autogenerated by Podman 3.2.1
# Thu Jul  1 09:17:00 CEST 2021

[Unit]
Description=Podman container-web01.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/container-web01.pid %t/container-web01.ctr-id

ExecStart=/usr/bin/podman run \
          --conmon-pidfile %t/container-web01.pid \
          --cidfile %t/container-web01.ctr-id \
          --cgroups=no-conmon \
          --replace \
          --detach \
          --tty \
          --name web01 \
          --publish 80:80/tcp \
          docker.io/library/httpd:latest

ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-web01.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-web01.ctr-id
PIDFile=%t/container-web01.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

container-web01.service

Update Containers

Now, that we are having a systemd managed container, we do have 3 easy options to update the containers.

Manual Update

You can update a container with only two commands.

# Update the image and restart a container
$ sudo podman pull docker.io/library/httpd:latest
$ sudo systemctl restart container-web01

This is very useful, if you want to keep control over the update process itself and update only a single container. Be aware, that all containers, using this image will be updated with the next restart, too.

Auto-Update on demand

A more automated way can be achieved, if you tweak the systemd unit file very slightly. In fact, you only have to add a label to the container. The below example shows the added line --label "io.containers.autoupdate=registry". This will ensure Podman is checking the fully qualified image path for new images and download them.

...
ExecStart=/usr/bin/podman run \
          --conmon-pidfile %t/container-web01.pid \
          --cidfile %t/container-web01.ctr-id \
          --cgroups=no-conmon \
          --replace \
          --detach \
          --tty \
          --name web01 \
          --publish 80:80/tcp \
          --label "io.containers.autoupdate=registry" \
          docker.io/library/httpd:latest
 ...

container-web01.service

After editing your unit file, you need to make systemd aware of the change and also restart the container service.

# Reload systemd unit files and restart the container
$ sudo systemctl daemon-reload
$ sudo systemctl restart container-web01.service

Now comes the fun part. Podman provides a new sub-command auto-update. This command will look for containers that are having the label and a systemd service file as described above. If the command finds one container, it will check for a new image, download it, restart the container service. So, just run the below command like you are running dnf update.

$ sudo podman auto-update

Be aware, that this only updates the services for the user executing it. If you are running it with privileges (root/sudo), it will update only rootfull containers. Same, for a user executing the command.

Auto-Update on a schedule

There is even more to it. In smaller environments, you may want to "don't care" about updating your containers. You can enable a systemd timer, which is taking care of running podman auto-update on a schedule. Let me show you, how to run automatic container updates every Monday morning for all rootfull containers.

First, you need to manage the container with systemd, as described above. In addition, you need to enable the podman-auto-update.timer. This can be done with the below command:

# Enable the auto update timer
$ sudo systemctl enable --now podman-auto-update.timer

You can have a short look at the timer via systemctl cat podman-auto-update.timer.

# /usr/lib/systemd/system/podman-auto-update.timer
[Unit]
Description=Podman auto-update timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

podman-auto-update.timer

As you can see, the timer will run every day. I want to adjust this to every Monday morning. Changing system default unit files is really easy and can be done with the below command.

# Edit the timer file
$ sudo systemctl edit podman-auto-update.timer

This will open your default editor and provide the original file as comments at the end. You just need to edit your changes, not provide a new file. In my case, I will edit the file with the below content.

[Timer]
OnCalendar=Mon *-*-* 05:00
Persistent=true

This will ensure, that the auto-update is running every Monday at ~05:00 in the morning. You can use other expressions, depending on your needs, as described in the man pages.

# Look at the manpage for systemd timers
$ man systemd.timer

Best Practices

Just blindly automating image updates and restarting containers may lead to unexpected behavior and issues. Here are some best practices, I want to provide to you. This is just my personal experience, so don't take them as industry standard.

only use the label for uncritical workloads, container images can be broken, too
don't use images with the latest tag, instead choose a versioned tag like 1, 1.1 or similar, if possible
check, if you can really restart every image automatically or if these depend on each other and may crash, if something gets lost
do backups of your volumes on a regular basis
introduce some kind of monitoring, that informs you, if a container does not come up automatically
be aware, that all containers using a specific image/tag will be updated with the next restart of the container

Furthermore, using automatic updates in a critical production environment should be avoided entirely. In most cases, you want to know when something is deployed and have full control over the deployment process.

Docs & Links

You can find a couple of articles, documentation and even videos about systemd timers or Podman auto-updates in the web.

podman-auto-update(1) — Podman documentation

Podman

Conclusion

With Podman auto-updates, you can simplify your maintenance work for many container workloads. Applying patches regularly and keep up-to-date in small environments is very easy.

You don't even need to deploy any additional workload to take care of these updates. The very good systemd support in Podman is doing the heavy lifting for you.

It would be nice to know if you consider to have auto-updates for your containers.