Following on from Managing Image mode for RHEL Using Satellite, here are some initial observations running a basic rhel9-bootc server that using Image Mode for RHEL.
Note
This post is not endorsed or affiliated with Red Hat – the information provided is based on experience, documentation and publicly available information. Feel free to leave feedback at the end of this page if anything needs correction.
For an up to date roadmap discussion on Image Mode for RHEL please contact your Red Hat Account rep.
What is Image Mode for Red Hat Enterprise Linux?
Where can I find out more?
- Red Hat Developers: Image mode for Red Hat Enterprise Linux
- Experience the AI-ready OS with image mode for Red Hat Enterprise Linux
- Using image mode for RHEL to build, deploy and manage operating systems
- Image mode for Red Hat Enterprise Linux: A quick start guide
How can I influence the future of Bootable Containers?
- Get Involved with Fedora Bootable Containers
- See the list of ‘What needs work’ at the upstream Bootable Container Images Repo
Can I register the server to the Red Hat Portal?
Yes, subscription-manager should allow this by default (as long as you have the network connectivity) as everything is setup in the rhel9/rhel-bootc image.
[root@bootc ~]# subscription-manager register
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: username
Password:
The system has been registered with ID: 394adef1-055d-4766-9ced-1234
The registered system name is: bootcCan I register the server to my Red Hat Satellite server without adding my certificates?
No, just as with a traditional package mode server, you will need to have installed the Satellite certificates, usually via the katello-ca-consumer-latest.noarch.rpm RPM
[root@bootc ~]# subscription-manager register --serverurl \ satellite.london.example.com:443/rhsm \
--activationkey RHEL9-Dev-Test --org 'Example_Organization'
Error: CA certificate for subscription service has not been installed.If you don’t have them installed, see next step.
How can I add the Red Hat Satellite certificate to the RHEL image mode server?
Ideally, the certificate will be baked into the container image. However, you can do this at runtime. If you followed Managing RHEL Image Mode Using Satellite and converted a traditional package mode server to image mode, you will find the satellite certificate in /sysroot/etc/pki/ca-trust/source/anchors/katello-server-ca.pem If so, you can copy it to the correct location on the running server:
[root@bootc ~]# cp /sysroot/etc/pki/ca-trust/source/anchors/katello-server-ca.pem /etc/pki/ca-trust/source/anchorIf you built the server from scratch, you can insert the certificate contents into /etc/pki/ca-trust/source/anchors/katello-server-ca.pem
After the certificate is in place, run update-ca-trust
[root@bootc ~]# update-ca-trustCan I ‘dnf install’ packages in Image Mode?
No, not by default as the /usr filesystem is read-only. To install an RPM package in image mode, you add the package in your containerfile/dockerfile, rebuild the image, publish it to the registry, perform a bootc upgrade and reboot.
[root@bootc ~]# dnf update
Updating Subscription Management repositories.
Last metadata expiration check: 0:13:57 ago on Wed May 29 09:55:47 2024.
Dependencies resolved.
========================================================================================================
Package Arch Version Repository Size
========================================================================================================
Installing:
kernel x86_64 5.14.0-427.18.1.el9_4 rhel-9-for-x86_64-baseos-rpms 5.7 M
kernel-core x86_64 5.14.0-427.18.1.el9_4 rhel-9-for-x86_64-baseos-rpms 21 M
kernel-modules x86_64 5.14.0-427.18.1.el9_4 rhel-9-for-x86_64-baseos-rpms 39 M
kernel-modules-core x86_64 5.14.0-427.18.1.el9_4 rhel-9-for-x86_64-baseos-rpms 34 M
Upgrading:
container-selinux noarch 3:2.229.0-1.el9_3 rhel-9-for-x86_64-appstream-rpms 58 k
glibc x86_64 2.34-100.el9_4.2 rhel-9-for-x86_64-baseos-rpms 2.0 M
glibc-common x86_64 2.34-100.el9_4.2 rhel-9-for-x86_64-baseos-rpms 313 k
glibc-gconv-extra x86_64 2.34-100.el9_4.2 rhel-9-for-x86_64-baseos-rpms 1.7 M
glibc-minimal-langpack x86_64 2.34-100.el9_4.2 rhel-9-for-x86_64-baseos-rpms 28 k
libxml2 x86_64 2.9.13-6.el9_4 rhel-9-for-x86_64-baseos-rpms 752 k
shim-x64 x86_64 15.8-4.el9_3 rhel-9-for-x86_64-baseos-rpms 476 k
Transaction Summary
========================================================================================================
Install 4 Packages
Upgrade 7 Packages
Total size: 104 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] kernel-5.14.0-427.18.1.el9_4.x86_64.rpm: Already downloaded
[SKIPPED] kernel-core-5.14.0-427.18.1.el9_4.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-5.14.0-427.18.1.el9_4.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-core-5.14.0-427.18.1.el9_4.x86_64.rpm: Already downloaded
[SKIPPED] shim-x64-15.8-4.el9_3.x86_64.rpm: Already downloaded
[SKIPPED] libxml2-2.9.13-6.el9_4.x86_64.rpm: Already downloaded
[SKIPPED] glibc-2.34-100.el9_4.2.x86_64.rpm: Already downloaded
[SKIPPED] glibc-common-2.34-100.el9_4.2.x86_64.rpm: Already downloaded
[SKIPPED] glibc-gconv-extra-2.34-100.el9_4.2.x86_64.rpm: Already downloaded
[SKIPPED] glibc-minimal-langpack-2.34-100.el9_4.2.x86_64.rpm: Already downloaded
[SKIPPED] container-selinux-2.229.0-1.el9_3.noarch.rpm: Already downloaded
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 3.5 MB/s | 3.6 kB 00:00
Importing GPG key 0xFD431D51:
Userid : "Red Hat, Inc. (release key 2) <security@redhat.com>"
Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
error: can't create transaction lock on /usr/share/rpm/.rpm.lock (Read-only file system)
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 3.5 MB/s | 3.6 kB 00:00
Importing GPG key 0xFD431D51:
Userid : "Red Hat, Inc. (release key 2) <security@redhat.com>"
Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
error: can't create transaction lock on /usr/share/rpm/.rpm.lock (Read-only file system)
Key import failed (code 2). Failing package is: kernel-5.14.0-427.18.1.el9_4.x86_64
GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Public key for kernel-core-5.14.0-427.18.1.el9_4.x86_64.rpm is not installed. Failing package is: kernel-core-5.14.0-427.18.1.el9_4.x86_64
GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Public key for kernel-modules-5.14.0-427.18.1.el9_4.x86_64.rpm is not installed. Failing package is: kernel-modules-5.14.0-427.18.1.el9_4.x86_64
GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Public key for kernel-modules-core-5.14.0-427.18.1.el9_4.x86_64.rpm is not installed. Failing package is: kernel-modules-core-5.14.0-427.18.1.el9_4.x86_64
GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
...However, see the next question regarding bootc usroverlay
What is bootc usroverlay?
The command bootc usroverlay is described in Fedora/CentOS bootc documentation – dnf as follows:
Using the
https://docs.fedoraproject.org/en-US/bootc/dnf/bootc usroverlaycommand, a transient writable overlay filesystem is created for/usr, anddnf installwill write to that overlay. This is a useful pattern for installing debugging tools.
This may provide an option for installing a temporary package update, but be aware that any changes made will be lost upon restart. To avoid drift between your running configuration and the image from which the server was last booted, one could argue it is best not to use bootc overlay and instead apply changes to the source contain and redeploy. See also Fedora bootc issue tracker – Local package layering story with bootc & dnf5
Can I make updates in /etc? Do these updates persist?
Yes, you can and yes they will persist a reboot
[root@bootc ~]# echo hello > /etc/welcome
[root@bootc ~]# systemctl reboot
...
[root@bootc ~]# cat /etc/welcome
helloWhen you say ‘image’, what do you mean?
Just as the term ‘server’ can have many different contexts (a process, an application, a physical machine, a cloud machine, etc) the term ‘image’ can be used in many different ways.
- Container Image. An unchangeable, static file that includes executable code. When running
podman buildyou are building a container image. - Disk Image. A snapshot of a storage device’s structure and data – typically stored in one more more files. When you run bootc-image-builder you are creating a disk image, usually an ISO, QCOW2, AMI or VMDK file.
How can I search for container images when I don’t have my custom CA Certificate in the running container?
If you are using the base rhel9/rhel-bootc image without any customisation, you won’t have any custom CA certificates installed. For example, this means you might not be able to use a custom registry, such as the one on your Satellite server.
[root@bootc ~]# podman search satellite.london.example.com/
Error: 1 error occurred:
* couldn't search registry "satellite.london.example.com": pinging container registry satellite.london.example.com: Get "https://satellite.london.example.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
You can use the --tls-verify=false to work around this (as an alternative to adding the katello-server-ca certificate) :
[root@bootc ~]# podman search satellite.london.example.com/
Error: 1 error occurred:
* couldn't search registry "satellite.london.example.com": pinging container registry satellite.london.example.com: Get "https://satellite.london.example.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
After a new image is available in the registry, how do I update?
Use the bootc upgrade or bootc switch to move to a new container image at the next boot.
[root@bootc-dev ~]# bootc switch satellite.london.example.com/example_organization-dev-test-rhel9-with-products-my-product-web-app:latest
layers already present: 66; layers needed: 10 (75.0 MB)
304 B [████████████████████] (0s) Fetched layer sha256:564ce5a0dd50 Loading usr/lib/ostree/prepare-root.conf
Queued for next boot: satellite.london.example.com/example_organization-dev-test-rhel9-with-products-my-product-web-app:latest
Version: 9.20240501.0
Digest: sha256:9e2d164ae5d6f67d1817a45adaeafb45f4edebe8b15e685ce696a76e97cdb971Perform a reboot and the new image will be take affect.
Certificate errors when performing a switch or upgrade
When performing a switch or upgrade, you’ll need to be able to connect securely to the registry. If the running image does trust the registry you may get an error such as:
[root@bootc ~]# bootc switch satellite.london.example.com/example_organization-dev-test-rhel9-with-products-my-product-web-app:latest
ERROR Switching: Pulling: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: pinging container registry satellite.london.example.com: Get "https://satellite.london.example.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
Add the CA certificate for the registry and the run update-ca-trust. Example
[root@bootc ~]# cp /sysroot/etc/pki/ca-trust/source/anchors/katello-server-ca.pem /etc/pki/ca-trust/source/anchors
[root@bootc ~]# update-ca-trustCan I make updates in /var? Do these updates persist?
Yes, these updates persist but remember that they are machine specific – ie servera.example.com and serverb.example.com will have different contents in /var.
[root@bootc ~]# echo "Running from base rhel9/rhel-bootc image" > /var/log/which_container
# Switch to a new container
[root@bootc ~]# bootc switch satellite.london.example.com/example_organization-dev-test-rhel9-with-products-my-product-web-app:latest
layers already present: 66; layers needed: 10 (75.0 MB)
304 B [████████████████████] (0s) Fetched layer sha256:564ce5a0dd50 Loading usr/lib/ostree/prepare-root.conf
Queued for next boot: satellite.london.example.com/example_organization-dev-test-rhel9-with-products-my-product-web-app:latest
Version: 9.20240501.0
Digest: sha256:9e2d164ae5d6f67d1817a45adaeafb45f4edebe8b15e685ce696a76e97cdb971
Perform a reboot, and then check the contents in /var – the contents should be persistent.
[root@bootc ~]# cat /var/log/which_container
Running from base rhel9/rhel-bootc imageHow do I use Red Hat Insights in RHEL Image Mode?
Install the insights-client package into your custom image, and set it to connect to Red Hat Insights (via Satellite or directly) on boot. See Build and distribute custom Image Mode for RHEL Containers for an example Containerfile which registers a server to Insights and Managing Image Mode for RHEL with Satellite and Insights for an overview of what you can see in Insights.
How do I manage compliance with RHEL Image Mode?
Install the openscap packages into your custom image (see Build and distribute custom Image Mode for RHEL Containers) and then manage compliance as you would with package mode RHEL (see Managing Image Mode for RHEL with Satellite and Insights).
Should I use SSH and Ansible to manage my Image Mode for RHEL Image server estate?
As with containers running in Kubernetes clusters or running on standalone hosts via podman or docker, one school of thought is that you never log in to a container, and instead rely on monitoring and logging to confirm the health of a server. If a configuration change or software update is needed, the container should be rebuilt and the update image deployed. However, with Image Mode for RHEL you can get the best of both worlds as you can treat some aspects of the Operating System as you do with a traditional one. For example, with SSH configured you are free to login, review logfiles, start and stop services and run (if they are installed) debugging tools such as strace and ss. These are tools that traditional system administrators are familiar with. With Ansible, you can automate tasks such as starting and stopping services or issuing bootc commands to upgrade the Operating System. They key difference with Image Mode is that changes to the running system will ideally be performed by deploying a new image. So logging in via SSH should only be for querying the current state or initiating a boot upgrade/switch, rather than configuration changes. Indeed, the read-only nature of the core operating system filesystems ensures that most changes require an updated image.
What are some of the advantages of Image Mode for RHEL?
Personal opinion – one of the nicest things about Image Mode for RHEL is the ability to fully codify the configuration of the Operating System and application(s) in one place – the Containerfile. This helps bring Developers, Security and Operations much closer together because everything is configured in a central location. The rollback feature of Image Mode is especially nice as the process is very quick and performed by one tool, rather than relying on LVM or virtual machine snapshots or backups of physical servers. Existing tools (monitoring, inventory management, Ansible Automation, etc) can for the most part, be used with Image Mode. Linux administrators should find learning and implementing Image mode for RHEL has a shallow learning curve compared to full container platforms like Kubernetes, whilst immediately getting the benefits that come with container-native tools and techniques.
Automated build and deploy pipelines (CI/CD) can be performed on the combined O/S and Application stack.
Are other bootc images available?
Yes, the above looks at Image Mode for RHEL but other containers are available by Fedora and CentOS. Here are the links for all three:
Where can I find sample Containerfiles?
Sample Containerfiles that use bootc are available here: