In the previous post – Creating Image Mode for RHEL Servers – we built and registered our development, stage and production servers to Red Hat Satellite and Insights. Let’s see how we can use these tools.
Satellite
We see that our three Image Mode for RHEL servers are all correctly registered with Red Hat Satellite and are in the correct Lifecycle.
If we look carefully, we notice that Satellite reports that the development server has a number of installable updates, whereas the production and stage servers have none. We’ve deployed from the same RHEL Image, so why would this be? Well, we have published some updates to the RHEL9-with-products Content View and begun to test them on our traditional package mode RHEL instances. The Lifecycle goes Dev-Test -> Stage -> Prod-DR. We’ve not yet pushed those updates to Stage or Prod-DR.
That explains the difference, but can we actually install those updates? No – as discussed in the Image Mode for RHEL FAQ by default we can’t actually dnf update the packages. The correct course of action is to update our container, synchronise it to Satellite, promote it through the Lifecycle and let bootc perform the upgrade (either automatically or manually).
Having the servers registered in Satellite provides administrators with a useful overview of their Image Mode and Package Mode RHEL installations. We can also setup remote execution from Satellite to connect to the Image Mode servers – a good way to control when updates and reboots are applied.
Insights
If we login to console.redhat.com and visit the inventory page, we can immediately make use of ‘tags’ to identify the new servers we have built.
If we select the tag ‘app_support_team=My Application Support Team’ we’ll now be managing only our three servers in Insights. Here we are at the inventory, systems view:
How about we look at recommendations, to see if there are any improvements we can make?
We see the Advisor has recommended we don’t use unencrypted HTTP traffic. Advisor also mentions we have a decrease in performance as we’re not running a recommended tuned service configuration. As this is just a test container we’ll leave the httpd warning for now, but take a look at the tuned recommendation by clicking on the link.
We see that Red Hat advises that we apply a tuned profile as described in https://access.redhat.com/solutions/369093. Great – this is a candidate for adding to the next version of our image deployment.
CIS Compliance
Let’s apply the CIS Red Hat Enterprise Linux 9 Benchmark for Level 1 – Server profile to our servers and run a scan. Organisations often have their own security team and use their own security scanning tools. However, using Insights we can empower the sysadmin/operations team to create security profiles and schedule their own scans without taking up the time of the security teams. This allows systems administrators to continually audit their own environment.
We’ve not done any hardening on the system or created custom filesystems (such as /var/log) so we’re expecting some red flags. We visit https://console.redhat.com/insights/compliance/scappolicies and apply the policy to our servers.
On our servers, we can then run insights-client --compliance. This should not take long run run (around 1 minute on my test server)
[root@bootc-dev ~]# insights-client --compliance
System uses SSG version 0.1.72
Running scan for xccdf_org.ssgproject.content_profile_cis_server_l1... this may take a while
Uploading Insights data.
Successfully uploaded report for bootc-dev.london.example.com.
Navigating over to https://console.redhat.com/insights/compliance/systems/ we can immediately see the report for our system and find we scored 71%. As a quick fix, I searched on ‘service’ and noticed that there were some services we can definitely remove. Again, we can add these fixes to the next version of our image.
Patching Advisories
Red Hat Satellite had already advised us that some errata is outstanding on our container image. This is also reflected in Red Hat Insights at https://console.redhat.com/insights/patch/advisories
As mentioned before, we’ll build a new Container Image with these updates and publish this in our Container Registry. If we are using Satellite, then we can promote the updates through our Lifecycle environments. How we can we then apply those updates? Again Insights can help us!
Remote Updates
Within the Automation Toolkit -> Tasks section of Red Hat Insights there is an option to initiate updates of Image Mode for RHEL servers.
As well as providing you with a playbook for you to adapt to your own environment, you can select servers and trigger a remote update. Using the Red Hat Insights tags we applied to our images we could identify all our ‘version 1.0.0’ application servers and target them for upgrades. This is a useful option for controlling the server upgrades remotely – we update the images in the container registry and perform the updates in a controlled manner.
You can find this option at https://console.redhat.com/insights/tasks/available/bootc-upgrade
Next Steps
Now we have a basic environment, we can build upon it:
- Synchronise the latest RHEL 9 package-mode content and apply it to our Dev-Test Lifecycle so that our build server is able to use it
- Synchronise the latest
bootcandbootc-image-buildercontainers and apply them to our Dev-Test Lifecycle - Update the Containerfile on the build server to include the tuned package and apply a profile, add fixes that apply to the CIS security profile and add in an SSH key for remote execution from Satellite
- Create a new version (eg 1.0.1) of our container image
- Synchronise this new update on Red Hat Satellite
- Publish and promote new content views within Satellite
- Deploy the updates to our Dev, Stage and Production servers as we make enhancements
- Consider automating this, as described below.
Automation
If you’ve followed from the first post Build and distribute custom Image Mode for RHEL Containers all the way to this one, you’ll notice that the tasks were somewhat manual. However, the process can be automated by creating Ansible roles and/or creating a Workflow in Ansible Automation Platform.
The tasks would be as follows, with automation happening after the first step, perhaps triggered by a webhook:
- Commit changes to the Containerfile in a git repository
- Check out the updated changes on the build host, perhaps with some additional checks for your organisation (eg, does the Containerfile contain a specific variable, is it pulling from an approved image, does it contain standard user accounts and keys, mandatory inventory management software, etc)
- Build a new Container Image using the new file
- Tag the Container Image with updated tags reflecting the new version of the application
- Push it to our internal Container registry
- Synchronise the registry with Satellite
- Promote the Satellite Content View to the Dev/Test environment
- Upgrade test servers using
bootc upgrade - (Optionally) perform testing
- (Optionally) promote the Content View onto the next environment
- (Optionally) upgrade servers in the next environments via
bootc upgrade
Summary
We’ve deployed our custom Image Mode for RHEL Dev-Test, Stage and Production servers using Red Hat Satellite as our container registry. We’ve registered our servers with Red Hat Satellite and Red Hat Insights and we’ve got some immediate feedback of things we can improve on our first Image Mode for RHEL deployment:
- We can apply a tuned profile to our image
- We can apply errata to our image
- We can improve the security profile of our image
Thanks to Red Hat Satellite, Red Hat Insights and Ansible we’re already well on the way to improving our Image Mode deployments!
Note
This post is not endorsed or affiliated with Red Hat – the information provided is based on experience, documentation and publicly available information. Feel free to leave feedback at the end of this page if anything needs correction.
For an up to date roadmap discussion on Image Mode for RHEL please contact your Red Hat Account rep.