Damn.
Something I didn't want to hear while showing up to work:
I have excellent news for you! You're gonna like that!
Usually, when my coworker says that with that grin. Its smells fishy.
Really fishy.
Soooo, we will soon change the base OS of the company and we will switch to Amazon Linux 2022. What do you think about that?
The smell of rotten dead rabbit swished to my nose instantly.
@daywork we maintain a digital factory. And one of it's legacy core part is based on the fact that our Puppet infrastructure just works.
But before we leave it to die in its own vomit, we need to keep it alive until the company find out a new way to deploy teams' components.
Shirt.
We need a plan
Okay, so, what do we know?
- The destination Operating System is Amazon Linux 2022
- We want a Puppet infrastructure to work without too much hassle.
- Our current version of Puppet is old. Really old. Kind of "EOLed in January 2021" old: Puppet 5
- Our current base OS is CentOS 7, not yet EOLed (Maintenance Updates until 2024-06-30)
- We have other tasks at the moment requiring our immediate attention at work
- I don't want to have some mental charge about "Hey, maybe it won't work and we'll be fracked?"
- I have too much time at home
- I like dirty things and to try unexpected things
Let's get dirty
What's in the box?
Before trying things, let's check what is Amazon Linux 2022 made of.
It's User Guide specifies that:
- It is based on Fedora as upstream
- The AMI uses XFS as root filesystem
- systemd-networkd service manages the network interface (The plague continue to spreads...)
- Core toolchains versions are upgraded
- It uses DNF rather than YUM for package managment
- Some changes about SSHD
- No EPEL or EPEL-like repositories currently work (oof)
There is no ISO to install outside of AWS. (This is something you could do with Amazon Linux 2), but there are container images availables on the Amazon ECR Public registry and the Docker Hub.
So... lets check that:
docker run -it amazonlinux:2022 /bin/bash
Unable to find image 'amazonlinux:2022' locally
2022: Pulling from library/amazonlinux
f692b7ceefbf: Pull complete
Digest: sha256:62cc92115a0ba23ce26eaefe295b448e8319f69956828b04a80fa10481c0bb42
Status: Downloaded newer image for amazonlinux:2022
curl --version
curl 7.86.0 (x86_64-amazon-linux-gnu) libcurl/7.86.0 OpenSSL/3.0.5 zlib/1.2.11 libidn2/2.3.2 nghttp2/1.45.1
Ahah. This is gonna be fun. Yeah, I like to check the curl version because the 7.86.0 has
issues with no_proxy...
How bad is the situation
Puppet 5 System Requirements
The Puppet 5 System Requirements states that it tested and published official puppet-agent package for:
Red Hat Enterprise Linux derivatives include Amazon Linux v1 (using RHEL 6 packages) and v2 (using RHEL 7 packages).
So Amazon Linux v2, the current AWS Linux optimized OS, was supported... that's one thing.
Also, Fedora was supported from version 26 to 29.
But as stated here (TODO), there is no EPEL repository available, so better check the others prerequisites:
- Ruby 2.4.X and some mandatories libraries
- Uh ... that is all? Well...
Even if we don't use the Enterprise version, the archived 2018.1 system requirements seems ok.
So... maybe it could work?
Enter Vagrant
I wanted to try Vagrant with the Docker provider.
Why? Why not? FOR SCIENCE!!!
# I have ~/.bin/ in my path
$ cd ~/.bin
$ wget https://releases.hashicorp.com/vagrant/2.3.4/vagrant_2.3.4_linux_amd64.zip
$ unzip vagrant_2.3.4_linux_amd64.zip
$ mv vagrant vagrant_2.3.4
$ ln -s vagrant_2.3.4 vagrant
$ vagrant --version
Vagrant 2.3.4
Init the Vagrant project for a Puppet Server:
$ mkdir puppet_server
$ cd puppet_server && vagrant init
Part 1: Master of puppets, I'm pulling your strings (Puppet Server)
Okay, let's try to set up a Puppet Server by using the official EOL Extra Linux Puppet repository RPM:
First try
Create the following Vagrantfile:
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.provider "docker" do |d|
d.build_dir = "."
end
end
And the following Dockerfile:
FROM amazonlinux:2022
RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm
# Sleeping beauty
CMD ["sleep", "infinity"]
Aaaand letsgo!
$ vagrant up
Bringing machine 'default' up with 'docker' provider...
==> default: Creating and configuring docker networks...
==> default: Building the container from a Dockerfile...
default: Sending build context to Docker daemon 9.728kB
default: Step 1/3 : FROM amazonlinux:2022
default: ---> 5fe27e80432b
default: Step 2/3 : RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm
default: ---> Running in 7e4ba6d822f9
default: Amazon Linux 2022 repository 25 MB/s | 11 MB 00:00
default: Last metadata expiration check: 0:00:02 ago on Wed Jan 25 21:15:35 2023.
default: puppet5-release-el-7.noarch.rpm 11 kB/s | 8.6 kB 00:00
default: (try to add '--skip-broken' to skip uninstallable packages)
default: Error:
default: Problem: conflicting requests
default: - nothing provides /bin/mkdir needed by puppet5-release-5.0.0-14.el7.noarch
default:
default: The command '/bin/sh -c dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm' returned a non-zero code: 1
A Docker command executed by Vagrant didn't complete successfully!
The command run along with the output from the command is shown
below.
Command: ["docker", "build", "/home/lenain/work/puppet_server", {:notify=>[:stdout, :stderr]}]
Stderr: The command '/bin/sh -c dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm' returned a non-zero code: 1
Stdout: Sending build context to Docker daemon 9.728kB
Step 1/3 : FROM amazonlinux:2022
---> 5fe27e80432b
Step 2/3 : RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm
---> Running in 7e4ba6d822f9
Amazon Linux 2022 repository 25 MB/s | 11 MB 00:00
Last metadata expiration check: 0:00:02 ago on Wed Jan 25 21:15:35 2023.
puppet5-release-el-7.noarch.rpm 11 kB/s | 8.6 kB 00:00
(try to add '--skip-broken' to skip uninstallable packages)
Error:
Problem: conflicting requests
- nothing provides /bin/mkdir needed by puppet5-release-5.0.0-14.el7.noarch
Mmmmokay.
The Puppet repository package expects /bin/mkdir to be provided, but there is no package that
provides this file, only /usr/bin/mkdir:
# dnf provides /bin/mkdir|grep Filename
Filename : /usr/bin/mkdir
Filename : /usr/bin/mkdir
Filename : /usr/bin/mkdir
Building an RPM
Lets create a fake RPM to artificially provide the already existing /bin/mkdir!
Install required tools and create an RPM build tree in a temporary container:
$ docker run -ti -v $PWD:/mnt amazonlinux:2022
$ dnf -y install rpm-build rpmdevtools
$ rpmdev-setuptree
Create an ./rpmbuild/SPECS/fake-mkdir.spec:
Name: fake-mkdir
Version: 0.1
Release: 1%{?dist}
Summary: A fake provider to /bin/mkdir for Puppet
License: AGPLv3+
Source0: %{name}-%{version}.tar.gz
Provides: /bin/mkdir
%description
A fake provider to /bin/mkdir for Puppet
%prep
%build
%install
%files
%changelog
Create a fake source file ./rpmbuild/SOURCES/fake-mkdir-0.1.tar.gz:
$ echo "Yes, you have a /bin/mkdir." > ./rpmbuild/SOURCES/README
$ tar cvzf ./rpmbuild/SOURCES/fake-mkdir-0.1.tar.gz -C ./rpmbuild/SOURCES/ README
Build the RPM:
$ rpmbuild -v -ba ~/rpmbuild/SPECS/fake-mkdir.spec
Check that the RPM does what it should do:
$ rpm -q --provides ./rpmbuild/RPMS/x86_64/fake-mkdir-0.1-1.amzn2022.x86_64.rpm
/bin/mkdir
fake-mkdir = 0.1-1.amzn2022
fake-mkdir(x86-64) = 0.1-1.amzn2022
Put it out of the RPM build container:
$ mv ./rpmbuild/RPMS/x86_64/fake-mkdir-0.1-1.amzn2022.x86_64.rpm /mnt/.
Second try
Update the Dockerfile:
FROM amazonlinux:2022
ARG FAKE_MKDIR_RPM=fake-mkdir-0.1-1.amzn2022.x86_64.rpm
COPY $FAKE_MKDIR_RPM /tmp
RUN dnf install -y /tmp/$FAKE_MKDIR_RPM
RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm
# Sleeping beauty
CMD ["sleep", "infinity"]
Lezgo!
Bringing machine 'default' up with 'docker' provider...
==> default: Creating and configuring docker networks...
==> default: Building the container from a Dockerfile...
default: Sending build context to Docker daemon 17.92kB
default: Step 1/6 : FROM amazonlinux:2022
default: ---> 5fe27e80432b
default: Step 2/6 : ARG FAKE_MKDIR_RPM=fake-mkdir-0.1-1.amzn2022.x86_64.rpm
default: ---> Running in e94eca9b8269
default: Removing intermediate container e94eca9b8269
default: ---> 487d5b5293f9
default: Step 3/6 : COPY $FAKE_MKDIR_RPM /tmp
default: ---> 9875d1d2d441
default: Step 4/6 : RUN dnf install -y /tmp/$FAKE_MKDIR_RPM
default: ---> Running in 0016e5f5c180
default: Amazon Linux 2022 repository 17 MB/s | 11 MB 00:00
default: Last metadata expiration check: 0:00:02 ago on Wed Jan 25 22:13:09 2023.
default: Dependencies resolved.
default: ================================================================================
default: Package Architecture Version Repository Size
default: ================================================================================
default: Installing:
default: fake-mkdir x86_64 0.1-1.amzn2022 @commandline 6.0 k
default:
default: Transaction Summary
default: ================================================================================
default: Install 1 Package
default:
default: Total size: 6.0 k
default: Installed size: 0
default: Downloading Packages:
default: Running transaction check
default: Transaction check succeeded.
default: Running transaction test
default: Transaction test succeeded.
default: Running transaction
default: Preparing : 1/1
default:
default: Installing : fake-mkdir-0.1-1.amzn2022.x86_64 1/1
default:
default: Verifying : fake-mkdir-0.1-1.amzn2022.x86_64 1/1
default:
default:
default: Installed:
default: fake-mkdir-0.1-1.amzn2022.x86_64
default:
default: Complete!
default: Removing intermediate container 0016e5f5c180
default: ---> 00154047b3b6
default: Step 5/6 : RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm
default: ---> Running in 3c943960f20e
default: Last metadata expiration check: 0:00:03 ago on Wed Jan 25 22:13:09 2023.
default: puppet5-release-el-7.noarch.rpm 103 kB/s | 8.6 kB 00:00
default: Dependencies resolved.
default: ================================================================================
default: Package Architecture Version Repository Size
default: ================================================================================
default: Installing:
default: puppet5-release noarch 5.0.0-14.el7 @commandline 8.6 k
default:
default: Transaction Summary
default: ================================================================================
default: Install 1 Package
default: Total size: 8.6 k
default: Installed size: 3.9 k
default: Downloading Packages:
default: Running transaction check
default: Transaction check succeeded.
default: Running transaction test
default: Transaction test succeeded.
default: Running transaction
default: Preparing : 1/1
default:
default: Running scriptlet: puppet5-release-5.0.0-14.el7.noarch 1/1
default:
default: Installing : puppet5-release-5.0.0-14.el7.noarch 1/1
default:
default: Running scriptlet: puppet5-release-5.0.0-14.el7.noarch 1/1
default:
default: Verifying : puppet5-release-5.0.0-14.el7.noarch 1/1
default:
default:
default: Installed:
default: puppet5-release-5.0.0-14.el7.noarch
default:
default: Complete!
default: Removing intermediate container 3c943960f20e
default: ---> d19dfb140f14
default: Step 6/6 : CMD ["sleep", "infinity"]
default: ---> Running in caebb71aac56
default: Removing intermediate container caebb71aac56
default: ---> 898257790bef
default: Successfully built 898257790bef
default:
default: Image: 898257790bef
==> default: Creating the container...
default: Name: puppet_server_default_1674684794
default: Image: 898257790bef
default: Volume: /home/lenain/work/puppet_server:/vagrant
default:
default: Container created: 917ddaac8b5d36bc
==> default: Enabling network interfaces...
==> default: Starting container...
Bwahahah! IT WORKED! (Doing the magic hand gesture)
Can I haz a Puppet Server now plz?
$ vagrant docker-exec -it -- /bin/sh
$ dnf list|grep puppetserver
puppetserver.noarch 5.3.16-1.el7 puppet5
$ dnf install -y puppetserver
Last metadata expiration check: 0:00:44 ago on Wed Jan 25 22:16:29 2023.
Error:
Problem: conflicting requests
...
- nothing provides java-1.8.0-openjdk-headless needed by puppetserver-5.3.9-1.el7.noarch
(try to add '--skip-broken' to skip uninstallable packages)
yum search java-1.8
Last metadata expiration check: 0:00:06 ago on Wed Jan 25 22:27:07 2023.
========================== Name Matched: java-1.8 =================================================
java-1.8.0-amazon-corretto.x86_64 : Amazon Corretto runtime environment
java-1.8.0-amazon-corretto-devel.x86_64 : Amazon Corretto development environment
Sad. Obviously Amazon only provides the corretto version of the JDK in their repositories, not the OpenJDK one.
Lets create a /etc/yum.repos.d/fedora.repo file with a fixed $releasever:
[fedora]
name=Fedora 37 - $basearch
#baseurl=http://download.example/pub/fedora/linux/releases/37/Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-37&arch=$basearch
enabled=1
countme=1
metadata_expire=7d
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-37-$basearch
skip_if_unavailable=False
Add Fedora GPG keys, update DNF, and try to install it this way.
$ dnf install -y https://kojipkgs.fedoraproject.org//packages/fedora-repos/37/1/noarch/fedora-gpg-keys-37-1.noarch.rpm
$ dnf update
$ dnf install -y puppetserver
...
Complete!
$ /opt/puppetlabs/server/apps/puppetserver/bin/puppetserver ruby --version
jruby 1.7.27 (1.9.3p551) 2017-05-11 8cdb01a on OpenJDK 64-Bit Server VM 1.8.0_345-b01 +jit [linux-amd64]
$ /opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start
$ tail -f /var/log/puppetlabs/puppetserver/puppetserver.log
...
2023-01-25 23:01:55,345 INFO [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
2023-01-25 23:01:55,577 INFO [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 4 of 4
Uuuaaaaah? ME LIKEY!
Lets fix our Dockerfile:
FROM amazonlinux:2022
# Required for Java 1.8 OpenJDK and not Corretto
COPY fedora.repo /etc/yum.repos.d/fedora.repo
RUN dnf install -y https://kojipkgs.fedoraproject.org//packages/fedora-repos/37/1/noarch/fedora-gpg-keys-37-1.noarch.rpm && \
dnf update
# Fake RPM to provide Puppet /bin/mkdir dependency
ARG FAKE_MKDIR_RPM=fake-mkdir-0.1-1.amzn2022.x86_64.rpm
COPY $FAKE_MKDIR_RPM /tmp
RUN dnf install -y /tmp/$FAKE_MKDIR_RPM && \
rm /tmp/$FAKE_MKDIR_RPM
# Puppet Specific
RUN dnf install -y https://yum.puppet.com/eol-releases/puppet5-release-el-7.noarch.rpm && \
dnf install -y puppetserver
# Puppet Server run as user puppet but the ssl directory is only writeable by root, fix this
RUN mkdir -p /etc/puppetlabs/puppet/ssl && \
chown puppet:puppet /etc/puppetlabs/puppet/ssl
# Start the Puppet Server and expose the port
CMD ["/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver", "foreground"]
EXPOSE 8140
We also need to expose the container port locally, so we add the forward_port directive to our
Vagrantfile:
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.provider "docker" do |d|
d.build_dir = "."
end
config.vm.network "forwarded_port", guest: 8140, host: 8140
end
We restart our Vagrant Puppet Server Container:
$ vagrant reload
Check the logs:
$ vagrant docker-logs --follow
==> default: 2023-01-25 23:23:47,487 INFO [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
==> default: 2023-01-25 23:23:47,683 INFO [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 4 of 4
Is it accessible from the local network?
curl -k "https://127.0.0.1:8140/status/v1/simple"
running
NOICE!
Part 2: Just call my name 'cause I'll hear you scream (Puppet Agent)
TODO