Bootstrap K3s Cluster With Ignition
In this walkthrough, we configure our FCOS powered raspberry pis,
to bootstrap K3s at each reboot effectively ensurely durability and reliability.
Each reboot will get nodes back at their desired state, allowing us to separate
the application leval from the data since the configuration is fully declarative.
The issue with K3s is the embedded datastore :
the power of this configuration is that each reboot nodes are fresh new,
but K3s has no possible way to add new master nodes if deployed with embedded sqlite.
The solution is an external datastore, we can use a mysql container.
Keep in mind that compatibility is up to certain version, more on this on the documentation,
The 5.7.9 and it works for sure : Rancher K3s
A docker container can be configured with the following parameters :
MYSQL_VERSION : 5.7.9debian8
MYSQL_USER: {{ USERNAME }}
MYSQL_PASSWORD: {{ PASSWORD }}
MYSQL_DATABASE: {{ DATABASE }}
MYSQL_ROOT_PASSWORD: {{ ROOT_PASSWORD }}
We will need the parameters to to write the ignition config.
First we need to download K3s, we are lucky is a single binary !
Important note, we need to verify the integrity of our binary,
simply use this command to extract the 512bit hash :
We need to store the hash in the igntion config file :
storage:
files:
- path: "/usr/local/bin/k3s"
contents:
source: https://github.com/k3s-io/k3s/releases/download/{{ K3S_VERSION }}%2Bk3s2/k3s
verification:
# Insert the generated hash
hash: sha512-{{ HASH_512 }}
mode: 0755
user:
name: root
group:
name: root
- path: /usr/local/etc/k3s/token
contents:
source: data:text/plain;charset=utf-8;base64,YWQ4NDE1NTUtZGMyOS00YTZkLTk1NjQtN2E5NzQ0NGUwM2Ux
mode: 0400
user:
name: root
The k3s binary and its token is downloaded now we need to run it as a systemd service.
Follows the config for the master nodes :
systemd:
units:
- name: k3s.service
enabled: true
contents: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
After=settimezone.service
[Install]
WantedBy=multi-user.target
[Service]
Type=notify
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
'server' \
'--disable-selinux' \
'--token-file=/usr/local/etc/k3s/token' \
'--cluster-cidr=10.10.0.0/16' \
'--service-cidr=10.20.0.0/16' \
'--token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\
'--datastore-endpoint=mysql://{{ USERNAME }}:{{ PASSWORD }}@tcp(10.10.130.100:12000)/{{ DATABASE }}'
- name: k3s.service
enabled: true
contents: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
After=settimezone.service
[Install]
WantedBy=multi-user.target
[Service]
Type=notify
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
'agent' \
'--server=https://{{ API_SERVER }}:6443' \
'--token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\
'--node-name=worker' \
'--with-node-id'
Generate the ignition configs with butane, and host it at the http-boot server.
Congratulations on your highly available rpi k3s cluster up and running!