Netboot
The first step, is to get get rid of the SDc card used as a storage. Is well known, such cards are not meant to be constantly performed reads/writes on, and are susceptible to corruption, due to it.
We'll setup, in a single shot, boot trogh NFS first, and ISCI later. our intent is to be able to run containerized applications in our devices. We wont be able to do that with our rootFS mounted trough NFS, due to storage driver incompatibility.
Lucky for us, PXE comes in our help. The boot process goes like this :
PXE Client powers on, sends DHCP discover packet
DCHP Server leases the client's IP, and TFPT Server's IP
Client download boot files via TFTP
TFTP Server sends data inside the requested folder
Client executes bootfile previously defined
With the following shell script, I automated the process to boot via NFS, all we have to do, is run this script once from a frash newly flashed SD. I picked RaspberryPiOS.
#!/bin/sh
echo "Insert NODEX number"
read NUM
echo "Insert TFTP server ip"
read TFTP_SERVER
hostnamectl set-hostname node$NUM
sudo apt update && sudo apt upgrade -y && sudo apt dist-upgrade -y
sudo apt-get full-upgrade -y
sudo apt install rpi-eeprom -y
apt install nfs-kernel-server -y
SERIAL=$(vcgencmd otp_dump | grep 28: | sed s/.*://g)
echo $SERIAL
cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2021-04-29.bin pieeprom.bin
rpi-eeprom-config pieeprom.bin > bootconf.txt
echo "[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=
TFTP_PREFIX=0
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0" > bootconf.txt
rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt pieeprom.bin
sudo rpi-eeprom-update -d -f ./pieeprom-new.bin
mkdir /nfs
mkdir /nfs/node$NUM
mkdir /nfs/tftp-boot
sudo mount $TFTP_SERVER:/volume1/node$NUM /nfs/node$NUM
sudo mount $TFTP_SERVER:/volume1/tftp-boot /nfs/tftp-boot
mkdir /nfs/tftp-boot/$SERIAL
cp /etc/fstab /etc/fstab.old
echo "proc /proc proc defaults 0 0
$TFTP_SERVER:/volume1/tftp-boot/$SERIAL /boot nfs defaults,vers=3,proto=tcp 0 0
" > /etc/fstab
rsync -xa --progress --exclude /nfs / /nfs/node$NUM/
sudo cp -r /boot/* /nfs/tftp-boot/$SERIAL
rm /etc/fstab
mv /etc/fstab.old /etc/fstab
echo "console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.4.100:/volume1/node$NUM,vers=3 rw ip=dhcp elevator=deadline rootwait cgroup_memory=1 cgroup_enable=memory" > /nfs/tftp-boot/$SERIAL/cmdline.txt