Setting up multi-node K3s cluster using Vagrant

What is K3s?
K3s is a simple, lightweight Kubernetes distribution designed for situations where compute resources may be limited or where a simple and easy Kubernetes solution is required.

Advantages of K3s?
1. Lightweight
K3s is designed to use half the memory of a regular k8s installation and uses a single <100MB binary.
2. Simple and Easy to use
K3s packages, dependencies and addons within a single binary. K3s provides sensible, secure defaults and automates complex cluster operations like distributing certificates.
Rancher built K3s.

Why is it called K3s?
Kubernetes has 10 letters and is stylized as K8s(the 8 represents 8 letters). A similar 5-letter word would be half the size of Kubernetes, with 3 letters between K and the S.
Kubernetes → K8s
K***s → K3s

K3s Use Cases
1. Edge
Edge devices often have resource constraints. K3s is lightweight and can help here. It’s simplicity also makes updates easier to manage.
2. Internet of Things(IoT)
IoT devices can also be resource-constrained and difficult to update. So, K3s can a great fit for it.
3. Continous Integration(CI)
Simple lightweight can be an easy addition to the CI process.

Technical Features
1. Lightweight: K3s uses the half memory of a regular Kubernetes cluster.
2. Single Binary: A single <100MB binary contains all Kubernetes control-plane components. K3s also bundles a container runtime and other dependencies.
3. sqlite3 Storage Backend: K3s uses a lightweight SQLite-based storage backend instead of the usual etcd. This is also included in the K3s binary.
4. Packaged Addons: K3s includes a few packaged add-ons to provide out-of-the-box functionality, such as flannel CNI, a Helm controller, and Traefik ingress controller.
5. Secure-by-Defalt: K3s includes sensible defaults for security and internally manages certificate distribution between components.

K3s Architecture Diagram

System Requirements
1. Minimum 512Mb RAM
2. Minimum 1 CPU
3. SSD(Solid State Disk) Recommended
4. Operating System(OS)
K3s is officially tested on the Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, Ubuntu 20.04 LTS and Ubuntu 22.04 LTS. Also, it is tased on Raspian Buster.
5. PORTs
API Server — All nodes need to be able to reach the K3s server on port 6443 via TCP.
Flannel VXLAN — All server and agent nodes need to be able to reach each other on port 8472 via UDP.
By default, only the root user can read the kubeconfig file generated by K3s installation. To allow other users to connect to the cluster, we need to allow them to read the kubeconfig file located at /etc/rancher/k3s/k3s.yaml.
So, not wasting time further let’s dive into the practical.
For my setup, I have used Oracle VirtualBox(https://www.virtualbox.org/wiki/Downloads) installed on Windows 11 OS. Enabled the virtualization as at BISOS to work seamlessly with vagrant(https://developer.hashicorp.com/vagrant/install?product_intent=vagrant).
After installing VirtualBox and Vagrant open the terminal(can use Powershell, and Command Prompt but for my demo, I have used cmder(https://cmder.app/)).

1. Clone my Github repo.(Install Git if not installed in your system https://git-scm.com/downloads)
https://github.com/infa-sasatapathy/vagrant-k3s

2. After cloning go inside the vagarant folder and check the Vagrantfile. The content of the Vagrantfile is as below.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64" # Ubuntu 22.04

  # Common configurations for all machines
  config.vm.provision "shell", inline: <<-SHELL
    # Fix DNS resolution
    echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf > /dev/null
    echo "nameserver 8.8.4.4" | sudo tee -a /etc/resolv.conf > /dev/null
 
  SHELL

  # Master node
  config.vm.define "k3s-master" do |master|
    master.vm.hostname = "k3s-master"
    master.vm.network "private_network", ip: "192.168.56.101"
    master.vm.provider "virtualbox" do |vb|
      vb.memory = 4096
      vb.cpus = 2
    end
  end

  # Worker node 1
  config.vm.define "k3s-worker1" do |worker1|
    worker1.vm.hostname = "k3s-worker1"
    worker1.vm.network "private_network", ip: "192.168.56.102"
    worker1.vm.provider "virtualbox" do |vb|
      vb.memory = 2048
      vb.cpus = 2
    end
  end

  # Worker node 2
  config.vm.define "k3s-worker2" do |worker2|
    worker2.vm.hostname = "k3s-worker2"
    worker2.vm.network "private_network", ip: "192.168.56.103"
    worker2.vm.provider "virtualbox" do |vb|
      vb.memory = 2048
      vb.cpus = 2
    end
  end
end

Adjust the private_network as per your system configuration.
Run the below command to spinup three VMs. One is K3s master and two worker nodes.
run vagrant status command to see the status of the nodes.

run vagrant up to spinup the three VMs.

If you want to explicitly provide VirtualBox as the provider then run the command instead,

$vagrant up - provider virtualbox

Check the status of the VMs by running, vagrant status

Now we will ssh into the k3s-matser node which will be our master node and set it up.

Update the systemd-resolved config file to make the nameserver changes persistent. Edit the /etc/systemd/resolved.conf file like this,

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
DNS=8.8.8.8 1.1.1.1
FallbackDNS=8.8.4.4 1.0.0.1

Now make the three node’s private IP entry in /etc/hosts entry.

127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost   ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
127.0.1.1       ubuntu-jammy    ubuntu-jammy

127.0.2.1 k3s-master k3s-master
192.168.56.101 k3s-master
192.168.56.102 k3s-worker1
192.168.56.103 k3s-worker2

After updating the two files, it’s time to install the K3s. Login into the K3s-master node and run the below commands,

$curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -
$sudo chown vagrant:vagrant /etc/rancher/k3s/k3s.yaml

After the installation copy the Token and save in some safe place.

$sudo cat /var/lib/rancher/k3s/server/node-token

Login into both the worker nodes and run the below command,

$curl -sfL https://get.k3s.io | K3S_URL=https://192.168.56.101:6443 K3S_TOKEN=(Token copied from the master node) sh -

Now both the Kubernetes Master and worker are set, can be verified by running,

$kubectl get nodes

For testing purposes run a nginx container and see if it’s working fine or not.

$cat > nginx.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
EOF

Run the kubectl apply -f nginx.yaml to create the pod.

Now, expose the pod using NodePort to access it outside.

$kubectl expose pod/nginx --type=NodePort

Check on which Node and what port it’s listening.

$kubectl get svc/nginx

kubectl get pod/nginx -o wideN

In this case, the pod is running in k3s-worker2 with port 31189. In my system browser Let’s open it and see if it’s working or not.

It’s working as expected. Now our multi-node K3s cluster is perfectly working.

Reference:

K3s

Github URL: https://github.com/infa-sasatapathy/vagrant-k3s