Kubernetes

Kubernetes Definitions

Cluster
The group of computers working as one single unit.

Master
Coordinates the cluster, all activities such as scheduling apps, maintaining state, scaling, updates.

Node
Serves as a worker machine in a cluster, each node has a kubelet and tools for handling containers.

Deployment
Instructs Kubernetes how to create/update instances of application, also monitors instances.

Pod
Hosts application instance, represents a group of one or more app containers, and shared resources.

Service
Exposes pods to external traffic and load balances traffic across multiple pods.

Installing Kubernetes

This guide will cover setting up a homelab for use with Kubernetes. Instead of using a third-party tool, it uses the official kubeadm tool.

Prerequisites before Installing

Turn swap off

Run this command to turn swap off now.

swapoff -a

Edit this file using nano.

sudo nano /etc/fstab

Remove the line with a reference to swap, then save and exit.

Let iptables see bridged traffic

Make sure br_netfilter is loaded. Run lsmod | grep br_netfilter to check. Run sudo modprobe br_netfilter if not loaded.
Run this command for iptables to correctly see bridged traffic.

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Installing Docker

Set up the repository and allow apt to use a repository over HTTPS.

sudo apt-get update && sudo apt-get install -y \
  apt-transport-https ca-certificates curl software-properties-common gnupg2

Add Docker's official GPG key.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -

Add the Docker apt repository.

sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

Install Docker CE

sudo apt-get update && sudo apt-get install -y \
  containerd.io=1.2.13-2 \
  docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs)

Set up the Docker daemon.

cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
  "max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

Create /etc/systemd/system/docker.service.d

sudo mkdir -p /etc/systemd/system/docker.service.d

Restart Docker

sudo systemctl daemon-reload
sudo systemctl restart docker

Enable Docker Service

sudo systemctl enable docker

Installing kubeadm, kubelet and kubectl

Run this command

sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Initalize the Kubernetes cluster

Run this command to initalize the cluster. This makes this system the master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

We will be using flannel as our CNI, flannel is a virtual network that attaches IP addresses to containers. When using kubeadm, you must set the pod network cidr during the kubeadm init command.

Save the output because we will need it to add workers to the Kubernetes cluster.
Run this command to save the kube configuration to your user directory. This will allow the user to access the cluster using the kubectl command.

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

On a local machine with SSH access to the master node and kubectl installed, we can run this command to pull down the kube configuration and manage the cluster remotely.

cd ~
scp -r username@remotehost:/home/username/.kube .

Install the flannel network plugin

Run this command to deploy flannel to the cluster.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Install Helm

Run this commmand on any system connected to the Kubernetes cluster. Helm is a Kubernetes package manager.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Install MetalLB

Run this command to deploy MetalLB, a load-balancer for bare metal Kubernetes clusters.

Change the IP address pool to your local network's subnet, but outside the DHCP pool. So my network's DHCP server assigns IPs between 192.168.10.5 and 192.168.10.30. I used an IP pool between 192.168.10.40 and 192.168.10.250.

kubectl create namespace metallb-system
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install metallb bitnami/metallb --namespace metallb-system \
--set configInline.address-pools[0].name=default \
--set configInline.address-pools[0].protocol=layer2 \
--set configInline.address-pools[0].addresses[0]=192.168.10.40-192.168.10.250

Install Traefik

Add Traefik's chart repository to Helm and update Helm's chart repositories.

helm repo add traefik https://helm.traefik.io/traefik
helm repo update

Run this command to deploy Traefik.

helm install traefik-ingress traefik/traefik

Install cert-manager

Add Jetstack's Helm repository and update Helm's repositories.

helm repo add jetstack https://charts.jetstack.io
helm repo update

Create cert-manager namespace and install cert-manager

kubectl create namespace cert-manager
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager --set installCRDs=true

Configure cert-manager

Create a file named letsencrypt-staging.yml and add the contents below, replace the email with your own.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-staging
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Create a file named letsencrypt-prod.yml and add the contents below, replace the email with your own.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Apply the certificate issuers.

kubectl apply -f letsencrypt-staging.yml
kubectl apply -f letsencrypt-prod.yml

Install Nextcloud

Add nextcloud repository and update Helm's repositories, then create nextcloud namespace.

helm repo add nextcloud https://nextcloud.github.io/helm/
helm repo update
kubectl create namespace nextcloud

Create the following file and then apply it to the cluster.

## nextcloud.persistentvolume.yml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "nextcloud-storage"
  labels:
    type: "local"
spec:
  storageClassName: "manual"
  capacity:
    storage: "5T"
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/pool0/nextcloud"
---

kubectl apply -f nextcloud.persistentvolume.yml

Create the following file and then apply it to the cluster.

## nextcloud.persistentvolumeclaim.yml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: "nextcloud"
  name: "nextcloud-storage"
spec:
  storageClassName: "manual"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "5T"
---

kubectl apply -f nextcloud.persistentvolumeclaim.yml

Download the chart values of the Nextcloud Helm Chart.

helm show values nextcloud/nextcloud >> nextcloud.values.yml

Update the values

## nextcloud.values.yml
nextcloud:
  host: "cloud.<domain.com>" # Host to reach NextCloud
  username: "admin" # Admin
  password: "<PASSWORD>" # Admin Password
(...)
persistence:
  enabled: true # Change to true
  existingClaim: "nextcloud-storage" # Persistent Volume Claim created earlier
  accessMode: ReadWriteOnce
  size: "5T"

Create nextcloud namespace and install nextcloud.

helm install nextcloud nextcloud/nextcloud \
  --namespace nextcloud \
  --values nextcloud.values.yml

Forward ports 80 and 443 to the LoadBalancer IP address of Traefik, which should be 192.168.10.40.
Map the subdomain cloud.example.com to your external IP address. This domain needs to resolve to your external IP address which will direct to your Traefik load balancer.
Create the ingress configuration file then deploy to cluster.

## nextcloud.ingress.yml
---
apiVersion: apps/v1
kind: Ingress
metadata:
  namespace: "nextcloud" # Same namespace as the deployment
  name: "nextcloud-ingress" # Name of the ingress (see kubectl get ingress -A)
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # Encrypt using the ClusterIssuer deployed while setting up Cert-Manager
    nginx.ingress.kubernetes.io/proxy-body-size:  "50m" # Increase the size of the maximum allowed size of the client request body
spec:
  tls:
  - hosts:
    - "cloud.<domain.com>" # Host to access nextcloud
    secretName: "nextcloud-prod-tls" # Name of the certifciate (see kubectl get certificate -A)
  rules:
  - host: "cloud.<domain.com>" # Host to access nextcloud
    http:
      paths:
        - path: /  # We will access NextCloud via the URL https://cloud.<domain.com>/
          backend:
            serviceName: "nextcloud" # Mapping to the service (see kubectl get services -n nextcloud)
            servicePort: 8080 # Mapping to the port (see kubectl get services -n nextcloud)

kubectl apply -f nextcloud.ingress.yml

To check certificate issuance, run this command.

kubectl get certificaterequest -n nextcloud -o wide

Kubernetes ​

Kubernetes Definitions ​

Installing Kubernetes ​

Prerequisites before Installing ​

Turn swap off ​

Let iptables see bridged traffic ​

Installing Docker ​

Installing kubeadm, kubelet and kubectl ​

Initalize the Kubernetes cluster ​

Install the flannel network plugin ​

Install Helm ​

Install MetalLB ​

Install Traefik ​

Install cert-manager ​

Configure cert-manager ​

Install Nextcloud ​