目錄

如何以 kubeadm 快速建立 K8s 叢集

Prerequisites

1. Edit /etc/hosts (all nodes)

1
2
3
4
10.10.10.166 master01-dev
10.10.10.167 master02-dev
10.10.10.168 master03-dev
10.10.10.169 vip.kube.dev

2. Enable bridge traffic

1
2
3
4
5
6
7
8
9
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

3. Enable VIP binding

1
2
3
4
5
sysctl -w net.ipv4.ip_nonlocal_bind=1
echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.conf

#check
sysctl net.ipv4.ip_nonlocal_bind

4. Disable swap

1
2
3
vim /etc/fstab 註解 swap.img
swapoff -a && sysctl -w vm.swappiness=0
reboot

5. Install **kubeadm, kubelet and kubectl**

1
2
3
4
5
6
7
8
9
sudo apt-mark unhold kubeadm kubectl kubelet

sudo apt-get install kubelet=1.22.8-00 kubeadm=1.22.8-00 kubectl=1.22.8-00

sudo apt-mark hold kubeadm kubectl kubelet

## If there is an network issud, just download the binary file and install
https://dl.k8s.io/v1.22.9/bin/linux/386/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/bin/kubectl

K8s 1.22.8 Installation

  • Generate/modify first node kube-vip config/manifest
  • init first node
  • join remaining nodes
  • Add remaining config/manifests

1. Create cluster config

create-cluster-config.sh

1
2
3
4
# 執行前,記得修改 create-cluster-config.sh 參數
chmod +x create-cluster-config.sh
vim create-cluster-config.sh
./create-cluster-config.sh

2. Run kubeadm init

1
2
kubeadm init --config kubeadm-configs.yaml --upload-certs --v=5
# Save the output!!
  • If pod crash/error, try docker rm -f
  • If kubeadm init/join fail, run reset script then REBOOT!

reset.sh

kubeadm output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 10.10.10.169:6443 --token qjfsuw.4ovqhe65b2lx59u2 \
        --discovery-token-ca-cert-hash sha256:8c28a32531adc4589fc885c24031a8c94e511f44e00caeffbd7c88e49bfff140 \
        --control-plane --certificate-key c0eb9d99e77f3c8f0994cb32b16f30ccc9c6f34e2f9836e1cc7d7d709281ac72

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.10.169:6443 --token qjfsuw.4ovqhe65b2lx59u2 \
        --discovery-token-ca-cert-hash sha256:8c28a32531adc4589fc885c24031a8c94e511f44e00caeffbd7c88e49bfff140

3. Install Calico

https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-50-nodes-or-less

安裝完後網路功能如 coredns 才會正常

calico-v3.22.2.yml

1
kubectl apply -f calico-v3.22.2.yaml

4. Join remaining controlplanes

先確認第一個 master node 沒有問題再執行

1
2
3
kubeadm join 10.10.10.169:6443 --token skev5a.tjtivikwvgrc7qur \
        --discovery-token-ca-cert-hash sha256:d4fd9a01c79f5d1cad2833cb4fbd026af376aa6bd0649cb4421029d456d86c17 \
        --control-plane --certificate-key 7d6ad10d2995e55063ad3493f34e5f6192bf45746ded783888576fcf3610be29 --v=7

完成後,將 kube-vip.yaml 複製到其他 Master 節點上的 /etc/kuberbetes/manifests/下

5. Copy kube-vip.yaml to /etc/kubernetes/manifests for remaining master nodes

6. Remove NoScheduale taint on controlplanes (研發環境 only)

1
kubectl taint nodes --all [node-role.kubernetes.io/master-](http://node-role.kubernetes.io/master-)

7. Join Worker Node

於 worker node 執行 kubeadm init 存下的指令即可

8. change pause image

vim /var/lib/kubelet/kubeadm-flags.env

systemctl daemon-reload

systemctl restart kubelet


參考資料

設定 Keepalived and HAProxy

按照此文件進行設定: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing

變數說明可參考: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#keepalived-and-haproxy

kube-vip: https://inductor.medium.com/say-good-bye-to-haproxy-and-keepalived-with-kube-vip-on-your-ha-k8s-control-plane-bb7237eca9fc