Prerequisites
1. Edit /etc/hosts (all nodes)
1
2
3
4
| 10.10.10.166 master01-dev
10.10.10.167 master02-dev
10.10.10.168 master03-dev
10.10.10.169 vip.kube.dev
|
2. Enable bridge traffic
1
2
3
4
5
6
7
8
9
| cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
|
3. Enable VIP binding
1
2
3
4
5
| sysctl -w net.ipv4.ip_nonlocal_bind=1
echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.conf
#check
sysctl net.ipv4.ip_nonlocal_bind
|
4. Disable swap
1
2
3
| vim /etc/fstab 註解 swap.img
swapoff -a && sysctl -w vm.swappiness=0
reboot
|
5. Install **kubeadm, kubelet and kubectl**
1
2
3
4
5
6
7
8
9
| sudo apt-mark unhold kubeadm kubectl kubelet
sudo apt-get install kubelet=1.22.8-00 kubeadm=1.22.8-00 kubectl=1.22.8-00
sudo apt-mark hold kubeadm kubectl kubelet
## If there is an network issud, just download the binary file and install
https://dl.k8s.io/v1.22.9/bin/linux/386/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/bin/kubectl
|
K8s 1.22.8 Installation
- Generate/modify first node
kube-vip config/manifest init first nodejoin remaining nodes- Add remaining config/manifests
1. Create cluster config
create-cluster-config.sh
1
2
3
4
| # 執行前,記得修改 create-cluster-config.sh 參數
chmod +x create-cluster-config.sh
vim create-cluster-config.sh
./create-cluster-config.sh
|
2. Run kubeadm init
1
2
| kubeadm init --config kubeadm-configs.yaml --upload-certs --v=5
# Save the output!!
|
- If pod crash/error, try
docker rm -f - If kubeadm init/join fail, run reset script then REBOOT!
reset.sh
kubeadm output:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 10.10.10.169:6443 --token qjfsuw.4ovqhe65b2lx59u2 \
--discovery-token-ca-cert-hash sha256:8c28a32531adc4589fc885c24031a8c94e511f44e00caeffbd7c88e49bfff140 \
--control-plane --certificate-key c0eb9d99e77f3c8f0994cb32b16f30ccc9c6f34e2f9836e1cc7d7d709281ac72
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.10.10.169:6443 --token qjfsuw.4ovqhe65b2lx59u2 \
--discovery-token-ca-cert-hash sha256:8c28a32531adc4589fc885c24031a8c94e511f44e00caeffbd7c88e49bfff140
|
3. Install Calico
https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-50-nodes-or-less
安裝完後網路功能如 coredns 才會正常
calico-v3.22.2.yml
1
| kubectl apply -f calico-v3.22.2.yaml
|
4. Join remaining controlplanes
先確認第一個 master node 沒有問題再執行
1
2
3
| kubeadm join 10.10.10.169:6443 --token skev5a.tjtivikwvgrc7qur \
--discovery-token-ca-cert-hash sha256:d4fd9a01c79f5d1cad2833cb4fbd026af376aa6bd0649cb4421029d456d86c17 \
--control-plane --certificate-key 7d6ad10d2995e55063ad3493f34e5f6192bf45746ded783888576fcf3610be29 --v=7
|
完成後,將 kube-vip.yaml 複製到其他 Master 節點上的 /etc/kuberbetes/manifests/下
5. Copy kube-vip.yaml to /etc/kubernetes/manifests for remaining master nodes
6. Remove NoScheduale taint on controlplanes (研發環境 only)
1
| kubectl taint nodes --all [node-role.kubernetes.io/master-](http://node-role.kubernetes.io/master-)
|
7. Join Worker Node
於 worker node 執行 kubeadm init 存下的指令即可
8. change pause image
vim /var/lib/kubelet/kubeadm-flags.env
systemctl daemon-reload
systemctl restart kubelet
參考資料
設定 Keepalived and HAProxy
按照此文件進行設定: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing
變數說明可參考: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#keepalived-and-haproxy
kube-vip: https://inductor.medium.com/say-good-bye-to-haproxy-and-keepalived-with-kube-vip-on-your-ha-k8s-control-plane-bb7237eca9fc