Oldbeef

Menu

oldbeef
Published on 2025-06-05 / 32 Visits
0
0

基于Kubeadm搭建Kubernetes集群

#虚拟化#Kubernetes

写在前面

在介绍环境的搭建前，有必要介绍一下容器、容器化、以及Kubernetes这些概念

容器与容器化

容器是一种轻量级、可移植的软件运行环境，它将应用程序及其依赖项打包在一起，形成一个独立的运行单元。容器的核心思想是通过隔离技术，为每个应用程序提供一个独立的运行空间，使其能够在不同的环境中保持一致的运行状态。容器化技术的出现，解决了传统软件部署中常见的“环境依赖”问题，即应用程序在开发环境中可以正常运行，但在生产环境中却可能出现各种兼容性或依赖性问题。通过容器化，应用程序及其运行环境被封装为一个镜像，这个镜像可以在任何支持容器技术的平台上运行，从而确保了开发、测试和生产环境的一致性。

容器化技术的兴起，得益于其在资源利用、部署效率和环境一致性方面的显著优势。与传统的虚拟化技术相比，容器共享宿主机的操作系统内核，无需为每个应用程序运行一个完整的虚拟机，因此启动速度快、资源占用少。容器化还简化了应用程序的部署过程，开发人员只需将应用程序及其依赖项打包为容器镜像，运维人员就可以直接在目标环境中运行这些镜像。此外，容器化还支持快速的水平扩展，通过启动多个容器实例，可以轻松应对高并发场景下的负载需求。

Kubernetes 集群

Kubernetes 是一个开源的容器编排平台，用于自动化容器的部署、扩展和管理。它为容器化应用程序提供了一个强大的运行环境，能够解决容器化带来的复杂性问题。在 Kubernetes 集群中，容器被组织为一组相关的单元，称为“Pod”。每个 Pod 包含一个或多个紧密相关的容器，它们共享网络和存储资源。Kubernetes 通过定义一系列的资源对象（如 Deployment、Service、Ingress 等），来管理容器的生命周期、网络通信、存储卷挂载以及负载均衡等任务。

Kubernetes 集群的核心功能之一是自动扩缩容。它可以根据预设的规则（如 CPU 使用率、内存使用率或自定义指标）自动调整容器的数量，以满足应用程序的负载需求。此外，Kubernetes 还提供了强大的故障恢复机制，当某个容器或 Pod 出现故障时，它可以自动重启或重新调度到其他节点上运行。Kubernetes 的另一个重要特性是其高度的可扩展性和灵活性。它支持多种存储后端（如本地存储、云存储、分布式存储等），并可以通过插件机制集成各种网络插件和存储插件，以满足不同场景下的需求。

Kubernetes 集群的管理通过命令行工具（如 kubectl）或图形化界面（如 Kubernetes Dashboard）进行操作。管理员可以通过这些工具定义资源对象的配置文件（通常是 YAML 格式），并将其应用到集群中。Kubernetes 还提供了丰富的 API，方便开发者通过编程接口与集群进行交互。随着容器化技术的普及，Kubernetes 已成为容器编排的事实标准，被广泛应用于云原生架构中。它不仅支持在公有云环境中运行，还可以在私有数据中心或混合云环境中部署。通过 Kubernetes，企业可以实现应用程序的快速迭代、高效部署和弹性扩展，从而更好地应对数字化转型中的挑战。

环境准备（仅做参考，请结合实际情况）

ubt22.04 node1 10.0.0.231

ubt22.04 node2 10.0.0.232

ubt22.04 node3 10.0.0.233

具体步骤

1.关闭swap分区

swapoff -a && sysctl -w vm.swappiness=0  # 临时关闭
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab  # 基于配置文件关闭

2.允许iptable检查桥接流量

cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

3.安装docker

auto-install-docker-v3.tar.gz
tar xvf auto-install-docker-v3.tar.gz
./install-docker.sh i
#此处使用的为我自己编写的安装脚本

#所需tar包已上传gitee，放心食用
git clone https://gitee.com/ooldan/docker-bash-install.git

4.安装kubectl kubeadm kubelet

#所有节点配置apt源
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update

5.检查支持的k8s版本

apt-cache madison kubeadm
   kubeadm |  1.28.2-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.28.1-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.28.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   ...
   kubeadm | 1.23.17-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm | 1.23.16-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm | 1.23.15-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm | 1.23.14-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   ...

6.所有节点安装kubelet kubeadm kubectl

apt-get -y install kubelet=1.23.17-00 kubeadm=1.23.17-00 kubectl=1.23.17-00

7.检查各种组件版本

root@node164:/kube# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:27:46Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
root@node164:/kube# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
root@node164:/kube# kubelet --version
Kubernetes v1.23.17

8.检查时区

 ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 
 ll /etc/localtime 
lrwxrwxrwx 1 root root 33 Feb 10 11:26 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai
date -R
Mon, 10 Feb 2025 11:26:38 +0800

9.基于kubeadm初始化k8s master组件

#使用docker pull 拉取以下镜像
docker images
REPOSITORY                                                                                                               TAG        IMAGE ID       CREATED         SIZE
registry.aliyuncs.com/google_containers/kube-apiserver                            v1.23.17   62bc5d8258d6   23 months ago   130MB
registry.aliyuncs.com/google_containers/kube-controller-manager          v1.23.17   1dab4fc7b6e0   23 months ago   120MB
registry.aliyuncs.com/google_containers/kube-scheduler                            v1.23.17   bc6794cb54ac   23 months ago   51.9MB
registry.aliyuncs.com/google_containers/kube-proxy                                   v1.23.17   f21c8d21558c   23 months ago   111MB
registry.aliyuncs.com/google_containers/etcd                                                3.5.6-0    fce326961ae2   2 years ago     299MB
registry.aliyuncs.com/google_containers/coredns                                           v1.8.6     a4ca41631cc7   3 years ago     46.8MB
registry.aliyuncs.com/google_containers/pause                                                3.6        6270bb605e12   3 years ago     683kB

10.初始化master节点

kubeadm init --kubernetes-version=v1.23.17 --image-repository registry.aliyuncs.com/google_containers  --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16  --service-dns-domain=oldan.cn
...
kubeadm join 10.0.0.164:6443 --token 342pfv.qphyg0g1xuwjwa2z \
	--discovery-token-ca-cert-hash sha256:a6e723d542d323b293154676939388038d7cf8c2c0a9fe917675039e5f8b05fd 
(该密钥是随机生成用于使从节点加入k8s集群)
...

11.拷贝授权文件，用于管理k8s集群

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

12.检查master组件是否正常工作

kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   

kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok                              
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}

13.查看工作节点

kubectl get nodes
NAME        STATUS     ROLES                  AGE     VERSION
node164   NotReady   control-plane,master   3m13s   v1.23.17

kubectl get no -o wide
NAME      STATUS   ROLES                  AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node164   Ready    control-plane,master   63m   v1.23.17   10.0.0.164    <none>        Ubuntu 22.04.4 LTS   5.15.0-140-generic   docker://20.10.24

14.基于kubeadm部署worker组件

#从节点拉取以下镜像
docker images
REPOSITORY                                           TAG               IMAGE ID       CREATED         SIZE
flannel/flannel                                      v0.24.3           f6f0ee58f497   11 months ago   78.6MB
flannel/flannel-cni-plugin                           v1.4.0-flannel1   77c1250c26d9   12 months ago   9.87MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.23.17          f21c8d21558c   23 months ago   111MB
registry.aliyuncs.com/google_containers/coredns      v1.8.6            a4ca41631cc7   3 years ago     46.8MB
registry.aliyuncs.com/google_containers/pause        3.6               6270bb605e12   3 years ago     683kB

15.将worker节点加入到master集群（该密钥为初始化master节点时获得）

[root@node165 ~]# kubeadm join 10.0.0.164:6443 --token 342pfv.qphyg0g1xuwjwa2z \
	--discovery-token-ca-cert-hash sha256:a6e723d542d323b293154676939388038d7cf8c2c0a9fe917675039e5f8b05fd 

[root@node166 ~]# kubeadm join 10.0.0.164:6443 --token 342pfv.qphyg0g1xuwjwa2z \
	--discovery-token-ca-cert-hash sha256:a6e723d542d323b293154676939388038d7cf8c2c0a9fe917675039e5f8b05fd

16.查看节点

kubectl get nodes
NAME        STATUS     	 ROLES 		                 AGE        VERSION
node164   NotReady   control-plane,master   9m57s   v1.23.17
node165   NotReady   <none>         	          93s         v1.23.17
node166   NotReady   <none>                           55s         v1.23.17

 kubectl get no -o wide
NAME        STATUS     ROLES                               AGE             VERSION      INTERNAL-IP   EXTERNAL-IP         OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node164   NotReady   control-plane,master   9m59s        v1.23.17          10.0.0.231         <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic       docker://20.10.24
node165   NotReady   <none>                           95s             v1.23.17          10.0.0.232         <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic       docker://20.10.24
node166   NotReady   <none>                           57s             v1.23.17          10.0.0.233         <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic       docker://20.10.24

17.部署flannel组件

vim kube-flannel。yml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    k8s-app: flannel
    pod-security.kubernetes.io/enforce: privileged
  name: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: flannel
  name: flannel
  namespace: kube-flannel
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: flannel
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: flannel
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-flannel
---
apiVersion: v1
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.100.0.0/16",
      "EnableNFTables": false,
      "Backend": {
        "Type": "vxlan"
      }
    }
kind: ConfigMap
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel-cfg
  namespace: kube-flannel
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel-ds
  namespace: kube-flannel
spec:
  selector:
    matchLabels:
      app: flannel
      k8s-app: flannel
  template:
    metadata:
      labels:
        app: flannel
        k8s-app: flannel
        tier: node
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - args:
        - --ip-masq
        - --kube-subnet-mgr
        command:
        - /opt/bin/flanneld
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        image: ghcr.io/flannel-io/flannel
        name: kube-flannel
        resources:
          requests:
            cpu: 100m
            memory: 50Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
          privileged: false
        volumeMounts:
        - mountPath: /run/flannel
          name: run
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
        - mountPath: /run/xtables.lock
          name: xtables-lock
      hostNetwork: true
      initContainers:
      - args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        command:
        - cp
        image: ghcr.io/flannel-io/flannel-cni-plugin
        name: install-cni-plugin
        volumeMounts:
        - mountPath: /opt/cni/bin
          name: cni-plugin
      - args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        command:
        - cp
        image: ghcr.io/flannel-io/flannel
        name: install-cni
        volumeMounts:
        - mountPath: /etc/cni/net.d
          name: cni
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
      priorityClassName: system-node-critical
      serviceAccountName: flannel
      tolerations:
      - effect: NoSchedule
        operator: Exists
      volumes:
      - hostPath:
          path: /run/flannel
        name: run
      - hostPath:
          path: /opt/cni/bin
        name: cni-plugin
      - hostPath:
          path: /etc/cni/net.d
        name: cni
      - configMap:
          name: kube-flannel-cfg
        name: flannel-cfg
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock

以该文件创建所需资源

kubectl apply -f kube-flannel.yml

18.验证flannel是否成功部署

kubectl get pods -o wide -n kube-flannel
NAME                              READY   STATUS    RESTARTS   AGE         IP               NODE        NOMINATED NODE           READINESS GATES
kube-flannel-ds-flnbm   1/1     Running                0          34s   10.0.0.231   master231            <none>                               <none>
kube-flannel-ds-s4vp6   1/1     Running                0          34s   10.0.0.233   worker233            <none>                               <none>
kube-flannel-ds-x6gv8   1/1     Running                0          34s   10.0.0.232   worker232            <none>                               <none>

19.查看节点信息

kubectl get no -o wide
NAME          STATUS   ROLES                             AGE    VERSION    INTERNAL-IP   EXTERNAL-IP          OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
master231   Ready    control-plane,master   168m   v1.23.17   10.0.0.231             <none>        Ubuntu 22.04.4 LTS      5.15.0-119-generic               docker://20.10.24
worker232   Ready    <none>                           159m   v1.23.17   10.0.0.232            <none>        Ubuntu 22.04.4 LTS      5.15.0-119-generic               docker://20.10.24
worker233   Ready    <none>                           159m   v1.23.17   10.0.0.233             <none>        Ubuntu 22.04.4 LTS     5.15.0-119-generic               docker://20.10.24

20.查看网卡信息

ip a
...
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 12:29:81:7a:44:2e brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::1029:81ff:fe7a:442e/64 scope link 
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ce:41:f3:8e:5e:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.1/24 brd 10.100.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::cc41:f3ff:fe8e:5eb8/64 scope link 
       valid_lft forever preferred_lft forever

21.其他节点并不具有cni网卡

---> 假设 master231的flannel.1是10.100.0.0网段。
ip link add cni0 type bridge
ip link set dev cni0 up
ip addr add 10.100.0.1/24 dev cni0

---> 假设 worker233的flannel.1是10.100.2.0网段。
ip link add cni0 type bridge
ip link set dev cni0 up
ip addr add 10.100.2.1/24 dev cni0

22.验证pod的flannel网络是否正常

cat > test.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: xiuxian-v1
spec:
  nodeName: node165
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1 
    name: xiuxian

---

apiVersion: v1
kind: Pod
metadata:
  name: xiuxian-v2
spec:
  nodeName: node166
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
    name: xiuxian
EOF

23.kubectl apply -f test.yaml

kubectl apply -f test.yaml
pod/xiuxian-v1 created
pod/xiuxian-v2 created

24.查看pod创建情况

kubectl get pods -o wide 
NAME         READY    STATUS    RESTARTS   AGE                IP                 NODE            NOMINATED NODE        READINESS GATES
xiuxian-v1    1/1       Running            0             14s          10.100.1.4     node165              <none>                              <none>
xiuxian-v2    1/1       Running            0             14s          10.100.2.2     node166              <none>                              <none>

curl 10.100.1.4
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v1</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: green">凡人修仙传 v1 </h1>
    <div>
      <img src="1.jpg">
    <div>
  </body>

</html>

25.配置环境变量

kubectl completion bash > ~/.kube/completion.bash.inc
echo source '$HOME/.kube/completion.bash.inc' >> ~/.bashrc 
source ~/.bashrc

26.增加自动补全功能

kubectl # 连续按2次tab键测试能否出现命令
alpha          auth           cordon         diff           get            patch          run            version
annotate       autoscale      cp             drain          help           plugin         scale          wait
api-resources  certificate    create         edit           kustomize      port-forward   set            
api-versions   cluster-info   debug          exec           label          proxy          taint          
apply          completion     delete         explain        logs           replace        top            
attach         config         describe       expose         options        rollout        uncordon

27.查看组件是否正常

root@node164:~# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok                              
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}

28.查看节点状态是否正常

root@node164:~# kubectl get nodes 
NAME      STATUS   ROLES                  AGE   VERSION
node164   Ready    control-plane,master   84m   v1.23.17
node165   Ready    <none>                 79m   v1.23.17
node166   Ready    <none>                 79m   v1.23.17

29.查看flannel组件是否正常

root@node164:~# kubectl get pods -n kube-flannel -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
kube-flannel-ds-9rnnw   1/1     Running   0          62m   10.0.0.165   node165   <none>           <none>
kube-flannel-ds-dzsd4   1/1     Running   0          62m   10.0.0.164   node164   <none>           <none>
kube-flannel-ds-h82k6   1/1     Running   0          62m   10.0.0.166   node166   <none>           <none>

至此，自用kubernetes集群配置完成。网络组件flannel适用于管理500台节点以下的集群环境，如果节点更多请使用calico网络组件。

Kubernetes服务类型
Neutron基本架构与数据流走向

Comment