• 嗯，整理K8s中pod在此与小伙伴分享调度相关笔记
• 博文内容包括：
  • kube-scheduler组件的简述
  • Pod调度(选择器、指定节点、主机亲和力)
  • 节点的coedon与drain标记
  • 节点的taint(污点)标记及pod的容忍污点(tolerations)定义
• 食用方式：
  • 需要了解K8s基础知识
  • 熟悉资源对象pod，deploy了解资源对象的定义yaml文件
  • 了解kubectl常用命令
• 理解不足的小伙伴帮忙指正

所谓成功，就是用自己的方式度过人生。-明朝的那些事

众多周知，Kubernetes Scheduler 是 Kubernetes中负责Pod调度运行在重要功能模块中的重要功能模块k8s 集群中的master节点。

作用 : Kubernetes Scheduler待调度的作用Pod (API新创建的Pod, Controller Manager为创建补充副本的Pod等)根据具体情况调度算法和调度策略绑定(Binding)适合集群Node写入绑定信息etcd中。

整个调度过程涉及三个对象，即

• 待调度Pod列表
• 可用Node列表
• 以及调度算法和策略

整体流程 :通过调度算法进行调度待调度Pod列表中的每个Pod从Node列表中选择一个最适合的Node随后, 目标node节点上的kubelet通过APIServer监听到Kubernetes Scheduler产生的Pod绑定事件,然后获得相应的Pod清单,下载Image镜像并启动容器。

kubelet进程通过与API Server每隔一个时间周期，互动就会被调用一次API Server的REST接口报告自身状态， API Server接收到此信息后，更新节点状态信息etcd中。 同时kubelet也通过API Server的Watch接口监听Pod信息,

• 假如监听到新的Pod如果将副本绑定到本节点，则执行Pod创建和启动相应容器的逻辑；
• 如果监听到Pod如果对象被删除，则在本节点上删除相应的对象Pod容器;
• 若监听到修改Pod信息,则kubelet听到变化后，节点将相应修改Pod容器。

所以说，kubernetes Schedule 承担整个系统承上启下的重要功能负责接收声明API或者创建新的控制器pod并为其安排适当的消息Node，对下，选择好node之后，交接工作node上的kubelet，由kubectl负责pod剩余生命周期。

Kubernetes Scheduler目前提供的默认调度流程分为以下两个步骤。这部分随着版本的变化而变化，小伙伴主要是官网

Kubernetes Scheduler通过插件加载调度过程"调度算法提供者” (AlgorithmProvider)具体实现AlgorithmProvider其实包括一组预选策略与一组优先选择策略的结构体.

Scheduler中可用的预选策略包含：NoDiskConflict、PodFitsResources、PodSelectorMatches、PodFitsHost、CheckNodeLabelPresence、CheckServiceAffinity 和PodFitsPorts策略等。

其默认的AlgorithmProvider返回布尔值的预选策略包括：

通过给node设置指定的标签，然后我们就可以创建了pod选择通过选择器选择node标签类似于前端DOM操作元素定位，或直接指定节点名

常用的节点标签命令

给节点设置标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl label node vms82.liruilongs.github.io disktype=node1
node/vms82.liruilongs.github.io labeled
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl label node vms83.liruilongs.github.io disktype=node2
node/vms83.liruilongs.github.io labeled

查看节点标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl get node --show-labels
NAME                         STATUS   ROLES                  AGE   VERSION   LABELS
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms81.liruilongs.github.io,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
vms82.liruilongs.github.io   Ready    <none>                 45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node1,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms82.liruilongs.github.io,kubernetes.io/os=linux
vms83.liruilongs.github.io   Ready    <none>                 45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node2,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms83.liruilongs.github.io,kubernetes.io/os=linux
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

特殊的内置标签node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,用于设置角色列roles

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl get node
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2
vms82.liruilongs.github.io   Ready    <none>                 45d   v1.22.2
vms83.liruilongs.github.io   Ready    <none>                 45d   v1.22.2

我们也可以做worker节点上设置标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl label nodes vms82.liruilongs.github.io node-role.kubernetes.io/worker1=
node/vms82.liruilongs.github.io labeled
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl label nodes vms83.liruilongs.github.io node-role.kubernetes.io/worker2=
node/vms83.liruilongs.github.io labeled

查看标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl get node
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2
vms82.liruilongs.github.io   Ready    worker1                45d   v1.22.2
vms83.liruilongs.github.io   Ready    worker2                45d   v1.22.2
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$

在特定节点上运行pod,给vms83.liruilongs.github.io节点打disktype=node2的标签，yaml文件选择器指定对应的标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes -l disktype=node2
NAME                         STATUS   ROLES     AGE   VERSION
vms83.liruilongs.github.io   Ready    worker2   45d   v1.22.2
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$vim pod-node2.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply  -f pod-node2.yaml
pod/podnode2 created

pod-node2.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnode2
  name: podnode2
spec:
  nodeSelector:
    disktype: node2
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnode2
    resources: { 
        }
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: { 
        }

pod被调度到了vms83.liruilongs.github.io

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
podnode2   1/1     Running   0          13m   10.244.70.60     vms83.liruilongs.github.io   <none>           <none>

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$vim pod-node1.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node1.yaml
pod/podnode1 created

pod-node1.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnode1
  name: podnode1
spec:
  nodeName: vms82.liruilongs.github.io
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnode1
    resources: { 
        }
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: { 
        }

需要注意 当pod资源文件指定的节点标签,或者节点名不存在时，这个pod资源是无法创建成功的

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
podnode1   1/1     Running   0          36s   10.244.171.165   vms82.liruilongs.github.io   <none>           <none>
podnode2   1/1     Running   0          13m   10.244.70.60     vms83.liruilongs.github.io   <none>           <none>

所谓主机亲和性，即在满足指定条件的节点上运行。分为硬策略(必须满足)，软策略(最好满足)

所调度节点的标签必须为其中的一个，这个标签是一个默认标签，会自动添加

  affinity:
    nodeAffinity: #主机亲和性
      requiredDuringSchedulingIgnoredDuringExecution: #硬策略
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node-a.yaml
pod/podnodea created

pod-node-a.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnodea
  name: podnodea
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnodea
    resources: { 
        }
  affinity:
    nodeAffinity: #主机亲和性
      requiredDuringSchedulingIgnoredDuringExecution: #硬策略
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: { 
        }

条件不满足，所以 Pending

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
podnodea   0/1     Pending   0          8s

我修改一下，修改为存在的标签

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$sed -i  's/vms84.liruilongs.github.io/vms83.liruilongs.github.io/' pod-node-a.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node-a.yaml
pod/podnodea created

可以发现pod调度成功

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
podnodea   1/1     Running   0          13s   10.244.70.61   vms83.liruilongs.github.io   <none>           <none>
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$

所调度节点尽量为其中一个

  affinity:
    nodeAffinity: #主机亲和性
      preferredDuringSchedulingIgnoredDuringExecution: # 软策略
      - weight: 2
        preference:
          matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$vim pod-node-a-r.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f  pod-node-a-r.yaml
pod/podnodea created

pod-node-a-r.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnodea
  name: podnodea
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnodea
    resources: { 
        }
  affinity:
    nodeAffinity: #主机亲和性
      preferredDuringSchedulingIgnoredDuringExecution: # 软策略
      - weight: 2
        preference:
          matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: { 
        }

检查一下,调度OK

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
podnodea   1/1     Running   0          28s   10.244.70.62   vms83.liruilongs.github.io   <none>           <none>

常见的标签运算符

如果想把某个节点设置为不可用的话，可以对节点实施cordon或者drain

如果一个node被标记为cordon，新创建的pod不会被调度到此node上，已经调度上去的不会被移走，coedon用于节点的维护，当不希望再节点分配pod，那么可以使用coedon把节点标记为不可调度。

这里我们为了方便，创建一个Deployment控制器用去用于演示，deploy拥有3个pod副本

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl create deployment nginx  --image=nginx  --dry-run=client -o yaml >nginx-dep.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$vim nginx-dep.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl apply  -f nginx-dep.yaml
deployment.apps/nginx created

nginx-dep.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  strategy: { 
        }
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        imagePullPolicy: IfNotPresent
        resources: { 
        }
status: { 
        }

可以看到pod调度到了两个工作节点

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE
      NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-hx96s   1/1     Running   0          2m16s   10.244.171.167   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-wshxp   1/1     Running   0          2m16s   10.244.70.1      vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-x78x4   1/1     Running   0          2m16s   10.244.70.63     vms83.liruilongs.github.io   <none>           <none>
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$

如果一个节点被设置为drain，则此节点不再被调度pod，且此节点上已经运行的pod会被驱逐(evicted)到其他节点

drain包含两种状态：cordon不可被调度，evicted驱逐当前节点所以pod

kubectl drain vms83.liruilongs.github.io   --ignore-daemonsets
kubectl uncordon vms83.liruilongs.github.io  

通过deployment添加4个nginx副本--replicas=4

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx --replicas=4
deployment.apps/nginx scaled
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide --one-output
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-2clnb   1/1     Running   0          22s   10.244.171.174   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-9p6g2   1/1     Running   0          22s   10.244.70.2      vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-ptqxm   1/1     Running   0          22s   10.244.171.173   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-zmdqm   1/1     Running   0          22s   10.244.70.4      vms83.liruilongs.github.io   <none>           <none>

添加一下drain 将节点vms82.liruilongs.github.io设置为drain

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl drain vms82.liruilongs.github.io --ignore-daemonsets --delete-emptydir-data
node/vms82.liruilongs.github.io cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-ntm7v, kube-system/kube-proxy-nzm24
evicting pod liruilong-pod-create/nginx-7cf7d6dbc8-ptqxm
evicting pod kube-system/metrics-server-bcfb98c76-wxv5l
evicting pod liruilong-pod-create/nginx-7cf7d6dbc8-2clnb
pod/nginx-7cf7d6dbc8-2clnb evicted
pod/nginx-7cf7d6dbc8-ptqxm evicted
pod/metrics-server-bcfb98c76-wxv5l evicted
node/vms82.liruilongs.github.io evicted

查看节点，状态为不可被调度

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS                     ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready                      control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready,SchedulingDisabled   worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready                      worker2                48d   v1.22.2

等一会，查看节点调度，所有pod调度到了vms83.liruilongs.github.io这台机器