kubeadm集群升级后，coredns的状态ContainerCreating
在使用kubeadm升级集群，版本升级跨度大，升级后发现coredns的pod状态一直为ContainerCreating。
[root@master kubernetes]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-64897985d-4zflq 0/1 ContainerCreating 0 11m coredns-64897985d-mzblm 0/1 ContainerCreating 0 11m etcd-master 1/1 Running 308 (2d16h ago) 20d kube-apiserver-master 1/1 Running 3 (2d16h ago) 20d kube-controller-manager-master 1/1 Running 12 (2d16h ago) 20d kube-proxy-4d46k 1/1 Running 1 (2d16h ago) 20d kube-proxy-s8gnc 1/1 Running 0 20d kube-scheduler-master 1/1 Running 11 (2d16h ago) 20d 用命令查看错误信息
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m2s default-scheduler Successfully assigned kube-system/coredns-64897985d-4zflq to master Warning FailedCreatePodSandBox 9m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "52e84109e69f863246b0c7b15acddf73d163722ac524e85a44dae15bfdd090a8" network for pod "coredns-64897985d-4zflq": networkPlugin cni failed to set up pod "coredns-64897985d-4zflq_kube-system" network: error getting ClusterInformation: Get "https://[10.1.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.1.0.1:443: connect: connection refused, failed to clean up sandbox container "52e84109e69f863246b0c7b15acddf73d163722ac524e85a44dae15bfdd090a8" network for pod "coredns-64897985d-4zflq": networkPlugin cni failed to teardown pod "coredns-64897985d-4zflq_kube-system" network: error getting ClusterInformation: Get "https://[10.1.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.1.0.1:443: connect: connection refused] Normal SandboxChanged 24s (x25 over 8m59s) kubelet Pod sandbox changed, it will be killed and re-created. 错误信息一直被提示创建sanbox失败， networkPlugin cni failed to set up pod “coredns-64897985d-4zflq_kube-system” network: error getting ClusterInformation: Get “https://[10.1.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.1.0.1:443: connect: connection refused 原来是network的问题。kubectl get pod -n kube-system 发现没有calico的相关pod信息。 安装calico 可根据不同场景到达calico相应的官网下载yaml文件。 https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises
[root@master ~]# curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 212k 100 212k 0 0 153k 0 0:00:01 0:00:01 --:--:-- 153k
[root@master k8s]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created
再一次查看pod状态
[root@master ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-566dc76669-hlt7c 0/1 ContainerCreating 0 89s
calico-node-2h59l 0/1 PodInitializing 0 89s
calico-node-j28bl 0/1 Init:2/3 0 89s
coredns-64897985d-4zflq 0/1 ContainerCreating 0 89m
coredns-64897985d-mzblm 0/1 ContainerCreating 0 89m
etcd-master 1/1 Running 308 (2d17h ago) 20d
kube-apiserver-master 1/1 Running 3 (2d17h ago) 20d
kube-controller-manager-master 1/1 Running 12 (2d17h ago) 20d
kube-proxy-4d46k 1/1 Running 1 (2d17h ago) 20d
kube-proxy-s8gnc 1/1 Running 0 20d
kube-scheduler-master 1/1 Running 11 (2d17h ago) 20d
最终状态均为running，修复完毕
[root@master ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-566dc76669-hlt7c 1/1 Running 0 9m54s 192.168.166.131 node1 <none> <none>
calico-node-2h59l 1/1 Running 0 9m54s 172.18.239.59 node1 <none> <none>
calico-node-j28bl 1/1 Running 0 9m54s 172.18.239.40 master <none> <none>
coredns-64897985d-4zflq 1/1 Running 0 98m 192.168.219.64 master <none> <none>
coredns-64897985d-mzblm 1/1 Running 0 98m 192.168.166.129 node1 <none> <none>
etcd-master 1/1 Running 308 (2d17h ago) 20d 172.18.239.40 master <none> <none>
kube-apiserver-master 1/1 Running 3 (2d17h ago) 20d 172.18.239.40 master <none> <none>
kube-controller-manager-master 1/1 Running 12 (2d17h ago) 20d 172.18.239.40 master <none> <none>
kube-proxy-4d46k 1/1 Running 1 (2d17h ago) 20d 172.18.239.40 master <none> <none>
kube-proxy-s8gnc 1/1 Running 0 20d 172.18.239.59 node1 <none> <none>
kube-scheduler-master 1/1 Running 11 (2d17h ago) 20d 172.18.239.40 master <none> <none>