云计算-使用Kubeadm在阿里云建单Master多Node的K8S
机器搭建阿里云
- 阿里云选择云服务器 ECS
- 单击管理控制台
- 创建我的ECS
- 开始创建，然后选择，其他的可以自己选择
机器环境配置
| ip | 节点 | hostname |
|---|---|---|
| 8.134.74.86 172.17.75.165 | master | kubeadm001 |
| 8.134.83.65 172.17.75.163 | node | kubeadm002 |
| 8.134.86.26 172.17.75.164 | node | kubeadm003 |
安装可能使用的基本包
yum install -y openssh-clients openssh-clients ntpdate yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet ipvsadm 修改主机hosts文件
修改每台机器/etc/hosts主要添加三行文件
172.17.75.162 kubeadm001 kubeadm001 172.17.75.160 kubeadm002 kubeadm002 172.17.75.161 kubeadm003 kubeadm003 主机间免密登录
- 输入ssh-keygen,不输入任何密码回车
ssk-keygen - 将本地生成的密钥文件复制到远程主机kubeadm002和kubeadm003上面
ssh-copy-id kubeadm002 ssh-copy-id kubeadm003
 ### 关闭交换区，提高性能 swapoff -a
#### 为何关闭交换区？ Swap它是交换区。当机器内存不足时，它会适应Swap，但是Swap性能不是很好，K8S默认不允许使用，以提高性能Swap分区。Kubeadm初始化时会进行检测Swap是否关闭，如果不关闭，初始化将失败。**如果不想关闭交换分区，安装k8s可指定时间-- ignore -preflight-errors=Swap**解决。 ### 修改机器内核参数 modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/profile cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF
sysctl -p /etc/sysctl.d/k8s.conf ``` ### 关闭防火墙 ``` systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld ```  若不习惯使用firewalld,可以安装`iptables`，在三台机器上做以下操作 * 安装`iptables`基础包 ``` yum install iptables-services -y ``` * 关闭`iptables` ``` service iptables stop && systemctl disable iptabl ``` * 清空防火墙规则 ``` iptables -F ``` ### 关闭selinux 使用`getenforce` 查看selinux是否关闭 ``` [root@kubeadm003 ~]# getenforce ``` **假如结果不是**Disabled,关闭以下命令selinux ``` sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config ``` **注**：修改selinux需要重启机器。 ### 时间同步 将三台机器上的时间与网络时间同步 * 同步机器时间和网络时间 ``` ntpdate cn.pool.ntp.org ``` * 每小时和网络时间同步一次定时任务 ``` crontab -e ``` * 定期添加任务 ``` * */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org ``` * 重启定时任务 ``` systemctl start crond ``` ### 开启ipvs * 在`/etc/sysconfig/modules`目录下添加ipvs.modules文件内容如下 ``` vi /etc/sysconfig/modules/ipvs.modules #!/bin/bash ipvs_modules=“ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack” for kernel_module in ${ipvs_modules}; do /sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1 if [ 0 -eq 0 ]; then /sbin/modprobe ${kernel_module} fi done
* 将`ipvs.modules`添加为可执行文件，并执行文件和查看执行情况 chmod x /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
#### 什么是ipvs
ipvs(IP Virtual Server)实现的是传输层负载均衡，是Linux内核的一部分。ipvs运行在宿主机上，充当负载均衡器。
### 配置K8S组件需要的repo源
* 使用vi 添加/etc/yum.repos.d/kubernetes.repo
vi /etc/yum.repos.d/kubernetes.repo
文件内容如下
[kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0
### 安装 docker 服务
#### 旧版本的卸载
在安装新的docker的时候，机器可能在以前安装过docker旧版本，我们需要将其进行卸载：
yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
#### 镜像仓的设置
一般我们在国内的话使用国内的镜像仓，我一般使用阿里的镜像仓
使用阿里国内源安装docker yum-config-manager –add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo sed -i ‘s+download.docker.com+mirrors.aliyun.com/docker-ce+’ /etc/yum.repos.d/docker-ce.repo
#### 更新并安装Docker-ce
docker-ce 社区版 , docker-ee企业版， 一般安装社区版，企业版需要授权
#更新并安装Docker-CE sudo yum makecache fast yum install -y docker-ce docker-ce-cli containerd.io
#### 启动安装并查看Docker状态
systemctl start docker && systemctl enable docker && systemctl status docker
#### 配置 docker 镜像加速器和驱动
vi /etc/docker/daemon.json
* 本文添加了多个镜像加速器，包含阿里镜像加速器、163镜像加速器等加速器
{ “registry-mirrors”:[“https://b5b7g6yt.mirror.aliyuncs.com”,“https://registry.docker-cn.com”,“https://docker.mirrors.ustc.edu.cn”,“https://dockerhub.azk8s.cn”,“http://hub-mirror.c.163.com”,“http://qtid6917.mirror.aliyuncs.com”,“https://rncxm540.mirror.aliyuncs.com”], “exec-opts”: [“native.cgroupdriver=systemd”] }
* 阿里云加速器每一个人的都不一样,添加如下
* 添加以后重启`docker`和`daemon-reload`
systemctl daemon-reload && systemctl restart docker && systemctl status docker
k8s安装
-----
### 安装初始化 k8s 需要的软件包
* 安装`kubelet`、`kubeadm`、`kubectl`
yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6
* 将`kubelet`设置为开启自启动以及查看`kubelet`状态
systemctl enable kubelet && systemctl status kubelet
现在看起来`kubelet`还不是`running`状态，现在这个状态是正常的，k8s正常启动启动以后，状态就正常了
**注**：
* `kubeadm`:`kubeadm`是一个工具,用于初始化k8s集群
* `kubelet`：`kubelet`**需要安装在集群的所有结点，作用是启动Pod**
* `kubectl`: `kubectl`可以部署和管理应用的工具
### 使用kubeadm初始化k8s集群
**这个操作只需要在master节点上操作就可以了**，本次搭建k8s的master内网节点是172.17.75.165
* \--kubernetes-version=1.20.6 表示版本
* \--apiserver-advertise-address=172.17.75.165 表示master节点在阿里云的内网IP
* \--image-repository registry.aliyuncs.com/google\_containers 指定镜像仓的地址
kubeadm init --kubernetes-version=1.20.6 --apiserver-advertise-address=172.17.75.165 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=SystemVerification
kubeadm默认拉取镜像的地址是k8s.grc.io，但是在国内访问效果差，所以设置为国内的镜像仓效果会更好。
* 如下说明在master上安装成功
* 配置kubectl的配置文件config就是上图标出的文件
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown ( i d − u ) : (id -u): (id−u):(id -g) $HOME/.kube/config
* 配置好以后查看nodes
kubectl get nodes
* 使用`kubeadm token create`命令生成添加节点的指令
kubeadm token create --print-join-command
**生成的指令如下，每次生成可能都会不同**
kubeadm join 172.17.75.165:6443 --token vsnwiz.9tdg2hklcmx26gbb --discovery-token-ca-cert-hash sha256:03e79bdddf8a15be78cfefb717275fda128de570bec60a1a332719c94bedc5c6
**我们需要将上面的`kubeadm join`指令在kubeadm002和kubeadm003两台机器上运行**
* 再次查看nodes节点
kubectl get nodes
我们发现新添加的node节点的**ROLES**是空的，也表示它是工作节点，但是我们可以将变成work
kubectl label node kubeadm002 node-role.kubernetes.io/worker=worker kubectl label node kubeadm003 node-role.kubernetes.io/worker=worker
然后再次查看结果变为
[root@kubeadm001 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubeadm001 NotReady control-plane,master 14m v1.20.6 kubeadm002 NotReady worker 5m48s v1.20.6 kubeadm003 NotReady worker 5m44s v1.20.6
**上面的STSTUS中的NotReady表示没有安装网卡插件**
### 安装k8s网络组件-Calico
链接：https://pan.baidu.com/s/1QMAAh6iL-jbke\_lM0dZxBg 提取码：1234 编写一个`calico.yaml`，我将它上传到百度云盘，可以自行下载，也可以到https://docs.projectcalico.org/manifests/calico.yaml进行下载
kubectl apply -f calico.yaml kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
**需要等待一会**
* 查看kube-system环境下的集群状态
kubectl get pod -n kube-system
* 再次查看k8s的集群状态
kubectl get nodes
我们发现现在集群的STATUS的状态变成了**Ready**,说明集群已经正常运行
#### 测试k8s创建pod是否可以正常访问网络
创建一个busybox容器,并进入到容器内部
kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox – sh
然后在里面输入`ping www.baidu.com`
通过ping我们发现网络可以访问，说明Calico安装正常
#### 测试coredns是否正常
还是上面的busybos容器,输入以下命令
nslookup kubernetes.default.svc.cluster.local
结果为以下，10.96.0.10就是我们coreDNS的clusterIP，说明coredns正常
Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default.svc.cluster.local Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
### k8s集群中安装tomcat服务
* 创建一个`tomcat.yaml`
vi tomcat.yaml
apiVersion: v1 #pod属于k8s核心组v1 kind: Pod #创建的是一个Pod资源 metadata: #元数据 name: tomcat-pod #pod名字 namespace: default #pod所属的名称空间 labels: app: myapp #pod具有的标签 env: dev #pod具有的标签 spec: containers: #定义一个容器，容器是对象列表，下面可以有多个name - name: tomcat-pod-test #容器的名字 ports: - containerPort: 8080 image: tomcat:8.5-jre8-alpine #容器使用的镜像 imagePullPolicy: IfNotPresent ``` * 使用kubect部署tomcat ``` kubectl apply -f tomcat.yaml ``` * 查看pods ``` [root@kubeadm001 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-pod 1/1 Running 0 58s ``` * 创建tomcat的Service， `tomcat-service.yaml` ``` vi tomcat-service.yaml ``` 内容如下 ``` apiVersion: v1 kind: Service metadata: name: tomcat spec: type: NodePort ports: - port: 8080 nodePort: 30080 selector: app: myapp env: dev ``` * 运行tomcat的`service` ``` kubectl apply -f tomcat-service.yaml ``` * 查看service的状态 ``` kubectl get svc ```  在浏览器中http://8.134.58.124:30080/ （8.134.58.124表示master节点）  安装k8s可视化UI界面dashboard ------------------------ * 创建kubernetes-dashboard.yaml文件 ``` # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. apiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard --- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard --- kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf: "" --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque --- kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.0.0-beta8 imagePullPolicy: IfNotPresent ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kubernetes-dashboard # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "beta.kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper --- kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' spec: containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.1 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "beta.kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {} ``` * 安装 kubernetes-dashboard ``` kubectl apply -f kubernetes-dashboard.yaml ``` * 查看dashboard的pod状态(需要等待一点时间) ``` kubectl get pods -n kubernetes-dashboard ``` 查看结果为如下，说明安装成功 ``` NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-778b77d469-r6cwm 1/1 Running 0 2m kubernetes-dashboard-648b6964bf-gnbw4 1/1 Running 0 2m ``` * 查看dashboard的前端service ``` kubectl get svc -n kubernetes-dashboard ``` 结果为 ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.97.73.249 <none> 8000/TCP 2m57s kubernetes-dashboard ClusterIP 10.106.69.73 <none> 443/TCP 2m57s ``` 我们发现kubernetes-dashboard的TYPE状态ClusterIP,我们需要将其修改为**NodePort**状态, ``` kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard ```  * 再次查看 kubernetes-dashboard的svc ``` [root@kubeadm001 ~]# kubectl get svc -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.97.73.249 <none> 8000/TCP 7m31s kubernetes-dashboard NodePort 10.106.69.73 <none> 443:31284/TCP 7m31s ``` 查看到端口是31284,然后使用任何一个node:31284进行查看  可看到出现了dashboard界面  #### 通过token令牌访问dashboard * 创建管理员token，具有查看任何空间的权限，可以管理所有资源对象 ``` kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard ``` * 查看kubernetes-dashboard名称空间下的secret ``` kubectl get secret -n kubernetes-dashboard ```  找到对应的带有oken的kubernetes-dashboard-token-66r22 ，**每次kubernetes-dashboard-token的name不同** ``` kubectl describe secret kubernetes-dashboard-token-9mwrn -n kubernetes-dashboard ```    #### 通过 kubernetes-dashboard 创建容器 打开 kubernetes 的 dashboard 界面点开右上角红色箭头标注的 “+”，如下图所示：  选择Create from form  **在dashboard的左侧选择Services**  **上图可看到刚才创建的nginx的service在宿主机映射的端口是32545，在浏览器访问：http://8.134.74.86:32545/**  **错误解决：**可能会出现dashboard打不开的情况，在谷歌浏览器，点击属性，在尾部添加\--disable-infobars --ignore-certificate-errors 