系列文章目录
第一章：? k8s入门:裸机部署 k8s 集群 第二章：? k8s入门:部署应用 k8s 集群 第三章：? k8s入门：service 简单使用 第四章：? k8s入门：StatefulSet 简单使用 第五章：? k8s入门：存储（storage） 第六章：? K8S 配置 storageclass 使用 nfs 动态应用本地磁盘空间 第七章：? k8s入门：配置 ConfigMap & Secret 第八章：? k8s入门：k8s入门：Helm 构建 MySQL 第九章：? k8s入门：kubernetes-dashboard 安装 第十章：? k8s入门：kube-prometheus-stack 全家桶搭建（Grafana Prometheus）
文章目录
-
-
-
- 系列文章目录
- 一、简介
- 二、配置文件
- 三、rbac.yaml
- 四、deployment.yaml
- 五、class.yaml
- 六、test-claim.yaml
- 七、test-pod.yaml
- 八、新增动态 PV
-
-
一、简介
本文使用 PersistentVolumeClaim （pvc） 结合资源对象 StorageClass （SC） 去动态申领 PersistentVolume （PV）本地 nfs 共享文件系统存储空间，SC 根本作用是基础 pvc 动态创建定义pv，不仅节省了我们管理员的时间，也可以包装不同类型的存储供应 pvc 选用。
每个 StorageClass 它们都包含以下三个重要的字段，它们将在sc需要动态分配pv时用：
-
每个 StorageClass 都有制备器（Provisioner），用于决定使用哪个卷插件 PV。 必须指定字段。
-
回收策略（reclaimPolicy）可以是 Delete 或者 Retain，如果 StorageClass 创建对象时没有指定 reclaimPolicy，它将默认为 Delete。
-
PersistentVolume 可配置为可扩展。将此功能设置为 true 时，允许用户通过编辑相应的 PVC 调整卷的大小。
注意： 首先，确保你的 nfs 共享文件系统可用，未安装可参考 https://blog.csdn.net/qq_41538097/article/details/125215989 步骤七
可参考配置文件：https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client/deploy StorageClass 详细使用可参考： https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes
二、配置文件
可以从链接中获得配置文件 https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes
也可以命令拉下整个项目git clone https://github.com/kubernetes-incubator/external-storage.git
我的配置文件如下
[root@master nfs]# pwd /root/StorageClass/nfs [root@master nfs]# ls class.yaml deployment.yaml rbac.yaml test-claim.yaml test-pod.yaml 三、rbac.yaml
创建 rbac配置文件(基于角色访问控制) rbac.yaml，就是用户通过角色与权限进行关联。
apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", , "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io 部署 rbac 配置文件
[root@master nfs]# kubectl apply -f rbac.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
四、deployment.yaml
新增 nfs 的 Deployment 配置 deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-diy # StorageClass 三个重要字段之一 Provisioner，名字自己指定
- name: NFS_SERVER
value: 192.168.25.100 # nfs 服务器地址
- name: NFS_PATH
value: /home/data # nfs 共享文件夹
volumes:
- name: nfs-client-root
nfs:
server: 192.168.25.100 # nfs 服务器地址
path: /home/data # nfs 共享文件夹
部署 nfs 的 Deployment
[root@master nfs]# kubectl apply -f deployment.yaml
deployment.apps/nfs-client-provisioner created
[root@master nfs]# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nfs-client-provisioner 1/1 1 1 10s
[root@master nfs]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nfs-client-provisioner-6b4c9588c6-9hs2q 1/1 Running 0 23s 10.244.2.46 node2 <none> <none>
五、class.yaml
定义了 存储类 class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-storage
provisioner: nfs-diy # 和 deployment.yaml 中 env.PROVISIONER_NAME 保持一致
reclaimPolicy: Retain
parameters:
archiveOnDelete: "false"
部署 SC
[root@master nfs]# kubectl apply -f class.yaml
storageclass.storage.k8s.io/nfs-storage created
[root@master nfs]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-storage nfs-diy Retain Immediate false 15s
六、test-claim.yaml
使用 PVC 申领本地磁盘
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
annotations:
volume.beta.kubernetes.io/storage-class: "nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
部署 PVC
[root@master nfs]# kubectl apply -f test-claim.yaml
persistentvolumeclaim/test-claim created
[root@master nfs]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Pending nfs-storage 9s
[root@master nfs]# kubectl describe pvc/test-claim
Name: test-claim
Namespace: default
StorageClass: nfs-storage
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-class: nfs-storage
volume.beta.kubernetes.io/storage-provisioner: nfs-diy
volume.kubernetes.io/storage-provisioner: nfs-diy
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 6s (x3 over 15s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator
如上可发现创建失败，报错 waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator
注意：如果是 v1.20 版本以上 apiserver 默认禁止使用 selfLink，需要手动配置- --feature-gates=RemoveSelfLink=false 开启。
修改kube-apiserver.yaml 文件：
[root@master nfs]# cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.25.100:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.25.100
- --feature-gates=RemoveSelfLink=false # 添加这条信息
- --allow-privileged=true
systemctl restart kubelet重启 kubelet 服务，然后查看 PVC，PV 信息，可以看到 PVC 申领 PV 卷成功。 注意： 如果不成功，仍然报这个错
- 可以使用
kubectl get pod -n kube-system查看kube-apiserver是否启动成功 - 查看上面 deployment 创建的 pod 是否启动失败
[root@master nfs]# systemctl restart kubelet
[root@master nfs]# kubectl describe pvc/test-claim
Name: test-claim
Namespace: default
StorageClass: nfs-storage
Status: Bound
Volume: pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-class: nfs-storage
volume.beta.kubernetes.io/storage-provisioner: nfs-diy
volume.kubernetes.io/storage-provisioner: nfs-diy
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Mi
Access Modes: RWX
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 12s (x5 over 72s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "nfs-diy" or manually created by system administrator
Normal Provisioning 5s nfs-diy_nfs-client-provisioner-5c58f4c86f-ndffx_15778b3c-f0a6-11ec-894c-ceb09bae6415 External provisioner is provisioning volume for claim "default/test-claim"
Normal ProvisioningSucceeded 5s nfs-diy_nfs-client-provisioner-5c58f4c86f-ndffx_15778b3c-f0a6-11ec-894c-ceb09bae6415 Successfully provisioned volume pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
[root@master nfs]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Bound pvc-c2ae61f8-7c40-4393-9110-114a0455afa1 1Mi RWX nfs-storage 107s
[root@master nfs]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-c2ae61f8-7c40-4393-9110-114a0455afa1 1Mi RWX Retain Bound default/test-claim nfs-storage 109s
[root@master nfs]# ls /home/data
default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
PVC 动态持久卷在 nfs 服务器持久卷路径被配置为 共享文件夹目录/${namespace}-${pvcName}-${pvName}，比如：/home/data/default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1
七、test-pod.yaml
创建 pod 测试
kind: Pod
apiVersion: v1
metadata:
name: test-pod-nginx
spec:
containers:
- name: test-pod-nginx-containers
image: nginx:latest
ports:
- containerPort: 80
name: "http-server"
hostPort: 80
volumeMounts:
- name: test-pod-nginx-storage
mountPath: "/usr/share/nginx/html"
restartPolicy: "Never"
volumes:
- name: test-pod-nginx-storage
persistentVolumeClaim:
claimName: test-claim
部署 pod
[root@master nfs]# kubectl apply -f test-pod.yaml
pod/test-pod-nginx created
[root@master nfs]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-pod-nginx 1/1 Running 0 33s 10.244.1.46 node1 <none> <none>
[root@master data]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-c2ae61f8-7c40-4393-9110-114a0455afa1 1Mi RWX Retain Bound default/test-claim nfs-storage 13m
[root@master nfs]# echo "K8S configures storageclass to dynamically claim local disk space using nfs" > /home/data/default-test-claim-pvc-c2ae61f8-7c40-4393-9110-114a0455afa1/index.html
因为上述 pod 部署到了 node1 节点（192.168.1.101），访问 http://192.168.25.101/ 测试成功
八、动态新增 PV
test-claim2.yaml，通过 nfs-storage 动态申领 100M PV 本地存储
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim2
annotations:
volume.beta.kubernetes.io/storage-class: "nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Mi
部署 PVC，可以看到申请的 PV 卷名字 pvc-9385f11d-2587-469b-bd13-3a756fe61acb，大小为 100 M，挂载目录：/home/data/default-test-claim2-pvc-9385f11d-2587-469b-bd13-3a756fe61acb
[root@master nfs]# kubectl apply -f test-claim2.yaml
persistentvolumeclaim/test-claim2 created
[root@master nfs]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Bound pvc-c2ae61f8-7c40-4393-9110-114a0455afa1 1Mi RWX nfs-storage 10h
test-claim2 Bound pvc-9385f11d-2587-469b-bd13-3a756fe61acb 100Mi RWX nfs-storage 5s
[root@master nfs]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-9385f11d-2587-469b-bd13-3a756fe61acb 100Mi RWX Retain Bound default/test-claim2 nfs-storage 17s
pvc-c2ae61f8-7c40-4393-9110-114a0455afa1 1Mi RWX Retain Bound default/test-claim nfs-storage 10h