基于NFS的PV动态供给(StorageClass)

一、PV&PVC简介

PersistentVolume（PV）是指由集群管理员配置提供的某存储系统上的段存储空间，它是对底层共享存储的抽象，将共享存储作为种可由用户申请使的资源，实现了“存储消费”机制。通过存储插件机制，PV支持使用多种网络存储系统或云端存储等多种后端存储系统，例如，NFS、RBD和Cinder等。PV是集群级别的资源，不属于任何名称空间，用户对PV资源的使需要通过PersistentVolumeClaim（PVC）提出的使申请（或称为声明）来完成绑定，是PV资源的消费者，它向PV申请特定大小的空间及访问模式（如rw或ro），从创建出PVC存储卷，后再由Pod资源通过PersistentVolumeClaim存储卷关联使用，如下图：

尽管PVC使得用户可以以抽象的方式访问存储资源，但很多时候还是会涉及PV的不少属性，例如，由于不同场景时设置的性能参数等。为此，集群管理员不得不通过多种方式提供多种不同的PV以满不同用户不同的使用需求，两者衔接上的偏差必然会导致用户的需求无法全部及时有效地得到满足。Kubernetes从1.4版起引入了一个新的资源对象StorageClass，可用于将存储资源定义为具有显著特性的类（Class）而不是具体的PV，例如fast slow或glod silver bronze等。用户通过PVC直接向意向的类别发出申请，匹配由管理员事先创建的PV，或者由其按需为用户动态创建PV，这样做甚至免去了需要先创建PV的过程。
PV对存储系统的支持可通过其插件来实现，目前，Kubernetes支持如下类型的插件。

官方地址：https://kubernetes.io/docs/concepts/storage/storage-classes/

二、安装NFS插件

注意：所有K8S worker节点需要安装NFS服务,不然kubelet去调度创建Pod会失败

yum install nfs-utils

注意：插件不提供NFS服务，需要外部的NFS服务

GitHub地址：https://github.com/kubernetes-retired/external-storage/blob/master/nfs/deploy/kubernetes；目前该项目已被作者归档

2.1：创建RBAC授权

[[email protected] ~]# mkdir nfs
[[email protected] ~]# cd nfs/
[[email protected] nfs]# vim rbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

2.2：创建Storageclass

[[email protected] nfs]# vim class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true"

2.3：创建deployment

修改相应的nfs服务器ip及挂载路径即可
注意：目前nfs-provisioner已经在k8s v1.20版本中弃用了SelfLink，所以之前nfs-provisioner的版本无法正常在k8s v1.20集群中创建PV，这里需要更新nfs-provisioner的镜像版本；国外维护的镜像地址：gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
在网上找到的国内的一个nfs-provisioner地址：registry.cn-hangzhou.aliyuncs.com/xzjs/nfs-subdir-external-provisioner:v4.0.0

[[email protected] nfs]# vim deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          #---由于quay.io仓库国内被墙，所以需要替换仓库地址
          image: registry.cn-hangzhou.aliyuncs.com/xzjs/nfs-subdir-external-provisioner:v4.0.0
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              #改成NFS Server地址
              value: 192.168.66.201 
            - name: NFS_PATH
              #NFS共享的存储目录
              value: /data/k8s/
      volumes:
        - name: nfs-client-root
          nfs:
            #改成NFS Server地址
            server: 192.168.66.201
            #NFS共享的存储目录
            path: /data/k8s/

2.4：执行所有定义文件

[[email protected] nfs]# kubectl apply -f .
storageclass.storage.k8s.io/managed-nfs-storage created
serviceaccount/nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
serviceaccount/nfs-client-provisioner unchanged
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created

2.5：查看资源创建信息

[[email protected] ~]# kubectl get deploy

三、创建Deployment

3.1：定义执行文件

[[email protected] ~]# vim nginx-deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: k8s-dev
data:
  default.conf: |-
    server {
        listen       80;
        listen  [::]:80;
        server_name  localhost;
        keepalive_timeout 60;
        client_header_timeout 120;
        client_body_timeout 120;
        send_timeout 120;
        client_max_body_size 10m;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
        location /file { 
            alias   /data/upload;
            autoindex on;
            autoindex_format html;
            autoindex_exact_size off;
            autoindex_localtime on;
            charset utf-8,gbk;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: client-pvc	#定义pvc资源名称
  namespace: k8s-dev
  annotations:
	#指定使用的sc资源名称
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany	#简写：RWX，读写权限
  resources:
    requests:
      storage: 1Gi	#定义所需pvc大小
---
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: client
  name: client
  namespace: k8s-dev
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: client
  template:
    metadata:
      labels:
        k8s-app: client
      namespace: k8s-dev
      name: client
    spec:
      containers:
      - name: client
        image: harbor.yupeiholdings.com/npm-projects/client:1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/conf.d
        - name: nginx-upload
          mountPath: /data/upload	#容器中挂载的目录
      imagePullSecrets:
      - name: regcred
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config
        - name: nginx-upload
          persistentVolumeClaim:
            claimName: client-pvc #绑定挂载到的pvc名称
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: client
  name: admin-dev
  namespace: k8s-dev

3.2：执行定义文件

[[email protected] ~]# kubectl apply -f nginx-deployment.yaml

3.3：查看PVC绑定

[[email protected] ~]# kubectl get pvc -n k8s-dev