Kubernetes / 云计算

二进制部署多Master高可用K8S集群-v1.18(二)

浅时光 · 7月26日 · 2020年 14285次已读

六、部署Master集群


1、部署环境说明

kubernetes master 节点运行如下组件:

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager

kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行:

  1. kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例,其它实例处于阻塞模式,当 leader 挂了后,重新选举产生新的 leader,从而保证服务可用性;
  2. kube-apiserver 是无状态的,可以通过 kube-nginx 进行代理访问从而保证服务可用性;
注意: 如果三台Master节点仅仅作为集群管理节点的话,那么则无需部署docker、kubelet、kube-proxy组件;但是如果后期要部署mertics-server、istio组件服务时会出现无法运行的情况,所以还是建议master节点也部署docker、kubelet、kube-proxy组件

1.1:下载程序包并解压

  • 将k8s-server压缩包上传至服务器/opt/k8s/work目录下,并进行解压
[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# tar -zxvf kubernetes-server-linux-amd64.tar.gz
[[email protected] work]# cd kubernetes/
[[email protected] kubernetes]# tar -zxvf kubernetes-src.tar.gz

1.文章来源(Source):https://www.dqzboy.com2:分发二进制文件

  • 将解压后的二进制文件拷贝到所有的K8S-Master集群的节点服务器上
  • kuberletkube-proxy文章来源(Source):https://www.dqzboy.com发给所有worker节点,存储目录/opt/k8s/bin
[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} [email protected]${node_ip}:/opt/k8s/bin/
    ssh [email protected]${node_ip} "chmod +x /opt/k8s/bin/*"
  done

[[email protected] work]# for node_ip in ${WORK_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kubernetes/server/bin/{kube-proxy,kubelet} [email protected]${node_ip}:/opt/k8s/bin/
    ssh [email protected]${node_ip} "chmod +x /opt/k8s/bin/*"
  done

2、部署kube-apiserver集群

2.1、创建 kubernetes-master 证书和私钥

  • hosts 字段指定授权使用该证书的 IP 和域名列表,这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名;
[[email protected] ~]# cd /opt/k8s/work/
[[email protected] work]# cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes-master",
  "hosts": [
    "127.0.0.1",
    "192.168.66.62",
    "192.168.66.63",
    "192.168.66.64",
    "${CLUSTER_KUBERNETES_SVC_IP}",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local.",
    "kubernetes.default.svc.${CLUSTER_DNS_DOMAIN}."
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ]
}
EOF
  • 生成证书和私钥
[[email protected] work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

[[email protected] work]# ls kubernetes*pem
kubernetes-key.pem  kubernetes.pem
  • 将生成的证书和私钥文件拷贝到所有 master 节点
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "mkdir -p /etc/kubernetes/cert"
    scp kubernetes*.pem [email protected]${node_ip}:/etc/kubernetes/cert/
  done

2.2、创建加密配置文件

[[email protected] work]# cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF
  • 将加密配置文件拷贝到 master 节点的 /etc/kubernetes 目录下
[[email protected]ster1 work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp encryption-config.yaml [email protected]${node_ip}:/etc/kubernetes/
  done

2.3、创建审计策略文件

[[email protected] work]# cat > audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses
  # for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
      
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF
  • 分发审计策略文件
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp audit-policy.yaml [email protected]${node_ip}:/etc/kubernetes/audit-policy.yaml
  done
文章来源(Source):https://www.dqzboy.com

2.4、创建后续访问 metrics-server 或 kube-prometheus 使用的证书

2.4.1:创建证书签名请求

[[email protected] work]# cat > proxy-client-csr.json <<EOF
{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ]
}
EOF

2.4.2:生成证书和私钥

[[email protected] work]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem  \
  -config=/etc/kubernetes/cert/ca-config.json  \
  -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client

[[email protected] work]# ls proxy-client*.pem
proxy-client-key.pem  proxy-client.pem

2.4.3:将生成的证书和私钥文件拷贝到所有 master 节点

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp proxy-client*.pem [email protected]${node_ip}:/etc/kubernetes/cert/
  done

2.5、创建 kube-apiserver systemd unit 模板文件

[[email protected] work]# cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
  --advertise-address=##MASTER_IP## \\
  --default-not-ready-toleration-seconds=360 \\
  --default-unreachable-toleration-seconds=360 \\
  --feature-gates=DynamicAuditing=true \\
  --max-mutating-requests-inflight=2000 \\
  --max-requests-inflight=4000 \\
  --default-watch-cache-size=200 \\
  --delete-collection-workers=2 \\
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --bind-address=##MASTER_IP## \\
  --secure-port=6443 \\
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
  --insecure-port=0 \\
  --audit-dynamic-configuration \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
  --profiling \\
  --anonymous-auth=false \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --enable-bootstrap-token-auth \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-admission-plugins=NodeRestriction \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --event-ttl=168h \\
  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2.6、为各节点创建和分发 kube-apiserver systemd unit 文件

文章来源(Source):https://www.dqzboy.com

2.6.1:创建systemd unit 文件

[[email protected] work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service 
  done

[[email protected] work]# ls kube-apiserver*.service
kube-apiserver-192.168.66.62.service  kube-apiserver-192.168.66.64.service
kube-apiserver-192.168.66.63.service

2.6.2:分发生成的 systemd unit 文件

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-apiserver-${node_ip}.service [email protected]${node_ip}:/etc/systemd/system/kube-apiserver.service
  done

2.7、启动 kube-apiserver 服务

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
    ssh [email protected]${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
  done

2.8、检查 kube-apiserver 运行状态

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "systemctl status kube-apiserver |grep 'Active:'"
  done

2.9、检查集群状态

[[email protected] work]# kubectl cluster-info
Kubernetes master is running at https://192.168.66.62:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客
[[email protected] work]# kubectl get all --all-namespaces
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.254.0.1   <none>        443/TCP   56s
[[email protected] work]# kubectl get componentstatuses
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客

3、部署高可用kube-controller-manager集群

该集群包含 3 个节点,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用时,阻塞的节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。

为保证通信安全,本文档先生成 x509 证书和私钥,kube-controller-manager 在如下两种情况下使用该证书:

  1. 与 kube-apiserver 的安全端口通信;
  2. 安全端口(https,10252) 输出 prometheus 格式的 metrics;

3.1、创建 kube-controller-manager 证书和私钥

3.1.1:创建证书签名请求

[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.66.62",
      "192.168.66.63",
      "192.168.66.64"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-controller-manager",
        "OU": "opsnull"
      }
    ]
}
EOF

3.1.2:生成证书和私钥

[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager


[[email protected] work]# ls kube-controller-manager*pem
kube-controller-manager-key.pem  kube-controller-manager.pem

3.文章来源(Source):https://www.dqzboy.com1.3:将生成的证书和私钥分发到所有文章来源(Source):https://www.dqzboy.com master 节点

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager*.pem [email protected]${node_ip}:/etc/kubernetes/cert/
  done

3.2、创建和分发 kubeconfig 文件

[[email protected] ~]# cd /opt/k8s/work

[[email protected] work]# kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server="https://##NODE_IP##:6443" \
  --kubeconfig=kube-controller-manager.kubeconfig

[[email protected] work]# kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

[[email protected] work]# kubectl config set-context system:kube-controller-manager \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

[[email protected] work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
  • 分发 kubeconfig 到所有 master 节点
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kube-controller-manager.kubeconfig > kube-controller-manager-${node_ip}.kubeconfig
    scp kube-controller-manager-${node_ip}.kubeconfig [email protected]${node_ip}:/etc/kubernetes/kube-controller-manager.kubeconfig
  done

3.3、创建 kube-controller-manager systemd unit 模板文件

[[email protected] work]# cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
  --profiling \\
  --cluster-name=kubernetes \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kube-api-qps=1000 \\
  --kube-api-burst=2000 \\
  --leader-elect \\
  --use-service-account-credentials\\
  --concurrent-service-syncs=2 \\
  --bind-address=##NODE_IP## \\
  --secure-port=10252 \\
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
  --port=0 \\
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --experimental-cluster-signing-duration=876000h \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --concurrent-deployment-syncs=10 \\
  --concurrent-gc-syncs=30 \\
  --node-cidr-mask-size=24 \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --pod-eviction-timeout=6m \\
  --terminated-pod-gc-threshold=10000 \\
  --root-ca-file=/etc/kubernetes/cert/ca.pem \\
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

3.3.1:为各Master节点创建和分发 kube-controller-mananger systemd unit 文件文章来源(Source):https://www.dqzboy.com

  • 替换模板文件中的变量
[[email protected] work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${NODE_IPS[i]}.service 
  done
  • 分发至给Master节点
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-controller-manager-${node_ip}.service [email protected]${node_ip}:/etc/systemd/system/kube-controller-manager.service
  done

3.4、启动kube-controller-manager服务

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
    ssh [email protected]${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
  done

3.5、检查服务运行状态

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "systemctl status kube-controller-manager|grep Active"
  done
  • kube-controller-manager 监听 10252 端口,接收 https 请求
[[email protected] work]# netstat -lnpt | grep kube-cont
tcp        0      0 192.168.66.62:10252     0.0.0.0:*               LISTEN      37321/kube-controll

3.6、查看输出的 metrics

注意:以下命令在 kube-controller-manager 节点上执行。

curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.66.62:10252/metrics |head
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客

3.7、查看当前的 leader

[[email protected] work]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
  • 可以看到当前的leader为k8s-master1
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客

4、部署高可用 kube-scheduler 集群

4.1、创建 kube-scheduler 证书和私钥

4文章来源(Source):https://www.dqzboy.com.1.1:创建证书签名请求

  • 注意hosts填写自己服务器master集群IP
[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# cat > kube-scheduler-csr.json <<EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.66.62",
      "192.168.66.63",
      "192.168.66.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "BeiJing",
        "L": "BeiJing",
        "O": "system:kube-scheduler",
        "OU": "opsnull "
      }
    ]
}
EOF

4.1.2:生成证书和私钥

[[email protected] work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

[[email protected] work]# ls kube-scheduler*pem
kube-scheduler-key.pem  kube-scheduler.pem

4.1.3:将生成的证书和私钥分发到所有 master文章来源(Source):https://www.dqzboy.com 节点

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler*.pem [email protected]${node_ip}:/etc/kubernetes/cert/
  done

4.2、创建和分发 kubeconfig 文件

kube-scheduler 使用 kubeconfig 文件访问 apiserver,该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书

4.2.1:创建kuberconfig文件

[[email protected] ~]# cd /opt/k8s/work

[[email protected] work]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server="https://##NODE_IP##:6443" \
--kubeconfig=kube-scheduler.kubeconfig


[[email protected] work]# kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.pem \
--client-key=kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig


[[email protected] work]# kubectl config set-context system:kube-scheduler \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig

[[email protected] work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

4.2.2:分发 kubeconfig 到所有 master 节点

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kube-scheduler.kubeconfig > kube-scheduler-${node_ip}.kubeconfig
    scp kube-scheduler-${node_ip}.kubeconfig [email protected]${node_ip}:/etc/kubernetes/kube-scheduler.kubeconfig
  done

4.3、创建 kube-scheduler 配置文件

4.3.1:创建kube-scheduler配置文件

[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# cat <<EOF | sudo tee kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF

4.3.2:分发 kube-scheduler 配置文件到所有 master 节点

  • 分发 kube-scheduler 配置文件到所有 master 节点
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler.yaml [email protected]${node_ip}:/etc/kubernetes/
  done

4.4、创建 kube-scheduler文章来源(Source):https://www.dqzboy.com systemd unit 模板文件

[[email protected] ~]# cd /opt/k8s/work
[[email protected] work]# cat > kube-scheduler.service.template <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \\
--config=/etc/kubernetes/kube-scheduler.yaml \\
--bind-address=##NODE_IP## \\
--secure-port=10259 \\
--port=0 \\
--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem
\\
--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconf
ig \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-allowed-names="" \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfi
g \\
--logtostderr=true \\
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF

4.5、为各节点创建和分发 kube-scheduler systemd unit 文件

[[email protected] work]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${MASTER_IPS[i]}.service 
  done
[[email protected] work]# ls kube-scheduler*.service
  • 分发 systemd unit 文件到所有 master 节点:
[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-scheduler-${node_ip}.service [email protected]${node_ip}:/etc/systemd/system/kube-scheduler.service
  done

4.7、启动kube-scheduler 服务

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "mkdir -p ${K8S_DIR}/kube-scheduler"
    ssh [email protected]${node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"
  done

4.8、检查服务运行状态

[[email protected] work]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh [email protected]${node_ip} "systemctl status kube-scheduler|grep Active"
  done

4.9、查看输出的 metrics

注意:以下命令在 kube-scheduler 节点上执行。

kube-scheduler 监听 10251 和 10259 端口:

  • 10251:接收 http 请求,非安全端口,不需要认证授权;
  • 10259:接收 https 请求,安全端口,需要认证授权;

两个接口都对外提供 /metrics/healthz 的访问。

[[email protected] work]# netstat -lnpt |grep kube-sch
tcp6       0      0 :::10251                :::*                    LISTEN      75717/kube-schedule 
tcp6       0      0 :::10259                :::*                    LISTEN      75717/kube-schedule

[[email protected] work]# curl -s http://192.168.66.62:10251/metrics |head

4.10、查看当前的 leader

[[email protected] work]# kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml
  • 可以看到当前的集群leader是k8s-master1节点
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客
二进制部署多Master高可用K8S集群-v1.18(二)-浅时光博客
0 条回应
    本站已安全运行: | 耗时 0.473 秒 | 查询 101 次 | 内存 22.16 MB