外部Prometheus监控K8s集群

第一步、创建RBAC对象

• 创建用于 Prometheus 访问Kubernetes资源对象的 RBAC 对象
• 因为我们后面需要获取所有资源的指标数据，最好使用权限大的认证用户，比如我这里直接使用admin权限，就不在单独创建RBAC资源了

kubectl -n kube-system get secret|grep admin-token


kubectl get secrets -n kube-system 
kubectl describe secret -n kube-system <secret_name>

• 创建RBAC资源清单文件文章来源(Source)：浅时光博客

~]# vim prometheus_rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: kube-monitoring

• 创建资源

kubectl create ns kube-monitoring
kubectl apply -f prometheus_rbac.yaml

第二步、获取Secret

• 获取上面的 Prometheus 对应的 Secret 的信息

#获取prometheus开头的secret资源token信息
kubectl get secrets -n kube-monitoring 
kubectl describe secret -n kube-monitoring prometheus-token-jxv4z

• 将 toke原文链接：https://www.dqzboy.comn 信息保存到一个名为 k8s.token 的文本文件中

第三步、配置Prometheus

• 现在我们添加一个 Prometheus 监控外部 Kubernetes 集群数据的任务，如下所示
• 在Prometheus的机器上进行添加

[root@k8s-master1 ~]# kubectl cluster-info

[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
  - job_name: "kube-node-kubelet"
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: k8s.token
    kubernetes_sd_configs:
    - role: node
      api_server: "https://192.168.66.62:6443"
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: k8s.token
    relabel_configs:
    - target_label: __address__
      # 使用replacement值替换__address__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      replacement: 192.168.66.62:6443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      # 使用replacement值替换__metrics_path__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:10250/proxy/metrics
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name

  # advisor    
  - job_name: "kube-node-cadvisor"
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: k8s.token
    kubernetes_sd_configs:
    - role: node
      api_server: "https://192.168.66.62:6443"
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: k8s.token
    relabel_configs:
    - target_label: __address__
      # 使用replacement值替换__address__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      replacement: 192.168.66.62:6443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      # 使用replacement值替换__metrics_path__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name

  - job_name: "kube-state-metrics"
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: k8s.token
    kubernetes_sd_configs:
    - role: endpoints
      api_server: "https://192.168.66.62:6443"
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: k8s.token
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      action: keep
      regex: '^(kube-state-metrics)$'
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__address__]
      action: replace
      target_label: instance
    - target_label: __address__
      # 使用replacement值替换__address__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      replacement: 192.168.66.62:6443
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
      regex: ([^;]+);([^;]+);([^;]+)
      # 使用replacement值替换__metrics_path__默认值；在同一个source_labels中可以通过replacement进行重新组合一个新的URL，并通过${1} ${2}匹配source_labels获取的参数对应的数据
      target_label: __metrics_path__
      replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name

• 通过promtool命令检查配置语法是否正确

[root@prometheus ~]# promtool check config /usr/local/prometheus/prometheus.yml

• 重启Prometheus查看targets里面是否正常有对应的节点

[root@prometheus ~]# systemctl restart prometheus

• 如果开启了hot reload 功能，则通过下面的方式热加载配置

curl -X POST http://127.0.0.1:9090/-/reload