第一步、创建RBAC对象
- 创建用于
Prometheus
访问Kubernetes资源对象的RBAC
对象 - 因为我们后面需要获取所有资源的指标数据,最好使用权限大的认证用户,比如我这里直接使用admin权限,就不在单独创建RBAC资源了
kubectl -n kube-system get secret|grep admin-token
kubectl get secrets -n kube-system
kubectl describe secret -n kube-system <secret_name>
- 创建
RBAC
资源清单文件文章来源(Source):浅时光博客
~]# vim prometheus_rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: kube-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-monitoring
- 创建资源
kubectl create ns kube-monitoring
kubectl apply -f prometheus_rbac.yaml
第二步、获取Secret
- 获取上面的 Prometheus 对应的
Secret
的信息
#获取prometheus开头的secret资源token信息
kubectl get secrets -n kube-monitoring
kubectl describe secret -n kube-monitoring prometheus-token-jxv4z

- 将 toke
原文链接:https://www.dqzboy.com n 信息保存到一个名为k8s.token
的文本文件中
第三步、配置Prometheus
- 现在我们添加一个 Prometheus 监控外部 Kubernetes 集群数据的任务,如下所示
- 在Prometheus的机器上进行添加
[root@k8s-master1 ~]# kubectl cluster-info
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: "kube-node-kubelet"
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
kubernetes_sd_configs:
- role: node
api_server: "https://192.168.66.62:6443"
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
relabel_configs:
- target_label: __address__
# 使用replacement值替换__address__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
replacement: 192.168.66.62:6443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
# 使用replacement值替换__metrics_path__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}:10250/proxy/metrics
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service_name
# advisor
- job_name: "kube-node-cadvisor"
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
kubernetes_sd_configs:
- role: node
api_server: "https://192.168.66.62:6443"
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
relabel_configs:
- target_label: __address__
# 使用replacement值替换__address__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
replacement: 192.168.66.62:6443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
# 使用replacement值替换__metrics_path__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service_name
- job_name: "kube-state-metrics"
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
kubernetes_sd_configs:
- role: endpoints
api_server: "https://192.168.66.62:6443"
tls_config:
insecure_skip_verify: true
bearer_token_file: k8s.token
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
action: keep
regex: '^(kube-state-metrics)$'
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__address__]
action: replace
target_label: instance
- target_label: __address__
# 使用replacement值替换__address__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
replacement: 192.168.66.62:6443
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
regex: ([^;]+);([^;]+);([^;]+)
# 使用replacement值替换__metrics_path__默认值;在同一个source_labels中可以通过replacement进行重新组合一个新的URL,并通过${1} ${2}匹配source_labels获取的参数对应的数据
target_label: __metrics_path__
replacement: /api/v1/namespaces/${1}/pods/http:${2}:${3}/proxy/metrics
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service_name
参数说明:
- 通过
promtool
命令检查配置语法是否正确
[root@prometheus ~]# promtool check config /usr/local/prometheus/prometheus.yml
将k8s.token
放到Prometheus的安装目录下,例如我这里为 /usr/local/prometheus
- 重启Prometheus查看targets里面是否正常有对应的节点
[root@prometheus ~]# systemctl restart prometheus
- 如果开启了
hot reload
功能,则通过下面的方式热加载配置
curl -X POST http://127.0.0.1:9090/-/reload

部署完了不出东西是咋回事
具体表现是啥?