搜索文章 Ctrl K

16-cadvisor监控Pod、daemonset部署node-exporter、deployment部署prometheus server

熊 0 阅读 监控平台

监控Pod指标数据需要使用cadvisor , cadvisor由谷歌开源 , cadvisor不仅可以收集一台机器上所有运行的容器信息 , 还提供基础查询界面和http接口 , 方便其他组件如Prometheus进行数据抓取 , cAdvisor可以对节点机器上的资源及容器进行实时监控和性能数据采集 , 包括CPU使用情况、内存使用情况、网络吞吐量及文件系统使用情况。

github链接: https://github.com/google/cadvisor

cAdvisor(容器顾问)让容器用户了解其运行容器的资源使用情况和性能状态 , cAdvisor用于收集、聚合、处理和导出有关正在运行的容器的信息 。 具体来说 , 对于每个容器都会保存资源隔离参数、历史资源使用情况、完整历史资源使用情况的直方图和网络统计信息 , 此数据按容器和机器范围导出。

1. 通过docker部署cadvisor

1.1 部署cadvisor

  1. 获取镜像 cadvisor镜像在谷歌镜像仓库 , 这里已经将镜像下载完毕并上传阿里云镜像 可以通过命令直接拉取镜像: nerdctl pull registry.cn-hangzhou.aliyuncs.com/tomikes/addon:cadvisor-v0.49.1

  2. 在部署cadvisor

    VERSION=v0.49.1 
sudo nerdctl run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/containerd/:/var/lib/containerd:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  registry.cn-hangzhou.aliyuncs.com/tomikes/addon:cadvisor-v0.49.1

  3. 访问cadvisor 浏览器输入: 192.168.71.7:8080# 192.168.71.7即部署cadvisor的主机ip

    image-20240626225313147

    访问http://192.168.71.7:8080/metrics , 可以查看容器的所有数据

    image-20240626225500323

1.2 cadvisor指标数据

指标名称

类型

含义

container_cpu_load_average_10s

gauge

过去10秒容器CPU的平均负载

container_cpu_usage_seconds_total

counter

容器在每个CPU内核上的累积占⽤时间 (单位: 秒)

container_cpu_system_seconds_total

counter System

CPU累积占⽤时间 ( 单位: 秒 )

container_cpu_user_seconds_total

counter User

CPU累积占⽤时间 ( 单位: 秒 )

container_fs_usage_bytes

gauge

容器中⽂件系统的使⽤量(单位: 字节)

container_fs_limit_bytes

gauge

容器可以使⽤的⽂件系统总量(单位: 字节)

container_fs_reads_bytes_total

counter

容器累积读取数据的总量(单位: 字节)

container_fs_writes_bytes_total

counter

容器累积写⼊数据的总量(单位: 字节)

container_memory_max_usage_bytes

gauge

容器的最⼤内存使⽤量 ( 单位: 字节 )

container_memory_usage_bytes

gauge

容器当前的内存使⽤量 ( 单位: 字节 )

container_spec_memory_limit_bytes

gauge

容器的内存使⽤量限制

machine_memory_bytes

gauge

当前主机的内存总量

container_network_receive_bytes_total

counter

容器⽹络累积接收数据总量 ( 单位: 字节 )

container_network_transmit_bytes_total

counter

容器⽹络累积传输数据总量 ( 单位: 字节 )

当能够正常采集到cAdvisor的样本数据后 , 可以通过以下表达式计算容器的CPU使用率:

  1. sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu) : 容器CPU使⽤率

  2. container_memory_usage_bytes{image!=""} : 查询容器内存使用量 ( 单位: 字节 ) :

  3. sum(rate(container_network_receive_bytes_total{image!=""}[1m])) without(interface) : 查询容器网络接收量 ( 速率 ) ( 单位: 字节/秒 ) :

  4. sum(rate(container_network_transmit_bytes_total{image!=""}[1m])) without(interface) : 容器网络传输量 字节/秒

  5. sum(rate(container_fs_reads_bytes_total{image!=""}[1m])) without (device) : 容器文件系统读取速率 字节/秒

  6. sum(rate(container_fs_writes_bytes_total{image!=""}[1m])) without (device) : 容器文件系统写⼊速率 字节/秒

cadvisor常用容器监控指标

  1. 网络流量 sum(rate(container_network_receive_bytes_total{name=~".+"}[1m])) by (name) : 容器网络接收的字节数 ( 1分钟内 ) , 根据名称查询 name=~".+" sum(rate(container_network_transmit_bytes_total{name=~".+"}[1m])) by (name) : 容器网络传输的字节数 ( 1分钟内 ) , 根据名称查询 name=~".+"

  2. 容器 CPU相关 sum(rate(container_cpu_system_seconds_total[1m])) : 所用容器system cpu的累计使用时间 ( 1min钟内 ) sum(irate(container_cpu_system_seconds_total{image!=""}[1m])) without (cpu) : 每个容器system cpu的使用时间 ( 1min钟内 ) sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name) * 100 : 每个容器的cpu使用率 sum(sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name) *100) : 总容器的cpu使用率

2. 通过daemonset部署cadvisor

2.1 部署cadvisor

  1. 获取cadvisor镜像并上传到harbor仓库

    root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor# nerdctl pull registry.cn-hangzhou.aliyuncs.com/tomikes/addon:cadvisor-v0.49.1
root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor# nerdctl tag registry.cn-hangzhou.aliyuncs.com/tomikes/addon:cadvisor-v0.49.1 harbor.cbear.top/baseimages/cadvisor:v0.49.1
root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor# nerdctl push harbor.cbear.top/baseimages/cadvisor:v0.49.1

  2. 创建监控专用的namespace

    kubectl create ns monitoring

  3. 创建cadvisor。这里创建的过程可以通过github页面的kustomize文件生成对应的yaml文件 。注意修改对应页面的namespace位置, 以为添加hostport端口(推荐使用hostnetwork, 只不过因为本地已经配置了localdns, 这个服务已经占用了8080端口), 修改对应的, 修改挂载的目录,因为本地使用的容器为containerd,需要修改挂载的目录。

    cadvisor/deploy/kubernetes/base at master · google/cadvisor (github.com) 

    root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor/base# kubectl kustomize ./
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: cadvisor
  name: monitoring
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: cadvisor
  name: cadvisor
  namespace: monitoring
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/pod: docker/default
  labels:
    app: cadvisor
  name: cadvisor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: cadvisor
      name: cadvisor
  template:
    metadata:
      labels:
        app: cadvisor
        name: cadvisor
    spec:
      automountServiceAccountToken: false
      containers:
      - image: harbor.cbear.top/baseimages/cadvisor:v0.49.1
        name: cadvisor
        ports:
        - containerPort: 8080
          hostPort: 8081
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 800m
            memory: 2000Mi
          requests:
            cpu: 400m
            memory: 400Mi
        volumeMounts:
        - mountPath: /rootfs
          name: rootfs
          readOnly: true
        - mountPath: /var/run
          name: var-run
          readOnly: true
        - mountPath: /sys
          name: sys
          readOnly: true
        - mountPath: /var/lib/containerd
          name: containerd
          readOnly: true
        - mountPath: /dev/disk
          name: disk
          readOnly: true
      serviceAccountName: cadvisor
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /
        name: rootfs
      - hostPath:
          path: /var/run
        name: var-run
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /var/lib/containerd
        name: containerd
      - hostPath:
          path: /dev/disk
        name: disk

    检查配置无误之后, 使用以下命令

    root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor/base# kubectl kustomize ./ | kubectl apply -f  -

  4. 查看pod , 每个node均存在一个cadvisor

    root@k8s-master01:/opt/k8s-data/yaml/cbear/cadvisor/base# kubectl get pods -n monitoring 
NAME             READY   STATUS    RESTARTS   AGE
cadvisor-4h9zz   1/1     Running   0          23m
cadvisor-54dpc   1/1     Running   0          23m
cadvisor-tj7d8   1/1     Running   0          23m
cadvisor-wtgsd   1/1     Running   0          23m
cadvisor-z9cq5   1/1     Running   0          23m
cadvisor-z9g5n   1/1     Running   0          23m
​

  5. 浏览访问cadvisor : 192.168.71.29:8081

    image-20240627230646650

2.2 修改prometheus配置 , 收集cadvisor数据

  1. 修改prometheus配置文件 , 添加k8s集群节点ip和端口

    root@prometheus01:/usr/local/src/prometheus# cat prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
​
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093
​
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
​
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "BJ Linux Server"
​
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
​
    static_configs:
      - targets: ["192.168.71.55:9100"]
        labels:
          idc: tongniu
          project: www
    metric_relabel_configs:
    - action: replace
      source_labels: ["instance"]
      regex: (.*):([0-9]+)  
      replacement: $1      
      target_label: "ip"
  - job_name: "Shanghai Linux Server"
    static_configs:
      - targets: ["192.168.71.56:9100","192.168.71.57:9100"]    
        labels:
          idc: sss
          project: blog
    relabel_configs:
      - action: labeldrop
        regex: "job"   
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ["192.168.71.54:9091"]
  - job_name: kubernetes          #以下几行到行尾为修改内容
    static_configs:
      - targets: ["192.168.71.29:8081","192.168.71.30:8081","192.168.71.31:8081","192.168.71.35:8081","192.168.71.36:8081","192.168.71.37:8081"]    
​
​

  2. 执行命令进行热加载

    root@prometheus01:/usr/local/src/prometheus# curl -X POST http://192.168.71.47:9090/-/reload

  3. 在prometheus监控控制台查看 , 已经获取到数据

    image-20240627231628960

2.3 Grafana导入pod模板

在grafana查看pod数据 , 要导入pod模板才可以查看 在grafana官网搜索container模板 , 获取模板id , 然后在grafana控制台导入模板id即可

image-20240627232110751

查看监控数据

image-20240627232236126

可以看到对应的容器数据。只不过这里看到的container名称为容器id, 而不是对应的kubernetes的容器实际名称。如果底层是docker则不会有这个问题

3. daemonset部署node-exporter

我们通过只对三台测试用的主机部署了node-exporter , 对三台机器做了监控 , 但并未对k8s集群内的node节点进行监控。 要想对k8s集群内node进行监控 , 要在node上部署node-exporter , 这里通过daemonset方式部署node-exporter对k8s集群的node节点进行监控。

3.1 k8s集群部署node-exporter

github链接 : prometheus/node_exporter: Exporter for machine metrics (github.com)

github中node-exporter的容器使用命令如下:

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

  1. 编写yaml文件

    root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/node-operator# cat node-operator.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
        k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      containers:
      - image: harbor.cbear.top/prometheus/node-exporter:v1.8.1
        imagePullPolicy: IfNotPresent
        name: prometheus-node-exporter
        ports:
        - containerPort: 9100        
          protocol: TCP
          name: metrics
        volumeMounts:
        - mountPath: /host/proc
          name: proc
        - mountPath: /host/sys
          name: sys
        - mountPath: /host
          name: rootfs
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /
      hostNetwork: true      
      hostPID: true

  2. 创建node-exporter资源

    root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/node-operator# kubectl apply -f node-operator.yaml 
daemonset.apps/node-exporter created

  3. 验证node-exporter资源

    root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/node-operator# kubectl get pods -n monitoring 
NAME                  READY   STATUS    RESTARTS   AGE
cadvisor-4h9zz        1/1     Running   0          135m
cadvisor-54dpc        1/1     Running   0          135m
cadvisor-tj7d8        1/1     Running   0          135m
cadvisor-wtgsd        1/1     Running   0          135m
cadvisor-z9cq5        1/1     Running   0          135m
cadvisor-z9g5n        1/1     Running   0          135m
node-exporter-4dkhb   1/1     Running   0          7s
node-exporter-c7s6l   1/1     Running   0          7s
node-exporter-dqdwt   1/1     Running   0          7s
node-exporter-fdwt2   1/1     Running   0          7s
node-exporter-kqm82   1/1     Running   0          7s
node-exporter-mpdt5   1/1     Running   0          7s

3.2 修改prometheus配置 , 收集node-exporter数据

  1. 修改prometheus配置文件 , 添加k8s集群节点ip和端口

    root@prometheus01:/usr/local/src/prometheus# cat prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "BJ Linux Server"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["192.168.71.55:9100"]
        labels:
          idc: tongniu
          project: www
    metric_relabel_configs:
    - action: replace
      source_labels: ["instance"]
      regex: (.*):([0-9]+)  
      replacement: $1      
      target_label: "ip"
  - job_name: "Shanghai Linux Server"
    static_configs:
      - targets: ["192.168.71.56:9100","192.168.71.57:9100"]    
        labels:
          idc: sss
          project: blog
    relabel_configs:
      - action: labeldrop
        regex: "job"   
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ["192.168.71.54:9091"]
  - job_name: kubernetes
    static_configs:
      - targets: ["192.168.71.29:8081","192.168.71.30:8081","192.168.71.31:8081","192.168.71.35:8081","192.168.71.36:8081","192.168.71.37:8081"]    
  - job_name: extral_docker
    static_configs:
      - targets: ["192.168.71.7:8080"]
  - job_name: prometheus-k8s-nodes    #以下几行到行尾为修改内容
    static_configs:
      - targets: ["192.168.71.29:9100","192.168.71.30:9100","192.168.71.31:9100","192.168.71.35:9100","192.168.71.36:9100","192.168.71.37:9100"]

  2. 执行命令进行热加载

    root@prometheus01:/usr/local/src/prometheus# curl -X POST http://192.168.71.47:9090/-/reload

  3. 在prometheus监控控制台查看 , 已经获取到数据

    image-20240628010456972

3.3 在grafana查看监控数据

由于node监控模板已经导入过了 , 可以直接查看node监控数据

  1. 导入1860和11074模板

    image-20240628010636339

  2. 查看dashboard

    image-20240628011654511
    image-20240628011729843

4. Deployment部署prometheus server

prometheus-server-deployment-1

请求过程 :

prometheus要提前配置好访问k8s集群的账号 , 获取可以访问k8s集群的权限

prometheus默认不知道k8s集群内要监控的pod地址、端口等信息

  1. prometheus配置监控信息 ( 要监控的目标pod、端口等 ) , 请求k8s集群的api-server发现想要获取的数据

  2. api-server查询etcd获取想要的数据 , etcd将数据返回给api-server

  3. api-server将数据返回给prometheus

  4. prometheus对获取到的数据进行加工 ( 即relabel , 重写 ) , 重写的原因是 , api-server返回的数据中有些数据并不是想要获取的 , 因此需要对数据进行重写

  5. 根据重写的数据 , 对目标pod进行数据抓取和监控

部署prometheus server

  1. 创建configmap , 作为prometheus的配置文件prometheus.yml

    • 编写yaml文件

      在master1上执行 : 

      root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/prometheus-server# cat prometheus-configmap.yaml 

---
kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:      #prometheus全局配置
      scrape_interval: 15s
      scrape_timeout: 10s
      evaluation_interval: 1m
    scrape_configs:    #数据抓取配置
    - job_name: 'kubernetes-node'      #配置job_name
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:9100'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
    - job_name: 'kubernetes-node-cadvisor-n66'
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:8080'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
    - job_name: 'kubernetes-node-cadvisor'
      kubernetes_sd_configs:
      - role:  node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    - job_name: 'kubernetes-apiserver'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
        namespaces: #可选指定namepace , 如果不指定就是发现所有的namespace中的pod
          names:
          - default
          - cbear
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

    • 根据yaml文件进行部署

      root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/prometheus-server# kubectl apply -f prometheus-configmap.yaml

    • 将prometheus运行在k8s-node01.cbear.top（192.168.71.35）并提前准备数据目录并授权

      在k8s-node01.cbear.top节点192.168.71.35上执行：

      mkdir -p /data/prometheusdata
chmod 777 /data/prometheusdata

  2. 创建监控账号, 在master1上执行

    kubectl create serviceaccount monitor -n monitoring

  3. 对monitoring 账号授权 与cluster-admin角色绑定 , 用于访问k8s集群 , 获取k8s集群信息(cluster-admin角色是默认存在的 , 具有集群管理员权限)

    kubectl create clusterrolebinding monitor-clusterrolebinding -n monitoring --clusterrole=cluster-admin --serviceaccount=monitoring:monitor

  4. 创建deployment控制器 , 通过deployment控制器创建prometheus

    ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: monitoring
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: server
    #matchExpressions:
    #- {key: app, operator: In, values: [prometheus]}
    #- {key: component, operator: In, values: [server]}
  template:
    metadata:
      labels:
        app: prometheus
        component: server
      annotations:
        prometheus.io/scrape: 'false'
    spec:
      nodeName: k8s-node01.cbear.top       #通过标签指定 , 把pod部署到node1节点上
      serviceAccountName: monitor
      containers:
      - name: prometheus
        image: harbor.cbear.top/prometheus/prometheus:v2.52.0     #试用官方镜像
        imagePullPolicy: IfNotPresent
        command:
          - prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --storage.tsdb.path=/prometheus
          - --storage.tsdb.retention=720h
          - --web.enable-lifecycle      #添加动态配置参数 , 可以通过传参方式热加载prometheus的配置
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts: 
        - mountPath: /etc/prometheus/prometheus.yml   #通过挂载configmap作为prometheus的配置文件
          name: prometheus-config
          subPath: prometheus.yml
        - mountPath: /prometheus/
          name: prometheus-storage-volume
      volumes:
        - name: prometheus-config
          configMap:       #指定configmap
            name: prometheus-config
            items:
              - key: prometheus.yml
                path: prometheus.yml
                mode: 0644
        - name: prometheus-storage-volume      #指定prometheus的数据存储目录
          hostPath:
           path: /data/prometheusdata
           type: Directory

  5. 根据yaml文件进行部署

    root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/prometheus-server# kubectl apply -f ./
configmap/prometheus-config created
deployment.apps/prometheus-server created

  6. 查看生成pod

    image-20240628021144818

  7. 为了可以通过外网访问k8s集群内的Prometheus , 需要为prometheus创建service并暴露端口

    ---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 39090
      protocol: TCP
  selector:
    app: prometheus
    component: server

    创建service

    root@k8s-master01:/opt/k8s-data/yaml/cbear/promethues/prometheus-server# kubectl apply -f prometheus-svc.yaml
service/prometheus created

  8. 验证: 访问prometheus

    浏览器访问192.168.71.35:30900 注意：prometheus监控页面会出现很多监控项，这是因为在prometheus的配置文件中添加了服务发现的配置，这会在后面讲解服务发现时进行说明

    image-20240628022403695

grafana 模板ID15760

image-20240630124511386

参考链接

四、cadvisor监控Pod、daemonset部署node-exporter、deployment部署prometheus server - 袁党生博客 (yuandangsheng.top)