Prometheus监控多K8S集群方式

3 solutions to achieve this: 

1. use prometheus federation. This will allow you to have a central prometheus server that will scrape samples from other prometheus servers.
2. use remote_write configuration. This will allow you to send your samples to a remote endpoint (and then eventually scrape that central endpoint). You'll also be able to apply relabeling rules with this configuration.
3. use another tool like thanos or Cortex. Those tools are great and allow you to do more stuff than just writing to a remote endpoint. You'll be able to implement horizontal scalling of your prometheus servers, long-term storage, etc.

我用的第二种方式，之在central集群部署一个prometheus，其他集群装kube-state-metrics

参考使用prometheus监控多k8s集群 - 云+社区 - 腾讯云 (tencent.com)

需要做的事，以AWS EKS为例

1.目标集群EKS API访问allow list。

2.目标集群取得token

3.prometheus添加监控job： (这里用了jinja模板定义了字典以循环定义多个集群的job)

{%- for aws_var in aws_aux_vars %}
  - job_name: kube-state-metrics-{{aws_var.region}}
    kubernetes_sd_configs:
      - role: endpoints
        api_server: https://{{aws_var.api}}
        tls_config:
          insecure_skip_verify: true
        bearer_token: {{aws_var.token}}
    tls_config:
      insecure_skip_verify: true
    bearer_token: {{aws_var.token}}
    scheme: https
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        action: keep
        regex: kube-system;fm22-test-kube-state-metrics
      - target_label: __address__
        replacement: {{aws_var.api}}
      - target_label: __metrics_path__
        replacement: /api/v1/namespaces/kube-system/services/http:fm22-test-kube-state-metrics:8080/proxy/metrics
{%- endfor %}

更新：上述方式涉及目标集群token的安全问题，不推荐，推荐直接用ingress暴露kube-state-metrics的服务，然后中央集群的prometheus通过static_config获得目标集群的kube-state-metrics数据