{"id":83428,"date":"2026-07-25T22:09:06","date_gmt":"2026-07-25T14:09:06","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/83428.html"},"modified":"2026-07-25T22:09:06","modified_gmt":"2026-07-25T14:09:06","slug":"%e3%80%90k8s-%e8%bf%90%e7%bb%b4%e5%ae%9e%e6%88%98%e3%80%9114-%e7%9b%91%e6%8e%a7%e5%91%8a%e8%ad%a6prometheus","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/83428.html","title":{"rendered":"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus"},"content":{"rendered":"<h2>\u76d1\u63a7\u544a\u8b66&#xff1a;Prometheus&#043;Grafana&#043;Alertmanager \u9ec4\u91d1\u7ec4\u5408<\/h2>\n<p>\u4e00\u53e5\u8bdd\u5b9a\u4f4d&#xff1a;K8s \u8be5\u76d1\u63a7\u54ea\u4e9b\u6307\u6807&#xff1f;\u544a\u8b66\u600e\u4e48\u4e0d\u6f0f\u4e0d\u566a&#xff1f;\u4e00\u5957\u65b9\u6cd5\u8bba\u52a0\u4e09\u4ef6\u5957&#xff08;Prometheus &#043; Grafana &#043; Alertmanager&#xff09;\u7ed9\u4f60\u7b54\u6848\u3002<\/p>\n<h3>\u5199\u5728\u524d\u9762<\/h3>\n<p>\u6211\u8bb0\u5f97\u6709\u4e00\u6b21\u51cc\u6668 2 \u70b9\u88ab On-Call \u7535\u8bdd\u53eb\u9192\u2014\u2014\u751f\u4ea7 Deploy \u6b63\u5728&#034;\u5065\u5eb7&#034;\u8fd0\u884c&#xff08;Readiness Probe \u8fd4\u56de 200&#xff09;&#xff0c;\u4f46 user \u7684\u8bf7\u6c42\u5168\u90e8\u8fd4\u56de 503\u3002\u540e\u6765\u590d\u76d8\u53d1\u73b0&#xff0c;\u76d1\u63a7\u53ea\u76ef\u4e86&#034;Pod \u662f\u5426\u6d3b\u7740&#034;&#xff0c;\u4f46 Pod \u5904\u7406\u8bf7\u6c42\u7684\u6210\u529f\u7387\u548c\u5ef6\u8fdf\u5b8c\u5168\u6ca1\u6709\u76d1\u63a7\u3002<\/p>\n<p>\u8fd9\u4e2a\u4e8b\u6545\u8ba9\u6211\u6df1\u523b\u7406\u89e3\u4e86\u90a3\u53e5\u8bdd&#xff1a;\u4f60\u65e0\u6cd5\u7ba1\u7406\u4f60\u6ca1\u6709\u5ea6\u91cf\u7684\u4e1c\u897f\u3002<\/p>\n<p>1 \u5e74\u4e0b\u6765&#xff0c;\u6211\u8fed\u4ee3\u4e86\u4e09\u7248\u76d1\u63a7\u4f53\u7cfb&#xff0c;\u6700\u7ec8\u6210\u578b\u7684\u5c31\u662f Prometheus &#043; Grafana &#043; Alertmanager \u8fd9\u5957\u7ec4\u5408\u3002\u8fd9\u7bc7\u6587\u7ae0\u628a\u6211\u8e29\u8fc7\u7684\u5751\u548c\u65b9\u6cd5\u8bba\u5b8c\u6574\u8f93\u51fa\u51fa\u6765\u3002<\/p>\n<p>\u8bfb\u5b8c\u4f60\u80fd\u5e26\u8d70&#xff1a;<\/p>\n<ul>\n<li>USE\/RED\/\u56db\u9ec4\u91d1\u4fe1\u53f7\u4e09\u5957\u76d1\u63a7\u65b9\u6cd5\u8bba<\/li>\n<li>K8s \u6838\u5fc3\u7ec4\u4ef6\u6307\u6807\u89e3\u8bfb<\/li>\n<li>Prometheus Operator \u751f\u4ea7\u90e8\u7f72\u65b9\u6848<\/li>\n<li>PromQL \u5e38\u7528\u67e5\u8be2\u6a21\u5f0f&#xff08;TOP N \/ \u5206\u4f4d \/ \u9884\u6d4b&#xff09;<\/li>\n<li>P0\/P1\/P2 \u544a\u8b66\u5206\u7ea7\u89c4\u8303\u548c Alertmanager \u8def\u7531\u914d\u7f6e<\/li>\n<\/ul>\n<h3>\u6838\u5fc3\u95ee\u9898<\/h3>\n<p>K8s \u8be5\u76d1\u63a7\u54ea\u4e9b\u6307\u6807&#xff1f;\u544a\u8b66\u600e\u4e48\u4e0d\u6f0f\u4e0d\u566a&#xff1f;<\/p>\n<p>\u5148\u8bf4\u7ed3\u8bba&#xff1a;\u4e0d\u6f0f\u9760\u65b9\u6cd5\u8bba&#xff0c;\u4e0d\u566a\u9760\u5206\u7ea7&#043;\u6291\u5236\u3002\u4e0b\u9762\u6211\u4eec\u5148\u8bb2\u65b9\u6cd5\u8bba&#xff0c;\u518d\u8bb2\u843d\u5730\u5de5\u5177\u3002<\/p>\n<hr \/>\n<h3>\u4e00\u3001\u539f\u7406\u5256\u6790<\/h3>\n<h4>1.1 \u4e09\u5957\u76d1\u63a7\u65b9\u6cd5\u8bba<\/h4>\n<p>\u65b9\u6cd5\u8bba\u4e0d\u662f\u5b66\u672f\u6982\u5ff5&#xff0c;\u662f\u5e2e\u4f60\u7f29\u5c0f\u6392\u67e5\u8303\u56f4\u7684\u5bfc\u822a\u3002\u51fa\u95ee\u9898\u65f6&#xff0c;\u4f60\u4e0d\u9700\u8981\u60f3&#034;\u8be5\u67e5\u4ec0\u4e48&#034;&#xff0c;\u65b9\u6cd5\u8bba\u7684\u68c0\u67e5\u6e05\u5355\u4f1a\u544a\u8bc9\u4f60\u3002<\/p>\n<p>USE \u65b9\u6cd5\u8bba&#xff08;Brendan Gregg \u63d0\u51fa&#xff09;\u2014\u2014 \u9762\u5411\u8d44\u6e90<\/p>\n<p>USE &#061; Utilization &#043; Saturation &#043; Errors<\/p>\n<p>\u68c0\u67e5\u6e05\u5355&#xff08;\u6bcf\u4e2a\u8d44\u6e90\u95ee\u4e09\u4e2a\u95ee\u9898&#xff09;&#xff1a;<br \/>\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502   \u8d44\u6e90    \u2502  Utilization \u2502  Saturation  \u2502    Errors    \u2502<br \/>\n\u2502           \u2502  \u4f7f\u7528\u7387\u591a\u5c11&#xff1f; \u2502  \u6392\u961f\u591a\u957f&#xff1f;   \u2502  \u6709\u9519\u8bef\u5417&#xff1f;   \u2502<br \/>\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524<br \/>\n\u2502   CPU     \u2502   \u4f7f\u7528\u7387%     \u2502  run queue   \u2502    N\/A       \u2502<br \/>\n\u2502   Memory  \u2502   \u4f7f\u7528\u7387%     \u2502  swap\/page   \u2502   OOM Kill   \u2502<br \/>\n\u2502   Disk    \u2502   \u4f7f\u7528\u7387%     \u2502   I\/O wait   \u2502   \u574f\u5757\/SMART \u2502<br \/>\n\u2502   Network \u2502   \u5e26\u5bbd%       \u2502   drop\/retry \u2502   checksum   \u2502<br \/>\n\u2502   FD      \u2502   \u5df2\u7528\/\u4e0a\u9650   \u2502   N\/A        \u2502   \u6253\u5f00\u5931\u8d25    \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n<p>\u9002\u7528\u573a\u666f&#xff1a;\u57fa\u7840\u8bbe\u65bd\u3001\u8282\u70b9\u3001\u4e2d\u95f4\u4ef6\u76d1\u63a7<\/p>\n<p>RED \u65b9\u6cd5\u8bba&#xff08;Tom Wilkie \u63d0\u51fa&#xff09;\u2014\u2014 \u9762\u5411\u670d\u52a1<\/p>\n<p>RED &#061; Rate &#043; Errors &#043; Duration<\/p>\n<p>\u6bcf\u4e2a\u670d\u52a1\u76d1\u63a7\u4e09\u4ef6\u4e8b&#xff1a;<br \/>\n  Rate     \u2014\u2014 \u6bcf\u79d2\u8bf7\u6c42\u6570&#xff08;QPS\/RPS&#xff09;<br \/>\n  Errors   \u2014\u2014 \u5931\u8d25\u8bf7\u6c42\u6570&#xff08;\u6309 HTTP 5xx \/ \u8d85\u65f6 \/ \u4e1a\u52a1\u9519\u8bef\u7801&#xff09;<br \/>\n  Duration \u2014\u2014 \u8bf7\u6c42\u5ef6\u8fdf\u5206\u5e03&#xff08;P50\/P90\/P99&#xff09;<\/p>\n<p>\u9002\u7528\u573a\u666f&#xff1a;\u5fae\u670d\u52a1\u3001API \u7f51\u5173\u3001RPC \u8c03\u7528\u76d1\u63a7<\/p>\n<p>\u56db\u9ec4\u91d1\u4fe1\u53f7&#xff08;Google SRE Book&#xff09;&#xff1a;<\/p>\n<p>\u56db\u9ec4\u91d1\u4fe1\u53f7 &#061; Latency &#043; Traffic &#043; Errors &#043; Saturation<\/p>\n<p>\u4e0e RED \u7684\u533a\u522b&#xff1a;\u628a Saturation&#xff08;\u9971\u548c\u5ea6&#xff09;\u52a0\u4e86\u8fdb\u6765<br \/>\n  Latency    &#061; \u8bf7\u6c42\u8017\u65f6&#xff08;\u5bf9\u5e94 RED \u7684 Duration&#xff09;<br \/>\n  Traffic    &#061; \u8bf7\u6c42\u91cf&#xff08;\u5bf9\u5e94 RED \u7684 Rate&#xff09;<br \/>\n  Errors     &#061; \u9519\u8bef\u7387&#xff08;\u5bf9\u5e94 RED \u7684 Errors&#xff09;<br \/>\n  Saturation &#061; \u670d\u52a1\u6709\u591a&#034;\u6ee1&#034;&#xff08;\u8fde\u63a5\u6570\u3001\u961f\u5217\u6df1\u5ea6\u3001goroutine \u6570\u7b49&#xff09;<\/p>\n<p>\u9002\u7528\u573a\u666f&#xff1a;SRE \u4f53\u7cfb\u4e0b\u7684\u670d\u52a1\u5065\u5eb7\u5ea6\u8bc4\u4f30<\/p>\n<p>\u5b9e\u6218\u4e2d\u6211\u600e\u4e48\u505a&#xff1a;\u57fa\u7840\u8bbe\u65bd\u5c42\u7528 USE&#xff0c;\u670d\u52a1\u5c42\u7528 RED &#043; Saturation\u3002K8s \u662f\u4e00\u4e2a\u5206\u5c42\u7cfb\u7edf&#xff0c;\u6240\u4ee5\u5206\u5c42\u76d1\u63a7&#xff1a;<\/p>\n<p>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n\u2502                     K8s \u76d1\u63a7\u5206\u5c42                              \u2502<br \/>\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524<br \/>\n\u2502                                                              \u2502<br \/>\n\u2502   \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510   \u2502<br \/>\n\u2502   \u2502  \u4e1a\u52a1\u5c42: \u8ba2\u5355\u91cf\/\u652f\u4ed8\u6210\u529f\u7387 (\u4e1a\u52a1\u57cb\u70b9\/Business Metrics)  \u2502   \u2502<br \/>\n\u2502   \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524   \u2502<br \/>\n\u2502   \u2502  \u5e94\u7528\u5c42: QPS\/\u5ef6\u8fdf\/\u9519\u8bef\u7387 (RED \u65b9\u6cd5\u8bba)                 \u2502   \u2502<br \/>\n\u2502   \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524   \u2502<br \/>\n\u2502   \u2502  K8s\u5c42: Pod \u72b6\u6001\/Deploy \u53ef\u7528\u526f\u672c\/\u8c03\u5ea6\u5ef6\u8fdf             \u2502   \u2502<br \/>\n\u2502   \u2502         kube-state-metrics                           \u2502   \u2502<br \/>\n\u2502   \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524   \u2502<br \/>\n\u2502   \u2502  \u8282\u70b9\u5c42: CPU\/Mem\/Disk\/Net (USE \u65b9\u6cd5\u8bba)                \u2502   \u2502<br \/>\n\u2502   \u2502         node-exporter \/ cAdvisor                     \u2502   \u2502<br \/>\n\u2502   \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518   \u2502<br \/>\n\u2502                                                              \u2502<br \/>\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n<h4>1.2 Prometheus \u6570\u636e\u6a21\u578b\u4e0e PromQL \u7cbe\u9ad3<\/h4>\n<p>Prometheus \u65f6\u5e8f\u6570\u636e\u6a21\u578b&#xff1a;<\/p>\n<p>\u6307\u6807\u540d{\u6807\u7b7e1&#061;&#034;\u503c1&#034;, \u6807\u7b7e2&#061;&#034;\u503c2&#034;} \u503c&#064;\u65f6\u95f4\u6233<\/p>\n<p>\u793a\u4f8b:<br \/>\nhttp_requests_total{method&#061;&#034;GET&#034;, handler&#061;&#034;\/api\/users&#034;, status&#061;&#034;200&#034;} 10467 &#064;1716000000<br \/>\n  \u2502                 \u2502                                                      \u2502        \u2502<br \/>\n  \u6307\u6807\u540d            \u6807\u7b7e\u7ec4                                                 \u91c7\u6837\u503c   \u65f6\u95f4\u6233<\/p>\n<p>PromQL \u4e2d\u6700\u91cd\u8981\u7684\u51e0\u4e2a\u6982\u5ff5&#xff0c;\u7406\u89e3\u4e86\u5c31\u901a\u4e86 80% \u7684\u67e5\u8be2&#xff1a;<\/p>\n<p>\u77ac\u65f6\u5411\u91cf vs \u8303\u56f4\u5411\u91cf:<\/p>\n<p>\u77ac\u65f6\u5411\u91cf (Instant Vector):<br \/>\n  http_requests_total             \u2190 \u8fd4\u56de\u6bcf\u4e2a\u65f6\u95f4\u5e8f\u5217\u5728\u67e5\u8be2\u77ac\u95f4\u7684\u6700\u65b0\u503c<\/p>\n<p>\u8303\u56f4\u5411\u91cf (Range Vector):<br \/>\n  http_requests_total[5m]         \u2190 \u8fd4\u56de\u6bcf\u4e2a\u65f6\u95f4\u5e8f\u5217\u5728\u8fc7\u53bb 5 \u5206\u949f\u7684\u6240\u6709\u91c7\u6837\u70b9<br \/>\n                                    \u8303\u56f4\u5411\u91cf\u4e0d\u80fd\u76f4\u63a5\u7528&#xff0c;\u5fc5\u987b\u914d\u5408 rate()\/avg() \u7b49\u51fd\u6570<\/p>\n<p>rate() vs irate():<br \/>\n  rate(http_requests_total[5m])        \u2190 5 \u5206\u949f\u5e73\u5747\u901f\u7387&#xff0c;\u5e73\u6ed1\u6ce2\u52a8<br \/>\n  irate(http_requests_total[5m])       \u2190 \u77ac\u65f6\u901f\u7387&#xff08;\u6700\u540e\u4e24\u4e2a\u91c7\u6837\u70b9&#xff09;&#xff0c;\u654f\u611f\u4f46\u6613\u6296<\/p>\n<p>  \u751f\u4ea7\u5efa\u8bae&#xff1a;\u544a\u8b66\u7528 rate&#xff0c;\u6392\u67e5\u95ee\u9898\u7528 irate<\/p>\n<h4>1.3 Prometheus Operator \u67b6\u6784<\/h4>\n<p>  #mermaid-svg-i9bIp3KxLyyYbMzf{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-i9bIp3KxLyyYbMzf .error-icon{fill:#552222;}#mermaid-svg-i9bIp3KxLyyYbMzf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-i9bIp3KxLyyYbMzf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-i9bIp3KxLyyYbMzf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-i9bIp3KxLyyYbMzf .marker.cross{stroke:#333333;}#mermaid-svg-i9bIp3KxLyyYbMzf svg{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-i9bIp3KxLyyYbMzf p{margin:0;}#mermaid-svg-i9bIp3KxLyyYbMzf .label{font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;color:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster-label text{fill:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster-label span{color:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster-label span p{background-color:transparent;}#mermaid-svg-i9bIp3KxLyyYbMzf .label text,#mermaid-svg-i9bIp3KxLyyYbMzf span{fill:#333;color:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf .node rect,#mermaid-svg-i9bIp3KxLyyYbMzf .node circle,#mermaid-svg-i9bIp3KxLyyYbMzf .node ellipse,#mermaid-svg-i9bIp3KxLyyYbMzf .node polygon,#mermaid-svg-i9bIp3KxLyyYbMzf .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-i9bIp3KxLyyYbMzf .rough-node .label text,#mermaid-svg-i9bIp3KxLyyYbMzf .node .label text,#mermaid-svg-i9bIp3KxLyyYbMzf .image-shape .label,#mermaid-svg-i9bIp3KxLyyYbMzf .icon-shape .label{text-anchor:middle;}#mermaid-svg-i9bIp3KxLyyYbMzf .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-i9bIp3KxLyyYbMzf .rough-node .label,#mermaid-svg-i9bIp3KxLyyYbMzf .node .label,#mermaid-svg-i9bIp3KxLyyYbMzf .image-shape .label,#mermaid-svg-i9bIp3KxLyyYbMzf .icon-shape .label{text-align:center;}#mermaid-svg-i9bIp3KxLyyYbMzf .node.clickable{cursor:pointer;}#mermaid-svg-i9bIp3KxLyyYbMzf .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-i9bIp3KxLyyYbMzf .arrowheadPath{fill:#333333;}#mermaid-svg-i9bIp3KxLyyYbMzf .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-i9bIp3KxLyyYbMzf .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-i9bIp3KxLyyYbMzf .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-i9bIp3KxLyyYbMzf .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-i9bIp3KxLyyYbMzf .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-i9bIp3KxLyyYbMzf .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster text{fill:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf .cluster span{color:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-i9bIp3KxLyyYbMzf .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-i9bIp3KxLyyYbMzf rect.text{fill:none;stroke-width:0;}#mermaid-svg-i9bIp3KxLyyYbMzf .icon-shape,#mermaid-svg-i9bIp3KxLyyYbMzf .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-i9bIp3KxLyyYbMzf .icon-shape p,#mermaid-svg-i9bIp3KxLyyYbMzf .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-i9bIp3KxLyyYbMzf .icon-shape .label rect,#mermaid-svg-i9bIp3KxLyyYbMzf .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-i9bIp3KxLyyYbMzf .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-i9bIp3KxLyyYbMzf .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-i9bIp3KxLyyYbMzf :root{&#8211;mermaid-font-family:\\&#8221;trebuchet ms\\&#8221;,verdana,arial,sans-serif;}<\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u6838\u5fc3 CRD<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>ServiceMonitor\u5b9a\u4e49\u5982\u4f55\u4ece Service \u91c7\u96c6\u6307\u6807<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>PodMonitor\u5b9a\u4e49\u5982\u4f55\u4ece Pod \u91c7\u96c6\u6307\u6807<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>PrometheusRule\u5b9a\u4e49\u544a\u8b66\/\u5f55\u5236\u89c4\u5219<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Prometheus Operator \u5de5\u4f5c\u6d41<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"edgeLabel\"><\/span><\/p>\n<p>           <span class=\"edgeLabel\"><\/span><\/p>\n<p>           <span class=\"edgeLabel\"><\/span><\/p>\n<p>           <span class=\"edgeLabel\"><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>\u5f00\u53d1\u8005\u7f16\u5199 ServiceMonitor YAML<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Operator \u76d1\u542c ServiceMonitor \u8d44\u6e90<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Operator \u751f\u6210 Prometheus \u914d\u7f6e<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>ConfigReloader \u70ed\u91cd\u8f7d\u914d\u7f6e<\/p>\n<p><\/span><\/p>\n<p>           <span class=\"nodeLabel\"><\/p>\n<p>Prometheus \u6309\u914d\u7f6e\u62c9\u53d6\u6307\u6807<\/p>\n<p><\/span><\/p>\n<p>ServiceMonitor \u5de5\u4f5c\u6d41&#xff08;ASCII \u793a\u610f&#xff09;&#xff1a;<\/p>\n<p>  ServiceMonitor<br \/>\n       \u2502<br \/>\n       \u2502 selector.matchLabels.app&#061;nginx<br \/>\n       \u25bc<br \/>\n  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n  \u2502    Service       \u2502<br \/>\n  \u2502  labels:         \u2502<br \/>\n  \u2502    app: nginx    \u2502<br \/>\n  \u2502  ports:          \u2502<br \/>\n  \u2502    &#8211; metrics:9113\u2502<br \/>\n  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n           \u2502<br \/>\n           \u2502 Endpoints<br \/>\n           \u25bc<br \/>\n  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n  \u2502         Pod 1               \u2502<br \/>\n  \u2502  labels:                    \u2502<br \/>\n  \u2502    app: nginx               \u2502<br \/>\n  \u2502  annotations:               \u2502<br \/>\n  \u2502    prometheus.io\/scrape&#061;true\u2502<br \/>\n  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<br \/>\n                 \u2502<br \/>\n                 \u2502 scrape http:\/\/10.1.2.3:9113\/metrics<br \/>\n                 \u25bc<br \/>\n         \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510<br \/>\n         \u2502 Prometheus   \u2502<br \/>\n         \u2502   Server     \u2502<br \/>\n         \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/p>\n<hr \/>\n<h3>\u4e8c\u3001\u5b9e\u6218\u64cd\u4f5c<\/h3>\n<h4>2.1 \u90e8\u7f72 kube-prometheus-stack<\/h4>\n<p><span class=\"token comment\"># \u6dfb\u52a0 Prometheus \u793e\u533a Helm \u4ed3\u5e93<\/span><br \/>\nhelm repo <span class=\"token function\">add<\/span> prometheus-community https:\/\/prometheus-community.github.io\/helm-charts<br \/>\nhelm repo update<\/p>\n<p><span class=\"token comment\"># \u521b\u5efa\u547d\u540d\u7a7a\u95f4<\/span><br \/>\nkubectl create namespace monitoring<\/p>\n<p>kube-prometheus-values.yaml \u2014\u2014 \u751f\u4ea7\u7ea7\u914d\u7f6e&#xff1a;<\/p>\n<p><span class=\"token comment\"># &#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;<\/span><br \/>\n<span class=\"token comment\"># kube-prometheus-stack \u751f\u4ea7\u90e8\u7f72\u914d\u7f6e<\/span><br \/>\n<span class=\"token comment\"># Helm Chart \u7248\u672c: 58.x (\u5bf9\u5e94 Prometheus 2.50&#043;, Grafana 10.x)<\/span><br \/>\n<span class=\"token comment\"># &#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;&#061;<\/span><\/p>\n<p><span class=\"token comment\"># \u2500\u2500 Prometheus Operator \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">prometheusOperator<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">image<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">tag<\/span><span class=\"token punctuation\">:<\/span> v0.72.0<br \/>\n  <span class=\"token key atrule\">admissionWebhooks<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n    <span class=\"token key atrule\">patch<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">tls<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">tolerations<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">key<\/span><span class=\"token punctuation\">:<\/span> node<span class=\"token punctuation\">&#8211;<\/span>role.kubernetes.io\/control<span class=\"token punctuation\">&#8211;<\/span>plane<br \/>\n      <span class=\"token key atrule\">operator<\/span><span class=\"token punctuation\">:<\/span> Exists<br \/>\n      <span class=\"token key atrule\">effect<\/span><span class=\"token punctuation\">:<\/span> NoSchedule<\/p>\n<p><span class=\"token comment\"># \u2500\u2500 Prometheus Server \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">prometheus<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>  <span class=\"token key atrule\">prometheusSpec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token comment\"># \u955c\u50cf\u7248\u672c<\/span><br \/>\n    <span class=\"token key atrule\">image<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">repository<\/span><span class=\"token punctuation\">:<\/span> quay.io\/prometheus\/prometheus<br \/>\n      <span class=\"token key atrule\">tag<\/span><span class=\"token punctuation\">:<\/span> v2.50.1<\/p>\n<p>    <span class=\"token comment\"># \u6570\u636e\u4fdd\u7559<\/span><br \/>\n    <span class=\"token key atrule\">retention<\/span><span class=\"token punctuation\">:<\/span> 30d<br \/>\n    <span class=\"token key atrule\">retentionSize<\/span><span class=\"token punctuation\">:<\/span> 45GB   <span class=\"token comment\"># \u8fbe\u5230 45GB \u81ea\u52a8\u6e05\u7406\u65e7\u6570\u636e<\/span><\/p>\n<p>    <span class=\"token comment\"># \u8d44\u6e90\u914d\u989d<\/span><br \/>\n    <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 500m<br \/>\n        <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 2Gi<br \/>\n      <span class=\"token key atrule\">limits<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 2000m<br \/>\n        <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 8Gi<\/p>\n<p>    <span class=\"token comment\"># \u6301\u4e45\u5316\u5b58\u50a8<\/span><br \/>\n    <span class=\"token key atrule\">storageSpec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">volumeClaimTemplate<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">spec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">storageClassName<\/span><span class=\"token punctuation\">:<\/span> ssd<span class=\"token punctuation\">&#8211;<\/span>sc<br \/>\n          <span class=\"token key atrule\">accessModes<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;ReadWriteOnce&#034;<\/span><span class=\"token punctuation\">]<\/span><br \/>\n          <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n              <span class=\"token key atrule\">storage<\/span><span class=\"token punctuation\">:<\/span> 100Gi<\/p>\n<p>    <span class=\"token comment\"># \u8fdc\u7a0b\u5199\u5165&#xff08;\u957f\u671f\u5b58\u50a8&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">remoteWrite<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">]<\/span><br \/>\n    <span class=\"token comment\"># \u751f\u4ea7\u5efa\u8bae\u914d\u7f6e Thanos \u6216 VictoriaMetrics:<\/span><br \/>\n    <span class=\"token comment\"># remoteWrite:<\/span><br \/>\n    <span class=\"token comment\">#   &#8211; url: http:\/\/victoriametrics.monitoring.svc:8428\/api\/v1\/write<\/span><\/p>\n<p>    <span class=\"token comment\"># \u91c7\u96c6\u95f4\u9694<\/span><br \/>\n    <span class=\"token key atrule\">scrapeInterval<\/span><span class=\"token punctuation\">:<\/span> 30s<br \/>\n    <span class=\"token key atrule\">evaluationInterval<\/span><span class=\"token punctuation\">:<\/span> 30s<\/p>\n<p>    <span class=\"token comment\"># \u5916\u90e8\u6807\u7b7e&#xff08;\u6240\u6709\u6307\u6807\u90fd\u4f1a\u5e26\u4e0a&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">externalLabels<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cluster<\/span><span class=\"token punctuation\">:<\/span> prod<span class=\"token punctuation\">&#8211;<\/span>k8s<span class=\"token punctuation\">&#8211;<\/span><span class=\"token number\">1<\/span><br \/>\n      <span class=\"token key atrule\">region<\/span><span class=\"token punctuation\">:<\/span> cn<span class=\"token punctuation\">&#8211;<\/span>east<span class=\"token punctuation\">&#8211;<\/span><span class=\"token number\">2<\/span><br \/>\n      <span class=\"token key atrule\">env<\/span><span class=\"token punctuation\">:<\/span> production<\/p>\n<p>    <span class=\"token comment\"># \u526f\u672c\u6570&#xff08;\u9ad8\u53ef\u7528&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">replicas<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">2<\/span><\/p>\n<p>    <span class=\"token comment\"># \u5f00\u542f Admin API&#xff08;\u6392\u67e5\u95ee\u9898\u7528&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">enableAdminAPI<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>    <span class=\"token comment\"># \u5173\u95ed\u4e0d\u9700\u8981\u7684 Etcd\/KubeScheduler\/KubeControllerManager \u76d1\u63a7<\/span><br \/>\n    <span class=\"token comment\"># &#xff08;\u6258\u7ba1 K8s \u5982 EKS\/GKE \u8fd9\u4e9b\u7ec4\u4ef6\u4e0d\u53ef\u8fbe&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">kubeEtcd<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">false<\/span><br \/>\n    <span class=\"token key atrule\">kubeScheduler<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">false<\/span><br \/>\n      <span class=\"token key atrule\">service<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">false<\/span><br \/>\n    <span class=\"token key atrule\">kubeControllerManager<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">false<\/span><br \/>\n      <span class=\"token key atrule\">service<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">false<\/span><\/p>\n<p>    <span class=\"token comment\"># ServiceMonitor \u9009\u62e9\u5668&#xff08;\u7a7a &#061; \u91c7\u96c6\u6240\u6709&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">serviceMonitorSelector<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">{<\/span><span class=\"token punctuation\">}<\/span><br \/>\n    <span class=\"token key atrule\">podMonitorSelector<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">{<\/span><span class=\"token punctuation\">}<\/span><\/p>\n<p>    <span class=\"token comment\"># \u544a\u8b66\u89c4\u5219\u9009\u62e9\u5668<\/span><br \/>\n    <span class=\"token key atrule\">ruleSelector<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">matchLabels<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">role<\/span><span class=\"token punctuation\">:<\/span> alert<span class=\"token punctuation\">&#8211;<\/span>rules<\/p>\n<p>    <span class=\"token comment\"># Alertmanager \u914d\u7f6e<\/span><br \/>\n    <span class=\"token key atrule\">alerting<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">alertmanagers<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">namespace<\/span><span class=\"token punctuation\">:<\/span> monitoring<br \/>\n          <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> alertmanager<span class=\"token punctuation\">&#8211;<\/span>operated<br \/>\n          <span class=\"token key atrule\">port<\/span><span class=\"token punctuation\">:<\/span> web<br \/>\n          <span class=\"token key atrule\">apiVersion<\/span><span class=\"token punctuation\">:<\/span> v2<\/p>\n<p>    <span class=\"token comment\"># \u5bb9\u5fcd\u5ea6&#xff08;\u5141\u8bb8\u8c03\u5ea6\u5230\u63a7\u5236\u8282\u70b9&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">tolerations<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">key<\/span><span class=\"token punctuation\">:<\/span> node<span class=\"token punctuation\">&#8211;<\/span>role.kubernetes.io\/control<span class=\"token punctuation\">&#8211;<\/span>plane<br \/>\n        <span class=\"token key atrule\">operator<\/span><span class=\"token punctuation\">:<\/span> Exists<br \/>\n        <span class=\"token key atrule\">effect<\/span><span class=\"token punctuation\">:<\/span> NoSchedule<\/p>\n<p>  <span class=\"token comment\"># \u2500\u2500 Service \u2500\u2500<\/span><br \/>\n  <span class=\"token key atrule\">service<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">type<\/span><span class=\"token punctuation\">:<\/span> ClusterIP<br \/>\n    <span class=\"token key atrule\">port<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">9090<\/span><br \/>\n    <span class=\"token key atrule\">annotations<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">prometheus.io\/scrape<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;true&#034;<\/span><\/p>\n<p><span class=\"token comment\"># \u2500\u2500 Alertmanager \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">alertmanager<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>  <span class=\"token key atrule\">alertmanagerSpec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">image<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">tag<\/span><span class=\"token punctuation\">:<\/span> v0.27.0<\/p>\n<p>    <span class=\"token key atrule\">replicas<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">3<\/span>                          <span class=\"token comment\"># \u96c6\u7fa4\u6a21\u5f0f<\/span><br \/>\n    <span class=\"token key atrule\">clusterAdvertiseAddress<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>    <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 100m<br \/>\n        <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 256Mi<br \/>\n      <span class=\"token key atrule\">limits<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 500m<br \/>\n        <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 1Gi<\/p>\n<p>    <span class=\"token comment\"># Alertmanager \u914d\u7f6e&#xff08;\u5916\u90e8\u5316\u5230 Secret&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">configSecret<\/span><span class=\"token punctuation\">:<\/span> alertmanager<span class=\"token punctuation\">&#8211;<\/span>config<\/p>\n<p>    <span class=\"token comment\"># \u6301\u4e45\u5316\u5b58\u50a8&#xff08;\u7528\u4e8e\u9759\u9ed8\u548c\u901a\u77e5\u72b6\u6001&#xff09;<\/span><br \/>\n    <span class=\"token key atrule\">storage<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">volumeClaimTemplate<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">spec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">storageClassName<\/span><span class=\"token punctuation\">:<\/span> ssd<span class=\"token punctuation\">&#8211;<\/span>sc<br \/>\n          <span class=\"token key atrule\">accessModes<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#034;ReadWriteOnce&#034;<\/span><span class=\"token punctuation\">]<\/span><br \/>\n          <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n              <span class=\"token key atrule\">storage<\/span><span class=\"token punctuation\">:<\/span> 10Gi<\/p>\n<p>    <span class=\"token key atrule\">tolerations<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">key<\/span><span class=\"token punctuation\">:<\/span> node<span class=\"token punctuation\">&#8211;<\/span>role.kubernetes.io\/control<span class=\"token punctuation\">&#8211;<\/span>plane<br \/>\n        <span class=\"token key atrule\">operator<\/span><span class=\"token punctuation\">:<\/span> Exists<br \/>\n        <span class=\"token key atrule\">effect<\/span><span class=\"token punctuation\">:<\/span> NoSchedule<\/p>\n<p><span class=\"token comment\"># \u2500\u2500 Grafana \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">grafana<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">image<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">tag<\/span><span class=\"token punctuation\">:<\/span> 10.4.0<\/p>\n<p>  <span class=\"token comment\"># Admin \u5bc6\u7801<\/span><br \/>\n  <span class=\"token key atrule\">adminPassword<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;&#034;<\/span> <span class=\"token comment\"># \u7559\u7a7a\u5219\u4f7f\u7528 Secret \u7ba1\u7406<\/span><\/p>\n<p>  <span class=\"token comment\"># \u6570\u636e\u6e90&#xff08;Loki\u3001Jaeger \u7b49\u5728\u4e0b\u4e00\u7bc7\u914d\u7f6e&#xff09;<\/span><br \/>\n  <span class=\"token key atrule\">additionalDataSources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> Prometheus<br \/>\n      <span class=\"token key atrule\">type<\/span><span class=\"token punctuation\">:<\/span> prometheus<br \/>\n      <span class=\"token key atrule\">url<\/span><span class=\"token punctuation\">:<\/span> http<span class=\"token punctuation\">:<\/span>\/\/prometheus<span class=\"token punctuation\">&#8211;<\/span>operated.monitoring.svc<span class=\"token punctuation\">:<\/span><span class=\"token number\">9090<\/span><br \/>\n      <span class=\"token key atrule\">access<\/span><span class=\"token punctuation\">:<\/span> proxy<br \/>\n      <span class=\"token key atrule\">isDefault<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n      <span class=\"token key atrule\">jsonData<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">timeInterval<\/span><span class=\"token punctuation\">:<\/span> 30s<\/p>\n<p>  <span class=\"token comment\"># \u8d44\u6e90<\/span><br \/>\n  <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 200m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 512Mi<br \/>\n    <span class=\"token key atrule\">limits<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 1000m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 1Gi<\/p>\n<p>  <span class=\"token comment\"># \u6301\u4e45\u5316<\/span><br \/>\n  <span class=\"token key atrule\">persistence<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n    <span class=\"token key atrule\">size<\/span><span class=\"token punctuation\">:<\/span> 20Gi<br \/>\n    <span class=\"token key atrule\">storageClassName<\/span><span class=\"token punctuation\">:<\/span> ssd<span class=\"token punctuation\">&#8211;<\/span>sc<\/p>\n<p>  <span class=\"token comment\"># \u9884\u88c5\u63d2\u4ef6<\/span><br \/>\n  <span class=\"token key atrule\">plugins<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> grafana<span class=\"token punctuation\">&#8211;<\/span>piechart<span class=\"token punctuation\">&#8211;<\/span>panel<br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> grafana<span class=\"token punctuation\">&#8211;<\/span>polystat<span class=\"token punctuation\">&#8211;<\/span>panel<\/p>\n<p>  <span class=\"token comment\"># Dashboard Providers<\/span><br \/>\n  <span class=\"token key atrule\">dashboardProviders<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">dashboardproviders.yaml<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">apiVersion<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1<\/span><br \/>\n      <span class=\"token key atrule\">providers<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> default<br \/>\n          <span class=\"token key atrule\">orgId<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1<\/span><br \/>\n          <span class=\"token key atrule\">folder<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;&#034;<\/span><br \/>\n          <span class=\"token key atrule\">type<\/span><span class=\"token punctuation\">:<\/span> file<br \/>\n          <span class=\"token key atrule\">disableDeletion<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n          <span class=\"token key atrule\">editable<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n          <span class=\"token key atrule\">options<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">path<\/span><span class=\"token punctuation\">:<\/span> \/var\/lib\/grafana\/dashboards\/default<\/p>\n<p>  <span class=\"token comment\"># \u6838\u5fc3\u770b\u677f&#xff08;K8s \u96c6\u7fa4\u3001\u8282\u70b9\u3001Pod&#xff09;<\/span><br \/>\n  <span class=\"token key atrule\">dashboards<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">default<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">k8s-cluster-overview<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">gnetId<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">15757<\/span>          <span class=\"token comment\"># K8s Cluster Overview<\/span><br \/>\n        <span class=\"token key atrule\">revision<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1<\/span><br \/>\n        <span class=\"token key atrule\">datasource<\/span><span class=\"token punctuation\">:<\/span> Prometheus<br \/>\n      <span class=\"token key atrule\">k8s-nodes<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">gnetId<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">15759<\/span>          <span class=\"token comment\"># K8s Nodes Detail<\/span><br \/>\n        <span class=\"token key atrule\">revision<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1<\/span><br \/>\n        <span class=\"token key atrule\">datasource<\/span><span class=\"token punctuation\">:<\/span> Prometheus<br \/>\n      <span class=\"token key atrule\">k8s-pods<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token key atrule\">gnetId<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">15760<\/span>          <span class=\"token comment\"># K8s Pods Detail<\/span><br \/>\n        <span class=\"token key atrule\">revision<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">1<\/span><br \/>\n        <span class=\"token key atrule\">datasource<\/span><span class=\"token punctuation\">:<\/span> Prometheus<\/p>\n<p><span class=\"token comment\"># \u2500\u2500 kube-state-metrics \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">kubeStateMetrics<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 100m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 200Mi<br \/>\n    <span class=\"token key atrule\">limits<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 200m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 400Mi<\/p>\n<p><span class=\"token comment\"># \u2500\u2500 node-exporter \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">nodeExporter<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">resources<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">requests<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 50m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 64Mi<br \/>\n    <span class=\"token key atrule\">limits<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">cpu<\/span><span class=\"token punctuation\">:<\/span> 200m<br \/>\n      <span class=\"token key atrule\">memory<\/span><span class=\"token punctuation\">:<\/span> 256Mi<\/p>\n<p><span class=\"token comment\"># \u2500\u2500 Prometheus Node Exporter \u2500\u2500<\/span><br \/>\n<span class=\"token key atrule\">prometheus-node-exporter<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">extraArgs<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token punctuation\">&#8211;<\/span><span class=\"token punctuation\">&#8211;<\/span>collector.filesystem.mount<span class=\"token punctuation\">&#8211;<\/span>points<span class=\"token punctuation\">&#8211;<\/span>exclude&#061;^\/(dev<span class=\"token punctuation\">|<\/span>proc<span class=\"token punctuation\">|<\/span>sys<span class=\"token punctuation\">|<\/span>var\/lib\/docker\/.&#043;<span class=\"token punctuation\">|<\/span>var\/lib\/kubelet\/.&#043;)($<span class=\"token punctuation\">|<\/span>\/)<br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token punctuation\">&#8211;<\/span><span class=\"token punctuation\">&#8211;<\/span>collector.filesystem.fs<span class=\"token punctuation\">&#8211;<\/span>types<span class=\"token punctuation\">&#8211;<\/span>exclude&#061;^(autofs<span class=\"token punctuation\">|<\/span>binfmt_misc<span class=\"token punctuation\">|<\/span>cgroup<span class=\"token punctuation\">|<\/span>configfs<span class=\"token punctuation\">|<\/span>debugfs<span class=\"token punctuation\">|<\/span>devpts<span class=\"token punctuation\">|<\/span>devtmpfs<span class=\"token punctuation\">|<\/span>fusectl<span class=\"token punctuation\">|<\/span>hugetlbfs<span class=\"token punctuation\">|<\/span>mqueue<span class=\"token punctuation\">|<\/span>overlay<span class=\"token punctuation\">|<\/span>proc<span class=\"token punctuation\">|<\/span>procfs<span class=\"token punctuation\">|<\/span>pstore<span class=\"token punctuation\">|<\/span>rpc_pipefs<span class=\"token punctuation\">|<\/span>securityfs<span class=\"token punctuation\">|<\/span>sysfs<span class=\"token punctuation\">|<\/span>tracefs)$<\/p>\n<p>\u90e8\u7f72\u547d\u4ee4&#xff1a;<\/p>\n<p><span class=\"token comment\"># \u90e8\u7f72 kube-prometheus-stack<\/span><br \/>\nhelm upgrade <span class=\"token parameter variable\">&#8211;install<\/span> kube-prometheus-stack <span class=\"token punctuation\">\\\\<\/span><br \/>\n  prometheus-community\/kube-prometheus-stack <span class=\"token punctuation\">\\\\<\/span><br \/>\n  <span class=\"token parameter variable\">&#8211;namespace<\/span> monitoring <span class=\"token punctuation\">\\\\<\/span><br \/>\n  <span class=\"token parameter variable\">&#8211;values<\/span> kube-prometheus-values.yaml <span class=\"token punctuation\">\\\\<\/span><br \/>\n  <span class=\"token parameter variable\">&#8211;timeout<\/span> 15m <span class=\"token punctuation\">\\\\<\/span><br \/>\n  <span class=\"token parameter variable\">&#8211;wait<\/span><\/p>\n<p><span class=\"token comment\"># \u9a8c\u8bc1\u6240\u6709\u7ec4\u4ef6\u8fd0\u884c\u6b63\u5e38<\/span><br \/>\nkubectl get pods <span class=\"token parameter variable\">-n<\/span> monitoring<\/p>\n<p><span class=\"token comment\"># \u9884\u671f\u8f93\u51fa\u793a\u4f8b:<\/span><br \/>\n<span class=\"token comment\"># NAME                                                       READY   STATUS<\/span><br \/>\n<span class=\"token comment\"># alertmanager-kube-prometheus-stack-alertmanager-0           2\/2     Running<\/span><br \/>\n<span class=\"token comment\"># kube-prometheus-stack-grafana-xxxxx                         3\/3     Running<\/span><br \/>\n<span class=\"token comment\"># kube-prometheus-stack-kube-state-metrics-xxxxx              1\/1     Running<\/span><br \/>\n<span class=\"token comment\"># kube-prometheus-stack-operator-xxxxx                        1\/1     Running<\/span><br \/>\n<span class=\"token comment\"># kube-prometheus-stack-prometheus-node-exporter-xxxxx        1\/1     Running<\/span><br \/>\n<span class=\"token comment\"># prometheus-kube-prometheus-stack-prometheus-0               2\/2     Running<\/span><\/p>\n<h4>2.2 \u6838\u5fc3\u6307\u6807\u89e3\u8bfb<\/h4>\n<p>\u4ee5\u4e0b\u662f\u6bcf\u4e2a K8s \u8fd0\u7ef4\u5fc5\u987b\u8bb0\u4f4f\u7684\u6838\u5fc3\u6307\u6807&#xff0c;\u6309\u76d1\u63a7\u5206\u5c42\u7ec4\u7ec7&#xff1a;<\/p>\n<p>\u8282\u70b9\u5c42&#xff08;node-exporter&#xff09;&#xff1a;<\/p>\n<table>\n<tr>\u6307\u6807\u542b\u4e49USE\u544a\u8b66\u9608\u503c\u5efa\u8bae<\/tr>\n<tbody>\n<tr>\n<td>node_cpu_seconds_total<\/td>\n<td>CPU \u5404\u6a21\u5f0f\u65f6\u95f4<\/td>\n<td>U<\/td>\n<td>&gt; 80% \u6301\u7eed 10min<\/td>\n<\/tr>\n<tr>\n<td>node_memory_MemAvailable_bytes<\/td>\n<td>\u53ef\u7528\u5185\u5b58<\/td>\n<td>U<\/td>\n<td>&lt; 10% \u603b\u5185\u5b58<\/td>\n<\/tr>\n<tr>\n<td>node_disk_io_time_seconds_total<\/td>\n<td>\u78c1\u76d8 I\/O \u65f6\u95f4<\/td>\n<td>S<\/td>\n<td>&gt; 80% \u6301\u7eed 10min<\/td>\n<\/tr>\n<tr>\n<td>node_filesystem_avail_bytes<\/td>\n<td>\u6587\u4ef6\u7cfb\u7edf\u53ef\u7528\u7a7a\u95f4<\/td>\n<td>U<\/td>\n<td>&lt; 20%<\/td>\n<\/tr>\n<tr>\n<td>node_network_receive_drop_total<\/td>\n<td>\u7f51\u5361\u4e22\u5305<\/td>\n<td>E<\/td>\n<td>&gt; 0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>K8s \u5bf9\u8c61\u5c42&#xff08;kube-state-metrics&#xff09;&#xff1a;<\/p>\n<table>\n<tr>\u6307\u6807\u542b\u4e49\u5173\u952e\u67e5\u8be2<\/tr>\n<tbody>\n<tr>\n<td>kube_deployment_status_replicas_available<\/td>\n<td>\u53ef\u7528\u526f\u672c\u6570<\/td>\n<td>\u4e0e desired \u5bf9\u6bd4<\/td>\n<\/tr>\n<tr>\n<td>kube_pod_status_phase<\/td>\n<td>Pod \u72b6\u6001<\/td>\n<td>\u8fc7\u6ee4 Pending\/Failed<\/td>\n<\/tr>\n<tr>\n<td>kube_node_status_condition<\/td>\n<td>\u8282\u70b9\u72b6\u6001<\/td>\n<td>condition&#061;Ready, status&#061;true<\/td>\n<\/tr>\n<tr>\n<td>kube_persistentvolume_status_phase<\/td>\n<td>PV \u72b6\u6001<\/td>\n<td>\u76d1\u63a7 Available\/Bound \u6bd4\u4f8b<\/td>\n<\/tr>\n<tr>\n<td>kube_job_status_failed<\/td>\n<td>Job \u5931\u8d25\u8ba1\u6570<\/td>\n<td>&gt; 0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u5bb9\u5668\u5c42&#xff08;cAdvisor&#xff09;&#xff1a;<\/p>\n<table>\n<tr>\u6307\u6807\u542b\u4e49<\/tr>\n<tbody>\n<tr>\n<td>container_cpu_usage_seconds_total<\/td>\n<td>\u5bb9\u5668 CPU \u603b\u4f7f\u7528<\/td>\n<\/tr>\n<tr>\n<td>container_memory_working_set_bytes<\/td>\n<td>\u5bb9\u5668\u5de5\u4f5c\u96c6\u5185\u5b58&#xff08;OOM \u5224\u65ad\u4f9d\u636e&#xff09;<\/td>\n<\/tr>\n<tr>\n<td>container_network_transmit_bytes_total<\/td>\n<td>\u5bb9\u5668\u7f51\u7edc\u53d1\u9001\u5b57\u8282<\/td>\n<\/tr>\n<tr>\n<td>container_fs_reads_bytes_total<\/td>\n<td>\u5bb9\u5668\u6587\u4ef6\u7cfb\u7edf\u8bfb\u53d6<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4>2.3 PromQL \u5b9e\u6218\u901f\u67e5<\/h4>\n<p># \u2500\u2500 TOP N \u67e5\u8be2 \u2500\u2500<\/p>\n<p># CPU \u4f7f\u7528 Top 5 Pod&#xff08;\u6309 namespace \u805a\u5408&#xff09;<br \/>\ntopk(5,<br \/>\n  sum(rate(container_cpu_usage_seconds_total{<br \/>\n    container!&#061;&#034;&#034;, namespace!&#061;&#034;kube-system&#034;<br \/>\n  }[5m])) by (namespace, pod)<br \/>\n)<\/p>\n<p># \u5185\u5b58\u4f7f\u7528 Top 10 Pod<br \/>\ntopk(10,<br \/>\n  sum(container_memory_working_set_bytes{<br \/>\n    container!&#061;&#034;&#034;, namespace!&#061;&#034;kube-system&#034;<br \/>\n  }) by (namespace, pod)<br \/>\n)<\/p>\n<p># \u2500\u2500 \u5206\u4f4d\u67e5\u8be2 \u2500\u2500<\/p>\n<p># P99 \u8bf7\u6c42\u5ef6\u8fdf&#xff08;\u5e94\u7528\u9700\u66b4\u9732 histogram \u6307\u6807&#xff09;<br \/>\nhistogram_quantile(0.99,<br \/>\n  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler)<br \/>\n)<\/p>\n<p># P50\/P90\/P99 \u4e00\u8d77\u770b&#xff08;\u591a\u5206\u4f4d&#xff09;<br \/>\nhistogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))<br \/>\nhistogram_quantile(0.90, rate(http_request_duration_seconds_bucket[5m]))<br \/>\nhistogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))<\/p>\n<p># \u2500\u2500 \u9519\u8bef\u7387 \u2500\u2500<\/p>\n<p># 5xx \u9519\u8bef\u7387<br \/>\nsum(rate(http_requests_total{status&#061;~&#034;5..&#034;}[5m])) by (service)<br \/>\n\/<br \/>\nsum(rate(http_requests_total[5m])) by (service)<\/p>\n<p># \u2500\u2500 \u53ef\u7528\u6027 \u2500\u2500<\/p>\n<p># \u670d\u52a1\u53ef\u7528\u526f\u672c\u7387<br \/>\nsum(kube_deployment_status_replicas_available{namespace&#061;&#034;production&#034;})<br \/>\n  by (deployment)<br \/>\n\/<br \/>\nsum(kube_deployment_spec_replicas{namespace&#061;&#034;production&#034;})<br \/>\n  by (deployment)<\/p>\n<p># \u2500\u2500 \u9884\u6d4b\u67e5\u8be2 \u2500\u2500<\/p>\n<p># \u78c1\u76d8\u7a7a\u95f4 24 \u5c0f\u65f6\u9884\u6d4b&#xff08;\u7ebf\u6027\u56de\u5f52&#xff09;<br \/>\npredict_linear(<br \/>\n  node_filesystem_avail_bytes{mountpoint&#061;&#034;\/&#034;}[6h],<br \/>\n  24 * 3600<br \/>\n) &lt; 0<\/p>\n<p># \u2500\u2500 \u53d8\u5316\u7387\u67e5\u8be2 \u2500\u2500<\/p>\n<p># Deployment \u6700\u8fd1 1 \u5c0f\u65f6\u6269\u5bb9\u6b21\u6570&#xff08;\u5224\u65ad\u662f\u5426\u6709\u5f02\u5e38&#xff09;<br \/>\nchanges(kube_deployment_spec_replicas[1h])<\/p>\n<p># Pod \u91cd\u542f\u6b21\u6570\u589e\u957f\u7387&#xff08;\u5224\u65ad CrashLoop \u8d8b\u52bf&#xff09;<br \/>\nrate(kube_pod_container_status_restarts_total[15m])<\/p>\n<p># \u2500\u2500 \u7f3a\u5931\u68c0\u6d4b \u2500\u2500<\/p>\n<p># \u670d\u52a1\u8d85\u8fc7 10 \u5206\u949f\u6ca1\u6709\u6536\u5230\u8bf7\u6c42&#xff08;\u53ef\u80fd\u6302\u4e86&#xff09;<br \/>\nabsent_over_time(http_requests_total{service&#061;&#034;payment&#034;}[10m])<\/p>\n<hr \/>\n<h3>\u4e09\u3001\u8e29\u5751\u4e0e\u6392\u67e5<\/h3>\n<h4>3.1 \u5751\u4e00&#xff1a;Prometheus \u5185\u5b58\u6301\u7eed\u589e\u957f\u76f4\u5230 OOM<\/h4>\n<p>\u73b0\u8c61&#xff1a;Prometheus \u8fd0\u884c\u51e0\u5c0f\u65f6\u540e\u5185\u5b58\u7a81\u7834 8Gi limit&#xff0c;\u88ab OOMKilled \u91cd\u542f&#xff0c;\u5faa\u73af\u5f80\u590d\u3002<\/p>\n<p>\u539f\u56e0&#xff1a;\u67d0\u4e2a ServiceMonitor \u914d\u7f6e\u7684\u91c7\u96c6\u95f4\u9694\u8fc7\u77ed&#xff08;\u5982 5 \u79d2&#xff09;&#xff0c;\u4e14\u8be5\u670d\u52a1\u66b4\u9732\u4e86\u5927\u91cf\u65f6\u95f4\u5e8f\u5217&#xff08;histogram \u6307\u6807\u9ed8\u8ba4\u6bcf\u4e2a bucket \u5bf9\u5e94\u4e00\u4e2a\u65f6\u95f4\u5e8f\u5217&#xff0c;\u4e00\u4e2a histogram \u53ef\u80fd\u6709 50&#043; \u4e2a\u5e8f\u5217&#xff09;\u3002<\/p>\n<p>\u5b9a\u4f4d&#xff1a;<\/p>\n<p><span class=\"token comment\"># 1. \u67e5\u770b Prometheus \u5f53\u524d\u5e8f\u5217\u6570<\/span><br \/>\nkubectl port-forward <span class=\"token parameter variable\">-n<\/span> monitoring svc\/prometheus-operated <span class=\"token number\">9090<\/span>:9090<br \/>\n<span class=\"token function\">curl<\/span> <span class=\"token parameter variable\">-s<\/span> http:\/\/localhost:9090\/api\/v1\/status\/tsdb <span class=\"token operator\">|<\/span> jq <span class=\"token string\">&#039;.data&#039;<\/span><\/p>\n<p><span class=\"token comment\"># \u8f93\u51fa\u793a\u4f8b&#xff08;\u89c2\u5bdf headSeries \u6570\u91cf&#xff09;:<\/span><br \/>\n<span class=\"token comment\"># {<\/span><br \/>\n<span class=\"token comment\">#   &#034;headStats&#034;: {<\/span><br \/>\n<span class=\"token comment\">#     &#034;numSeries&#034;: 8500000,     \u2190 \u5e8f\u5217\u6570 850 \u4e07&#xff01;\u660e\u663e\u5f02\u5e38<\/span><br \/>\n<span class=\"token comment\">#     &#034;chunkCount&#034;: 12300000,<\/span><br \/>\n<span class=\"token comment\">#     &#8230;<\/span><br \/>\n<span class=\"token comment\">#   }<\/span><br \/>\n<span class=\"token comment\"># }<\/span><\/p>\n<p><span class=\"token comment\"># 2. \u627e\u5230\u5e8f\u5217\u6570\u6700\u591a\u7684\u76ee\u6807<\/span><br \/>\n<span class=\"token function\">curl<\/span> <span class=\"token parameter variable\">-s<\/span> http:\/\/localhost:9090\/api\/v1\/targets <span class=\"token operator\">|<\/span> jq <span class=\"token string\">&#039;.data.activeTargets[] | {labels: .labels, scrapeUrl: .scrapeUrl}&#039;<\/span> <span class=\"token operator\">|<\/span> <span class=\"token function\">head<\/span> <span class=\"token parameter variable\">-20<\/span><\/p>\n<p><span class=\"token comment\"># 3. \u67e5\u8be2\u524d 10 \u4e2a\u5e8f\u5217\u6570\u6700\u591a\u7684\u6307\u6807<\/span><br \/>\n<span class=\"token function\">curl<\/span> <span class=\"token parameter variable\">-s<\/span> <span class=\"token string\">&#039;http:\/\/localhost:9090\/api\/v1\/query?query&#061;topk(10, count by(__name__)({__name__&#061;~&#034;.&#043;&#034;}))&#039;<\/span> <span class=\"token operator\">|<\/span> jq<\/p>\n<p>\u89e3\u51b3\u65b9\u6848&#xff1a;<\/p>\n<p><span class=\"token comment\"># \u5728 ServiceMonitor \u6216 PodMonitor \u4e2d\u914d\u7f6e metric relabeling<\/span><br \/>\n<span class=\"token key atrule\">apiVersion<\/span><span class=\"token punctuation\">:<\/span> monitoring.coreos.com\/v1<br \/>\n<span class=\"token key atrule\">kind<\/span><span class=\"token punctuation\">:<\/span> ServiceMonitor<br \/>\n<span class=\"token key atrule\">metadata<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> my<span class=\"token punctuation\">&#8211;<\/span>app<br \/>\n<span class=\"token key atrule\">spec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">endpoints<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">port<\/span><span class=\"token punctuation\">:<\/span> metrics<br \/>\n      <span class=\"token key atrule\">interval<\/span><span class=\"token punctuation\">:<\/span> 30s<br \/>\n      <span class=\"token comment\"># \u5173\u952e&#xff1a;\u53ea\u4fdd\u7559\u9700\u8981\u7684\u6307\u6807<\/span><br \/>\n      <span class=\"token key atrule\">metricRelabelings<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token comment\"># \u4ec5\u4fdd\u7559 http_ \u5f00\u5934\u7684\u6307\u6807<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">sourceLabels<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span>__name__<span class=\"token punctuation\">]<\/span><br \/>\n          <span class=\"token key atrule\">regex<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;http_.*&#039;<\/span><br \/>\n          <span class=\"token key atrule\">action<\/span><span class=\"token punctuation\">:<\/span> keep<br \/>\n        <span class=\"token comment\"># \u5220\u9664\u9ad8\u57fa\u6570\u7684 histogram bucket&#xff08;\u4fdd\u7559 summary \u66ff\u4ee3&#xff09;<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">sourceLabels<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span>__name__<span class=\"token punctuation\">]<\/span><br \/>\n          <span class=\"token key atrule\">regex<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;.*_bucket&#039;<\/span><br \/>\n          <span class=\"token key atrule\">action<\/span><span class=\"token punctuation\">:<\/span> drop<\/p>\n<p>\u540c\u65f6\u5728 Prometheus \u5168\u5c40\u914d\u7f6e\u4e2d\u9650\u5236&#xff1a;<\/p>\n<p><span class=\"token key atrule\">prometheus<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">prometheusSpec<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token comment\"># \u9650\u5236\u6bcf\u4e2a\u76ee\u6807\u7684\u5e8f\u5217\u6570<\/span><br \/>\n    <span class=\"token key atrule\">enforcedSampleLimit<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">50000<\/span><br \/>\n    <span class=\"token comment\"># \u76ee\u6807\u6807\u7b7e\u6570\u91cf\u9650\u5236<\/span><br \/>\n    <span class=\"token key atrule\">enforcedLabelLimit<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">30<\/span><\/p>\n<h4>3.2 \u5751\u4e8c&#xff1a;\u544a\u8b66\u98ce\u66b4<\/h4>\n<p>\u73b0\u8c61&#xff1a;\u96c6\u7fa4\u7f51\u7edc\u6296\u52a8 2 \u5206\u949f&#xff0c;\u6240\u6709\u670d\u52a1\u7684 P99 \u5ef6\u8fdf\u544a\u8b66\u540c\u65f6\u89e6\u53d1&#xff0c;On-Call \u624b\u673a\u88ab\u6253\u7206&#xff0c;\u5b9e\u9645\u53ea\u6709 2 \u4e2a\u670d\u52a1\u771f\u6b63\u53d7\u5f71\u54cd\u3002<\/p>\n<p>\u539f\u56e0&#xff1a;\u544a\u8b66\u89c4\u5219\u6ca1\u6709\u5206\u7ea7&#xff0c;\u6ca1\u6709\u505a\u4f9d\u8d56\u94fe\u8def\u7684\u544a\u8b66\u6291\u5236\u3002\u7f51\u7edc\u95ee\u9898\u662f\u6839\u56e0&#xff0c;\u670d\u52a1\u5ef6\u8fdf\u662f\u75c7\u72b6\u3002<\/p>\n<p>\u89e3\u51b3\u65b9\u6848\u2014\u2014Alertmanager \u6291\u5236\u89c4\u5219&#xff1a;<\/p>\n<p><span class=\"token comment\"># alertmanager-config.yaml<\/span><br \/>\n<span class=\"token key atrule\">apiVersion<\/span><span class=\"token punctuation\">:<\/span> v1<br \/>\n<span class=\"token key atrule\">kind<\/span><span class=\"token punctuation\">:<\/span> Secret<br \/>\n<span class=\"token key atrule\">metadata<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> alertmanager<span class=\"token punctuation\">&#8211;<\/span>config<br \/>\n  <span class=\"token key atrule\">namespace<\/span><span class=\"token punctuation\">:<\/span> monitoring<br \/>\n<span class=\"token key atrule\">stringData<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token key atrule\">alertmanager.yaml<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">|<\/span><span class=\"token scalar string\"><br \/>\n    global:<br \/>\n      resolve_timeout: 5m<br \/>\n      smtp_smarthost: &#039;smtp.company.com:587&#039;<br \/>\n      smtp_from: &#039;alertmanager&#064;company.com&#039;<br \/>\n      smtp_auth_username: &#039;alertmanager&#064;company.com&#039;<br \/>\n      smtp_auth_password: &#039;xxx&#039;<\/span><\/p>\n<p>    <span class=\"token comment\"># \u2500\u2500 \u8def\u7531\u6811 \u2500\u2500<\/span><br \/>\n    <span class=\"token key atrule\">route<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">receiver<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;default-receiver&#039;<\/span><br \/>\n      <span class=\"token key atrule\">group_by<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#039;alertname&#039;<\/span><span class=\"token punctuation\">,<\/span> <span class=\"token string\">&#039;severity&#039;<\/span><span class=\"token punctuation\">]<\/span><br \/>\n      <span class=\"token key atrule\">group_wait<\/span><span class=\"token punctuation\">:<\/span> 30s<br \/>\n      <span class=\"token key atrule\">group_interval<\/span><span class=\"token punctuation\">:<\/span> 5m<br \/>\n      <span class=\"token key atrule\">repeat_interval<\/span><span class=\"token punctuation\">:<\/span> 4h    <span class=\"token comment\"># \u540c\u4e00\u6761\u544a\u8b66\u6700\u591a 4 \u5c0f\u65f6\u91cd\u590d\u4e00\u6b21<\/span><\/p>\n<p>      <span class=\"token key atrule\">routes<\/span><span class=\"token punctuation\">:<\/span><br \/>\n        <span class=\"token comment\"># P0 \u544a\u8b66&#xff1a;\u7acb\u5373\u901a\u77e5<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">severity<\/span><span class=\"token punctuation\">:<\/span> P0<br \/>\n          <span class=\"token key atrule\">receiver<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;pagerduty-critical&#039;<\/span><br \/>\n          <span class=\"token key atrule\">continue<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>        <span class=\"token comment\"># P1 \u544a\u8b66&#xff1a;5 \u5206\u949f\u5206\u7ec4\u540e\u901a\u77e5<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">severity<\/span><span class=\"token punctuation\">:<\/span> P1<br \/>\n          <span class=\"token key atrule\">receiver<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;wechat-p1&#039;<\/span><br \/>\n          <span class=\"token key atrule\">group_wait<\/span><span class=\"token punctuation\">:<\/span> 10s<br \/>\n          <span class=\"token key atrule\">continue<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>        <span class=\"token comment\"># P2 \u544a\u8b66&#xff1a;\u4ec5\u8bb0\u5f55<\/span><br \/>\n        <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n            <span class=\"token key atrule\">severity<\/span><span class=\"token punctuation\">:<\/span> P2<br \/>\n          <span class=\"token key atrule\">receiver<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;null&#039;<\/span><\/p>\n<p>    <span class=\"token comment\"># \u2500\u2500 \u6291\u5236\u89c4\u5219 \u2500\u2500<\/span><br \/>\n    <span class=\"token key atrule\">inhibit_rules<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token comment\"># \u89c4\u52191: \u8282\u70b9 NotReady \u65f6&#xff0c;\u6291\u5236\u8be5\u8282\u70b9\u4e0a\u6240\u6709 Pod \u7684\u544a\u8b66<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">source_match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">alertname<\/span><span class=\"token punctuation\">:<\/span> NodeNotReady<br \/>\n          <span class=\"token key atrule\">severity<\/span><span class=\"token punctuation\">:<\/span> P0<br \/>\n        <span class=\"token key atrule\">target_match_re<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">severity<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;P[12]&#039;<\/span><br \/>\n        <span class=\"token key atrule\">equal<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#039;node&#039;<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>      <span class=\"token comment\"># \u89c4\u52192: \u96c6\u7fa4\u7ea7\u522b\u7684\u7f51\u7edc\u544a\u8b66\u89e6\u53d1\u65f6&#xff0c;\u6291\u5236\u670d\u52a1\u7ea7\u522b\u7684\u5ef6\u8fdf\u544a\u8b66<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">source_match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">alertname<\/span><span class=\"token punctuation\">:<\/span> ClusterNetworkIssue<br \/>\n        <span class=\"token key atrule\">target_match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">alertname<\/span><span class=\"token punctuation\">:<\/span> HighLatency<br \/>\n        <span class=\"token key atrule\">equal<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#039;cluster&#039;<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>      <span class=\"token comment\"># \u89c4\u52193: namespace \u7ea7\u522b\u7684\u544a\u8b66\u89e6\u53d1\u65f6&#xff0c;\u6291\u5236\u8be5 ns \u4e0b\u6240\u6709\u670d\u52a1\u7684\u544a\u8b66<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">source_match<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">alertname<\/span><span class=\"token punctuation\">:<\/span> NamespaceQuotaExceeded<br \/>\n        <span class=\"token key atrule\">target_match_re<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token key atrule\">alertname<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;PodCrashLooping|DeploymentReplicasMismatch&#039;<\/span><br \/>\n        <span class=\"token key atrule\">equal<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token string\">&#039;namespace&#039;<\/span><span class=\"token punctuation\">]<\/span><\/p>\n<p>    <span class=\"token comment\"># \u2500\u2500 \u63a5\u6536\u5668 \u2500\u2500<\/span><br \/>\n    <span class=\"token key atrule\">receivers<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;default-receiver&#039;<\/span><br \/>\n        <span class=\"token key atrule\">webhook_configs<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">url<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;http:\/\/alert-webhook.company.com\/alert&#039;<\/span><\/p>\n<p>      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;pagerduty-critical&#039;<\/span><br \/>\n        <span class=\"token comment\"># \u63a5\u5165 PagerDuty \/ \u7535\u8bdd\u901a\u77e5<\/span><\/p>\n<p>      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;wechat-p1&#039;<\/span><br \/>\n        <span class=\"token key atrule\">wechat_configs<\/span><span class=\"token punctuation\">:<\/span><br \/>\n          <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">corp_id<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;ww1234567890&#039;<\/span><br \/>\n            <span class=\"token key atrule\">to_party<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;2&#039;<\/span><br \/>\n            <span class=\"token key atrule\">agent_id<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;1000001&#039;<\/span><br \/>\n            <span class=\"token key atrule\">api_secret<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;xxx&#039;<\/span><\/p>\n<p>      <span class=\"token punctuation\">&#8211;<\/span> <span class=\"token key atrule\">name<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#039;null&#039;<\/span><\/p>\n<h4>3.3 \u5751\u4e09&#xff1a;node-exporter \u91c7\u96c6\u5931\u8d25&#xff08;\u6258\u7ba1 K8s&#xff09;<\/h4>\n<p>\u73b0\u8c61&#xff1a;\u5728 EKS\/GKE\/ACK \u7b49\u6258\u7ba1 K8s \u4e0a&#xff0c;node-exporter \u90e8\u7f72\u6b63\u5e38\u4f46 Prometheus \u65e0\u6cd5\u91c7\u96c6&#xff0c;targets \u9875\u9762\u663e\u793a context deadline exceeded\u3002<\/p>\n<p>\u539f\u56e0&#xff1a;\u6258\u7ba1 K8s \u7684\u8282\u70b9 IP \u662f\u4e91\u5382\u5546 VPC \u5185\u7f51 IP&#xff0c;Prometheus \u53ef\u80fd\u65e0\u6cd5\u76f4\u63a5\u901a\u8fc7 Pod \u7f51\u7edc\u8bbf\u95ee\u3002\u6b64\u5916&#xff0c;\u5b89\u5168\u7ec4\/\u9632\u706b\u5899\u53ef\u80fd\u6ca1\u6709\u653e\u884c node-exporter \u7aef\u53e3&#xff08;9100&#xff09;\u3002<\/p>\n<p>\u5b9a\u4f4d&#xff1a;<\/p>\n<p><span class=\"token comment\"># 1. \u624b\u52a8\u6d4b\u8bd5 node-exporter \u53ef\u8fbe\u6027<\/span><br \/>\n<span class=\"token assign-left variable\">NODE_EXPORTER_POD<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token variable\"><span class=\"token variable\">$(<\/span>kubectl get pod <span class=\"token parameter variable\">-n<\/span> monitoring <span class=\"token parameter variable\">-l<\/span> app.kubernetes.io\/name<span class=\"token operator\">&#061;<\/span>prometheus-node-exporter <span class=\"token parameter variable\">-o<\/span> <span class=\"token assign-left variable\">jsonpath<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#039;{.items[0].metadata.name}&#039;<\/span><span class=\"token variable\">)<\/span><\/span><br \/>\n<span class=\"token assign-left variable\">NODE_IP<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token variable\"><span class=\"token variable\">$(<\/span>kubectl get pod <span class=\"token parameter variable\">-n<\/span> monitoring $NODE_EXPORTER_POD <span class=\"token parameter variable\">-o<\/span> <span class=\"token assign-left variable\">jsonpath<\/span><span class=\"token operator\">&#061;<\/span><span class=\"token string\">&#039;{.status.hostIP}&#039;<\/span><span class=\"token variable\">)<\/span><\/span><\/p>\n<p><span class=\"token comment\"># 2. \u4ece Prometheus Pod \u5185\u6d4b\u8bd5\u7f51\u7edc<\/span><br \/>\nkubectl <span class=\"token builtin class-name\">exec<\/span> <span class=\"token parameter variable\">-n<\/span> monitoring prometheus-kube-prometheus-stack-prometheus-0 &#8212; <span class=\"token punctuation\">\\\\<\/span><br \/>\n  <span class=\"token function\">wget<\/span> -qO- http:\/\/<span class=\"token variable\">${NODE_IP}<\/span>:9100\/metrics<\/p>\n<p>\u89e3\u51b3\u65b9\u6848&#xff1a;<\/p>\n<p><span class=\"token key atrule\">prometheus-node-exporter<\/span><span class=\"token punctuation\">:<\/span><br \/>\n  <span class=\"token comment\"># \u4f7f\u7528 hostNetwork \u6a21\u5f0f&#xff08;DaemonSet \u9ed8\u8ba4\u4f7f\u7528&#xff09;<\/span><br \/>\n  <span class=\"token key atrule\">hostNetwork<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n  <span class=\"token key atrule\">hostPID<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><\/p>\n<p>  <span class=\"token comment\"># \u6216\u6539\u7528 Service \u7aef\u70b9&#xff08;\u63a8\u8350&#xff09;<\/span><br \/>\n  <span class=\"token key atrule\">service<\/span><span class=\"token punctuation\">:<\/span><br \/>\n    <span class=\"token key atrule\">enabled<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token boolean important\">true<\/span><br \/>\n    <span class=\"token key atrule\">type<\/span><span class=\"token punctuation\">:<\/span> ClusterIP<br \/>\n    <span class=\"token key atrule\">port<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">9100<\/span><br \/>\n    <span class=\"token key atrule\">targetPort<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">9100<\/span><br \/>\n    <span class=\"token key atrule\">annotations<\/span><span class=\"token punctuation\">:<\/span><br \/>\n      <span class=\"token key atrule\">prometheus.io\/scrape<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token string\">&#034;true&#034;<\/span><\/p>\n<hr \/>\n<h3>\u56db\u3001\u6700\u4f73\u5b9e\u8df5<\/h3>\n<h4>\u544a\u8b66\u5206\u7ea7\u6e05\u5355<\/h4>\n<p>P0&#xff08;5 \u5206\u949f\u5185\u54cd\u5e94&#xff09;         P1&#xff08;30 \u5206\u949f\u5185\u54cd\u5e94&#xff09;        P2&#xff08;\u5de5\u4f5c\u65f6\u95f4\u5904\u7406&#xff09;<br \/>\n\u251c\u2500 \u96c6\u7fa4\u6838\u5fc3\u7ec4\u4ef6\u4e0d\u53ef\u7528        \u251c\u2500 \u670d\u52a1\u53ef\u7528\u526f\u672c &lt; \u671f\u671b      \u251c\u2500 \u8bc1\u4e66\u5373\u5c06\u8fc7\u671f<br \/>\n\u251c\u2500 Node NotReady &gt; 5min     \u251c\u2500 P99 \u5ef6\u8fdf &gt; 1s(5min)     \u251c\u2500 PV \u5bb9\u91cf &gt; 80%<br \/>\n\u251c\u2500 \u6838\u5fc3\u670d\u52a1\u9519\u8bef\u7387 &gt; 5%       \u251c\u2500 \u78c1\u76d8\u53ef\u7528 &lt; 20%          \u251c\u2500 HPA \u8fbe\u5230\u6700\u5927\u526f\u672c<br \/>\n\u251c\u2500 \u8bc1\u4e66\u5df2\u8fc7\u671f               \u251c\u2500 Pod CrashLoopBackOff    \u251c\u2500 \u5185\u5b58\u4f7f\u7528 &gt; 80%(\u975e\u7d27\u6025)<br \/>\n\u2514\u2500 PV \u65e0\u6cd5\u7ed1\u5b9a               \u2514\u2500 CPU Throttling &gt; 50%    \u2514\u2500 ConfigMap\/Secret \u8d85\u989d<\/p>\n<h4>\u544a\u8b66\u89c4\u5219\u8bbe\u8ba1\u539f\u5219<\/h4>\n<ul>\n<li class=\"task-list-item\"> \u6bcf\u6761\u89c4\u5219\u5fc5\u987b\u6709 for \u6301\u7eed\u65f6\u95f4\u2014\u2014\u8fc7\u6ee4\u77ac\u65f6\u6296\u52a8&#xff0c;\u5efa\u8bae P0 &gt;&#061; 1min&#xff0c;P1 &gt;&#061; 5min&#xff0c;P2 &gt;&#061; 10min<\/li>\n<li class=\"task-list-item\"> summary \u5199\u6e05\u695a\u5f71\u54cd\u8303\u56f4&#xff0c;\u201c\u67d0\u670d\u52a1\u9519\u8bef\u7387\u9ad8\u201d \u4e0d\u5982 \u201cpayment \u670d\u52a1 P99 \u5ef6\u8fdf 2.3s&#xff08;\u9608\u503c 1s&#xff09;\u201d<\/li>\n<li class=\"task-list-item\"> description \u5199\u6392\u67e5\u7b2c\u4e00\u6b65&#xff0c;\u201c\u68c0\u67e5 payment-* Pod \u65e5\u5fd7\u548c DB \u8fde\u63a5\u201d<\/li>\n<li class=\"task-list-item\"> \u591a\u7528 rate()&#xff0c;\u5c11\u7528 irate()\u2014\u2014\u5e73\u6ed1\u544a\u8b66&#xff0c;\u51cf\u5c11\u566a\u70b9<\/li>\n<li class=\"task-list-item\"> \u544a\u8b66\u89c4\u5219\u505a\u51cf\u6cd5\u2014\u2014\u5982\u679c\u4f60\u4ece\u6765\u6ca1\u54cd\u5e94\u8fc7\u67d0\u6761\u544a\u8b66&#xff0c;\u5220\u6389\u5b83<\/li>\n<li class=\"task-list-item\"> \u4e00\u6b21\u544a\u8b66\u98ce\u66b4\u540e\u5fc5\u987b\u505a RCA \u548c\u89c4\u5219\u4f18\u5316<\/li>\n<\/ul>\n<h4>\u770b\u677f\u8bbe\u8ba1\u6e05\u5355<\/h4>\n<ul>\n<li class=\"task-list-item\"> \u4f7f\u7528\u53d8\u91cf&#xff08;Variables&#xff09;&#xff1a;namespace\u3001deployment\u3001pod \u505a\u6210\u4e0b\u62c9\u9009\u62e9<\/li>\n<li class=\"task-list-item\"> \u7b2c\u4e00\u884c\u653e RED \u6307\u6807&#xff1a;Rate \/ Errors \/ Duration \u4e00\u773c\u770b\u5230\u670d\u52a1\u5065\u5eb7<\/li>\n<li class=\"task-list-item\"> \u4e0b\u65b9\u653e USE \u6307\u6807&#xff1a;CPU\/Mem\/Disk\/Network \u65f6\u95f4\u5e8f\u5217<\/li>\n<li class=\"task-list-item\"> \u53f3\u4e0a\u89d2\u52a0\u90e8\u7f72\u6807\u6ce8&#xff08;Annotations&#xff09;&#xff1a;\u53d1\u5e03\u4e8b\u4ef6\u5728 Grafana \u4e0a\u6807\u7ad6\u7ebf<\/li>\n<li class=\"task-list-item\"> \u544a\u8b66\u9608\u503c\u753b\u6c34\u5e73\u7ebf&#xff1a;Prometheus Alert \u9608\u503c\u5728\u56fe\u8868\u4e0a\u53ef\u89c6\u5316<\/li>\n<li class=\"task-list-item\"> \u6bcf\u4e2a\u9762\u677f\u6709 Description&#xff1a;\u89e3\u91ca\u6307\u6807\u542b\u4e49\u548c\u6392\u67e5\u601d\u8def<\/li>\n<\/ul>\n<h4>\u957f\u671f\u5b58\u50a8\u9009\u578b<\/h4>\n<table>\n<tr>\u65b9\u6848\u4f18\u70b9\u7f3a\u70b9\u9002\u7528\u89c4\u6a21<\/tr>\n<tbody>\n<tr>\n<td>Thanos<\/td>\n<td>Prometheus \u539f\u751f\u517c\u5bb9&#xff0c;Sidecar \u6a21\u5f0f\u65e0\u4fb5\u5165<\/td>\n<td>\u7ec4\u4ef6\u591a&#xff0c;\u8fd0\u7ef4\u590d\u6742<\/td>\n<td>\u5927\u578b\u96c6\u7fa4(&gt;100\u8282\u70b9)<\/td>\n<\/tr>\n<tr>\n<td>VictoriaMetrics<\/td>\n<td>\u5355\u4e8c\u8fdb\u5236&#xff0c;\u538b\u7f29\u7387\u6781\u9ad8(7x)&#xff0c;\u67e5\u8be2\u5feb<\/td>\n<td>\u975e Prometheus \u5b98\u65b9\u9879\u76ee<\/td>\n<td>\u4e2d\u5c0f\u89c4\u6a21\u9996\u9009<\/td>\n<\/tr>\n<tr>\n<td>Mimir<\/td>\n<td>Grafana \u5b98\u65b9&#xff0c;\u5bf9\u8c61\u5b58\u50a8\u539f\u751f&#xff0c;\u591a\u79df\u6237<\/td>\n<td>\u8f83\u65b0&#xff0c;\u751f\u6001\u4e0d\u5982 Thanos<\/td>\n<td>\u591a\u79df\u6237 SaaS<\/td>\n<\/tr>\n<tr>\n<td>Cortex<\/td>\n<td>\u6210\u719f&#xff0c;\u6c34\u5e73\u6269\u5c55<\/td>\n<td>\u8fd0\u7ef4\u590d\u6742&#xff0c;Mimir \u66ff\u4ee3\u4e2d<\/td>\n<td>\u4e0d\u63a8\u8350\u65b0\u9879\u76ee<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u63a8\u8350\u65b9\u6848&#xff1a;\u4e2d\u5c0f\u89c4\u6a21\u7528 VictoriaMetrics&#xff0c;\u5927\u89c4\u6a21\u5f0f\u7528 Thanos\u3002<\/p>\n<hr \/>\n<h3>\u4e94\u3001\u5c0f\u7ed3<\/h3>\n<p>\u76d1\u63a7\u4f53\u7cfb\u7684\u5efa\u8bbe\u662f\u8fed\u4ee3\u51fa\u6765\u7684&#xff0c;\u4e0d\u662f\u8bbe\u8ba1\u51fa\u6765\u7684\u3002\u6211\u7684\u5efa\u8bae&#xff1a;<\/p>\n<li>\u7b2c\u4e00\u5468&#xff1a;\u90e8\u7f72 Prometheus &#043; Grafana &#043; kube-state-metrics &#043; node-exporter&#xff0c;\u5148\u770b\u5230\u6570\u636e<\/li>\n<li>\u7b2c\u4e8c\u5468&#xff1a;\u6309 RED \u65b9\u6cd5\u8bba\u7ed9\u4f60\u7684\u6838\u5fc3\u670d\u52a1\u914d\u4e0a\u544a\u8b66&#xff0c;\u544a\u8b66\u5148\u8d70\u90ae\u4ef6<\/li>\n<li>\u7b2c\u4e09\u5468&#xff1a;\u6839\u636e\u544a\u8b66\u53cd\u9988\u8c03\u6574\u9608\u503c\u548c\u5206\u7ea7&#xff0c;\u63a5\u5165 Alertmanager \u8def\u7531<\/li>\n<li>\u7b2c\u56db\u5468&#xff1a;\u4f18\u5316\u770b\u677f\u3001\u6dfb\u52a0\u957f\u671f\u5b58\u50a8\u3001\u5efa\u7acb On-Call \u6d41\u7a0b<\/li>\n<p>\u53e6\u5916\u8bb0\u4f4f\u4e00\u53e5\u8bdd&#xff1a;\u544a\u8b66\u7684\u4ef7\u503c &#061; \u54cd\u5e94\u884c\u52a8\u7684\u6709\u6548\u6027\u3002\u5982\u679c\u4f60\u770b\u5230\u4e00\u6761\u544a\u8b66\u540e\u4e0d\u77e5\u9053\u8be5\u505a\u4ec0\u4e48&#xff0c;\u8bf4\u660e\u8fd9\u6761\u544a\u8b66\u5b9a\u4e49\u6709\u95ee\u9898\u3002<\/p>\n<hr \/>\n<h3>\u601d\u8003\u9898<\/h3>\n<li>Prometheus \u7684 rate() \u51fd\u6570\u8981\u6c42\u81f3\u5c11\u4e24\u4e2a\u91c7\u6837\u70b9\u624d\u80fd\u8ba1\u7b97&#xff0c;\u5982\u679c\u4f60\u7684\u76ee\u6807 scrape \u95f4\u9694\u662f 30s&#xff0c;rate(&#8230;[1m]) \u4e3a\u4ec0\u4e48\u6bd4 rate(&#8230;[30s]) \u66f4\u53ef\u9760&#xff1f;<\/li>\n<li>\u5982\u679c\u4f60\u7684\u670d\u52a1\u4f7f\u7528 HPA \u81ea\u52a8\u6269\u7f29\u5bb9&#xff0c;\u544a\u8b66\u89c4\u5219\u4e2d\u7684 sum by (pod) \u4f1a\u5728\u6269\u7f29\u5bb9\u65f6\u51fa\u73b0\u4ec0\u4e48\u73b0\u8c61&#xff1f;\u5982\u4f55\u907f\u514d&#xff1f;<\/li>\n<li>\u4e00\u4e2a\u8282\u70b9 CPU \u4f7f\u7528\u7387 95% \u4f46\u8d1f\u8f7d&#xff08;load average&#xff09;\u53ea\u6709 2&#xff0c;\u53e6\u4e00\u4e2a\u8282\u70b9 CPU \u4f7f\u7528\u7387 60% \u4f46\u8d1f\u8f7d 32&#xff0c;\u54ea\u4e2a\u8282\u70b9\u66f4&#034;\u5fd9&#034;&#xff1f;\u4e3a\u4ec0\u4e48&#xff1f;<\/li>\n<hr \/>\n<h3>\u5ef6\u4f38\u9605\u8bfb<\/h3>\n<ul>\n<li>Prometheus \u5b98\u65b9\u6587\u6863 &#8211; \u6700\u4f73\u5b9e\u8df5<\/li>\n<li>Google SRE Book &#8211; \u76d1\u63a7\u5206\u5e03\u5f0f\u7cfb\u7edf<\/li>\n<li>Robust Perception &#8211; PromQL \u8fdb\u9636<\/li>\n<li>USE \u65b9\u6cd5\u8bba\u539f\u6587&#xff08;Brendan Gregg&#xff09;<\/li>\n<li>RED \u65b9\u6cd5\u8bba&#xff08;Tom Wilkie&#xff09;<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>\u76d1\u63a7\u544a\u8b66&#xff1a;PrometheusGrafanaAlertmanager \u9ec4\u91d1\u7ec4\u5408 \u4e00\u53e5\u8bdd\u5b9a\u4f4d&#xff1a;K8s \u8be5\u76d1\u63a7\u54ea\u4e9b\u6307\u6807&#xff1f;\u544a\u8b66\u600e\u4e48\u4e0d\u6f0f\u4e0d\u566a&#xff1f;\u4e00\u5957\u65b9\u6cd5\u8bba\u52a0\u4e09\u4ef6\u5957&#xff08;Prometheus  Grafana  Alertmanager&#xff09;\u7ed9\u4f60\u7b54\u6848\u3002 \u5199\u5728\u524d\u9762<br \/>\n\u6211\u8bb0\u5f97\u6709\u4e00\u6b21\u51cc\u6668 2 \u70b9\u88ab On-Call \u7535\u8bdd\u53eb\u9192\u2014\u2014\u751f\u4ea7 Deploy \u6b63\u5728\\&#8221;\u5065\u5eb7\\&#8221;\u8fd0\u884c&#xff08;Readiness Probe \u8fd4\u56de 20<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[120,678,44],"topic":[],"class_list":["post-83428","post","type-post","status-publish","format-standard","hentry","category-server","tag-kubernetes","tag-prometheus","tag-44"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/83428.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"\u76d1\u63a7\u544a\u8b66&#xff1a;PrometheusGrafanaAlertmanager \u9ec4\u91d1\u7ec4\u5408 \u4e00\u53e5\u8bdd\u5b9a\u4f4d&#xff1a;K8s \u8be5\u76d1\u63a7\u54ea\u4e9b\u6307\u6807&#xff1f;\u544a\u8b66\u600e\u4e48\u4e0d\u6f0f\u4e0d\u566a&#xff1f;\u4e00\u5957\u65b9\u6cd5\u8bba\u52a0\u4e09\u4ef6\u5957&#xff08;Prometheus Grafana Alertmanager&#xff09;\u7ed9\u4f60\u7b54\u6848\u3002 \u5199\u5728\u524d\u9762 \u6211\u8bb0\u5f97\u6709\u4e00\u6b21\u51cc\u6668 2 \u70b9\u88ab On-Call \u7535\u8bdd\u53eb\u9192\u2014\u2014\u751f\u4ea7 Deploy \u6b63\u5728&quot;\u5065\u5eb7&quot;\u8fd0\u884c&#xff08;Readiness Probe \u8fd4\u56de 20\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/83428.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-07-25T14:09:06+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/83428.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/83428.html\",\"name\":\"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-07-25T14:09:06+00:00\",\"dateModified\":\"2026-07-25T14:09:06+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/83428.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/83428.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/83428.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/83428.html","og_locale":"zh_CN","og_type":"article","og_title":"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"\u76d1\u63a7\u544a\u8b66&#xff1a;PrometheusGrafanaAlertmanager \u9ec4\u91d1\u7ec4\u5408 \u4e00\u53e5\u8bdd\u5b9a\u4f4d&#xff1a;K8s \u8be5\u76d1\u63a7\u54ea\u4e9b\u6307\u6807&#xff1f;\u544a\u8b66\u600e\u4e48\u4e0d\u6f0f\u4e0d\u566a&#xff1f;\u4e00\u5957\u65b9\u6cd5\u8bba\u52a0\u4e09\u4ef6\u5957&#xff08;Prometheus Grafana Alertmanager&#xff09;\u7ed9\u4f60\u7b54\u6848\u3002 \u5199\u5728\u524d\u9762 \u6211\u8bb0\u5f97\u6709\u4e00\u6b21\u51cc\u6668 2 \u70b9\u88ab On-Call \u7535\u8bdd\u53eb\u9192\u2014\u2014\u751f\u4ea7 Deploy \u6b63\u5728\"\u5065\u5eb7\"\u8fd0\u884c&#xff08;Readiness Probe \u8fd4\u56de 20","og_url":"https:\/\/www.wsisp.com\/helps\/83428.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-07-25T14:09:06+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"11 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/83428.html","url":"https:\/\/www.wsisp.com\/helps\/83428.html","name":"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-07-25T14:09:06+00:00","dateModified":"2026-07-25T14:09:06+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/83428.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/83428.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/83428.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u3010K8S \u8fd0\u7ef4\u5b9e\u6218\u301114-\u76d1\u63a7\u544a\u8b66Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/83428","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=83428"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/83428\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=83428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=83428"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=83428"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=83428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}