实战：混沌工程入门，模拟服务器宕机的故障演练

核心提要

混沌工程是通过主动注入可控故障，验证系统韧性与故障自愈能力的工程实践，核心目标是“在故障发生前发现系统脆弱点”。本次入门实战以模拟服务器宕机为核心场景，基于Chaos Mesh（轻量级混沌工程工具）+ Kubernetes 技术栈，从环境准备、故障定义、演练执行、结果分析到复盘优化，完整覆盖混沌工程演练全流程，零基础掌握混沌工程核心操作，验证系统在服务器宕机后的服务可用性、流量切换、数据一致性能力，最终实现“发现脆弱点→优化系统→提升韧性”的目标。

本次实战为入门级，无需复杂的分布式系统基础，基于单节点K8s（Minikube）搭建演练环境，所有操作均提供可直接执行的命令，快速上手混沌工程核心思路。

一、混沌工程核心概念（入门必知）

📊 实战总览思维导图

快速把握本次混沌演练的全局结构：

#mermaid-svg-yMy776qjT2XM7vHm{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yMy776qjT2XM7vHm .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yMy776qjT2XM7vHm .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yMy776qjT2XM7vHm .error-icon{fill:#552222;}#mermaid-svg-yMy776qjT2XM7vHm .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yMy776qjT2XM7vHm .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yMy776qjT2XM7vHm .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yMy776qjT2XM7vHm .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yMy776qjT2XM7vHm .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yMy776qjT2XM7vHm .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yMy776qjT2XM7vHm .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yMy776qjT2XM7vHm .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yMy776qjT2XM7vHm .marker.cross{stroke:#333333;}#mermaid-svg-yMy776qjT2XM7vHm svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yMy776qjT2XM7vHm p{margin:0;}#mermaid-svg-yMy776qjT2XM7vHm .edge{stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .section–1 rect,#mermaid-svg-yMy776qjT2XM7vHm .section–1 path,#mermaid-svg-yMy776qjT2XM7vHm .section–1 circle,#mermaid-svg-yMy776qjT2XM7vHm .section–1 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section–1 text{fill:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth–1{stroke-width:17;}#mermaid-svg-yMy776qjT2XM7vHm .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-0 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-0 path,#mermaid-svg-yMy776qjT2XM7vHm .section-0 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-0 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yMy776qjT2XM7vHm .section-0 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-0{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-0{stroke-width:14;}#mermaid-svg-yMy776qjT2XM7vHm .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-1 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-1 path,#mermaid-svg-yMy776qjT2XM7vHm .section-1 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-1 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-1 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-1{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-1{stroke-width:11;}#mermaid-svg-yMy776qjT2XM7vHm .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-2 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-2 path,#mermaid-svg-yMy776qjT2XM7vHm .section-2 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-2 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-2 text{fill:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-2{stroke-width:8;}#mermaid-svg-yMy776qjT2XM7vHm .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-3 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-3 path,#mermaid-svg-yMy776qjT2XM7vHm .section-3 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-3 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-3 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-3{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-3{stroke-width:5;}#mermaid-svg-yMy776qjT2XM7vHm .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-4 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-4 path,#mermaid-svg-yMy776qjT2XM7vHm .section-4 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-4 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-4 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-4{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-4{stroke-width:2;}#mermaid-svg-yMy776qjT2XM7vHm .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-5 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-5 path,#mermaid-svg-yMy776qjT2XM7vHm .section-5 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-5 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-5 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-5{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-5{stroke-width:-1;}#mermaid-svg-yMy776qjT2XM7vHm .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-6 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-6 path,#mermaid-svg-yMy776qjT2XM7vHm .section-6 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-6 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-6 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-6{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-6{stroke-width:-4;}#mermaid-svg-yMy776qjT2XM7vHm .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-7 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-7 path,#mermaid-svg-yMy776qjT2XM7vHm .section-7 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-7 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-7 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-7{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-7{stroke-width:-7;}#mermaid-svg-yMy776qjT2XM7vHm .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-8 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-8 path,#mermaid-svg-yMy776qjT2XM7vHm .section-8 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-8 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-8 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-8{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-8{stroke-width:-10;}#mermaid-svg-yMy776qjT2XM7vHm .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-9 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-9 path,#mermaid-svg-yMy776qjT2XM7vHm .section-9 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-9 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-9 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-9{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-9{stroke-width:-13;}#mermaid-svg-yMy776qjT2XM7vHm .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-10 rect,#mermaid-svg-yMy776qjT2XM7vHm .section-10 path,#mermaid-svg-yMy776qjT2XM7vHm .section-10 circle,#mermaid-svg-yMy776qjT2XM7vHm .section-10 polygon,#mermaid-svg-yMy776qjT2XM7vHm .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-10 text{fill:black;}#mermaid-svg-yMy776qjT2XM7vHm .node-icon-10{font-size:40px;color:black;}#mermaid-svg-yMy776qjT2XM7vHm .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .edge-depth-10{stroke-width:-16;}#mermaid-svg-yMy776qjT2XM7vHm .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yMy776qjT2XM7vHm .disabled,#mermaid-svg-yMy776qjT2XM7vHm .disabled circle,#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:lightgray;}#mermaid-svg-yMy776qjT2XM7vHm .disabled text{fill:#efefef;}#mermaid-svg-yMy776qjT2XM7vHm .section-root rect,#mermaid-svg-yMy776qjT2XM7vHm .section-root path,#mermaid-svg-yMy776qjT2XM7vHm .section-root circle,#mermaid-svg-yMy776qjT2XM7vHm .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-yMy776qjT2XM7vHm .section-root text{fill:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .section-root span{color:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .section-2 span{color:#ffffff;}#mermaid-svg-yMy776qjT2XM7vHm .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-yMy776qjT2XM7vHm .edge{fill:none;}#mermaid-svg-yMy776qjT2XM7vHm .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-yMy776qjT2XM7vHm :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

混沌工程：模拟服务器宕机演练

🔍 前置认知

混沌工程核心原则

本次实验目标

🛠️ 环境准备

工具栈选型

Docker集群搭建

🧪 混沌实验

基础版：手动模拟宕机

专业版：ChaosBlade工具注入

📈 观测与分析

故障场景表现

系统脆弱点发现

✅ 恢复与优化

故障快速回滚

Nginx健康检查优化

优化后效果验证

📝 复盘与进阶

实验闭环总结

入门避坑指南

进阶场景拓展

1. 核心原则（5大核心原则）

#mermaid-svg-ViUOEFEofRLRPYPA{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ViUOEFEofRLRPYPA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ViUOEFEofRLRPYPA .error-icon{fill:#552222;}#mermaid-svg-ViUOEFEofRLRPYPA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ViUOEFEofRLRPYPA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ViUOEFEofRLRPYPA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ViUOEFEofRLRPYPA .marker.cross{stroke:#333333;}#mermaid-svg-ViUOEFEofRLRPYPA svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ViUOEFEofRLRPYPA p{margin:0;}#mermaid-svg-ViUOEFEofRLRPYPA .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-ViUOEFEofRLRPYPA .cluster-label text{fill:#333;}#mermaid-svg-ViUOEFEofRLRPYPA .cluster-label span{color:#333;}#mermaid-svg-ViUOEFEofRLRPYPA .cluster-label span p{background-color:transparent;}#mermaid-svg-ViUOEFEofRLRPYPA .label text,#mermaid-svg-ViUOEFEofRLRPYPA span{fill:#333;color:#333;}#mermaid-svg-ViUOEFEofRLRPYPA .node rect,#mermaid-svg-ViUOEFEofRLRPYPA .node circle,#mermaid-svg-ViUOEFEofRLRPYPA .node ellipse,#mermaid-svg-ViUOEFEofRLRPYPA .node polygon,#mermaid-svg-ViUOEFEofRLRPYPA .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ViUOEFEofRLRPYPA .rough-node .label text,#mermaid-svg-ViUOEFEofRLRPYPA .node .label text,#mermaid-svg-ViUOEFEofRLRPYPA .image-shape .label,#mermaid-svg-ViUOEFEofRLRPYPA .icon-shape .label{text-anchor:middle;}#mermaid-svg-ViUOEFEofRLRPYPA .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ViUOEFEofRLRPYPA .rough-node .label,#mermaid-svg-ViUOEFEofRLRPYPA .node .label,#mermaid-svg-ViUOEFEofRLRPYPA .image-shape .label,#mermaid-svg-ViUOEFEofRLRPYPA .icon-shape .label{text-align:center;}#mermaid-svg-ViUOEFEofRLRPYPA .node.clickable{cursor:pointer;}#mermaid-svg-ViUOEFEofRLRPYPA .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ViUOEFEofRLRPYPA .arrowheadPath{fill:#333333;}#mermaid-svg-ViUOEFEofRLRPYPA .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ViUOEFEofRLRPYPA .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ViUOEFEofRLRPYPA .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ViUOEFEofRLRPYPA .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ViUOEFEofRLRPYPA .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ViUOEFEofRLRPYPA .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ViUOEFEofRLRPYPA .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ViUOEFEofRLRPYPA .cluster text{fill:#333;}#mermaid-svg-ViUOEFEofRLRPYPA .cluster span{color:#333;}#mermaid-svg-ViUOEFEofRLRPYPA div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ViUOEFEofRLRPYPA .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ViUOEFEofRLRPYPA rect.text{fill:none;stroke-width:0;}#mermaid-svg-ViUOEFEofRLRPYPA .icon-shape,#mermaid-svg-ViUOEFEofRLRPYPA .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ViUOEFEofRLRPYPA .icon-shape p,#mermaid-svg-ViUOEFEofRLRPYPA .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ViUOEFEofRLRPYPA .icon-shape rect,#mermaid-svg-ViUOEFEofRLRPYPA .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ViUOEFEofRLRPYPA .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ViUOEFEofRLRPYPA .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ViUOEFEofRLRPYPA :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

定义稳态明确系统正常基准

提出假设预判故障下的系统表现

受控注入故障严格隔离影响范围

验证假设发现系统脆弱点

恢复与复盘优化系统并闭环

• 可控故障：仅在指定环境（测试/预发）注入故障，严格控制故障范围、持续时间，避免影响生产；

• 先定基准：明确系统正常运行的基准指标（如服务可用率99.9%、接口响应时间<500ms）；

• 假设驱动：先提出假设（如“服务器宕机后，服务可自动切换至备用节点，可用率保持99%以上”），再通过演练验证；

• 复盘优化：无论演练是否符合假设，均需复盘，针对脆弱点优化系统；

• 闭环迭代：将演练发现的问题、优化方案纳入迭代，持续提升系统韧性。

2. 本次演练核心目标

我们将搭建极简分布式架构，验证单服务器宕机后系统的容错能力，核心目标如下：

#mermaid-svg-KZD2z10qMJi4Xbzp{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KZD2z10qMJi4Xbzp .error-icon{fill:#552222;}#mermaid-svg-KZD2z10qMJi4Xbzp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KZD2z10qMJi4Xbzp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KZD2z10qMJi4Xbzp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-KZD2z10qMJi4Xbzp .marker.cross{stroke:#333333;}#mermaid-svg-KZD2z10qMJi4Xbzp svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-KZD2z10qMJi4Xbzp p{margin:0;}#mermaid-svg-KZD2z10qMJi4Xbzp .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster-label text{fill:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster-label span{color:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster-label span p{background-color:transparent;}#mermaid-svg-KZD2z10qMJi4Xbzp .label text,#mermaid-svg-KZD2z10qMJi4Xbzp span{fill:#333;color:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp .node rect,#mermaid-svg-KZD2z10qMJi4Xbzp .node circle,#mermaid-svg-KZD2z10qMJi4Xbzp .node ellipse,#mermaid-svg-KZD2z10qMJi4Xbzp .node polygon,#mermaid-svg-KZD2z10qMJi4Xbzp .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-KZD2z10qMJi4Xbzp .rough-node .label text,#mermaid-svg-KZD2z10qMJi4Xbzp .node .label text,#mermaid-svg-KZD2z10qMJi4Xbzp .image-shape .label,#mermaid-svg-KZD2z10qMJi4Xbzp .icon-shape .label{text-anchor:middle;}#mermaid-svg-KZD2z10qMJi4Xbzp .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-KZD2z10qMJi4Xbzp .rough-node .label,#mermaid-svg-KZD2z10qMJi4Xbzp .node .label,#mermaid-svg-KZD2z10qMJi4Xbzp .image-shape .label,#mermaid-svg-KZD2z10qMJi4Xbzp .icon-shape .label{text-align:center;}#mermaid-svg-KZD2z10qMJi4Xbzp .node.clickable{cursor:pointer;}#mermaid-svg-KZD2z10qMJi4Xbzp .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-KZD2z10qMJi4Xbzp .arrowheadPath{fill:#333333;}#mermaid-svg-KZD2z10qMJi4Xbzp .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-KZD2z10qMJi4Xbzp .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-KZD2z10qMJi4Xbzp .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KZD2z10qMJi4Xbzp .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-KZD2z10qMJi4Xbzp .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KZD2z10qMJi4Xbzp .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster text{fill:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp .cluster span{color:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-KZD2z10qMJi4Xbzp .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-KZD2z10qMJi4Xbzp rect.text{fill:none;stroke-width:0;}#mermaid-svg-KZD2z10qMJi4Xbzp .icon-shape,#mermaid-svg-KZD2z10qMJi4Xbzp .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KZD2z10qMJi4Xbzp .icon-shape p,#mermaid-svg-KZD2z10qMJi4Xbzp .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-KZD2z10qMJi4Xbzp .icon-shape rect,#mermaid-svg-KZD2z10qMJi4Xbzp .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KZD2z10qMJi4Xbzp .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-KZD2z10qMJi4Xbzp .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-KZD2z10qMJi4Xbzp :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

实验验证目标

负载均衡是否自动剔除故障节点

核心服务可用性是否保持100%

用户是否无感知故障发生

3. 实验架构可视化

本次实战基于Docker搭建隔离环境，结合K8s与Chaos Mesh实现，简化版架构与流量走向如下：

#mermaid-svg-dPygP6k5L1hgFc2W{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dPygP6k5L1hgFc2W .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dPygP6k5L1hgFc2W .error-icon{fill:#552222;}#mermaid-svg-dPygP6k5L1hgFc2W .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dPygP6k5L1hgFc2W .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dPygP6k5L1hgFc2W .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dPygP6k5L1hgFc2W .marker.cross{stroke:#333333;}#mermaid-svg-dPygP6k5L1hgFc2W svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dPygP6k5L1hgFc2W p{margin:0;}#mermaid-svg-dPygP6k5L1hgFc2W .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-dPygP6k5L1hgFc2W .cluster-label text{fill:#333;}#mermaid-svg-dPygP6k5L1hgFc2W .cluster-label span{color:#333;}#mermaid-svg-dPygP6k5L1hgFc2W .cluster-label span p{background-color:transparent;}#mermaid-svg-dPygP6k5L1hgFc2W .label text,#mermaid-svg-dPygP6k5L1hgFc2W span{fill:#333;color:#333;}#mermaid-svg-dPygP6k5L1hgFc2W .node rect,#mermaid-svg-dPygP6k5L1hgFc2W .node circle,#mermaid-svg-dPygP6k5L1hgFc2W .node ellipse,#mermaid-svg-dPygP6k5L1hgFc2W .node polygon,#mermaid-svg-dPygP6k5L1hgFc2W .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dPygP6k5L1hgFc2W .rough-node .label text,#mermaid-svg-dPygP6k5L1hgFc2W .node .label text,#mermaid-svg-dPygP6k5L1hgFc2W .image-shape .label,#mermaid-svg-dPygP6k5L1hgFc2W .icon-shape .label{text-anchor:middle;}#mermaid-svg-dPygP6k5L1hgFc2W .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dPygP6k5L1hgFc2W .rough-node .label,#mermaid-svg-dPygP6k5L1hgFc2W .node .label,#mermaid-svg-dPygP6k5L1hgFc2W .image-shape .label,#mermaid-svg-dPygP6k5L1hgFc2W .icon-shape .label{text-align:center;}#mermaid-svg-dPygP6k5L1hgFc2W .node.clickable{cursor:pointer;}#mermaid-svg-dPygP6k5L1hgFc2W .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dPygP6k5L1hgFc2W .arrowheadPath{fill:#333333;}#mermaid-svg-dPygP6k5L1hgFc2W .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dPygP6k5L1hgFc2W .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dPygP6k5L1hgFc2W .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dPygP6k5L1hgFc2W .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dPygP6k5L1hgFc2W .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dPygP6k5L1hgFc2W .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dPygP6k5L1hgFc2W .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dPygP6k5L1hgFc2W .cluster text{fill:#333;}#mermaid-svg-dPygP6k5L1hgFc2W .cluster span{color:#333;}#mermaid-svg-dPygP6k5L1hgFc2W div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dPygP6k5L1hgFc2W .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dPygP6k5L1hgFc2W rect.text{fill:none;stroke-width:0;}#mermaid-svg-dPygP6k5L1hgFc2W .icon-shape,#mermaid-svg-dPygP6k5L1hgFc2W .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dPygP6k5L1hgFc2W .icon-shape p,#mermaid-svg-dPygP6k5L1hgFc2W .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dPygP6k5L1hgFc2W .icon-shape rect,#mermaid-svg-dPygP6k5L1hgFc2W .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dPygP6k5L1hgFc2W .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dPygP6k5L1hgFc2W .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dPygP6k5L1hgFc2W :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

K8s集群环境

宿主机环境（Minikube节点）

访问节点30080端口

轮询转发流量

轮询转发流量

注入宕机故障

用户/测试工具 curl/ab

Chaos Mesh故障注入工具

负载均衡 Ingress-Nginx

Web节点1 nginx-demo-1

Web节点2 nginx-demo-2

4. 技术栈选型

二、演练环境准备

1. 环境规划

2. 基础环境部署（一步到位命令）

（1）安装Minikube（单节点K8s）

# 下载并安装Minikube（Linux/macOS通用）
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# 启动Minikube（分配2C4G资源，驱动为Docker，开启Ingress）
minikube start –cpus=2 –memory=4096 –driver=docker –addons=ingress

# 验证Minikube状态（显示Ready即为正常）
minikube status
kubectl get nodes

（2）安装Chaos Mesh（核心混沌工程工具）

使用官方脚本一键部署，自动适配Minikube环境：

# 下载并执行Chaos Mesh安装脚本
curl -sSL https://mirrors.chaos-mesh.org/v2.5/install.sh | bash -s — –namespace=chaos-mesh

# 验证Chaos Mesh部署状态（所有Pod均为Running即为正常）
kubectl get pods -n chaos-mesh -w

关键验证：出现 chaos-controller-manager、chaos-daemon、chaos-dashboard 三个Pod且状态为Running，说明部署成功。

（3）暴露Chaos Mesh可视化面板（Dashboard）

# 端口转发，访问本地8080端口即可进入Dashboard
kubectl port-forward -n chaos-mesh svc/chaos-dashboard 8080:2333 –address 0.0.0.0

访问地址：http://服务器IP:8080，无需账号密码（入门环境），生产环境需配置认证。

（4）部署目标应用（Nginx Web服务，带多副本）

为了验证服务器宕机后的服务韧性，将目标应用部署为2个副本（模拟多实例部署），配置K8s Service暴露服务：

# 创建目标应用命名空间
kubectl create namespace demo-app

# 部署Nginx应用（2个副本）
kubectl apply -f – <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-demo
namespace: demo-app
spec:
replicas: 2
selector:
matchLabels:
app: nginx-demo
template:
metadata:
labels:
app: nginx-demo
spec:
containers:
– name: nginx
image: nginx:1.25-alpine
ports:
– containerPort: 80
resources:
requests:
cpu: 100m
memory: 128Mi
—
apiVersion: v1
kind: Service
metadata:
name: nginx-demo-svc
namespace: demo-app
spec:
type: NodePort
selector:
app: nginx-demo
ports:
– port: 80
targetPort: 80
nodePort: 30080
EOF

# 验证应用部署状态（2个Pod为Running即为正常）
kubectl get pods -n demo-app

（5）验证目标应用正常访问

# 获取Minikube节点IP
MINIKUBE_IP=$(minikube ip)

# 测试访问（返回Nginx默认页面即为正常）
curl http://$MINIKUBE_IP:30080

预期输出：Nginx的HTML默认页面，说明应用正常运行。

三、演练前置准备

1. 定义系统基准指标

在注入故障前，明确系统正常运行的基准指标，作为故障演练的对比依据：

2. 提出演练假设

本次演练核心假设：当运行目标应用的K8s节点发生宕机故障后，目标应用的服务可用率保持≥99%，接口响应时间仍<500ms，K8s可自动将Pod调度至健康节点（若有）。

3. 制定故障注入规则（可控故障核心）

为避免故障扩散，严格定义故障注入的范围、持续时间、恢复方式：

4. 启动指标监控（实时查看故障期间变化）

# 实时监控应用Pod状态
kubectl get pods -n demo-app -w

# 新开终端，实时监控K8s节点状态
kubectl get nodes -w

# 新开终端，持续测试服务可用性（每2秒请求一次）
while true; do
curl -o /dev/null -s -w "时间：%{time_now} 响应码：%{http_code} 响应时间：%{time_total}s\\n" http://$MINIKUBE_IP:30080
sleep 2
done

四、实战步骤：注入服务器宕机故障（两种方式）

实验全流程时序图

完整还原从环境搭建、故障注入到恢复优化的全流程时序：

Web2Web1Chaos Mesh工具Web2（nginx-demo-2）Web1（nginx-demo-1）LB负载均衡（Ingress）测试工具工程师Web2Web1Chaos Mesh工具Web2（nginx-demo-2）Web1（nginx-demo-1）LB负载均衡（Ingress）测试工具工程师#mermaid-svg-AvHEgKKF88dmdT9t{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-AvHEgKKF88dmdT9t .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-AvHEgKKF88dmdT9t .error-icon{fill:#552222;}#mermaid-svg-AvHEgKKF88dmdT9t .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-AvHEgKKF88dmdT9t .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-AvHEgKKF88dmdT9t .marker{fill:#333333;stroke:#333333;}#mermaid-svg-AvHEgKKF88dmdT9t .marker.cross{stroke:#333333;}#mermaid-svg-AvHEgKKF88dmdT9t svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-AvHEgKKF88dmdT9t p{margin:0;}#mermaid-svg-AvHEgKKF88dmdT9t .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-AvHEgKKF88dmdT9t text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-AvHEgKKF88dmdT9t .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-AvHEgKKF88dmdT9t .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-AvHEgKKF88dmdT9t .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-AvHEgKKF88dmdT9t .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-AvHEgKKF88dmdT9t #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-AvHEgKKF88dmdT9t .sequenceNumber{fill:white;}#mermaid-svg-AvHEgKKF88dmdT9t #sequencenumber{fill:#333;}#mermaid-svg-AvHEgKKF88dmdT9t #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-AvHEgKKF88dmdT9t .messageText{fill:#333;stroke:none;}#mermaid-svg-AvHEgKKF88dmdT9t .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-AvHEgKKF88dmdT9t .labelText,#mermaid-svg-AvHEgKKF88dmdT9t .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-AvHEgKKF88dmdT9t .loopText,#mermaid-svg-AvHEgKKF88dmdT9t .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-AvHEgKKF88dmdT9t .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-AvHEgKKF88dmdT9t .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-AvHEgKKF88dmdT9t .noteText,#mermaid-svg-AvHEgKKF88dmdT9t .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-AvHEgKKF88dmdT9t .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-AvHEgKKF88dmdT9t .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-AvHEgKKF88dmdT9t .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-AvHEgKKF88dmdT9t .actorPopupMenu{position:absolute;}#mermaid-svg-AvHEgKKF88dmdT9t .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-AvHEgKKF88dmdT9t .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-AvHEgKKF88dmdT9t .actor-man circle,#mermaid-svg-AvHEgKKF88dmdT9t line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-AvHEgKKF88dmdT9t :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}🛠️ 阶段1：环境搭建与稳态验证✅ 确认稳态：请求成功率100%，流量轮询分配🧪 阶段2：注入宕机故障📈 阶段3：观测故障表现❌ 发现脆弱点：无健康检查导致部分请求失败✅ 阶段4：故障恢复与系统优化✅ 优化验证：服务可用性保持100%📝 阶段5：复盘总结，完成闭环部署Nginx应用（2副本）部署Nginx应用（2副本）配置NodePort暴露服务多次发起请求转发请求转发请求返回Nginx默认页面返回Nginx默认页面配置节点宕机故障（持续60s）模拟服务器宕机，Pod状态异常持续发起请求尝试转发请求无响应/连接失败返回502错误转发请求返回Nginx默认页面故障自动恢复（60s后）配置存活/就绪探针（健康检查）再次模拟Web1宕机周期性健康探测探测失败将节点从集群中剔除转发全部流量全部请求正常返回

Chaos Mesh支持可视化界面（Dashboard） 和命令行（YAML） 两种故障注入方式，入门推荐使用可视化界面，操作更直观；命令行方式适合自动化演练，本次两种方式均实现。

方式1：可视化界面（Dashboard）注入故障（推荐）

步骤1：进入Chaos Mesh Dashboard

访问 http://服务器IP:8080，左侧导航栏选择混沌实验 → 点击新建实验。

步骤2：选择故障类型

• 实验范围：选择集群内；

• 故障类型：点击节点故障（对应服务器宕机）；

• 实验名称：自定义（如node-down-demo-01）；

• 实验描述：模拟Minikube节点宕机，验证Nginx服务韧性。

步骤3：配置故障参数（核心）

步骤4：提交实验并启动

点击提交 → 实验列表中找到新建的实验 → 点击启动，故障开始注入。

方式2：命令行（YAML）注入故障（适合自动化）

直接通过kubectl apply创建Chaos Mesh的故障资源，实现无界面故障注入：

# 注入节点宕机故障，持续60秒
kubectl apply -f – <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: NodeChaos
metadata:
name: node-down-demo-01
namespace: chaos-mesh
spec:
action: node-down # 故障类型：节点宕机
mode: one # 故障范围：单个节点
selector:
nodes:
– minikube # 目标节点（Minikube节点名）
duration: 60s # 故障持续时间
scheduler:
cron: "@once" # 执行方式：立即执行一次
EOF

验证故障启动：

# 查看故障状态（phase为Running即为注入成功）
kubectl get nodechaos -n chaos-mesh

五、故障期间：实时观察系统状态

故障注入后，观察之前启动的监控终端，记录核心现象（入门级典型现象）：

1. K8s节点状态变化

节点状态从Ready快速变为NotReady，表示服务器宕机故障注入成功。

2. 应用Pod状态变化

Pod状态从Running变为Unknown/NotReady，因为节点宕机，K8s无法监控Pod状态。

3. 服务可用性变化

• 单节点K8s环境下：服务暂时不可用（curl返回连接失败，响应码非200）；

• 多节点K8s环境下：K8s会将Pod自动调度至健康节点，服务中断时间极短（秒级），随后恢复可用。

4. 指标变化

• 接口响应时间：故障期间无响应（超时）；

• 服务可用率：故障期间降至0%；

• 故障恢复后：节点状态恢复Ready，Pod状态恢复Running，服务可用率回到100%。

六、故障恢复：自动+人工兜底

1. 自动恢复

故障持续时间（60s）结束后，Chaos Mesh会自动清理故障，节点状态从NotReady恢复为Ready，Pod状态也会随之恢复，服务自动恢复可用。

2. 人工兜底恢复（若自动恢复失败）

方式1：可视化界面停止故障

Dashboard → 混沌实验 → 找到对应实验 → 点击停止，故障立即恢复。

方式2：命令行删除故障资源

# 删除节点宕机故障资源，立即恢复
kubectl delete nodechaos node-down-demo-01 -n chaos-mesh

# 验证节点状态恢复
kubectl get nodes

3. 验证故障完全恢复

# 测试服务访问（返回Nginx页面即为恢复）
curl http://$MINIKUBE_IP:30080

# 验证Pod状态（2个Running）
kubectl get pods -n demo-app

七、演练结果分析与复盘

故障场景对比可视化

直观展示「有无健康检查」两种情况下，服务器宕机后的系统表现差异：

#mermaid-svg-9WjmCzJ1PZe4fx1y{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9WjmCzJ1PZe4fx1y .error-icon{fill:#552222;}#mermaid-svg-9WjmCzJ1PZe4fx1y .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9WjmCzJ1PZe4fx1y .marker{fill:#333333;stroke:#333333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .marker.cross{stroke:#333333;}#mermaid-svg-9WjmCzJ1PZe4fx1y svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-9WjmCzJ1PZe4fx1y p{margin:0;}#mermaid-svg-9WjmCzJ1PZe4fx1y .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster-label text{fill:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster-label span{color:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster-label span p{background-color:transparent;}#mermaid-svg-9WjmCzJ1PZe4fx1y .label text,#mermaid-svg-9WjmCzJ1PZe4fx1y span{fill:#333;color:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .node rect,#mermaid-svg-9WjmCzJ1PZe4fx1y .node circle,#mermaid-svg-9WjmCzJ1PZe4fx1y .node ellipse,#mermaid-svg-9WjmCzJ1PZe4fx1y .node polygon,#mermaid-svg-9WjmCzJ1PZe4fx1y .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .rough-node .label text,#mermaid-svg-9WjmCzJ1PZe4fx1y .node .label text,#mermaid-svg-9WjmCzJ1PZe4fx1y .image-shape .label,#mermaid-svg-9WjmCzJ1PZe4fx1y .icon-shape .label{text-anchor:middle;}#mermaid-svg-9WjmCzJ1PZe4fx1y .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .rough-node .label,#mermaid-svg-9WjmCzJ1PZe4fx1y .node .label,#mermaid-svg-9WjmCzJ1PZe4fx1y .image-shape .label,#mermaid-svg-9WjmCzJ1PZe4fx1y .icon-shape .label{text-align:center;}#mermaid-svg-9WjmCzJ1PZe4fx1y .node.clickable{cursor:pointer;}#mermaid-svg-9WjmCzJ1PZe4fx1y .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .arrowheadPath{fill:#333333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9WjmCzJ1PZe4fx1y .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-9WjmCzJ1PZe4fx1y .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9WjmCzJ1PZe4fx1y .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster text{fill:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y .cluster span{color:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9WjmCzJ1PZe4fx1y .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9WjmCzJ1PZe4fx1y rect.text{fill:none;stroke-width:0;}#mermaid-svg-9WjmCzJ1PZe4fx1y .icon-shape,#mermaid-svg-9WjmCzJ1PZe4fx1y .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9WjmCzJ1PZe4fx1y .icon-shape p,#mermaid-svg-9WjmCzJ1PZe4fx1y .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-9WjmCzJ1PZe4fx1y .icon-shape rect,#mermaid-svg-9WjmCzJ1PZe4fx1y .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9WjmCzJ1PZe4fx1y .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9WjmCzJ1PZe4fx1y .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9WjmCzJ1PZe4fx1y :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

🟢 配置健康检查后

Web1服务器宕机

LB周期性执行健康探测

连续探测失败，将Web1从集群剔除

所有流量自动转发到正常节点Web2

服务可用性保持100%，用户无感知

🔴 未配置健康检查

Web1服务器宕机

LB仍按轮询规则转发流量到Web1

部分请求出现502错误

服务可用性下降，用户感知故障

1. 验证演练假设

本次演练为单节点K8s环境，核心假设验证结果：

• ❌ 服务可用率：故障期间降至0%，未达到≥99%的目标；

• ❌ 接口响应时间：故障期间无响应，超出<500ms的目标；

• ✅ 自动恢复：故障超时后，系统自动恢复至正常状态；

• 多节点K8s环境下：✅ 服务可用率可保持≥99%，Pod自动调度至健康节点。

2. 分析系统脆弱点（核心）

从演练结果中提炼系统韧性的脆弱点，这是混沌工程的核心价值：

3. 针对性优化建议（入门级可落地）

针对以上脆弱点，给出低成本、可落地的优化方案，从根本上提升系统韧性：

# 为Nginx添加健康检查（更新Deployment）
kubectl patch deployment nginx-demo -n demo-app –type='json' -p '[
{
"op": "add",
"path": "/spec/template/spec/containers/0/livenessProbe",
"value": {
"httpGet": {"path": "/", "port": 80},
"initialDelaySeconds": 5,
"periodSeconds": 10
}
},
{
"op": "add",
"path": "/spec/template/spec/containers/0/readinessProbe",
"value": {
"httpGet": {"path": "/", "port": 80},
"initialDelaySeconds": 5,
"periodSeconds": 10
}
}
]'

4. 实验闭环总结

#mermaid-svg-vLwPqdBspxi31L6l{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-vLwPqdBspxi31L6l .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-vLwPqdBspxi31L6l .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-vLwPqdBspxi31L6l .error-icon{fill:#552222;}#mermaid-svg-vLwPqdBspxi31L6l .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-vLwPqdBspxi31L6l .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-vLwPqdBspxi31L6l .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-vLwPqdBspxi31L6l .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-vLwPqdBspxi31L6l .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-vLwPqdBspxi31L6l .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-vLwPqdBspxi31L6l .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-vLwPqdBspxi31L6l .marker{fill:#333333;stroke:#333333;}#mermaid-svg-vLwPqdBspxi31L6l .marker.cross{stroke:#333333;}#mermaid-svg-vLwPqdBspxi31L6l svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-vLwPqdBspxi31L6l p{margin:0;}#mermaid-svg-vLwPqdBspxi31L6l .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-vLwPqdBspxi31L6l .cluster-label text{fill:#333;}#mermaid-svg-vLwPqdBspxi31L6l .cluster-label span{color:#333;}#mermaid-svg-vLwPqdBspxi31L6l .cluster-label span p{background-color:transparent;}#mermaid-svg-vLwPqdBspxi31L6l .label text,#mermaid-svg-vLwPqdBspxi31L6l span{fill:#333;color:#333;}#mermaid-svg-vLwPqdBspxi31L6l .node rect,#mermaid-svg-vLwPqdBspxi31L6l .node circle,#mermaid-svg-vLwPqdBspxi31L6l .node ellipse,#mermaid-svg-vLwPqdBspxi31L6l .node polygon,#mermaid-svg-vLwPqdBspxi31L6l .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-vLwPqdBspxi31L6l .rough-node .label text,#mermaid-svg-vLwPqdBspxi31L6l .node .label text,#mermaid-svg-vLwPqdBspxi31L6l .image-shape .label,#mermaid-svg-vLwPqdBspxi31L6l .icon-shape .label{text-anchor:middle;}#mermaid-svg-vLwPqdBspxi31L6l .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-vLwPqdBspxi31L6l .rough-node .label,#mermaid-svg-vLwPqdBspxi31L6l .node .label,#mermaid-svg-vLwPqdBspxi31L6l .image-shape .label,#mermaid-svg-vLwPqdBspxi31L6l .icon-shape .label{text-align:center;}#mermaid-svg-vLwPqdBspxi31L6l .node.clickable{cursor:pointer;}#mermaid-svg-vLwPqdBspxi31L6l .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-vLwPqdBspxi31L6l .arrowheadPath{fill:#333333;}#mermaid-svg-vLwPqdBspxi31L6l .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-vLwPqdBspxi31L6l .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-vLwPqdBspxi31L6l .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-vLwPqdBspxi31L6l .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-vLwPqdBspxi31L6l .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-vLwPqdBspxi31L6l .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-vLwPqdBspxi31L6l .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-vLwPqdBspxi31L6l .cluster text{fill:#333;}#mermaid-svg-vLwPqdBspxi31L6l .cluster span{color:#333;}#mermaid-svg-vLwPqdBspxi31L6l div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-vLwPqdBspxi31L6l .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-vLwPqdBspxi31L6l rect.text{fill:none;stroke-width:0;}#mermaid-svg-vLwPqdBspxi31L6l .icon-shape,#mermaid-svg-vLwPqdBspxi31L6l .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-vLwPqdBspxi31L6l .icon-shape p,#mermaid-svg-vLwPqdBspxi31L6l .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-vLwPqdBspxi31L6l .icon-shape rect,#mermaid-svg-vLwPqdBspxi31L6l .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-vLwPqdBspxi31L6l .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-vLwPqdBspxi31L6l .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-vLwPqdBspxi31L6l :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

未配置健康检查，故障节点无法自动剔除

Nginx添加健康检查规则+多节点部署

单节点宕机后服务可用性100%

分布式架构需配套健康检查与容灾机制

八、进阶演练：扩展故障场景（入门后尝试）

掌握基础的服务器宕机演练后，可基于本次环境，扩展以下常见混沌工程故障场景，进一步验证系统韧性：

1. 容器级故障

模拟单个应用Pod宕机（而非整个服务器），验证K8s的Pod重启/重建能力：

• 故障类型：容器故障（container-kill）；

• 核心验证：Pod被杀死后，K8s是否自动重建Pod，服务是否无感知。

2. 网络故障

模拟服务器网络中断/延迟，验证系统的网络韧性：

• 故障类型：网络故障（network-partition/network-latency）；

• 核心验证：网络中断后，服务是否有降级策略，网络恢复后是否正常。

3. 资源耗尽故障

模拟服务器CPU/内存耗尽，验证应用的资源隔离能力：

• 故障类型：压力故障（stress-cpu/stress-memory）；

• 核心验证：单个应用资源耗尽后，是否影响其他应用，K8s是否触发资源限制。

4. 持久化存储故障

模拟服务器存储卷挂载失败，验证应用的数据一致性能力：

• 故障类型：存储故障（pod-failure）；

• 核心验证：存储故障后，应用是否有数据备份，恢复后数据是否一致。

九、混沌工程入门核心经验总结

十、常见问题排查（入门踩坑指南）

混沌工程入门避坑核心要点

❌ 禁止直接在生产环境开展入门实验
✅ 从隔离环境（本地/测试环境）开始
✅ 从小故障（单节点宕机）到复杂故障（网络分区、资源耗尽）逐步进阶
✅ 实验前必须明确故障回滚预案
✅ 配套监控工具，提升故障观测能力

1. Chaos Mesh部署后Pod状态为ImagePullBackOff

原因：镜像拉取失败，国内网络无法访问官方镜像；

解决：使用国内镜像源重新部署：

# 卸载原有Chaos Mesh
curl -sSL https://mirrors.chaos-mesh.org/v2.5/uninstall.sh | bash –

# 使用国内镜像源部署
curl -sSL https://mirrors.chaos-mesh.org/v2.5/install.sh | bash -s — –namespace=chaos-mesh –image-registry=registry.aliyuncs.com/chaos-mesh

2. 注入节点宕机故障后，节点状态未变为NotReady

原因：Minikube节点未开启特权模式，Chaos Mesh无权限修改节点状态；

解决：重新启动Minikube并开启特权模式：

minikube stop && minikube delete
minikube start –cpus=2 –memory=4096 –driver=docker –addons=ingress –privileged

3. 故障恢复后，应用Pod仍为Unknown状态

原因：K8s节点缓存未刷新；

解决：手动重启应用Deployment：

kubectl rollout restart deployment nginx-demo -n demo-app

4. Chaos Mesh Dashboard无法访问

原因：端口转发失败，或服务器防火墙未开放8080端口；

解决：

5. 注入故障后，服务未按预期恢复

原因：故障持续时间配置过长，或故障资源未被清理；

解决：

可视化内容扩展方案

若需提升文档可视化效果，可参考以下方式优化：