K8s服务器由于机房维护，关机后启动问题

一、背景

服务器由于升级原因，临时断电，断电前手动停止程序并手动关机（k8s环境没有理会）。恢复供电后，问题处理.
环境: centos7 docker k8s
[root@wdy01 ~]# kubectl get node
E0727 13:54:51.638488 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.638859 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.640220 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.641583 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
E0727 13:54:51.642889 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout=32s”: dial tcp 10.1.1.1:6443: connect: connection refused
The connection to the server 10.1.1.1:6443 was refused – did you specify the right host or port?
[root@wdy01 ~]#

二、解决过程

2.1 先检查集群有效期，发现正常

[root@wdy01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 07, 2027 00:21 UTC 344d ca no
apiserver Jul 07, 2027 00:21 UTC 344d ca no
apiserver-etcd-client Jul 07, 2027 00:21 UTC 344d etcd-ca no
apiserver-kubelet-client Jul 07, 2027 00:21 UTC 344d ca no
controller-manager.conf Jul 07, 2027 00:21 UTC 344d ca no
etcd-healthcheck-client Jul 07, 2027 00:21 UTC 344d etcd-ca no
etcd-peer Jul 07, 2027 00:21 UTC 344d etcd-ca no
etcd-server Jul 07, 2027 00:21 UTC 344d etcd-ca no
front-proxy-client Jul 07, 2027 00:21 UTC 344d front-proxy-ca no
scheduler.conf Jul 07, 2027 00:21 UTC 344d ca no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 02, 2035 01:45 UTC 8y no
etcd-ca Jul 02, 2035 01:45 UTC 8y no
front-proxy-ca Jul 02, 2035 01:45 UTC 8y no

2.2 查看kubelet日志

Jul 27 14:22:56 wdy01 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Jul 27 14:22:56 wdy01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jul 27 14:22:56 wdy01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jul 27 14:22:56 wdy01 kubelet[237447]: Flag –container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's –config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 14:22:56 wdy01 kubelet[237447]: Flag –pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.600080 237447 server.go:203] "–pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.605137 237447 server.go:467] "Kubelet version" kubeletVersion="v1.2x.2"
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.605175 237447 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.606111 237447 server.go:895] "Client rotation is on, will bootstrap in background"
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.607831 237447 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Jul 27 14:22:56 wdy01 kubelet[237447]: I0727 14:22:56.608791 237447 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Jul 27 14:22:56 wdy01 kubelet[237447]: W0727 14:22:56.609247 237447 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
Jul 27 14:22:56 wdy01 kubelet[237447]: "Addr": "/var/run/cri-dockerd.sock",
Jul 27 14:22:56 wdy01 kubelet[237447]: "ServerName": "/var/run/cri-dockerd.sock",
Jul 27 14:22:56 wdy01 kubelet[237447]: "Attributes": null,
Jul 27 14:22:56 wdy01 kubelet[237447]: "BalancerAttributes": null,
Jul 27 14:22:56 wdy01 kubelet[237447]: "Type": 0,
Jul 27 14:22:56 wdy01 kubelet[237447]: "Metadata": null
Jul 27 14:22:56 wdy01 kubelet[237447]: }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refused"
Jul 27 14:22:56 wdy01 kubelet[237447]: E0727 14:22:56.610057 237447 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint \\"unix:///var/run/cri-dockerd.sock\\": rpc error: code = Unavailable desc = connection error: desc = \\"transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refused\\""
Jul 27 14:22:56 wdy01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Jul 27 14:22:56 wdy01 systemd[1]: Unit kubelet.service entered failed state.
Jul 27 14:22:56 wdy01 systemd[1]: kubelet.service failed.

如上日志：发现kubelet 启动失败核心原因：
dial unix /var/run/cri-dockerd.sock: connect: connection refused
cri-dockerd 服务没启动 / 未安装，kubelet 无法对接 Docker 的 CRI 接口，直接崩溃退出

2.3 解决

1）查看服务状态

systemctl status cri-dockerd

2）查看套接字文件是否存在

ls -l /var/run/cri-dockerd.sock

3）发现断电重启后，cri-dockerd服务丢失了

我下载过 cri-dockerd-0.3.14-3.el7.x86_64.rpm，执行rpm -ivh cri-dockerd-0.3.14-3.el7.x86_64.rpm安装。

4）重启并确认

systemctl daemon-reload
# 开机自启并启动
systemctl enable –now cri-docker
# 查看状态确认running
systemctl status cri-docker
# 检查socket文件是否生成
ls /var/run/cri-dockerd.sock

5）重启并确认

修复 kubelet 配置，消除 cri-dockerd.sock 连接拒绝报错

vi /var/lib/kubelet/config.yaml

追加一行（没有就加，有就确认一致）：

containerRuntimeEndpoint: unix:///var/run/cri-dockerd.sock

6) 重启kubelet

systemctl daemon-reload
systemctl restart kubelet
# 实时观察日志
journalctl -u kubelet -f

# 如果是docker作为容器的话，可执行如下命令。其余容器方法类似
docker ps |grep kube-apiserver|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-controller-manage|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-scheduler|grep -v pause|awk '{print $1}'|xargs -i docker restart {}

三、补充

补充1：根据实际情况，docker必须保障正常。
systemctl restart docker
systemctl enable docker

补充2：如启动后发现，worker节点也没有正常ready，则worker节点也需要排查并安装cri-dockerd。

四、结果

如下图，执行kubectl get nodes查看集群状态已正常。

五、新问题

后端程序链接nacos报错：java.net.NoRouteToHostException: No route to host (Host unreachable)
测试发现，宿主机worker是可以连通01部署的nacos的。但程序pod内不通。关闭防火墙也不行。
最后发现：02 节点上 docker/flannel 残留 iptables nat 规则 拦截了 Pod 访问宿主机网段的回包，清空 iptables 后立刻恢复连通。
最后是在02 清空所有 iptables 规则（nat+filter） 解决的。

# 清空过滤链
iptables -F
# 清空nat转发链（关键）
iptables -t nat -F
# 设置默认转发允许
iptables -P FORWARD ACCEPT

END