一、kubesphere v4.1.2 离线包制作

为什么选择kubesphere v4.1.2是一个社区版，并且支持管理多集群，在高的社区版就支持单集群。需要多集群的也可以采购商业版。本系统将从离线包制作、安装、组建（开发、测试、uat、生产）联邦集群，到项目在联邦集群如何部署及常见的问题处理。

1、离线部署资源制作

本文增加了一台能联网的 ksp-deploy 节点，用来制作离线部署资源包。

在该节点下载 KubeKey 最新版（v3.1.7）。具体 KubeKey 版本号可以在 KubeKey 发行页面 查看。

1.1 下载 KubeKey

• 下载最新版的 KubeKey

从https://github.com/kubesphere/kubekey/releases/download/v3.1.7/kubekey-v3.1.7-linux-amd64.tar.gz 下载 KubeKey

#把下载的kubekey-v3.1.7-linux-amd64.tar.gz 上传到/opt/kubesphere
mkdir -p /opt/kubesphere
cd /opt/kubesphere

tar -zxvf kubekey-v3.1.7-linux-amd64.tar.gz

1.2 创建 manifests 文件

KubeKey v3.1.0 之前 manifests 文件需要根据模板手动编写， 现在可以通过 KubeKey 的

create manifest 命令自动生成一份 manifests 模板。

• create manifest 支持的参数如下

$ ./kk create manifest –help
Create an offline installation package configuration file
Usage:
kk create manifest [flags]
Flags:
–arch stringArray Specify a supported arch (default [amd64])
–debug Print detailed information
-f, –filename string Specify a manifest file path
-h, –help help for manifest
–ignore-err Ignore the error message, remove the host which reported error and force to continue
–kubeconfig string Specify a kubeconfig file
–name string Specify a name of manifest object (default "sample")
–namespace string KubeKey namespace to use (default "kubekey-system")
–with-kubernetes string Specify a supported version of kubernetes
–with-registry Specify a supported registry components
-y, –yes Skip confirm check

• 官方示例（支持多集群、多架构）

# 如需使用 kk 离线部署镜像仓库，添加 –with-registry 打包镜像仓库的安装文件

./kk create manifest –with-kubernetes v1.28.12 –with-registry

cp manifest-sample.yaml manifest.yaml

生成的配置文件manifest.yaml,需要修改内容如下

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
name: sample
spec:
arches:
– amd64
operatingSystems: []
kubernetesDistributions:
– type: kubernetes
version: v1.28.12
components:
helm:
version: v3.14.3
cni:
version: v1.2.0
etcd:
version: v3.5.13
containerRuntimes:
– type: docker
version: 24.0.9
– type: containerd
version: 1.7.13
calicoctl:
version: v3.27.4
crictl:
version: v1.29.0
docker-registry:
version: "2"
harbor:
version: v2.12.2
docker-compose:
version: v2.34.0
images:
– registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.28.12
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.28.12
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.28.12
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.28.12
– registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3
– registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4
– registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
– registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
– registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4
– registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.4
– registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
– registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
– registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.9.6-alpine
– registry.cn-beijing.aliyuncs.com/kubesphereio/kube-vip:v0.7.2
## ks-core
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-apiserver:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-console:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-controller-manager:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.16
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/redis:7.2.4-alpine
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/haproxy:2.9.6-alpine
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/ks-extensions-museum:v1.1.2
## devops
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/devops-apiserver:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/devops-controller:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/devops-tools:v4.1.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/devops-jenkins:v4.1.2-2.346.3
– swr.cn-southwest-2.myhuaweicloud.com/ks/jenkins/inbound-agent:4.10-2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-base:v3.2.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-nodejs:v3.2.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-maven:v3.2.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-maven:v3.2.1-jdk11
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-python:v3.2.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.16
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.17
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.18
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-base:v3.2.2-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-nodejs:v3.2.0-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-maven:v3.2.0-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-maven:v3.2.1-jdk11-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-python:v3.2.0-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.0-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.16-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.17-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/builder-go:v3.2.2-1.18-podman
– swr.cn-southwest-2.myhuaweicloud.com/ks/argoproj/argocd:v2.3.3
– swr.cn-southwest-2.myhuaweicloud.com/ks/argoproj/argocd-applicationset:v0.4.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/dexidp/dex:v2.30.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/library/redis:6.2.6-alpine
## gatekeeper
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/gatekeeper-extension-apiserver:v1.0.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/openpolicyagent/gatekeeper:v3.14.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/openpolicyagent/gatekeeper-crds:v3.14.0
## gateway
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/nginx-ingress-controller:v1.4.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/gateway-apiserver:v1.0.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/gateway-controller-manager:v1.0.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.16
## grafana
– swr.cn-southwest-2.myhuaweicloud.com/ks/curlimages/curl:7.85.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/grafana/grafana:10.4.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/library/busybox:1.31.1
## kubeedge
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubeedge/iptables-manager:v1.13.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubeedge/cloudcore:v1.13.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubeedge/controller-manager:v1.13.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubeedge-proxy:v0.4.1
## kubefed
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubefed-extension:v1.0.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubefed:v0.8.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.4
## loki
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/grafana/loki:3.0.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/grafana/loki-helm-test:ewelch-distributed-helm-chart-17db5ee
– swr.cn-southwest-2.myhuaweicloud.com/ks/grafana/loki-canary:3.0.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/nginxinc/nginx-unprivileged:1.24-alpine
– swr.cn-southwest-2.myhuaweicloud.com/ks/library/memcached:1.6.23-alpine
– swr.cn-southwest-2.myhuaweicloud.com/ks/prom/memcached-exporter:v0.14.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kiwigrid/k8s-sidecar:1.24.3
– swr.cn-southwest-2.myhuaweicloud.com/ks/minio/minio:RELEASE.2022-09-17T00-09-45Z
– swr.cn-southwest-2.myhuaweicloud.com/ks/minio/mc:RELEASE.2022-09-16T09-16-47Z
## metrics-server
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/metrics-server:v0.7.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/addon-resizer:1.8.20
## network
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/network-extension-apiserver:v1.1.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/network-extension-controller:v1.1.0
## openpitrix
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/apps-manage:v2.0.1
## opensearch
– swr.cn-southwest-2.myhuaweicloud.com/ks/opensearchproject/opensearch:2.8.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/library/busybox:1.35.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/opensearch-curator:v0.0.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/opensearchproject/opensearch-dashboards:2.8.0
## servicemesh
– swr.cn-southwest-2.myhuaweicloud.com/ks/istio/pilot:1.16.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/istio/proxyv2:1.16.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/istio/istioctl:1.16.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.4
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kiali-operator:v1.59.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kiali:v1.59
– swr.cn-southwest-2.myhuaweicloud.com/ks/jaegertracing/jaeger-operator:1.35.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/jaegertracing/jaeger-agent:1.35
– swr.cn-southwest-2.myhuaweicloud.com/ks/jaegertracing/jaeger-collector:1.35
– swr.cn-southwest-2.myhuaweicloud.com/ks/jaegertracing/jaeger-query:1.35
– swr.cn-southwest-2.myhuaweicloud.com/ks/jaegertracing/jaeger-es-index-cleaner:1.35
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/servicemesh-apiserver:v0.1.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/servicemesh-controller-manager:v0.1.0
## storage-utils
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/storageclass-accessor:v0.2.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/snapshot-controller:v4.2.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/snapshotclass-controller:v0.0.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/pvc-autoresizer:v0.3.1
## tower
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/tower:v0.2.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/tower-extension:v1.0.0
## vector
– swr.cn-southwest-2.myhuaweicloud.com/ks/timberio/vector:0.39.0-debian
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/vector-config:v0.2.1
## whizard-alerting
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/whizard-alerting-apiserver:v1.0.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/whizard-alerting-controller-manager:v1.0.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/thanosio/thanos:v0.36.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/cortex-tenant:v1.12.5
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus-operator/prometheus-config-reloader:v0.75.1
## whizard-events
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kube-events-exporter:v0.8.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/jimmidyson/configmap-reload:v0.9.0
## whizard-logging
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/log-sidecar-injector:v1.3.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/jimmidyson/configmap-reload:v0.9.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/elastic/filebeat:6.7.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/timberio/vector:0.39.0-debian
– swr.cn-southwest-2.myhuaweicloud.com/ks/library/alpine:3.14
## whizard-monitoring
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kube-state-metrics:v2.12.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubespheredev/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
– swr.cn-southwest-2.myhuaweicloud.com/ks/thanosio/thanos:v0.36.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/brancz/kube-rbac-proxy:v0.18.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus-operator/prometheus-config-reloader:v0.75.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus-operator/prometheus-operator:v0.75.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus/node-exporter:v1.8.1
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus/prometheus:v2.51.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/process-exporter:0.5.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/nginxinc/nginx-unprivileged:1.24
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/calico-exporter:v0.3.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/whizard-monitoring-helm-init:v0.1.0
## whizard-notification
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kubectl:v1.27.12
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/kube-rbac-proxy:v0.11.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/alertmanager-proxy:v0.2.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/notification-manager-operator:v2.5.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/notification-manager:v2.5.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/notification-tenant-sidecar:v4.0.2
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus/alertmanager:v0.27.0
– swr.cn-southwest-2.myhuaweicloud.com/ks/prometheus-operator/prometheus-config-reloader:v0.75.1
## whizard-telemetry
– swr.cn-southwest-2.myhuaweicloud.com/ks/kubesphere/whizard-telemetry-apiserver:v1.2.2

registry:
auths: {}

完整的镜像一共 120 个，受限于篇幅，这里不做展示，请查看下文完整的 manifest 文件。

本实验环境使用的操作系统是 x64 的 openEuler 22.03 LTS SP3，需要自己制作安装 Kubernetes 需要的操作系统依赖包镜像 openEuler-22.03-rpms-amd64.iso，其他操作系统请读者在 KubeKey releases 页面下载。

个人建议在离线环境用 openEuler 的安装 ISO，制做一个完整的离线软件源。在利用 KubeKey 安装离线集群时，就不需要考虑操作系统依赖包的问题。

manifest 修改说明

从https://github.com/kubesphere/kubekey/releases/download/v3.0.10/centos7-rpms-amd64.iso下载

上传到"/data/kubekey/"

• 默认创建的 manifest 模板中的镜像是从 docker.io 获取，替换前缀为 registry.cn-beijing.aliyuncs.com/kubesphereio

• operatingSystems 配置删除默认的 ubuntu，新增 openEuler 配置

注意：使用 registry.cn-beijing.aliyuncs.com/kubesphereio加速下载的镜像，在上传到harbor的时候会上传到一个仓库项目，在安装kubesphere的时候需要，安装创建项目并把镜像拷贝到对应的项目

1.5 导出制品 artifact

根据生成的 manifest，执行下面的命令制作制品（artifact）。

export KKZONE=cn
./kk artifact export -m manifest.yaml -o ksp-v4.1.2-v1.28.12.tar.gz

正确执行后，输出结果如下 :

制品制作完成后，查看制品大小（全镜像，制品包居然达到了 13G，生产环境还是有选择的裁剪吧）。

$ ls -lh ksp-v4.1.2-v1.28.12.tar.gz -rw-r–r– 1 root root 13G May 20 06:05 ksp-v3.4.1-v1.23.15.tar.gz

1.6 导出 KubeKey 离线安装包

把 KubeKey 工具也制作成压缩包，便于拷贝到离线节点。

tar zcvf kubekey-offline-v4.1.2-v1.28.12.tar.gz kk kubekey-v3.1.7-linux-amd64.tar.gz ksp-v4.1.2-v1.28.12.tar.gz