2-4-基础ansible部署k8s
根据公司业务场景的不同, 你可能需要部署几套或几十套 二进制K8S高可用集群, 面对繁琐的部署步骤, 你可能望而却步, 急需自动化的K8S部署工具, kubeasz就可以很好的解决你的问题。
k8s 集群环境主要是 kubernetes 管理端服务(kube-apiserver、kube-controller-manager、kube-scheduler)的⾼可⽤实现 , 以及 node 节点上的(kubelet、kube-proxy)客户端服务的部署。
如果是公司托管的 IDC 环境 , 建议直接将 harbor 和 node 节点部署在物理机环境 , master 节点、etcd、负载均衡等可以是虚拟机。
本次安装使用的服务器资源为: harbor 1台( 高可用环境可以2台做harbor镜像同步) 、haproxy+keepalived 2台、master 节点3台( 高可用环境最少2台) 、node 节点3台( 高可用环境2~N台) 、etcd 3台( 高可用环境最少3台才能选举leader) 。
kubeasz是一个利用ansible-playbook基于二进制方式自动化部署和运维k8s集群的开源项目 , 目前该项目最新发布版本为3.6.3 , 基于该版本我们可以快速实现部署最高版本为1.29的k8s集群。
基于kubeasz安装的k8s高可用架构方案如上 , kubeasz以前的官方文档中Node节点还使用的是haproxy处理自身高可用 , 但实际上kubeasz-3.3.0是使用kube-lb代替了haproxy。
对于没有安装ansible的部署服务器。可以通过kubeasz容器。实现整个部署流程
#启动kubeasz容器
./ezdown -S安装过程中默认会创建如下别名
root@harbor:~# cat .bashrc
alias dk='docker exec -it kubeasz' # generated by kubeasz之后直接使用dk 切换到容器，运行安装过程中的命令即可。通过以下类似命令实现部署过程。整个
dk ezctl setup k8s-cluster all 1. 基础环境准备
1.1 服务器配置
192.168.71.34服务器安装 ansible , 初始化配置。
安装
ansibleroot@harbor:~# apt install ansible准备
ansible环境root@harbor:~# mkdir /etc/ansible root@harbor:/etc# cd /etc/kubeasz/ root@harbor:/etc/kubeasz# ls ansible.cfg bin docs down example ezctl ezdown manifests pics playbooks README.md roles tools root@harbor:/etc/kubeasz# cp example/ config.yml hosts.allinone hosts.multi-node root@harbor:/etc/kubeasz# cp example/hosts.multi-node /etc/ansible/hosts配置
ssh密钥登录这里有一个集群互通的
shell脚本, 通过读取文件cluster_config.txt中的IP 和主机名。这个脚本主要用于配置ssh互通和主机名设置root@harbor:~# cat cluster_config.txt 192.168.71.29 k8s-master01.cbear.top 192.168.71.30 k8s-master02.cbear.top 192.168.71.31 k8s-master03.cbear.top 192.168.71.32 ha1.cbear.top 192.168.71.33 ha2.cbear.top 192.168.71.33 ha2.cbear.top 192.168.71.34 harbor.cbear.top 192.168.71.35 k8s-node01.cbear.top 192.168.71.36 k8s-node02.cbear.top 192.168.71.37 k8s-node03.cbear.top 192.168.71.42 k8s-etcd01.cbear.top 192.168.71.43 k8s-etcd02.cbear.top 192.168.71.44 k8s-etcd03.cbear.top#!/bin/bash # 集群配置文件路径 CONFIG_FILE="cluster_config.txt" REMOTE_PORT="22" REMOTE_USER="root" REMOTE_PASS="daemon" PASSPHRASE="" declare -A NODES_MAP SSH_OPTIONS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null) # 检查配置文件是否存在 if [ ! -f "$CONFIG_FILE" ]; then echo "Configuration file not found: $CONFIG_FILE" exit 1 fi function read_env() { echo "--------------------------" echo "开始读取配置文件中的主机名和IP" while IFS=' ' read -r ip hostname; do # 忽略空行 if [[ -z "$ip" || -z "$hostname" ]]; then continue fi # 将IP和hostname存储到映射中 NODES_MAP["$ip"]="$hostname" done < "$CONFIG_FILE" } # 安装sshpass软件包 function install_sshpass() { echo "--------------------------" echo "开始检查并安装sshpass软件包" for ip in "${!NODES_MAP[@]}"; do echo "检查节点 ${NODES_MAP[$ip]} ($ip) 上的sshpass安装状态" # 检查sshpass是否已经安装 if sshpass -p "${REMOTE_PASS}" ssh "${SSH_OPTIONS[@]}" "${REMOTE_USER}@${ip}" "dpkg -s sshpass" &> /dev/null; then echo "节点 $ip 上已经安装sshpass" continue else echo "节点 $ip 上未安装sshpass, 开始安装..." # 如果sshpass未安装, 则执行安装 if sshpass -p "${REMOTE_PASS}" ssh "${SSH_OPTIONS[@]}" "${REMOTE_USER}@${ip}" "apt-get update && apt-get install -y sshpass"; then echo "节点 $ip 上sshpass安装成功" else echo "节点 $ip 上sshpass安装失败" >&2 # 安装失败可以选择退出脚本或继续 # exit 1 # 或者使用 continue 来继续其他节点的安装 continue fi fi done } function remove_ssh_file() { echo "--------------------------" echo "初始化远程主机ssh配置信息" for ip in "${!NODES_MAP[@]}" ; do echo “删除节点$ip ssh信息” sshpass -p "${REMOTE_PASS}" ssh "${REMOTE_USER}@${ip}" "${SSH_OPTIONS[@]}" "rm -rf /root/.ssh" done } #每个节点生成公钥函数 function keygen_rsa() { echo "--------------------------" echo "开始为每个节点生成SSH公钥" for ip in "${!NODES_MAP[@]}"; do echo "为节点 $ip 生成公钥" # 使用sshpass传递密码进行远程命令执行, 但请注意这并不安全 # 推荐使用SSH密钥认证 sshpass -p "$REMOTE_PASS" ssh "${SSH_OPTIONS[@]}" "${REMOTE_USER}@${ip}" \ "ssh-keygen -f /root/.ssh/id_rsa -t rsa -N \"$PASSPHRASE\"" || \ { echo "为 $ip 生成公钥失败"; continue; } done } #配置hosts文件 function set_hosts() { echo "--------------------------" echo "配置hosts文件" > /etc/hosts cat > /etc/hosts << EOF 127.0.0.1 localhost 127.0.1.1 ubuntu2204 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters EOF for ip in "${!NODES_MAP[@]}" ;do echo "$ip ${NODES_MAP[$ip]}" >> /etc/hosts done } #获取集群中每个节点的公钥信息 function collect_pub_to_knowns(){ echo "--------------------------" echo "配置knowns_hosts文件" > /root/.ssh/known_hosts for ip in "${!NODES_MAP[@]}" ; do ssh-keyscan -t rsa ${NODES_MAP[$ip]} >> /root/.ssh/known_hosts done } function transferring_hosts_files { echo "--------------------------" echo "传输known_hosts和hosts文件" for ip in "${!NODES_MAP[@]}" ;do sshpass -p ${REMOTE_PASS} scp /root/.ssh/known_hosts root@${NODES_MAP[$ip]}:/root/.ssh/ echo "给${NODES_MAP[$ip]}节点传递known_hosts文件" sshpass -p ${REMOTE_PASS} scp /etc/hosts root@${NODES_MAP[$ip]}:/etc/hosts echo "给${NODES_MAP[$ip]}节点传递hosts文件" done } function node_interconnection() { echo "--------------------------" echo "配置节点间的SSH互通" for LOGIN_HOST in "${!NODES_MAP[@]}"; do LOGIN_IP=${NODES_MAP[$LOGIN_HOST]} for REMOTE_HOST in "${!NODES_MAP[@]}"; do REMOTE_IP=${NODES_MAP[$REMOTE_HOST]} # 使用sshpass传递密码执行ssh-copy-id, 但请注意这并不安全 # 推荐使用SSH密钥认证 sshpass -p "$REMOTE_PASS" ssh "${SSH_OPTIONS[@]}" "${REMOTE_USER}@$LOGIN_IP" \ "sshpass -p "$REMOTE_PASS" ssh-copy-id -i ~/.ssh/id_rsa.pub ${REMOTE_USER}@$REMOTE_IP" || \ { echo "SSH互通设置失败: $LOGIN_IP 到 $REMOTE_IP"; continue; } done done } function set_hostname(){ echo "--------------------------" echo "设置节点主机名" for ip in "${!NODES_MAP[@]}" ; do # 设置远程节点的主机名 echo "将节点$ip 设置为主机名 ${NODES_MAP[$ip]} " ; sshpass -p "${REMOTE_PASS}" ssh root@"${NODES_MAP[$ip]}" "sudo hostnamectl set-hostname ${NODES_MAP[$ip]}" ; done } function transferring_harbor_certs_files(){ echo "--------------------------" echo "传输harbor证书文件" for ip in "${!NODES_MAP[@]}" ; do sshpass -p "${REMOTE_PASS}" ssh "${REMOTE_USER}@${NODES_MAP[$ip]}" "mkdir -p /etc/docker/certs.d/harbor.cbear.top" done } read_env install_sshpass remove_ssh_file keygen_rsa sleep 2 set_hosts sleep 2 set_hostname sleep 2 collect_pub_to_knowns transferring_hosts_files node_interconnection echo "Cluster setup completed."
1.2 harbor配置
为
harbor生成对应证书证书中参数
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=yourdomain.com"#生成CA私钥信息 root@harbor:/usr/local/harbor/certs# openssl genrsa -out ca.key 4096 #生成CA公钥信息 root@harbor:/usr/local/harbor/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/CN=harbor.cbear.top" \ -key ca.key \ -out ca.crt #生成服务器私钥信息 root@harbor:/usr/local/harbor/certs# openssl genrsa -out harbor.cbear.top.key 4096 #生成服务器证书请求 root@harbor:/usr/local/harbor/certs# openssl req -sha512 -new \ -subj "/CN=harbor.cbear.top" \ -key harbor.cbear.top.key \ -out harbor.cbear.top.csr #生成一个x509 v3扩展文件 root@harbor:/usr/local/harbor/certs# cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1=harbor.cbear.top DNS.2=harbor EOF # 使用该v3.ext文件为您的Harbor主机生成证书 root@harbor:/usr/local/harbor/certs# openssl x509 -req -sha512 -days 3650 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in harbor.cbear.top.csr \ -out harbor.cbear.top.crt #将服务器证书yourdomain.com.crt的编码格式转换为yourdomain.com.cert, 提供Docker使用 root@harbor:/usr/local/harbor/certs# openssl x509 -inform PEM -in harbor.cbear.top.crt -out harbor.cbear.top.cert安装
harbor, 并为harbor配置证书root@harbor:/usr/local/harbor# cat harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: harbor.cbear.top # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /usr/local/harbor/certs/harbor.cbear.top.crt private_key: /usr/local/harbor/certs/harbor.cbear.top.key ..... root@harbor:/usr/local/harbor# ./prepare root@harbor:/usr/local/harbor# ./install.sh配置docker证书
root@harbor:/etc/docker# mkdir -p /etc/docker/certs.d/harbor.cbear.top root@harbor:/etc/docker# cp /usr/local/harbor/certs/ca.crt /etc/docker/certs.d/harbor.cbear.top/ root@harbor:/etc/docker# cp /usr/local/harbor/certs/harbor.cbear.top.cert /etc/docker/certs.d/harbor.cbear.top/ root@harbor:/etc/docker# cp /usr/local/harbor/certs/harbor.cbear.top.key /etc/docker/certs.d/harbor.cbear.top/ root@harbor:~# ll /etc/docker/certs.d/harbor.cbear.top/ total 20 drwxr-xr-x 2 root root 4096 May 20 11:32 ./ drwxr-xr-x 3 root root 4096 May 20 11:24 ../ -rw-r--r-- 1 root root 1826 May 20 11:23 ca.crt -rw-r--r-- 1 root root 1915 May 20 11:23 harbor.cbear.top.cert -rw------- 1 root root 3268 May 20 11:23 harbor.cbear.top.key
1.3 服务器安装 containerd 和 nerdctl
临时配置ansible的
hosts文件root@harbor:~# cat /etc/ansible/hosts [k8s-master] k8s-master01.cbear.top k8s-master02.cbear.top k8s-master03.cbear.top [haproxy] ha1.cbear.top ha2.cbear.top [harbor] harbor.cbear.top [k8s-node] k8s-node01.cbear.top k8s-node02.cbear.top k8s-node03.cbear.top [k8s-etcd] k8s-etcd01.cbear.top k8s-etcd02.cbear.top k8s-etcd03.cbear.top准备二进制包和安装脚本
wget https://github.com/containerd/containerd/releases/download/v1.7.17/cri-containerd-cni-1.7.17-linux-amd64.tar.gz wget https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-1.7.6-linux-amd64.tar.gz#!/bin/bash #程序包名称 CONTAINERD_VERSION=1.7.17 NETDCTL_VERSION=1.7.6 containerd_package_name=cri-containerd-cni-${CONTAINERD_VERSION}-linux-amd64.tar.gz netdctl_package_name=nerdctl-${NETDCTL_VERSION}-linux-amd64.tar.gz #程序包解压目录 package_dir=/usr/local/containerd echo 正在创建程序包目录: ${package_dir} #创建程序包解压目录 mkdir -p ${package_dir} echo 正在解压${containerd_package_name}到${package_dir} #解压程序包 tar xf ${containerd_package_name} -C ${package_dir} echo 正在安装 cri、containerd、cni、runc #拷贝文件到指定目录 cp -r ${package_dir}/usr/local/bin /usr/local/ cp -r ${package_dir}/usr/local/sbin /usr/local/ cp -r ${package_dir}/opt/* /opt cp -r ${package_dir}/etc/cni /etc/ cp -r ${package_dir}/etc/crictl.yaml /etc/ cp -r ${package_dir}/etc/systemd/system/containerd.service /lib/systemd/system/ echo 正在初始化 containerd 配置 #创建 containerd 配置文件目录 mkdir -p /etc/containerd #生成 containerd 默认配置文件 #可以用这个命令生成默认配置文件 #containerd config default > /etc/containerd/config.toml #以下配置文件中, 主要修改了容器仓库的位置。 cat > /etc/containerd/config.toml << EOF disabled_plugins = [] imports = [] oom_score = 0 plugin_dir = "" required_plugins = [] root = "/var/lib/containerd" state = "/run/containerd" temp = "" version = 2 [cgroup] path = "" [debug] address = "" format = "" gid = 0 level = "" uid = 0 [grpc] address = "/run/containerd/containerd.sock" gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 tcp_address = "" tcp_tls_ca = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0 [metrics] address = "" grpc_histogram = false [plugins] [plugins."io.containerd.gc.v1.scheduler"] deletion_threshold = 0 mutation_threshold = 100 pause_threshold = 0.02 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] device_ownership_from_security_context = false disable_apparmor = false disable_cgroup = false disable_hugetlb_controller = true disable_proc_mount = false disable_tcp_service = true enable_selinux = false enable_tls_streaming = false enable_unprivileged_icmp = false enable_unprivileged_ports = false ignore_image_defined_volumes = false max_concurrent_downloads = 3 max_container_log_line_size = 16384 netns_mounts_under_state_dir = false restrict_oom_score_adj = false sandbox_image = "harbor.cbear.top/easzlab/pause:3.9" selinux_category_range = 1024 stats_collect_period = 10 stream_idle_timeout = "4h0m0s" stream_server_address = "127.0.0.1" stream_server_port = "0" systemd_cgroup = false tolerate_missing_hugetlb_controller = true unset_seccomp_profile = "" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" conf_template = "/etc/cni/net.d/10-default.conf" max_conf_num = 1 [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" disable_snapshot_annotations = true discard_unpacked_layers = false ignore_rdt_not_enabled_errors = false no_pivot = false snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 IoUid = 0 NoNewKeyring = false NoPivotRoot = false Root = "" ShimCgroup = "" SystemdCgroup = true [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "node" [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.auths] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.cbear.top".tls] ca_file = "/etc/containerd/certs.d/harbor.cbear.top/ca.crt" cert_file = "/etc/containerd/certs.d/harbor.cbear.top/harbor.cbear.top.cert" key_file = "/etc/containerd/certs.d/harbor.cbear.top/harbor.cbear.top.key" insecure_skip_verify = false [plugins."io.containerd.grpc.v1.cri".registry.headers] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.cbear.top"] endpoint = ["https://harbor.cbear.top"] [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] no_shim = false runtime = "runc" runtime_root = "" shim = "containerd-shim" shim_debug = false [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.snapshotter.v1.aufs"] root_path = "" [plugins."io.containerd.snapshotter.v1.btrfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.devmapper"] async_remove = false base_image_size = "" pool_name = "" root_path = "" [plugins."io.containerd.snapshotter.v1.native"] root_path = "" [plugins."io.containerd.snapshotter.v1.overlayfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.zfs"] root_path = "" [proxy_plugins] [stream_processors] [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"] accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar" [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"] accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar+gzip" [timeouts] "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [ttrpc] address = "" gid = 0 uid = 0 EOF echo 正在启动 containerd #启动 containerd 并设置服务开机启动 systemctl daemon-reload systemctl restart containerd.service systemctl enable containerd.service echo 正在安装 nerdctl package_dir=/etc/local/netdctl echo 正在创建程序包目录: ${package_dir} #创建程序包解压目录 mkdir -p ${package_dir} echo 正在解压 ${netdctl_package_name} 到 ${package_dir} #解压程序包 tar xf ${netdctl_package_name} -C ${package_dir} echo 正在安装 nerdctl #拷贝文件到指定目录 cp -r ${package_dir}/nerdctl /usr/local/bin/ echo 正在初始化 nerdctl 配置 mkdir -p /etc/nerdctl cat > /etc/nerdctl/nerdctl.toml <<EOF namespace = "k8s.io" debug = false debug_full = false insecure_registry = true EOF echo 安装完成配置
ansible的yamlroot@harbor:~# cat install-containerd.yaml - hosts: k8s-master,k8s-etcd,k8s-node remote_user: root tasks: - name: "拷贝 nerdctl-1.3.1-linux-amd64.tar.gz 文件" copy: src: /root/nerdctl-1.7.6-linux-amd64.tar.gz dest: /tmp owner: root group: root mode: 0644 - name: "拷贝 cri-containerd-cni-1.6.20-linux-amd64.tar.gz 文件" copy: src: /root//cri-containerd-cni-1.7.17-linux-amd64.tar.gz dest: /tmp/ owner: root group: root mode: 0644 - name: "拷贝 install.sh 文件" copy: src: /root/install.sh dest: /tmp/ owner: root group: root mode: 0744 - name: "执行安装脚本" shell: cd /tmp/ && bash install.sh运行
ansible脚本root@harbor:~# ansible-playbook install-containerd.yaml检查
containerd和nerdctl版本信息root@harbor:~# ansible k8s-master,k8s-etcd,k8s-node -m shell -a "containerd -v && nerdctl version"在
master和work节点中为containerd配置证书- hosts: k8s-master,k8s-etcd,k8s-node remote_user: root tasks: - name: "创建containerd证书目录:/etc/containerd/harbor.cbear.top" file: path: /etc/containerd/harbor.cbear.top state: directory owner: root group: root mode: 0644 - name: "拷贝ca.crt 证书文件" copy: src: /usr/local/harbor/certs/ca.crt dest: /etc/containerd/harbor.cbear.top owner: root group: root mode: 0644 - name: "拷贝 cri-containerd-cni-1.6.20-linux-amd64.tar.gz 文件" copy: src: /root//cri-containerd-cni-1.7.17-linux-amd64.tar.gz dest: /tmp/ owner: root group: root mode: 0644 - name: "拷贝 install.sh 文件" copy: src: /root/install.sh dest: /tmp/ owner: root group: root mode: 0744 - name: "执行安装脚本" shell: cd /tmp/ && bash install.sh以下配置文件为
kubeasz自动生成的配置文件。这个文件可以作为参考disabled_plugins = [] imports = [] oom_score = 0 plugin_dir = "" required_plugins = [] root = "/var/lib/containerd" state = "/run/containerd" temp = "" version = 2 [cgroup] path = "" [debug] address = "" format = "" gid = 0 level = "" uid = 0 [grpc] address = "/run/containerd/containerd.sock" gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 tcp_address = "" tcp_tls_ca = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0 [metrics] address = "" grpc_histogram = false [plugins] [plugins."io.containerd.gc.v1.scheduler"] deletion_threshold = 0 mutation_threshold = 100 pause_threshold = 0.02 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] device_ownership_from_security_context = false disable_apparmor = false disable_cgroup = false disable_hugetlb_controller = true disable_proc_mount = false disable_tcp_service = true enable_selinux = false enable_tls_streaming = false enable_unprivileged_icmp = false enable_unprivileged_ports = false ignore_image_defined_volumes = false max_concurrent_downloads = 3 max_container_log_line_size = 16384 netns_mounts_under_state_dir = false restrict_oom_score_adj = false sandbox_image = "harbor.cbear.top/easzlab/pause:3.9" selinux_category_range = 1024 stats_collect_period = 10 stream_idle_timeout = "4h0m0s" stream_server_address = "127.0.0.1" stream_server_port = "0" systemd_cgroup = false tolerate_missing_hugetlb_controller = true unset_seccomp_profile = "" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" conf_template = "/etc/cni/net.d/10-default.conf" max_conf_num = 1 [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" disable_snapshot_annotations = true discard_unpacked_layers = false ignore_rdt_not_enabled_errors = false no_pivot = false snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 IoUid = 0 NoNewKeyring = false NoPivotRoot = false Root = "" ShimCgroup = "" SystemdCgroup = true [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] base_runtime_spec = "" container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "node" [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.auths] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."easzlab.io.local:5000".tls] insecure_skip_verify = true [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.cbear.top".tls] insecure_skip_verify = true [plugins."io.containerd.grpc.v1.cri".registry.headers] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."easzlab.io.local:5000"] endpoint = ["http://easzlab.io.local:5000"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.cbear.top"] endpoint = ["https://harbor.cbear.top"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://docker.nju.edu.cn/", "https://kuamavit.mirror.aliyuncs.com"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"] endpoint = ["https://gcr.nju.edu.cn"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"] endpoint = ["https://gcr.nju.edu.cn/google-containers/"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"] endpoint = ["https://quay.nju.edu.cn"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."ghcr.io"] endpoint = ["https://ghcr.nju.edu.cn"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nvcr.io"] endpoint = ["https://ngc.nju.edu.cn"] [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] no_shim = false runtime = "runc" runtime_root = "" shim = "containerd-shim" shim_debug = false [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.snapshotter.v1.aufs"] root_path = "" [plugins."io.containerd.snapshotter.v1.btrfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.devmapper"] async_remove = false base_image_size = "" pool_name = "" root_path = "" [plugins."io.containerd.snapshotter.v1.native"] root_path = "" [plugins."io.containerd.snapshotter.v1.overlayfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.zfs"] root_path = "" [proxy_plugins] [stream_processors] [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"] accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar" [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"] accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar+gzip" [timeouts] "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [ttrpc] address = "" gid = 0 uid = 0
1.4 镜像构建软件准备
社区中目前有三款工具可以支持无 docker daemon 化的构建：kaniko，img 和 buildkit。
Kaniko是由Google开发的在k8s上做docker构建的命令行工具，使用非常简洁，只需要build一个二进制工具即可，支持dockerfile构建、push、credentail文件读取。Buildkit是docker公司开发，目前由社区和docker公司合理维护的“含着金钥匙出生”的新一代构建工具，拥有良好的扩展性、极大地提高了构建速度，以及更好的安全性，功能上配合docker使用还是没问题的，独立使用功能其实有残缺，这个放到后面来讲。img是社区贡献者开发，基于buildkit封装的类docker化命令行工具，无需daemon进程，无需privilege，可以独立运行的二进制工具，非常小巧易用，而且有着和buildkit一样的性能优势。
下载
buildkitd二进制包wget https://github.com/moby/buildkit/releases/download/v0.13.2/buildkit-v0.13.2.linux-amd64.tar.gz解压对应软件包
root@k8s-master01:~# mkdir /usr/local/src/buildkit root@k8s-master01:~# tar -xf buildkit-v0.13.2.linux-amd64.tar.gz -C /usr/local/src/buildkit创建对应二进制软链接文件
root@k8s-master01:~# ln -sv /usr/local/src/buildkit/bin/* /usr/local/bin/ '/usr/local/bin/buildctl' -> '/usr/local/src/buildkit/bin/buildctl' '/usr/local/bin/buildkit-cni-bridge' -> '/usr/local/src/buildkit/bin/buildkit-cni-bridge' '/usr/local/bin/buildkit-cni-firewall' -> '/usr/local/src/buildkit/bin/buildkit-cni-firewall' '/usr/local/bin/buildkit-cni-host-local' -> '/usr/local/src/buildkit/bin/buildkit-cni-host-local' '/usr/local/bin/buildkit-cni-loopback' -> '/usr/local/src/buildkit/bin/buildkit-cni-loopback' '/usr/local/bin/buildkitd' -> '/usr/local/src/buildkit/bin/buildkitd' '/usr/local/bin/buildkit-qemu-aarch64' -> '/usr/local/src/buildkit/bin/buildkit-qemu-aarch64' '/usr/local/bin/buildkit-qemu-arm' -> '/usr/local/src/buildkit/bin/buildkit-qemu-arm' '/usr/local/bin/buildkit-qemu-i386' -> '/usr/local/src/buildkit/bin/buildkit-qemu-i386' '/usr/local/bin/buildkit-qemu-mips64' -> '/usr/local/src/buildkit/bin/buildkit-qemu-mips64' '/usr/local/bin/buildkit-qemu-mips64el' -> '/usr/local/src/buildkit/bin/buildkit-qemu-mips64el' '/usr/local/bin/buildkit-qemu-ppc64le' -> '/usr/local/src/buildkit/bin/buildkit-qemu-ppc64le' '/usr/local/bin/buildkit-qemu-riscv64' -> '/usr/local/src/buildkit/bin/buildkit-qemu-riscv64' '/usr/local/bin/buildkit-qemu-s390x' -> '/usr/local/src/buildkit/bin/buildkit-qemu-s390x' '/usr/local/bin/buildkit-runc' -> '/usr/local/src/buildkit/bin/buildkit-runc'准备
systemd服务文件。对应文件可以在github文件中下载:https://github.com/moby/buildkit/tree/master/examples/systemd/system
root@k8s-master01:~# cat /lib/systemd/system/buildkit.service [Unit] Description=BuildKit Requires=buildkit.socket After=buildkit.socket Documentation=https://github.com/moby/buildkit [Service] ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true [Install] WantedBy=multi-user.target root@k8s-master01:~# cat /lib/systemd/system/buildkit.socket [Unit] Description=BuildKit Documentation=https://github.com/moby/buildkit [Socket] ListenStream=%t/buildkit/buildkitd.sock [Install] WantedBy=sockets.target
2. 使用kubeasz部署kubernetes
2.1 部署环境准备
准备脚本, 二进制和镜像文件
export release=3.6.3 wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown chmod +x ./ezdown修改
ezdownshell脚本中的默认本地仓库。这里这个脚本默认下载镜像的过程中, 会在本地启动一个镜像仓库https://easzlab.io.local:5000。这个需要将这个默认仓库修改为手动创建仓库harbor.cbear.topcd /etc/kubeasz sed -i "s/easzlab.io.local:5000/harbor.cbear.top/g" /etc/kubeasz/ezdown sed -i "s/easzlab.io.local/harbor.cbear.top/g" /etc/kubeasz/ezdown下载镜像, 这里会将下载的镜像传递到指定的镜像库中。这里需要在
harbor创建calico、coredns、easzlab、kubenetesui这几个仓库./ezdown -D下载离线系统包
root@harbor:/etc/kubeasz# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" root@harbor:/etc/kubeasz# ./ezdown -P ubuntu_22 2024-05-20 12:12:18 INFO Action begin: get_sys_pkg ubuntu_22 2024-05-20 12:12:18 INFO downloading system packages kubeasz-sys-pkg:1.0.0_ubuntu_22 1.0.0_ubuntu_22: Pulling from easzlab/kubeasz-sys-pkg 91d30c5bc195: Pull complete 651615f46b5f: Pull complete Digest: sha256:4e715196e5e84e679955eb0f38129d2c976e6698f990219b4a9804bc56c822ed Status: Downloaded newer image for easzlab/kubeasz-sys-pkg:1.0.0_ubuntu_22 docker.io/easzlab/kubeasz-sys-pkg:1.0.0_ubuntu_22 2024-05-20 12:12:26 DEBUG run a temporary container 07baf343cbaa1b1ad87f9e3d1a97e0d464d0b2ec075d73af7c9a46f3307cf66d 2024-05-20 12:12:27 DEBUG cp system packages Successfully copied 9.72MB to /etc/kubeasz/down 2024-05-20 12:12:27 DEBUG stop&remove temporary container temp_sys_pkg 2024-05-20 12:12:28 INFO Action successed: get_sys_pkg ubuntu_22单独为
flannel功能下载镜像。需要再harbor中创建flannel仓库。下载额外容器镜像(cilium,flannel,prometheus等)root@harbor:/etc/kubeasz# ./ezdown -X flannel
上述脚本运行成功后, 所有文件(kubeasz代码、二进制、离线镜像)均已整理好放入目录/etc/kubeasz, /etc/kubeasz包 kubeasz版本为 ${release} 的发布代码, /etc/kubeasz/bin 包含 k8s/etcd/docker/cni 等二进制文件, /etc/kubeasz/down 包含集群安装时需要的离线容器镜像, /etc/kubeasz/down/packages 包含集群安装时需要的系统基础软件
2.2 创建集群配置实例
使用
ezctl命令生成集群配置root@harbor:/etc/kubeasz# ./ezctl new k8s-cluster 2024-05-20 07:38:25 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-cluster 2024-05-20 07:38:25 DEBUG set versions 2024-05-20 07:38:25 DEBUG disable registry mirrors 2024-05-20 07:38:25 DEBUG cluster k8s-cluster: files successfully created. 2024-05-20 07:38:25 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-cluster/hosts' 2024-05-20 07:38:25 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-cluster/config.yml'说明: 然后根据提示配置
/etc/kubeasz/clusters/k8s-cluster/hosts和/etc/kubeasz/clusters/k8s-cluster/config.yml; 根据前面节点规划修改hosts文件和其他集群层面的主要配置选项; 其他集群组件等配置项可以在config.yml文件中修改。配置
hosts文件hosts和config.yml文件需要根据自己部署需求和实际环境情况进行修改。
root@harbor:/etc/kubeasz/clusters/k8s-cluster# cat hosts # 'etcd' cluster should have odd member(s) (1,3,5,...) [etcd] 192.168.71.42 192.168.71.43 192.168.71.44 # master node(s), set unique 'k8s_nodename' for each node # CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.', # and must start and end with an alphanumeric character [kube_master] 192.168.71.29 k8s_nodename='k8s-master01.cbear.top' 192.168.71.30 k8s_nodename='k8s-master02.cbear.top' # work node(s), set unique 'k8s_nodename' for each node # CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.', # and must start and end with an alphanumeric character [kube_node] 192.168.71.35 k8s_nodename='k8s-node01.cbear.top' 192.168.71.36 k8s_nodename='k8s-node02.cbear.top' # [optional] harbor server, a private docker registry # 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one [harbor] #192.168.1.8 NEW_INSTALL=false # [optional] loadbalance for accessing k8s from outside [ex_lb] 192.168.71.32 LB_ROLE=backup EX_APISERVER_VIP=192.168.71.150 EX_APISERVER_PORT=8443 192.168.71.33 LB_ROLE=master EX_APISERVER_VIP=192.168.71.150 EX_APISERVER_PORT=8443 # [optional] ntp server for the cluster [chrony] #192.168.1.1 [all:vars] # --------- Main Variables --------------- # Secure port for apiservers SECURE_PORT="6443" # Cluster container-runtime supported: docker, containerd # if k8s version >= 1.24, docker is not supported CONTAINER_RUNTIME="containerd" # Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn CLUSTER_NETWORK="flannel" # Service proxy mode of kube-proxy: 'iptables' or 'ipvs' PROXY_MODE="ipvs" # K8S Service CIDR, not overlap with node(host) networking SERVICE_CIDR="10.68.0.0/16" # Cluster CIDR (Pod CIDR), not overlap with node(host) networking CLUSTER_CIDR="172.20.0.0/16" # NodePort Range NODE_PORT_RANGE="30000-32767" # Cluster DNS Domain CLUSTER_DNS_DOMAIN="cbear.local" # -------- Additional Variables (don't change the default value right now) --- # Binaries Directory bin_dir="/opt/kube/bin" # Deploy Directory (kubeasz workspace) base_dir="/etc/kubeasz" # Directory for a specific cluster cluster_dir="{{ base_dir }}/clusters/k8s-cluster" # CA and other components cert/key Directory ca_dir="/etc/kubernetes/ssl" # Default 'k8s_nodename' is empty k8s_nodename='' # Default python interpreter ansible_python_interpreter=/usr/bin/python3配置
config.ymlroot@harbor:/etc/kubeasz/clusters/k8s-cluster# cat config.yml ############################ # prepare ############################ # 可选离线安装系统软件包 (offline|online) INSTALL_SOURCE: "offline" # 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening # (deprecated) 未更新上游项目, 未验证最新k8s集群安装, 不建议启用 OS_HARDEN: false ############################ # role:deploy ############################ # default: ca will expire in 100 years # default: certs issued by the ca will expire in 50 years CA_EXPIRY: "876000h" CERT_EXPIRY: "438000h" # force to recreate CA and other certs, not suggested to set 'true' CHANGE_CA: false # kubeconfig 配置参数 CLUSTER_NAME: "cluster1" CONTEXT_NAME: "context-{{ CLUSTER_NAME }}" # k8s version K8S_VER: "1.29.0" # set unique 'k8s_nodename' for each node, if not set(default:'') ip add will be used # CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.', # and must start and end with an alphanumeric character (e.g. 'example.com'), # regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*' K8S_NODENAME: "{%- if k8s_nodename != '' -%} \ {{ k8s_nodename|replace('_', '-')|lower }} \ {%- else -%} \ {{ inventory_hostname }} \ {%- endif -%}" ############################ # role:etcd ############################ # 设置不同的wal目录, 可以避免磁盘io竞争, 提高性能 ETCD_DATA_DIR: "/var/lib/etcd" ETCD_WAL_DIR: "" ############################ # role:runtime [containerd,docker] ############################ # [.]启用拉取加速镜像仓库 ENABLE_MIRROR_REGISTRY: true # [.]添加信任的私有仓库 INSECURE_REG: - "http://easzlab.io.local:5000" - "https://harbor.cbear.top" # [.]基础容器镜像 SANDBOX_IMAGE: "harbor.cbear.top/easzlab/pause:3.9" # [containerd]容器持久化存储目录 CONTAINERD_STORAGE_DIR: "/var/lib/containerd" # [docker]容器存储目录 DOCKER_STORAGE_DIR: "/var/lib/docker" # [docker]开启Restful API DOCKER_ENABLE_REMOTE_API: false ############################ # role:kube-master ############################ # k8s 集群 master 节点证书配置, 可以添加多个ip和域名( 比如增加公网ip和域名) MASTER_CERT_HOSTS: - "192.168.71.150" - "k8s.easzlab.io" #- "www.test.com" # node 节点上 pod 网段掩码长度( 决定每个节点最多能分配的pod ip地址) # 如果flannel 使用 --kube-subnet-mgr 参数, 那么它将读取该设置为每个节点分配pod网段 # https://github.com/coreos/flannel/issues/847 NODE_CIDR_LEN: 24 ############################ # role:kube-node ############################ # Kubelet 根目录 KUBELET_ROOT_DIR: "/var/lib/kubelet" # node节点最大pod 数 MAX_PODS: 110 # 配置为kube组件( kubelet,kube-proxy,dockerd等) 预留的资源量 # 数值设置详见templates/kubelet-config.yaml.j2 KUBE_RESERVED_ENABLED: "no" # k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控, 了解系统的资源占用状况; # 并且随着系统运行时间, 需要适当增加资源预留, 数值设置详见templates/kubelet-config.yaml.j2 # 系统预留设置基于 4c/8g 虚机, 最小化安装系统服务, 如果使用高性能物理机可以适当增加预留 # 另外, 集群安装时候apiserver等资源占用会短时较大, 建议至少预留1g内存 SYS_RESERVED_ENABLED: "no" ############################ # role:network [flannel,calico,cilium,kube-ovn,kube-router] ############################ # ------------------------------------------- flannel # [flannel]设置flannel 后端"host-gw","vxlan"等 FLANNEL_BACKEND: "vxlan" DIRECT_ROUTING: false # [flannel] flannel_ver: "v0.22.2" # ------------------------------------------- calico # [calico] IPIP隧道模式可选项有: [Always, CrossSubnet, Never],跨子网可以配置为Always与CrossSubnet(公有云建议使用always比较省事, 其他的话需要修改各自公有云的网络配置, 具体可以参考各个公有云说明) # 其次CrossSubnet为隧道+BGP路由混合模式可以提升网络性能, 同子网配置为Never即可. CALICO_IPV4POOL_IPIP: "Always" # [calico]设置 calico-node使用的host IP, bgp邻居通过该地址建立, 可手工指定也可以自动发现 IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}" # [calico]设置calico 网络 backend: bird, vxlan, none CALICO_NETWORKING_BACKEND: "bird" # [calico]设置calico 是否使用route reflectors # 如果集群规模超过50个节点, 建议启用该特性 CALICO_RR_ENABLED: false # CALICO_RR_NODES 配置route reflectors的节点, 如果未设置默认使用集群master节点 # CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"] CALICO_RR_NODES: [] # [calico]更新支持calico 版本: ["3.19", "3.23"] calico_ver: "v3.26.4" # [calico]calico 主版本 calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}" # ------------------------------------------- cilium # [cilium]镜像版本 cilium_ver: "1.14.5" cilium_connectivity_check: true cilium_hubble_enabled: false cilium_hubble_ui_enabled: false # ------------------------------------------- kube-ovn # [kube-ovn]离线镜像tar包 kube_ovn_ver: "v1.11.5" # ------------------------------------------- kube-router # [kube-router]公有云上存在限制, 一般需要始终开启 ipinip; 自有环境可以设置为 "subnet" OVERLAY_TYPE: "full" # [kube-router]NetworkPolicy 支持开关 FIREWALL_ENABLE: true # [kube-router]kube-router 镜像版本 kube_router_ver: "v1.5.4" ############################ # role:cluster-addon ############################ # coredns 自动安装 dns_install: "yes" corednsVer: "1.11.1" ENABLE_LOCAL_DNS_CACHE: true dnsNodeCacheVer: "1.22.23" # 设置 local dns cache 地址 LOCAL_DNS_CACHE: "169.254.20.10" # metric server 自动安装 metricsserver_install: "yes" metricsVer: "v0.6.4" # dashboard 自动安装 dashboard_install: "yes" dashboardVer: "v2.7.0" dashboardMetricsScraperVer: "v1.0.8" # prometheus 自动安装 prom_install: "no" prom_namespace: "monitor" prom_chart_ver: "45.23.0" # kubeapps 自动安装, 如果选择安装, 默认同时安装local-storage( 提供storageClass: "local-path") kubeapps_install: "no" kubeapps_install_namespace: "kubeapps" kubeapps_working_namespace: "default" kubeapps_storage_class: "local-path" kubeapps_chart_ver: "12.4.3" # local-storage (local-path-provisioner) 自动安装 local_path_provisioner_install: "no" local_path_provisioner_ver: "v0.0.24" # 设置默认本地存储路径 local_path_provisioner_dir: "/opt/local-path-provisioner" # nfs-provisioner 自动安装 nfs_provisioner_install: "no" nfs_provisioner_namespace: "kube-system" nfs_provisioner_ver: "v4.0.2" nfs_storage_class: "managed-nfs-storage" nfs_server: "192.168.1.10" nfs_path: "/data/nfs" # network-check 自动安装 network_check_enabled: false network_check_schedule: "*/5 * * * *" ############################ # role:harbor ############################ # harbor version, 完整版本号 HARBOR_VER: "v2.8.4" HARBOR_DOMAIN: "harbor.easzlab.io.local" HARBOR_PATH: /var/data HARBOR_TLS_PORT: 8443 HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}" # if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down' HARBOR_SELF_SIGNED_CERT: true # install extra component HARBOR_WITH_NOTARY: false HARBOR_WITH_TRIVY: false root@harbor:/etc/kubeasz/clusters/k8s-cluster#
2.3 部署k8s环境
部署k8s环境过程中,主要是通过ezctl命令实现
root@harbor:/etc/kubeasz# ./ezctl upgrade
Usage: ezctl COMMAND [args]
-------------------------------------------------------------------------------------
Cluster setups:
list to list all of the managed clusters
checkout <cluster> to switch default kubeconfig of the cluster
new <cluster> to start a new k8s deploy with name 'cluster'
setup <cluster> <step> to setup a cluster, also supporting a step-by-step way
start <cluster> to start all of the k8s services stopped by 'ezctl stop'
stop <cluster> to stop all of the k8s services temporarily
upgrade <cluster> to upgrade the k8s cluster
destroy <cluster> to destroy the k8s cluster
backup <cluster> to backup the cluster state (etcd snapshot)
restore <cluster> to restore the cluster state from backups
start-aio to quickly setup an all-in-one cluster with default settings
Cluster ops:
add-etcd <cluster> <ip> to add a etcd-node to the etcd cluster
add-master <cluster> <ip> to add a master node to the k8s cluster
add-node <cluster> <ip> to add a work node to the k8s cluster
del-etcd <cluster> <ip> to delete a etcd-node from the etcd cluster
del-master <cluster> <ip> to delete a master node from the k8s cluster
del-node <cluster> <ip> to delete a work node from the k8s cluster
Extra operation:
kca-renew <cluster> to force renew CA certs and all the other certs (with caution)
kcfg-adm <cluster> <args> to manage client kubeconfig of the k8s cluster
Use "ezctl help <command>" for more information about a given command.安装主要通过ezctl setup 命令安装集群
root@harbor:/etc/kubeasz# ./ezctl setup --help
Usage: ezctl setup <cluster> <step>
available steps:
01 prepare to prepare CA/certs & kubeconfig & other system settings
02 etcd to setup the etcd cluster
03 container-runtime to setup the container runtime(docker or containerd)
04 kube-master to setup the master nodes
05 kube-node to setup the worker nodes
06 network to setup the network plugin
07 cluster-addon to setup other useful plugins
90 all to run 01~07 all at once
10 ex-lb to install external loadbalance for accessing k8s from outside
11 harbor to install a new harbor server or to integrate with an existed one
examples: ./ezctl setup test-k8s 01 (or ./ezctl setup test-k8s prepare)
./ezctl setup test-k8s 02 (or ./ezctl setup test-k8s etcd)
./ezctl setup test-k8s all
./ezctl setup test-k8s 04 -t restart_masterkubeasz 使用的相关 ansible-playbook 文件
root@harbor:/etc/kubeasz/playbooks# ls
01.prepare.yml 05.kube-node.yml 11.harbor.yml 31.deletcd.yml 91.start.yml 95.restore.yml
02.etcd.yml 06.network.yml 21.addetcd.yml 32.delnode.yml 92.stop.yml 96.update-certs.yml
03.runtime.yml 07.cluster-addon.yml 22.addnode.yml 33.delmaster.yml 93.upgrade.yml 99.clean.yml
04.kube-master.yml 10.ex-lb.yml 23.addmaster.yml 90.setup.yml 94.backup.yml准备
CA和基础环境初始化root@harbor:/etc/kubeasz# ./ezctl setup k8s-cluster 01 ansible-playbook -i clusters/k8s-cluster/hosts -e @clusters/k8s-cluster/config.yml playbooks/01.prepare.yml *** Component Version ********************* ******************************************* * kubernetes: v1.29.0 * etcd: v3.5.10 * flannel: v0.22.2 ******************************************* 2024-05-20 14:53:15 INFO cluster:k8s-cluster setup step:01 begins in 5s, press any key to abort: PLAY [kube_master,kube_node,etcd,ex_lb,chrony] *********************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************** ok: [192.168.71.29] ok: [192.168.71.30] ....部署
etcd集群root@harbor:/etc/kubeasz# ./ezctl setup k8s-cluster 02 ansible-playbook -i clusters/k8s-cluster/hosts -e @clusters/k8s-cluster/config.yml playbooks/02.etcd.yml *** Component Version ********************* ******************************************* * kubernetes: v1.29.0 * etcd: v3.5.10 * flannel: v0.22.2 ******************************************* 2024-05-20 15:21:02 INFO cluster:k8s-cluster setup step:02 begins in 5s, press any key to abort: ....验证
etcd集群配置export NODE_IPS="192.168.1.1 192.168.1.2 192.168.1.3" for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl \ --endpoints=https://${ip}:2379 \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/kubernetes/ssl/etcd.pem \ --key=/etc/kubernetes/ssl/etcd-key.pem \ endpoint health; done root@k8s-etcd01:~# export NODE_IPS="192.168.71.42 192.168.71.43 192.168.71.44" root@k8s-etcd01:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done https://192.168.71.42:2379 is healthy: successfully committed proposal: took = 25.595102ms https://192.168.71.43:2379 is healthy: successfully committed proposal: took = 25.620095ms https://192.168.71.44:2379 is healthy: successfully committed proposal: took = 21.691977ms部署
master节点配置
root@harbor:/etc/kubeasz# ./ezctl setup k8s-cluster 04 ansible-playbook -i clusters/k8s-cluster/hosts -e @clusters/k8s-cluster/config.yml playbooks/04.kube-master.yml *** Component Version ********************* ******************************************* * kubernetes: v1.29.0 * etcd: v3.5.10 * flannel: v0.22.2 ******************************************* 2024-05-20 15:54:30 INFO cluster:k8s-cluster setup step:04 begins in 5s, press any key to abort: ....检查
master节点服务状态root@k8s-master01:~# systemctl status kube-apiserver root@k8s-master01:~#systemctl status kube-controller-manager root@k8s-master01:~#systemctl status kube-scheduler root@harbor:/etc/kubeasz# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy ok部署
kube-node节点root@harbor:/etc/kubeasz# ./ezctl setup k8s-cluster 05 ansible-playbook -i clusters/k8s-cluster/hosts -e @clusters/k8s-cluster/config.yml playbooks/05.kube-node.yml *** Component Version ********************* ******************************************* * kubernetes: v1.29.0 * etcd: v3.5.10 * flannel: v0.22.2 ******************************************* 2024-05-20 16:23:24 INFO cluster:k8s-cluster setup step:05 begins in 5s, press any key to abort:检查安装之后的效果
root@harbor:/etc/kubeasz# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01.cbear.top Ready,SchedulingDisabled master 4m17s v1.29.0 k8s-master02.cbear.top Ready,SchedulingDisabled master 4m17s v1.29.0 k8s-node01.cbear.top Ready node 63s v1.29.0 k8s-node02.cbear.top Ready node 63s v1.29.0部署网络插件
flannel这里需要注意,
kubeasz默认使用的仓库为部署节点中的easzlab.io.local:5000。这里这个仓库是使用ezdown -D默认在本地配置的仓库。由于已经迁移到harbor.cbear.top中。这里需要修改安装配置文件中的yaml配置文件中的仓库位置。由easzlab.io.local:5000修改为harbor.cbear.top。需要注意,kubeasz安装后续插件的过程中均需要有类型修改仓库操作。root@harbor:~# sed -i.bak "s/easzlab.io.local:5000/harbor.cbear.top/g" /etc/kubeasz/roles/flannel/templates/kube-flannel.yaml.j2这一步就可以直接使用
kubectl apply -f .yaml文件安装这个网络插件即可。另外可以使用以下命令实现同样操作。推荐使用以下命令root@harbor:/etc/kubeasz# ./ezctl setup k8s-cluster 06 ansible-playbook -i clusters/k8s-cluster/hosts -e @clusters/k8s-cluster/config.yml playbooks/06.network.yml *** Component Version ********************* ******************************************* * kubernetes: v1.29.0 * etcd: v3.5.10 * flannel: v0.22.2 ******************************************* 2024-05-21 08:01:11 INFO cluster:k8s-cluster setup step:06 begins in 5s, press any key to abort:使用以下命令测试网络插件已经安装完成
root@harbor:/etc/kubeasz# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-64xfx 1/1 Running 0 95s kube-system kube-flannel-ds-dsbxg 1/1 Running 0 95s kube-system kube-flannel-ds-nz7pl 1/1 Running 0 95s kube-system kube-flannel-ds-xbpvz 1/1 Running 0 95s至此基本已经都安装完成
root@harbor:/etc/kubeasz# kubectl run net-test1 --image=alpine sleep 36000 pod/net-test1 created root@harbor:/etc/kubeasz# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default net-test1 1/1 Running 0 11s kube-system kube-flannel-ds-64xfx 1/1 Running 0 3m39s kube-system kube-flannel-ds-dsbxg 1/1 Running 0 3m39s kube-system kube-flannel-ds-nz7pl 1/1 Running 0 3m39s kube-system kube-flannel-ds-xbpvz 1/1 Running 0 3m39s root@harbor:/etc/kubeasz# kubectl get pods NAME READY STATUS RESTARTS AGE net-test1 1/1 Running 0 17s root@harbor:/etc/kubeasz# kubectl exec -it net-test1 sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. / # ping baidu.com ^C / # ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=48 time=64.044 ms 64 bytes from 8.8.8.8: seq=1 ttl=48 time=63.291 ms 64 bytes from 8.8.8.8: seq=2 ttl=48 time=62.878 ms 64 bytes from 8.8.8.8: seq=3 ttl=48 time=62.761 ms 64 bytes from 8.8.8.8: seq=4 ttl=48 time=62.797 ms 64 bytes from 8.8.8.8: seq=5 ttl=48 time=62.698 ms ^C --- 8.8.8.8 ping statistics --- 6 packets transmitted, 6 packets received, 0% packet loss round-trip min/avg/max = 62.698/63.078/64.044 ms剩余的插件基本可以通过直接使用
kubectl命令一个一个安装。另外可以使用./ezctl setup k8s-cluster 07直接按照配置文件中的要求一起安装。接下来的所有步骤。均是按照一个一个插件的安装来。可以查看
kubeasz推荐安装步骤。这里可以看到kubeasz首先复制对应的yaml文件到指定位置之后。直接使用kubectl apply -f .yaml文件root@harbor:/etc/kubeasz/roles/cluster-addon/tasks# cat coredns.yml - block: - name: 准备 DNS的部署文件 template: src=dns/coredns.yaml.j2 dest={{ cluster_dir }}/yml/coredns.yaml - name: 删除coredns部署 shell: "{{ base_dir }}/bin/kubectl delete -f {{ cluster_dir }}/yml/coredns.yaml || echo true; sleep 3" tags: force_change_certs when: 'CHANGE_CA|bool' - name: 创建coredns部署 shell: "{{ base_dir }}/bin/kubectl apply -f {{ cluster_dir }}/yml/coredns.yaml" tags: force_change_certs when: 'dns_install == "yes"'剩余步骤大致按照
kubeasz插件流程来处理。将对应coredns.yaml.j2文件复制到集群目录的yml文件中root@harbor:~# cp /etc/kubeasz/roles/cluster-addon/templates/dns/coredns.yaml.j2 /etc/kubeasz/clusters/k8s-cluster/yml/coredns.yaml切换到指定集群的
yaml目录中root@harbor:~#cp /etc/ root@harbor:~# cd /etc/kubeasz/clusters/k8s-cluster/yml/ root@harbor:/etc/kubeasz/clusters/k8s-cluster/yml#修改指定
yaml文件中的镜像仓库位置root@harbor:/etc/kubeasz/clusters/k8s-cluster/yml# sed -i.bak "s/easzlab.io.local:5000/harbor.cbear.top/g" coredns.yaml默认给的配置文件中的
coredns.yaml文件中引用了一个CLUSTER_DNS_SVC_IP变量。默认这个变量的值为SERVICE_CIDR。这里为10.68.0.2root@harbor:/etc/kubeasz# cat ./roles/cluster-addon/vars/main.yml # default values # coredns 服务地址, 根据SERVICE_CIDR 设置, 默认选择网段第二个地址 CLUSTER_DNS_SVC_IP: "{{ SERVICE_CIDR.split('.')[0] }}.{{ SERVICE_CIDR.split('.')[1] }}.{{ SERVICE_CIDR.split('.')[2] }}.{{ SERVICE_CIDR.split('.')[3]|int + 2 }}"修改完
yaml文件中的版本信息之后, 直接运行kubectl apply -f .yaml。这里只是简单介绍一下。直接直接使用
./ezctl setup k8s-cluster 07更方便一点。在主节点中配置
kubectl使用环境参数。这一步之后基本整体安装完成了。这里只有harbor节点中才能使用kubectl命令。这里需要为master节点配置kubectl使用运行环境。配置完成之后重新登录即可#将/etc/kubeasz/clusters/k8s-cluster/目录中的各个节点的kubectl 的kubeconfig文件拷贝到各个master节点中 root@harbor:/etc/kubeasz/clusters/k8s-cluster# scp k8s-master01.cbear.top-kubectl.kubeconfig root@k8s-master01.cbear.top:/root/ k8s-master01.cbear.top-kubectl.kubeconfig 100% 6197 4.0MB/s 00:00 root@harbor:/etc/kubeasz/clusters/k8s-cluster# scp k8s-master02.cbear.top-kubectl.kubeconfig root@k8s-master02.cbear.top:/root/ k8s-master02.cbear.top-kubectl.kubeconfig 100% 6197 4.1MB/s 00:00 root@harbor:/etc/kubeasz/clusters/k8s-cluster# scp k8s-master03.cbear.top-kubectl.kubeconfig root@k8s-master03.cbear.top:/root/ k8s-master03.cbear.top-kubectl.kubeconfig 100% 6197 4.0MB/s 00:00 #将拷贝到各个节点的kubeconfig文件拷贝到/root/.kube/目录中 root@k8s-master01:~# cp k8s-master01.cbear.top-kubectl.kubeconfig .kube/config root@k8s-master02:~# cp k8s-master02.cbear.top-kubectl.kubeconfig .kube/config root@k8s-master03:~# cp k8s-master03.cbear.top-kubectl.kubeconfig .kube/config
3. 集群管理
3.1 管理master节点
扩容节点分为扩容master节点和work节点
3.1.1 扩容master节点
新增kube_master节点大致流程为: (参考ezctl 中add-master函数和playbooks/23.addmaster.yml)
[可选]新节点安装
chrony时间同步新节点预处理
prepare新节点安装
container runtime新节点安装
kube_master服务新节点安装
kube_node服务新节点安装网络插件相关
禁止业务
pod调度到新master节点更新 node 节点 haproxy 负载均衡并重启( 这个实验中, 整体流程中没有配置
ex_lb。因此最好是在2.2j节中, 设置[ex_lb]注释掉对应内容 , 避免扩容节点过程中报错)
root@harbor:/etc/kubeasz# ./ezctl add-master k8s-cluster 192.168.71.31 k8s_nodename=·k8s-master03.cbear.top这里新扩容的节点需要提前安装对应的containerd环境, 不然会默认自动安装kubeasz默认的containerd容器环境
另外需将配置免密码登录
# ssh 免密码登录
$ ssh-copy-id k8s-master03.cbear.top另外这个命令的使用示例如下:
# 新增节点
$ ezctl add-master test-k8s 192.168.1.11
# 同理, 重复上面步骤再新增节点并自定义nodename
$ ezctl add-master test-k8s 192.168.1.12 k8s_nodename=master-03通过以下命令验证
# 在新节点master 服务状态
$ systemctl status kube-apiserver
$ systemctl status kube-controller-manager
$ systemctl status kube-scheduler
# 查看新master的服务日志
$ journalctl -u kube-apiserver -f
# 查看集群节点, 可以看到新 master节点 Ready, 并且禁止了POD 调度功能
root@harbor:/etc/kubeasz# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.cbear.top Ready,SchedulingDisabled master 4h28m v1.29.0
k8s-master02.cbear.top Ready,SchedulingDisabled master 4h28m v1.29.0
k8s-master03.cbear.top Ready,SchedulingDisabled master 11m v1.29.0
k8s-node01.cbear.top Ready node 4h27m v1.29.0
k8s-node02.cbear.top Ready node 4h27m v1.29.03.1.2 缩容master节点
删除kube_master节点大致流程为: (参考ezctl 中del-master函数和playbooks/33.delmaster.yml)
检测是否可以删除
迁移节点
pod删除
master相关服务及文件删除
node相关服务及文件从集群删除
node节点从
ansible hosts移除节点在
ansible控制端更新kubeconfig更新
node节点haproxy配置
命令如下:
root@harbor:/etc/kubeasz# ./ezctl del-master k8s-cluster 192.168.71.31 #指定删除的节点为192.168.71.313.2 管理node节点
3.2.1 扩容节点
新增kube_node节点大致流程为: (参考ezctl 里面add-node函数 和 playbooks/22.addnode.yml)
[可选]新节点安装
chrony时间同步新节点预处理
prepare新节点安装
container runtime新节点安装
kube_node服务新节点安装网络插件相关
使用用法如下
# ssh 免密码登录
$ ssh-copy-id 192.168.1.11
# 新增节点
$ ezctl add-node test-k8s 192.168.1.11
# 同理, 重复上面步骤再新增节点并自定义nodename
$ ezctl add-node test-k8s 192.168.1.12 k8s_nodename=worker-03整体的扩容思路与扩容master类似
root@harbor:/etc/kubeasz# ./ezctl add-node k8s-cluster 192.168.71.37 k8s_nodename='k8s-node03.cbear.top'用以下命令验证扩容结果
# 验证新节点状态
$ kubectl get node
# 验证新节点的网络插件calico 或flannel 的Pod 状态
$ kubectl get pod -n kube-system3.2.2 缩容节点
删除 node 节点流程: (参考ezctl 里面del-node函数 和 playbooks/32.delnode.yml)
检测是否可以删除
迁移节点上的
pod删除
node相关服务及文件从集群删除
node
操作步骤
$ ezctl del-node k8s-cluster 192.168.71.37 # 假设待删除节点为 192.168.71.373.3 升级版本
升级的主要原理是直接在master/work节点中, 直接下载二进制包。并将对应二进制包, 直接替换原本/opt/kube/bin/中的二进制文件。并重启对应服务即可
#该目录为kubeasz安装之后的二进制包, 每个节点中配置的二进制文件存放位置
root@k8s-master01:~# ll /opt/kube/bin/
total 556360
drwxr-xr-x 3 root root 4096 May 21 03:23 ./
drwxr-xr-x 4 root root 4096 May 20 14:58 ../
drwxr-xr-x 2 root root 4096 May 21 03:23 containerd-bin/
-rwxr-xr-x 1 root root 54939628 May 21 03:23 crictl*
-rwxr-xr-x 1 root root 123715584 May 20 15:54 kube-apiserver*
-rwxr-xr-x 1 root root 118345728 May 20 15:55 kube-controller-manager*
-rwxr-xr-x 1 root root 49704960 May 20 15:55 kubectl*
-rwxr-xr-x 1 root root 111808512 May 20 15:55 kubelet*
-rwxr-xr-x 1 root root 55234560 May 20 15:56 kube-proxy*
-rwxr-xr-x 1 root root 55939072 May 20 15:55 kube-scheduler*集群升级存在一定风险, 请谨慎操作。
支持
k8s相同大版本基础上升级任意小版本, 比如当前安装集群为1.25.0, 你可以方便的升级到任何1.25.x版本不建议跨大版本升级, 一般大版本更新时
k8sapi有一些变动
3.3.1 备份etcd数据
自动备份
kubeasz项目也可以如下方便执行备份(假设集群名为k8s-01),dk ezctl backup k8s-01手动备份
etcd数据, 在任意etcd节点上执行:# snapshot备份 $ ETCDCTL_API=3 etcdctl snapshot save backup.db # 查看备份 $ ETCDCTL_API=3 etcdctl --write-out=table snapshot status backup.db
3.3.2 小版本升级
快速升级k8s小版本, 比较常见如Bug修复 特性发布时使用。
首先去官网
release下载待升级的k8s版本, 例如https://dl.k8s.io/v1.25.4/kubernetes-server-linux-amd64.tar.gz直接在对应
github下载kubernetes/CHANGELOG/CHANGELOG-1.29.md at master · kubernetes/kubernetes · GitHub
root@harbor:~# wget https://dl.k8s.io/v1.29.5/kubernetes-server-linux-amd64.tar.gz解压下载的
tar.gz文件, 找到如下kube*开头的二进制, 复制替换kubeasz控制端目录/etc/kubeasz/bin对应文件kube-apiserverkube-controller-managerkubectlkubeletkube-proxykube-scheduler
切换当前所在集群为升级集群, 在
kubeasz控制端执行dk ezctl checkout k8s-01root@harbor:/etc/kubeasz# ezctl checkout k8s-cluster 2024-05-21 16:45:59 INFO set default kubeconfig: cluster k8s-cluster (current)在
kubeasz控制端执行dk ezctl upgrade k8s-01即可完成k8s升级, 不会中断业务应用#先拷贝对应的二进制文件到/etc/kubeasz/bin目录中 root@harbor:~# cd kubernetes/server/bin/ root@harbor:~/kubernetes/server/bin# cp kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler /etc/kubeasz/bin/ #验证拷贝之后的版本信息 root@harbor:/etc/kubeasz/bin# ./kube-apiserver --version Kubernetes v1.29.5 root@harbor:/etc/kubeasz/bin# ./kube-controller-manager --version Kubernetes v1.29.5 root@harbor:/etc/kubeasz/bin# ./kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.0 root@harbor:/etc/kubeasz/bin# ./kubelet --version Kubernetes v1.29.5 root@harbor:/etc/kubeasz/bin# ./kube-proxy --version Kubernetes v1.29.5 root@harbor:/etc/kubeasz/bin# ./kube-scheduler --version Kubernetes v1.29.5 #直接运行升级命令 root@harbor:/etc/kubeasz# ezctl upgrade k8s-cluster
3.3.3 其他升级说明
其他升级是指升级k8s组件包括: etcd版本 docker版本, 一般不需要用到, 不建议升级, 以下仅作说明。
1.下载所有组件相关新的二进制解压并替换
/etc/kubeasz/bin/目录下文件2.升级 etcd:
ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml -t upgrade_etcd playbooks/02.etcd.yml3.升级 docker ( 建议使用k8s官方支持的docker稳定版本)
如果可以接受短暂业务中断, 执行
ansible-playbook -t upgrade_docker 03.docker.yml如果要求零中断升级, 执行
ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml -t download_docker playbooks/03.runtime.yml, 然后手动执行如下
待升级节点, 先应用
kubectl cordon和kubectl drain命令迁移业务pod待升级节点执行
systemctl restart docker恢复节点可调度
kubectl uncordon
3.4 管理 etcd 集群
Etcd集群支持在线改变集群成员节点, 可以增加、修改、删除成员节点; 不过改变成员数量仍旧需要满足集群成员多数同意原则quorum), 另外请记住集群成员数量变化的影响:
注意: 如果
etcd集群有故障节点, 务必先删除故障节点, 然后添加新节点增加
etcd集群节点, 提高集群稳定性增加
etcd集群节点, 提高集群读性能 ( 所有节点数据一致, 客户端可以从任意节点读取数据)增加
etcd集群节点, 降低集群写性能 ( 所有节点数据一致, 每一次写入会需要所有节点数据同步)
3.4.1 备份 etcd 数据
手动在任意正常
etcd节点上执行备份:# snapshot备份 $ ETCDCTL_API=3 etcdctl snapshot save backup.db # 查看备份 $ ETCDCTL_API=3 etcdctl --write-out=table snapshot status backup.db使用
kubeasz备份cluster_name为k8s-01ezctl backup k8s-01
使用 crontab 定时备份示例(使用容器化的 kubeasz , 每日01:01 备份)
1 1 * * * /usr/bin/docker exec -i kubeasz ezctl backup k8s-01备份文件在
{{ base_dir }}/clusters/k8s-01/backup3.4.2 etcd集群节点操作
执行如下 (假设待操作节点为 192.168.1.11 , 集群名称test-k8s):
增加
etcd节点:# ssh 免密码登录 $ ssh-copy-id 192.168.1.11 # 新增节点 $ ezctl add-etcd test-k8s 192.168.1.11删除
etcd节点:$ ezctl del-etcd test-k8s 192.168.1.11
具体操作流程参考 ezctl中 add-etcd/del-etcd 相关函数和playbooks/目录的操作剧本
3.4.3 验证 etcd 集群
# 登录任意etcd节点验证etcd集群状态
$ export ETCDCTL_API=3
$ etcdctl member list
# 验证所有etcd节点服务状态和日志
$ systemctl status etcd
$ journalctl -u etcd -f3.5 K8S 集群备份与恢复
虽然 K8S 集群可以配置成多主多节点的高可用的部署 , 还是有必要了解下集群的备份和容灾恢复能力 ; 在高可用k8s集群中 etcd集群保存了整个集群的状态 , 因此这里的备份与恢复重点就是:
从运行的
etcd集群备份数据到磁盘文件从
etcd备份文件恢复数据 , 从而使集群恢复到备份时状态
备份与恢复操作说明
首先搭建一个测试集群 , 部署几个测试
deployment, 验证集群各项正常后 , 进行一次备份(假设集群名为k8s-01):$ ezctl backup k8s-01 # 或者如下手动执行ansible命令 # ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/94.backup.yml执行完毕可以在部署主机的备份目录下检查备份情况 , 示例如下:
/etc/kubeasz/clusters/k8s-01/backup/ ├── snapshot_202106201205.db ├── snapshot_202106211406.db └── snapshot.db其中 ,
snapshot.db始终为最近一次备份文件模拟误删除操作( 略)
恢复集群及验证
可以在
roles/cluster-restore/defaults/main.yml文件中配置需要恢复的etcd备份版本( 从上述备份目录中选取) , 默认使用最近一次备份 ; 执行恢复后 , 需要一定时间等待pod/svc等资源恢复重建。$ ezctl restore k8s-01 # 或者如下手动执行ansible命令 # ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/95.restore.yml
如果集群主要组件( master/etcd/node) 等出现不可恢复问题 , 可以尝试使用如下步骤 清理--> 创建--> 恢复
$ ezctl clean k8s-01
# 或者如下手动执行ansible命令
# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/99.clean.yml
$ ezctl setup k8s-01 01
$ ezctl setup k8s-01 02
$ ezctl setup k8s-01 03
$ ezctl setup k8s-01 04
$ ezctl setup k8s-01 05
...
$ ezctl restore k8s-01
# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/95.restore.yml3.6 管理客户端kubeconfig
默认 k8s集群安装成功后生成客户端kubeconfig，它拥有集群管理的所有权限（不要将这个admin权限、50年期限的kubeconfig流露出去） ; 而我们经常需要将限定权限、限定期限的kubeconfig 分发给普通用户 ; 利用cfssl签发自定义用户证书和k8s灵活的rbac权限绑定机制，ezctl 工具封装了这个功能。
使用帮助
ezctl help kcfg-adm
Usage: ezctl kcfg-adm <cluster> <args>
available <args>:
-A to add a client kubeconfig with a newly created user
-D to delete a client kubeconfig with the existed user
-L to list all of the users
-e to set expiry of the user certs in hours (ex. 24h, 8h, 240h)
-t to set a user-type (admin or view)
-u to set a user-name prefix
examples: ./ezctl kcfg-adm test-k8s -L
./ezctl kcfg-adm default -A -e 240h -t admin -u jack
./ezctl kcfg-adm default -D -u jim-202101162141可以设置过期时间
可以设置权限：管理员权限（
admin）和只读权限（view）
使用举例
查看集群
k8s-01当前自定义kubeconfigezctl kcfg-adm k8s-01 -L 2021-01-24 16:32:43 INFO list-kcfg k8s-01 2021-01-24 16:32:43 INFO list-kcfg in cluster:k8s-01 USER TYPE EXPIRY(+8h if in Asia/Shanghai) --------------------------------------------------------------------------------- 2021-01-24 16:32:43 INFO list-kcfg k8s-01 success初始情况下列表为空
增加集群
k8s-01一个自定义用户kubeconfig，用户名user01，期限24h，只读权限ezctl kcfg-adm k8s-01 -A -u user01 -e 24h -t view 2021-01-24 17:32:33 INFO add-kcfg k8s-01 2021-01-24 17:32:33 INFO add-kcfg in cluster:k8s-01 with user:user01-202101241732 PLAY [localhost] ***************************************************************************************************** ...（此处省略输出） TASK [deploy : debug] ************************************************************************************************ ok: [localhost] => { "msg": "查看user01-202101241732自定义kubeconfig：/etc/kubeasz/clusters/k8s-01/ssl/users/user01-202101241732.kubeconfig" } PLAY RECAP *********************************************************************************************************** localhost : ok=12 changed=10 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 2021-01-24 17:32:41 INFO add-kcfg k8s-01 success生成的
kubeconfig位于/etc/kubeasz/clusters/k8s-01/ssl/users/user01-202101241732.kubeconfig再增加一个用户
user02，期限240h，admin权限ezctl kcfg-adm k8s-01 -A -u user02 -e 240h -t admin 2021-01-24 18:38:47 INFO add-kcfg k8s-01 2021-01-24 18:38:47 INFO add-kcfg in cluster:k8s-01 with user:user02-202101241838 PLAY [localhost] ***************************************************************************************************** ...（此处省略输出） TASK [deploy : debug] ************************************************************************************************ ok: [localhost] => { "msg": "查看user02-202101241838自定义kubeconfig：/etc/kubeasz/clusters/k8s-01/ssl/users/user02-202101241838.kubeconfig" } PLAY RECAP *********************************************************************************************************** localhost : ok=12 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 2021-01-24 18:38:55 INFO add-kcfg k8s-01 success再次查看集群
k8s-01当前自定义kubeconfigezctl kcfg-adm k8s-01 -L 2021-01-24 18:40:30 INFO list-kcfg k8s-01 2021-01-24 18:40:30 INFO list-kcfg in cluster:k8s-01 USER TYPE EXPIRY(+8h if in Asia/Shanghai) --------------------------------------------------------------------------------- user02-202101241838 cluster-admin 2021-02-03T10:34:00Z user01-202101241732 view 2021-01-25T09:28:00Z 2021-01-24 18:40:31 INFO list-kcfg k8s-01 success删除
user01-202101241732权限ezctl kcfg-adm k8s-01 -D -u user01-202101241732 2021-01-24 21:41:50 INFO del-kcfg k8s-01 2021-01-24 21:41:50 INFO del-kcfg in cluster:k8s-01 with user:user01-202101241732 clusterrolebinding.rbac.authorization.k8s.io "crb-user01-202101241732" deleted 2021-01-24 21:41:50 INFO del-kcfg k8s-01 success ezctl kcfg-adm k8s-01 -L 2021-01-24 21:42:02 INFO list-kcfg k8s-01 2021-01-24 21:42:02 INFO list-kcfg in cluster:k8s-01 USER TYPE EXPIRY(+8h if in Asia/Shanghai) --------------------------------------------------------------------------------- user02-202101241838 cluster-admin 2021-02-03T10:34:00Z 2021-01-24 21:42:02 INFO list-kcfg k8s-01 success
3.7 修改 APISERVER（MASTER）证书
kubeasz 创建集群后，APISERVER（MASTER）证书默认 CN 包含如下域名和IP：参见roles/kube-master/templates/kubernetes-csr.json.j2
"hosts": [
"127.0.0.1",
{% if groups['ex_lb']|length > 0 %}
"{{ hostvars[groups['ex_lb'][0]]['EX_APISERVER_VIP'] }}",
{% endif %}
{% for host in groups['kube_master'] %}
"{{ host }}",
{% endfor %}
"{{ CLUSTER_KUBERNETES_SVC_IP }}",
{% for host in MASTER_CERT_HOSTS %}
"{{ host }}",
{% endfor %}
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],有的时候（比如apiserver地址通过边界防火墙的NAT转换成公网IP访问，或者需要添加公网域名访问）我们需要在 APISERVER（MASTER）证书中添加一些域名或者IP，可以方便操作如下：
修改配置文件
/etc/kubeasz/clusters/${集群名}/config.yaml# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名） MASTER_CERT_HOSTS: - "10.1.1.1" - "k8s.test.io" #- "www.test.com"执行新证书生成并重启
apiserver$ ezctl setup ${集群名} 04 -t change_cert,restart_master
3.8 强制更新CA和所有证书
WARNNING: 此命令使用需要小心谨慎，确保了解功能背景和可能的结果 ; 执行后，它会重新创建集群CA证书以及由它颁发的所有其他证书 ; 一般适合于集群admin.conf不小心泄露，为了避免集群被非法访问，重新创建CA，从而使已泄漏的admin.conf失效。如果需要分发受限的
kubeconfig，强烈建议使用自定义权限和期限的kubeconfig
3.8.1 使用帮助
确认需要强制更新后，在ansible 控制节点使用如下命令：(xxx 表示需要操作的集群名)
docker exec -it kubeasz ezctl kca-renew xxx
# 或者使用 dk ezctl kca-renew xxx上述命令执行后，按序进行以下的操作：详见playbooks/96.update-certs.yml
重新生成
CA证书，以及各种kubeconfig签发新
etcd证书，并使用新证书重启etcd服务签发新
kube-apiserver证书，并重启kube-apiserver/kube-controller-manager/kube-scheduler服务签发新
kubelet证书，并重启kubelet/kube-proxy服务重启网络组件
pod重启其他集群组件
pod特别注意： 如果集群中运行的业务负载
pod需要访问apiserver，需要重启这些pod
3.8.2 检查验证
更新完毕，注意检查集群组件日志和容器pod日志，确认集群处于正常状态
集群组件日志：使用
journalctl -u xxxx.service -f依次检查etcd.service/kube-apiserver.service/kube-controller-manager.service/kube-scheduler.service/kubelet.service/kube-proxy.service容器
pod日志：使用kubectl logs方式检查容器日志
3.9 配置负载转发 ingress nodeport
向集群外暴露 ingress-controller 本身的服务端口(80/443/8080)一般有以下三种方法：
部署
ingress-controller时使用hostNetwork: true，这样就可以直接使用上述端口，可能与host已listen端口冲突部署
ingress-controller时使用LoadBalancer类型服务，需要集群支持LoadBalancer部署
ingress-controller时使用nodePort类型服务，然后在集群外使用haproxy/f5等配置virtual server集群
本文档讲解使用 haproxy 配置 ingress的 VS 集群，前提是配置了自建ex_lb节点
配置
ex_lb参数开启转发ingress nodeport# 编辑 roles/ex-lb/defaults/main.yml，配置如下变量 INGRESS_NODEPORT_LB: "yes" INGRESS_TLS_NODEPORT_LB: "yes"重新配置启动LB节点服务
$ ezctl setup ${集群名} ex-lb验证
ex_lb节点的haproxy服务配置/etc/haproxy/haproxy.cfg包含如下配置... 前文省略 listen kube_master bind 0.0.0.0:8443 mode tcp option tcplog balance roundrobin server 192.168.1.1 192.168.1.1:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.2 192.168.1.2:6443 check inter 2000 fall 2 rise 2 weight 1 listen ingress-node bind 0.0.0.0:80 mode tcp option tcplog balance roundrobin server 192.168.1.3 192.168.1.3:23456 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.4 192.168.1.4:23456 check inter 2000 fall 2 rise 2 weight 1 listen ingress-node-tls bind 0.0.0.0:443 mode tcp option tcplog balance roundrobin server 192.168.1.3 192.168.1.3:23457 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.4 192.168.1.4:23457 check inter 2000 fall 2 rise 2 weight 1