2-3-基于kubeadm工具部署的k8s集群证书到期更新方法

基于kubeadm部署的 Kubernetes 集群，默认证书有效期是一年，如果不及时更新证书，就会导致一系列问题和影响，包括：

1. 节点通信受阻：证书到期后，节点之间的安全通信可能会受到影响，这可能导致节点无法相互通信或者无法加入集群。

2. API服务器访问受限：API服务器使用证书进行身份验证和安全通信。如果证书过期，可能会导致客户端无法连接到API服务器，从而影响管理和监控集群的能力。

3. TLS连接问题：集群中的各种服务和组件可能会使用TLS证书进行加密通信。证书过期可能导致服务之间的通信出现问题，从而影响应用程序的正常运行。

4. 自动扩展和自动修复受影响：某些自动化功能，如水平扩展和自动修复，可能依赖于与Kubernetes API服务器的通信。证书过期可能会影响这些功能的正常运行。

5. 安全风险增加：证书过期可能会增加安全风险，因为过期的证书可能会被恶意利用，攻击者可以使用它们来尝试入侵集群或窃取数据。

为了减轻证书到期可能带来的影响，建议在证书到期之前进行更新或者续订，并确保集群中的所有节点和组件都能够使用最新的有效证书。

当前环境是基于etcd集群部署在k8s集群内的操作方法。如果是部署的外置etcd集群，则建议使用手动更新方式。

执行以下命令查看证书到期时间

root@kubeadm-master1:~# kubeadm certs check-expiration

方法一: 脚本自动更新

k8s集群1.15版本以上支持此脚本自动化更新。(此证书更新后有效期10年)

下载脚本

# git clone https://github.com/yuyicai/update-kube-cert.git
# cd update-kubeadm-cert 
# chmod 755 update-kubeadm-cert.sh

同时更新etcd证书和master证书(每个master节点都需要执行一次)

#在每台master上执行以下命令
#./update-kubeadm-cert.sh all

方法二: 手动更新

1. 备份老的证书及配置

   root@kubeadm-master1:~# cp -r /etc/kubernetes /etc/kubernetes_bak

2. 续期证书

   root@kubeadm-master1:~# kubeadm certs renew all --config=kubeadm.yaml 
   certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
   certificate for serving the Kubernetes API renewed
   certificate the apiserver uses to access etcd renewed
   certificate for the API server to connect to kubelet renewed
   certificate embedded in the kubeconfig file for the controller manager to use renewed
   certificate for liveness probes to healthcheck etcd renewed
   certificate for etcd nodes to communicate with each other renewed
   certificate for serving etcd renewed
   certificate for the front proxy client renewed
   certificate embedded in the kubeconfig file for the scheduler manager to use renewed
   certificate embedded in the kubeconfig file for the super-admin renewed
   ​
   Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
   #执行上面命令后，需要重启对应pod才能生效kube-apiserver, kube-controller-manager, kube-scheduler and etcd

3. 需要拷贝新证书到其他master节点

   root@kubeadm-master1:~# rsync -avp  /etc/kubernetes/pki/ root@192.168.71.30:/etc/kubernetes/pki/
   root@kubeadm-master1:~# rsync -avp  /etc/kubernetes/pki/ root@192.168.71.31:/etc/kubernetes/pki/

4. 重新生成配置文件

   root@kubeadm-master1:~# cd /etc/kubernetes
   root@kubeadm-master1:/etc/kubernetes# ls
   admin.conf  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf  super-admin.conf  tmp
   root@kubeadm-master1:/etc/kubernetes# ls *conf
   admin.conf  controller-manager.conf  kubelet.conf  scheduler.conf  super-admin.conf
   root@kubeadm-master1:/etc/kubernetes# rm -rf *conf
   root@kubeadm-master1:/etc/kubernetes# kubeadm init phase kubeconfig all --config=/root/kubeadm.yaml 
   [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
   [kubeconfig] Writing "admin.conf" kubeconfig file
   [kubeconfig] Writing "super-admin.conf" kubeconfig file
   [kubeconfig] Writing "kubelet.conf" kubeconfig file
   [kubeconfig] Writing "controller-manager.conf" kubeconfig file
   [kubeconfig] Writing "scheduler.conf" kubeconfig file

5. 生成其他主节点的配置文件

   root@kubeadm-master1:/tmp/master2# kubeadm init phase kubeconfig all --config=/root/kubeadm.yaml --node-name=kubeadm-master2.cbear.top  --kubeconfig-dir /tmp/master2/
   [kubeconfig] Using kubeconfig folder "/tmp/master2/"
   [kubeconfig] Writing "admin.conf" kubeconfig file
   [kubeconfig] Writing "super-admin.conf" kubeconfig file
   [kubeconfig] Writing "kubelet.conf" kubeconfig file
   [kubeconfig] Writing "controller-manager.conf" kubeconfig file
   [kubeconfig] Writing "scheduler.conf" kubeconfig file
   root@kubeadm-master1:/tmp/master2# vi admin.conf 
   root@kubeadm-master1:/tmp/master2# ls
   admin.conf  controller-manager.conf  kubelet.conf  scheduler.conf  super-admin.conf
   ​
   root@kubeadm-master1:/tmp/master2# scp * root@192.168.71.30:/etc/kubernetes/
   ​

6. 复制新的认证文件

   root@kubeadm-master1:~# rm -rf ~/.kube
   root@kubeadm-master1:~# mkdir -p $HOME/.kube
   root@kubeadm-master1:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   root@kubeadm-master1:~#  chown $(id -u):$(id -g) $HOME/.kube/config
   ​

7. 直接在所有master节点中删除所有容器

   root@kubeadm-master1:~# kubectl -n kube-system delete pod -l 'component=kube-apiserver'
   root@kubeadm-master1:~# kubectl -n kube-system delete pod -l 'component=kube-controller-manager'
   root@kubeadm-master1:~#kubectl -n kube-system delete pod -l 'component=kube-scheduler'
   root@kubeadm-master1:~# kubectl -n kube-system delete pod -l 'component=etcd'
   ​

8. 为防止 kubelet 客户端证书轮换失败，我们需要将(此处坑有点大)kubelet-client-* 进行删除，在kubelet服务重启时又会自动生成, 如果此轮换过程失败，你可能会在 kube-apiserver 日志中看到诸如 x509: certificate has expired or is not yet valid 之类的错误

   对 kubeadm 进行故障排查 | Kubernetes

   # 重启节点上 kubelet 服务
   systemctl restart kubelet.service
   ​
   # 查看 kubelet 服务状态
   systemctl status -l kubelet.service 
   ​
   # 查看 kubelet 服务运行日志有无异常（有异常请依次解决）
   journalctl -xefu kubelet
   ​
   # 重启 kubelet 后 kubelet 客户端证书轮换自动生成新的 pem 证书
   ls -alh /var/lib/kubelet/pki/kubelet-client-current.pem
     # lrwxrwxrwx 1 root root 59 Jan 31 22:29 /var/lib/kubelet/pki/kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2023-01-31-22-29-20.pe

参考链接

运维案例之记一次Kubernetes集群证书过期或延期操作处理实践指南（干货分享） (qq.com)

基于kubeadm工具部署的k8s集群证书到期更新方法 (qq.com)