搜索文章 Ctrl K

4-2-k8s组件-etcd

熊 0 阅读 容器

1. etcd基础

etcd 是一个基于 Raft 协议实现。开源的、分布式的键值存储系统。主要用于在分布式系统中提供强一致性和高可用性的数据存储。

etcd 在 Kubernetes 中的作用如下 :

  1. 集群状态数据存储: 集群配置 , 集群状态信息等

  2. 保证集群一致性和高可用: 多实例的数据同步

  3. 服务发现和配置共享

  4. 集群数据备份和恢复

作为 Kubernetes 的核心组件 , etcd 为集群的稳定性、可靠性和一致性提供了支撑。

root@k8s-etcd01:~# systemctl cat etcd
# /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
​
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd   #数据保存目录 
ExecStart=/opt/kube/bin/etcd \
  --name=etcd-192.168.71.42 \   #当前node名称
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://192.168.71.42:2380 \           #通告自己的集群端口
  --listen-peer-urls=https://192.168.71.42:2380 \                      #集群之间通讯端口 
  --listen-client-urls=https://192.168.71.42:2379,http://127.0.0.1:2379 \    #客户端访问地址
  --advertise-client-urls=https://192.168.71.42:2379 \                       #通告自己的客户端端口
  --initial-cluster-token=etcd-cluster-0 \                   #创建集群使用的token，一个集群内的节点保持一致 
  --initial-cluster=etcd-192.168.71.42=https://192.168.71.42:2380,etcd-192.168.71.43=https://192.168.71.43:2380,etcd-192.168.71.44=https://192.168.71.44:2380 \   #集群所有的节点信息 
  --initial-cluster-state=new \                      #新建集群的时候的值为new,如果是已经存在的集群为existing。 
  --data-dir=/var/lib/etcd \                         #数据目录路径
  --wal-dir= \
  --snapshot-count=50000 \
  --auto-compaction-retention=1 \
  --auto-compaction-mode=periodic \
  --max-request-bytes=10485760 \
  --quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999
​
[Install]
WantedBy=multi-user.target
​

2. etcd使用

2.1 查看成员信息

etcd有多个不同的API访问版本，v1版本已经废弃，etcd v2 和 v3 本质上是共享同一套 raft 协议代码的两个独立的应用，接口不一样，存储不一样，数据互相隔离。也就是说如果从 Etcd v2 升级到 Etcd v3，原来v2 的数据还是只能用 v2 的接口访问，v3 的接口创建的数据也只能访问通过 v3 的接口访问。

root@k8s-etcd01:~# ETCDCTL_API=3 etcdctl --help
root@k8s-etcd01:~# ETCDCTL_API=3 etcdctl member --help 
NAME:
    member - Membership related commands
​
USAGE:
    etcdctl member <subcommand> [flags]
​
API VERSION:
    3.5
​
​
COMMANDS:
    add Adds a member into the cluster
    list    Lists all members in the cluster
    promote Promotes a non-voting member in the cluster
    remove  Removes a member from the cluster
    update  Updates a member in the cluster
​
OPTIONS:
  -h, --help[=false]    help for member
​
GLOBAL OPTIONS:
      --cacert=""               verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""                 identify secure client using this TLS certificate file
      --command-timeout=5s          timeout for short running command (excluding dial timeout)
      --debug[=false]               enable client-side debug logging
      --dial-timeout=2s             dial timeout for client connections
  -d, --discovery-srv=""            domain name to query for SRV records describing cluster endpoints
      --discovery-srv-name=""           service name to query when using DNS discovery
      --endpoints=[127.0.0.1:2379]      gRPC endpoints
      --hex[=false]             print byte strings as hex encoded strings
      --insecure-discovery[=true]       accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]    skip server certificate verification (CAUTION: this option should be enabled only for testing purposes)
      --insecure-transport[=true]       disable transport security for client connections
      --keepalive-time=2s           keepalive time for client connections
      --keepalive-timeout=6s            keepalive timeout for client connections
      --key=""                  identify secure client using this TLS key file
      --password=""             password for authentication (if this option is used, --user option shouldn't include password)
      --user=""                 username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"          set the output format (fields, json, protobuf, simple, table)
​
root@k8s-etcd01:~# etcdctl  member list 
1e37acd9c09e84ed, started, etcd-192.168.71.42, https://192.168.71.42:2380, https://192.168.71.42:2379, false
753516f1b797b04c, started, etcd-192.168.71.44, https://192.168.71.44:2380, https://192.168.71.44:2379, false
c39c6b17103e2555, started, etcd-192.168.71.43, https://192.168.71.43:2380, https://192.168.71.43:2379, false
​
​
​

2.2 验证当前etcd所有成员状态

root@k8s-etcd01:~# export NODE_IPS="192.168.71.42 192.168.71.43 192.168.71.44"
root@k8s-etcd01:~# for ip in ${NODE_IPS}; do   ETCDCTL_API=3 etcdctl   --endpoints=https://${ip}:2379    --cacert=/etc/kubernetes/ssl/ca.pem   --cert=/etc/kubernetes/ssl/etcd.pem   --key=/etc/kubernetes/ssl/etcd-key.pem   endpoint health; done
https://192.168.71.42:2379 is healthy: successfully committed proposal: took = 25.595102ms
https://192.168.71.43:2379 is healthy: successfully committed proposal: took = 25.620095ms
https://192.168.71.44:2379 is healthy: successfully committed proposal: took = 21.691977ms

2.3 查看etcd数据信息

root@k8s-etcd01:~#  ETCDCTL_API=3 etcdctl get / --prefix --keys-only    #以路径的方式所有key信息
/registry/apiregistration.k8s.io/apiservices/v1.
​
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.apps
​
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
​
/registry/apiregistration.k8s.io/apiservices/v1.batch
​
/registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.events.k8s.io
​
/registry/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io
......
​
​
​
root@k8s-etcd01:~#  ETCDCTL_API=3 etcdctl get /registry/services/specs/kube-system/node-local-dns
/registry/services/specs/kube-system/node-local-dns
k8s
​
v1Service«  
¡
​
root@k8s-etcd01:~#  ETCDCTL_API=3 etcdctl  get --prefix /registry | less
/registry/apiregistration.k8s.io/apiservices/v1.
{"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1","metadata":{"name":"v1.","uid":"a747e704-3ddc-4606-bde5-89cf8b21151b","creationTimestamp":"2024-05-21T10:05:23Z","labels":{"kube-aggregator.kubernetes.io/automanaged":"onstart"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiregistration.k8s.io/v1","time":"2024-05-21T10:05:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:kube-aggregator.kubernetes.io/automanaged":{}}},"f:spec":{"f:groupPriorityMinimum":{},"f:version":{},"f:versionPriority":{}}}}]},"spec":{"version":"v1","groupPriorityMinimum":18000,"versionPriority":1},"status":{"conditions":[{"type":"Available","status":"True","lastTransitionTime":"2024-05-21T10:05:23Z","reason":"Local","message":"Local APIServices are always available"}]}}
​
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
{"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1","metadata":{"name":"v1.admissionregistration.k8s.io","uid":"12e3cbfb-7407-411e-a3e9-d043ce2e759d","creationTimestamp":"2024-05-21T10:05:23Z","labels":{"kube-aggregator.kubernetes.io/automanaged":"onstart"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiregistration.k8s.io/v1","time":"2024-05-21T10:05:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:kube-aggregator.kubernetes.io/automanaged":{}}},"f:spec":{"f:group":{},"f:groupPriorityMinimum":{},"f:version":{},"f:versionPriority":{}}}}]},"spec":{"group":"admissionregistration.k8s.io","version":"v1","groupPriorityMinimum":16700,"versionPriority":15},"status":{"conditions":[{"type":"Available","status":"True","lastTransitionTime":"2024-05-21T10:05:23Z","reason":"Local","message":"Local APIServices are always available"}]}}
​

2.4 etcd增删改查数据

root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl put  /testkey "test for k8s" 
OK
root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl get   /testkey 
/testkey
test for k8s
root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl put  /testkey "test for k8s-new"
OK
root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl get   /testkey 
/testkey
test for k8s-new
root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl del   /testkey 
1
root@k8s-etcd01:~# ETCDCTL_API=3  etcdctl get   /testkey 
​

2.5 etcd数据watch机制

基于不断监看数据，发生变化就主动触发通知客户端，Etcd v3 的watch机制支持watch某个固定的key，也支持 watch一个范围。 相比Etcd v2, Etcd v3的一些主要变化：

接口通过grpc提供rpc接口，放弃了v2的http接口，优势是长连接效率提升明显，缺点是使用不如以前方便，尤其对不方便维护长连接的场景。 
废弃了原来的目录结构，变成了纯粹的kv，用户可以通过前缀匹配模式模拟目录。 
内存中不再保存value，同样的内存可以支持存储更多的key。 
watch机制更稳定，基本上可以通过watch机制实现数据的完全同步。 
提供了批量操作以及事务机制，用户可以通过批量事务请求来实现Etcd v2的CAS机制（批量事务支持if条件判断）。

watch测试：

#在etcd node1上watch一个key，没有此key也可以执行watch，后期可以再创建： 
root@k8s-etcd2:~# ETCDCTL_API=3  /usr/bin/etcdctl watch  /testkey 
#在etcd node2修改数据，验证etcd node1是否能够发现数据变化 
root@k8s-etcd1:~# ETCDCTL_API=3  /usr/bin/etcdctl put  /testkey "test for linux36-new" 
OK

2.6 etcd数据备份与恢复机制

WAL是write ahead log的缩写，顾名思义，也就是在执行真正的写操作之前先写一个日志。

wal : 存放预写式日志,最大的作用是记录了整个数据变化的全部历程。在etcd中，所有数据的修改在提交前，都要先写入到WAL中。

2.6.1 etcd v2版本数据备份与恢复

V2版本帮助信息:
root@k8s-etcd01:~# ETCDCTL_API=2  etcdctl backup --help
NAME:
   etcdctl backup - --data-dir=... --backup-dir={output}
​
USAGE:
   [deprecated] offline backup an etcd directory.
​
DESCRIPTION:
   Performs an offline backup of etcd directory.
​
Moved to `./etcdutl backup` and going to be decomissioned in v3.5
​
The recommended (online) backup command is: `./etcdctl snapshot save ...`.
​
​
​
OPTIONS:
   --data-dir value        Path to the etcd data dir
   --wal-dir value         Path to the etcd wal dir
   --backup-dir value      Path to the backup dir
   --backup-wal-dir value  Path to the backup wal dir
   --with-v3               Backup v3 backend data
​
#v2版本备份数据
root@k8s-etcd2:~# ETCDCTL_API=2 etcdctl backup  --data-dir /var/lib/etcd/ --backup-dir 
/opt/etcd_backup 
2019-07-11 18:59:57.674432 I | wal: segmented wal file 
/opt/etcd_backup/member/wal/0000000000000001-0000000000017183.wal is created 
​
#v2版本恢复数据
root@k8s-etcd01:~# etcd  --help | grep force
  --force-new-cluster 'false'
    Force to create a new one-member cluster.
​
etcd --data-dir=/var/lib/etcd/default.etcd --force-new-cluster & 
root@k8s-etcd2:~# vim /etc/systemd/system/etcd.service 
[Unit] 
Description=Etcd Server 
After=network.target 
After=network-online.target 
Wants=network-online.target 
Documentation=https://github.com/coreos 
[Service] 
Type=notify 
WorkingDirectory=/var/lib/etcd/ 
ExecStart=/usr/bin/etcd  \ 
--name=etcd2 \ 
................................. 
--data-dir=/opt/etcd_backup  -force-new-cluster #强制设置为为新集群 
Restart=on-failure 
RestartSec=5 
LimitNOFILE=65536 
[Install] 
WantedBy=multi-user.target

2.6.2 etcd v3版本数据备份与恢复

V3版本备份数据： 
root@k8s-etcd2:~# ETCDCTL_API=3 etcdctl  snapshot save  snapshot.db 
Snapshot saved at snapshot.db 
V3版本恢复数据： 
root@k8s-etcd2:~#  ETCDCTL_API=3  etcdctl snapshot restore  snapshot.db  --data-dir=/opt/etcd-testdir #将数据恢复到一个新的不存在的目录中 
2019-07-11 18:57:46.526757 I | mvcc: restore compact to 74541 
2019-07-11 18:57:46.536114 I | etcdserver/membership: added member 8e9e05c52164694d 
[http://localhost:2380] to cluster cdf818194e3a8c32 
#自动备份数据 
root@k8s-etcd2:~# mkdir /data/etcd-backup-dir/ -p 
root@k8s-etcd2:~# cat  script.sh  
#!/bin/bash 
source /etc/profile 
DATE=`date +%Y-%m-%d_%H-%M-%S` 
ETCDCTL_API=3 /usr/bin/etcdctl  snapshot save  /data/etcd-backup-dir/etcd-snapshot-${DATE}.db

Kubernetes 漫游: etcd (qq.com)