2021-07-26-etcd入门 [[etcd]] [[Kubernetes]] etcd 入门 etcd(读作 et-see-dee)是一种开源的分布式统一键值存储,用于分布式系统或计算机集群的共享配置、服务发现和的调度协调。etcd 有助于促进更加安全的自动更新,协调向主机调度的工作,并帮助设置容器的覆盖网络。
etcd 是许多其他项目的核心组件。最值得注意的是,它是 Kubernetes 的首要数据存储,也是容器编排的实际标准系统。使用 etcd, 云原生应用可以保持更为一致的运行时间,而且在个别服务器发生故障时也能正常工作。应用从 etcd 读取数据并写入到其中;通过分散配置数据,为节点配置提供冗余和弹性。
一种键值对分布式数据库。
通过 raft 共识机制,使得 etcd 具备高可用性。
raft visualization
安装并启动 etcd 作为 k8s 的常见组建,使用 Go 开发,可以方便的构建并安装到大多数的系统中。在 Linux 中只需要一个二进制程序即可启动。
cd /tmp wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz tar -zxvf etcd-v3.5.2-linux-amd64.tar.gz -C etcd cd etcd/ ./etcd ## 下面就是启动日志 对于 etcd 集群的启动,通过环境变量的方式进行参数配置并启动。
官方也提供了单机集群的方案,需要用到 goreman,然后修改 Procfile 文件,把 etcd 执行路径修改为真实路径以及网络配置的变动,通过 goreman -f Procfile start 即可启动一个 etcd 集群。
使用 常见两种方式:
命令行通过 etcdctl 进行使用 程序包调用 增删改查 # put etcdctl --endpoints=$ENDPOINTS put foo "Hello World!" # get etcdctl --endpoints=$ENDPOINTS get foo etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo # get keys by prefix etcdctl --endpoints=$ENDPOINTS put web1 value1 etcdctl --endpoints=$ENDPOINTS put web2 value2 etcdctl --endpoints=$ENDPOINTS put web3 value3 etcdctl --endpoints=$ENDPOINTS get web --prefix # delete keys etcdctl --endpoints=$ENDPOINTS put key myvalue etcdctl --endpoints=$ENDPOINTS del key # delete keys by prefix etcdctl --endpoints=$ENDPOINTS put k1 value1 etcdctl --endpoints=$ENDPOINTS put k2 value2 etcdctl --endpoints=$ENDPOINTS del k --prefix 事务 # use transaction # etcd transaction mode # if compare # then op # else op # commit etcdctl --endpoints=$ENDPOINTS put user1 bad etcdctl --endpoints=$ENDPOINTS txn --interactive compares: value("user1") = "bad" success requests (get, put, delete): del user1 failure requests (get, put, delete): put user1 good SUCCESS 1 # in this case, key:user1 could be delete 监测 key # watch keys # in this case,you should open two terminal # terminal 1, create a watcher, wait key:stock1 appear etcdctl --endpoints=$ENDPOINTS watch stock1 # after terminal2 put stock1,could happen: PUT stock1 1000 # terminal 2, put stock1 etcdctl --endpoints=$ENDPOINTS put stock1 1000 # watch keys by prefix # terminal 1,wait etcdctl --endpoints=$ENDPOINTS watch stock --prefix # after terminal2 put keys,could happen: PUT stock1 10 PUT stock2 20 # terminal 2,put keys etcdctl --endpoints=$ENDPOINTS put stock1 10 etcdctl --endpoints=$ENDPOINTS put stock2 20 # or watch keys history etcdctl --endpoints=$ENDPOINTS watch --rev=1 foo 租约 # lease ## create a lease etcdctl --endpoints=$ENDPOINTS lease grant 300 lease 49527f598bbfe014 granted with TTL(300s) ## key binding to lease etcdctl --endpoints=$ENDPOINTS put sample value --lease=49527f598bbfe014 etcdctl --endpoints=$ENDPOINTS get sample sample value ## if set keep-alive, than the lease could never timeout etcdctl --endpoints=$ENDPOINTS lease keep-alive 49527f598bbfe014 ## revoke lease manual etcdctl --endpoints=$ENDPOINTS lease revoke 49527f598bbfe014 lease 49527f598bbfe014 revoked # or after 300 seconds,could get null data etcdctl --endpoints=$ENDPOINTS get sample 分布式锁 # create lock # in this case,you should open two terminal # terminal 1, etcdctl --endpoints=$ENDPOINTS lock mutex1 mutex1/49527f598bbfe018 # interrupt this lock ^C # terminal 2, wait terminal 1 release lock etcdctl --endpoints=$ENDPOINTS lock mutex1 # will get lock after terminal 1 interrupt mutex1/49527f598bbfe01b 领导选举 # How to conduct leader election in etcd cluster # but i can't unserstand # tips: need think more 查看集群状态 # check cluster status ─ ~ ──────────────────────────────────────────────────────── INT ✘ ╰─ etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 172.16.242.3:2379 | 45d97e7a24a2c952 | 3.5.2 | 20 kB | false | false | 2 | 31 | 31 | | | 172.16.242.3:22379 | 1803770bd57fe7f5 | 3.5.2 | 20 kB | false | false | 2 | 31 | 31 | | | 172.16.242.3:32379 | 180470dceeda2be0 | 3.5.2 | 20 kB | true | false | 2 | 31 | 31 | | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ╭─ ~ ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ✔ ╰─ etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint health +--------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +--------------------+--------+-------------+-------+ | 172.16.242.3:2379 | true | 10.182483ms | | | 172.16.242.3:32379 | true | 11.098445ms | | | 172.16.242.3:22379 | true | 13.308361ms | | +--------------------+--------+-------------+-------+ 快照 # save the database # must set endpoint to one host ENDPOINTS=$HOST_1:2379 etcdctl --endpoints=$ENDPOINTS snapshot save my.db {"level":"info","ts":1646487341.2083359,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"my.db.part"} {"level":"info","ts":"2022-03-05T21:35:41.210+0800","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"} {"level":"info","ts":1646487341.211041,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"172.16.242.3:2379"} {"level":"info","ts":"2022-03-05T21:35:42.438+0800","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"} {"level":"info","ts":1646487342.465489,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"172.16.242.3:2379","size":"44 MB","took":1.257066939} {"level":"info","ts":1646487342.4674938,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"my.db"} Snapshot saved at my.db etcdctl --write-out=table --endpoints=$ENDPOINTS snapshot status my.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 79ec4d35 | 18024 | 36026 | 44 MB | +----------+----------+------------+------------+ # in this case, I found keys is too much and revision too high. Because I use `ectdctl --endpoints=$ENDPOINTS check perf`.Every perfomance check could raise too much keys and revision. 合并v2到v3 # how to migrate etcd from v2 to v3 # let it pass # write key in etcd version 2 store export ETCDCTL_API=2 etcdctl --endpoints=http://$ENDPOINT set foo bar # read key in etcd v2 etcdctl --endpoints=$ENDPOINTS --output="json" get foo # stop etcd node to migrate, one by one # migrate v2 data export ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal" # restart etcd node after migrate, one by one # confirm that the key got migrated etcdctl --endpoints=$ENDPOINTS get /foo 增减集群节点 # deal with membership https://etcd.io/docs/v3.5/tutorials/how-to-deal-with-membership/ 权限、用户与角色 # create role etcdctl --endpoints=${ENDPOINTS} role add root Role root created ## can be grant to every key,need use / to replace foo etcdctl --endpoints=${ENDPOINTS} role grant-permission root readwrite foo Role root updated etcdctl --endpoints=${ENDPOINTS} role get root Role root KV Read: foo KV Write: foo # create user and grant role ## set password is 123 etcdctl --endpoints=${ENDPOINTS} user add root Password of root: Type password of root again for confirmation: User root created etcdctl --endpoints=${ENDPOINTS} user grant-role root root Role root is granted to user root etcdctl --endpoints=${ENDPOINTS} user get root User: root Roles: root # enable auth etcdctl --endpoints=${ENDPOINTS} auth enable Authentication Enabled # now all client requests go through auth etcdctl --endpoints=${ENDPOINTS} --user=root:123 put foo bar OK etcdctl --endpoints=${ENDPOINTS} get foo {"level":"warn","ts":"2022-03-05T22:19:54.108+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-7b0e89ed-5f03-4f6d-b44c-ce59520a4c07/172.16.242.3:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"} Error: etcdserver: user name is empty etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo foo bar etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo1 tips: root 在内部可能有独立定义(或者uid很高),除非先关闭auth,否则无法直接删除该用户
...