安装一个etcd cluster

lx1036

2020年10月10日 18:47

背景

etcd是golang写的一个分布式key-value数据存储服务，用来存储配置数据。同时，还可以用来作为分布式锁、分布式队列、消息发布和订阅、服务注册和发现等等功能。

部署节点一般是奇数方便选主，节点之间数据通过raft协议保持强一致性。K8s使用etcd作为数据库，主要是kube-apiserver组件和etcd数据通信。

为了方便深入学习了解etcd，先本地部署一个带有证书认证的etcd cluster，方便调试开发。

步骤

(1) 下载etcd和证书工具cfssl

# etcdctl是etcd官方提供的CLI工具，方便与etcd交互
wget https://github.com/etcd-io/etcd/releases/download/v3.4.10/etcd-v3.4.10-darwin-amd64.zip
unzip etcd-v3.4.10-darwin-amd64.zip
mv etcd-v3.4.10-darwin-amd64/etcd /usr/local/bin/etcd
mv etcd-v3.4.10-darwin-amd64/etcdctl /usr/local/bin/etcdctl
# 或者
macos的为
brew install etcd

# 证书工具cfssl
curl -o cfssl https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64
curl -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64
chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/local/bin/
# 或者macos的为
brew install cfssl

# linux安装cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
复制代码

(2) 生成etcd服务端和客户端证书

执行根证书ca.sh脚本：


# Docs: https://kubernetes.io/zh/docs/tasks/tls/managing-tls-in-a-cluster/

# Certificate Authority
## cfssl: https://github.com/cloudflare/cfssl
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

####
# CN: Common Name
# https://blog.cloudflare.com/introducing-cfssl/
####
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
复制代码

执行服务端证书脚本 etcd-server.sh：

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "etcd",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert \
  -ca=./ca.pem \
  -ca-key=./ca-key.pem \
  -config=./ca-config.json \
  -hostname=127.0.0.1,kubernetes.default \
  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd # -> etcd-key.pem, etcd.pem
复制代码

执行客户端证书脚本 etcd-client.sh：

cat > etcd-client-csr.json <<EOF
{
  "CN": "etcd-client",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "etcd",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert \
  -ca=./ca.pem \
  -ca-key=./ca-key.pem \
  -config=./ca-config.json \
  -hostname=127.0.0.1,kubernetes.default \
  -profile=kubernetes \
  etcd-client-csr.json | cfssljson -bare etcd-client # -> etcd-client-key.pem, etcd-client.pem
复制代码

最后会生成根证书、etcd启动时需要的服务端证书和etcdctl交互时需要的客户端证书：

其中，客户端证书也可以直接复用服务端证书，所以可以不用再生成个客户端证书。为了更安全，这里还是生成个客户端证书。

(3)开启一个etcd cluster

在Makefile文件内写上：

PWD := $(shell pwd)

etcd1-auth:
	etcd \
      --name=infra1 \
      --client-cert-auth=true \
      --cert-file=$(PWD)/tls/etcd.pem \
      --key-file=$(PWD)/tls/etcd-key.pem \
      --peer-client-cert-auth=true \
      --peer-cert-file=$(PWD)/tls/etcd.pem \
      --peer-key-file=$(PWD)/tls/etcd-key.pem \
      --trusted-ca-file=$(PWD)/tls/ca.pem \
      --peer-trusted-ca-file=$(PWD)/tls/ca.pem \
      --initial-advertise-peer-urls=https://127.0.0.1:42380 \
      --listen-peer-urls=https://127.0.0.1:42380 \
      --listen-client-urls=https://127.0.0.1:42379 \
      --advertise-client-urls=https://127.0.0.1:42379 \
      --initial-cluster-token=etcd-cluster-0 \
      --initial-cluster 'infra1=https://127.0.0.1:42380,infra2=https://127.0.0.1:52380,infra3=https://127.0.0.1:62380' \
      --initial-cluster-state=new \
      --data-dir=$(PWD)/infra1.etcd \
      --logger=zap \
      --log-outputs=stderr

etcd2-auth:
	etcd \
      --name=infra2 \
      --client-cert-auth=true \
      --cert-file=$(PWD)/tls/etcd.pem \
      --key-file=$(PWD)/tls/etcd-key.pem \
      --peer-client-cert-auth=true \
      --peer-cert-file=$(PWD)/tls/etcd.pem \
      --peer-key-file=$(PWD)/tls/etcd-key.pem \
      --trusted-ca-file=$(PWD)/tls/ca.pem \
      --peer-trusted-ca-file=$(PWD)/tls/ca.pem \
      --initial-advertise-peer-urls=https://127.0.0.1:52380 \
      --listen-peer-urls=https://127.0.0.1:52380 \
      --listen-client-urls=https://127.0.0.1:52379 \
      --advertise-client-urls=https://127.0.0.1:52379 \
      --initial-cluster-token=etcd-cluster-0 \
      --initial-cluster 'infra1=https://127.0.0.1:42380,infra2=https://127.0.0.1:52380,infra3=https://127.0.0.1:62380' \
      --initial-cluster-state=new \
      --data-dir=$(PWD)/infra2.etcd \
      --logger=zap \
      --log-outputs=stderr

etcd3-auth:
	etcd \
      --name=infra3 \
      --client-cert-auth=true \
      --cert-file=$(PWD)/tls/etcd.pem \
      --key-file=$(PWD)/tls/etcd-key.pem \
      --peer-client-cert-auth=true \
      --peer-cert-file=$(PWD)/tls/etcd.pem \
      --peer-key-file=$(PWD)/tls/etcd-key.pem \
      --trusted-ca-file=$(PWD)/tls/ca.pem \
      --peer-trusted-ca-file=$(PWD)/tls/ca.pem \
      --initial-advertise-peer-urls=https://127.0.0.1:62380 \
      --listen-peer-urls=https://127.0.0.1:62380 \
      --listen-client-urls=https://127.0.0.1:62379 \
      --advertise-client-urls=https://127.0.0.1:62379 \
      --initial-cluster-token=etcd-cluster-0 \
      --initial-cluster 'infra1=https://127.0.0.1:42380,infra2=https://127.0.0.1:52380,infra3=https://127.0.0.1:62380' \
      --initial-cluster-state=new \
      --data-dir=$(PWD)/infra3.etcd \
      --logger=zap \
      --log-outputs=stderr


list:
	ETCDCTL_API=3 etcdctl member list \
		--write-out=table \
		--endpoints=https://127.0.0.1:42379 \
		--cacert $(PWD)/tls/ca.pem \
		--cert $(PWD)/tls/etcd-client.pem \
		--key $(PWD)/tls/etcd-client-key.pem
复制代码