etcd在线更新
在线恢复的前提是etcd正常工作（少部分节点丢失仍可以正常工作），etcd的ca.crt和ca.key都存在。etcd开启了https，kube-apiserver连接上来需要进行server authentication和client authentication。server authentication是指etcd表明自己的身份，client可以信任server；client authentication是指client可以访问本server。如果ca.crt或者ca.key丢失，则无法进行在线迁移。因此在部署完毕后，一定要保存ca.key和ca.crt。同样的，k8s的ca.crt和ca.key要需要保存。
查询K8s证书的有效期
for i in `ls /etc/kubernetes/pki/*.crt`;
do
echo -------------$i---------------
openssl x509 -in $i -noout -dates
echo
done
在线迁移的具体步骤如下：
生成证书
使用etcd的ca.crt和ca.key生产server.key, server.crt, peer.crt, peer.key, healthcheck-client.key, healthcheck-client.crt。假设需要恢复的主机的IP和HostName为：
export ETCD_IP=10.222.33.44
export ETCD_HOSTNAME=yz-222-33-44.h.chinabank.com.cn
生成server.key, server.crt, peer.crt, peer.key的命令为：
cat <<EOF > csr.conf
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = ${ETCD_HOSTNAME}
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = ${ETCD_HOSTNAME}
DNS.2 = localhost
IP.1 = 127.0.0.1
IP.2 = ${ETCD_IP}
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
EOF
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -config csr.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 10000 -extensions v3_ext -extfile csr.conf
openssl genrsa -out peer.key 2048
openssl req -new -key peer.key -out peer.csr -config csr.conf
openssl x509 -req -in peer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out peer.crt -days 10000 -extensions v3_ext -extfile csr.conf
生成healthcheck-client.key， healthcheck-client.crt的命令为：
cat <<EOF > healthcheck-client-csr.conf
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = ${ETCD_HOSTNAME}
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = ${ETCD_HOSTNAME}
DNS.2 = localhost
IP.1 = 127.0.0.1
IP.2 = ${ETCD_IP}
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=clientAuth
subjectAltName=@alt_names
EOF
openssl genrsa -out healthcheck-client.key 2048
openssl req -new -key healthcheck-client.key -out healthcheck-client.csr -config healthcheck-client-csr.conf
openssl x509 -req -in healthcheck-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out healthcheck-client.crt -days 10000 -extensions v3_ext -extfile healthcheck-client-csr.conf
然后替换相应的server.key, server.crt, peer.crt, peer.key, healthcheck-client.key, healthcheck-client.crt即可。
apiserver更新
cat <<EOF > apiserver-csr.conf
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = CN
ST = BeiJing
L = BeiJing
O = k8s
OU = System
CN = kubernetes
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = localhost
DNS.2 = TX-220-54-4.h.chinabank.com.cn
DNS.3 = TX-220-54-7.h.chinabank.com.cn
DNS.4 = TX-220-54-9.h.chinabank.com.cn
DNS.5 = TX-220-48-92.h.chinabank.com.cn
DNS.6 = kubernetes
DNS.7 = kubernetes.default
DNS.8 = kubernetes.default.svc
DNS.9 = kubernetes.default.svc.cluster
DNS.10 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 10.220.54.4
IP.3 = 10.220.54.7
IP.4 = 10.220.54.9
IP.5 = 10.220.48.92
IP.6 = 10.96.0.1
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
EOF
openssl genrsa -out apiserver.key 2048
openssl req -new -key apiserver.key -out apiserver.csr -config apiserver-csr.conf
openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out apiserver.crt -days 10000 -extensions v3_ext -extfile apiserver-csr.conf
openssl genrsa -out apiserver-kubelet-client.key 2048
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -config apiserver-csr.conf
openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out apiserver-kubelet-client.crt -days 10000 -extensions v3_ext -extfile apiserver-csr.conf
openssl genrsa -out front-proxy-client.key 2048
openssl req -new -key front-proxy-client.key -out front-proxy-client.csr -config apiserver-csr.conf
openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key -CAcreateserial -out front-proxy-client.crt -days 10000 -extensions v3_ext -extfile apiserver-csr.conf
kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--embed-certs=true \
--client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--kubeconfig=scheduler.conf
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--embed-certs=true \
--client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--kubeconfig=controller-manager.conf
证书签名与kubernetes的RBAC角色绑定的关系。
由于后续 kube-apiserver 在启用RBAC模式之后， 客户端(如 kubelet、kube-proxy、Pod)请求进行授权的时候会需要认证用户名、以及用户组； CN=admin/O=system:masters/OU=System就是在CN定义用户为admin，O定义用户组为system:masters，OU 指定该证书的 Group 为 system:masters。
kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings（角色），如 cluster-admin （角色）将 Group（组） system:masters与 Role（角色） cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限； 那么当然的，我们创建admin的证书的时候，就要按照该上面的说明定义好证书的组、用户。
kubectl create clusterrolebinding system:kubernetes --clusterrole=cluster-admin --user=system:kubernetes
X509: What's the difference between digital signature and non-repudiation
I realise this question is a bit old, but I think I can shed some much-needed light on the question.The non-repudiation value in the keyUsage attribute relates to the whole certificate, not any purpose in particular. The presence of the non-repudiation flag indicates that the private key has sufficient protections in place that the entity named in the certificate cannot later repudiate—deny—actions they take with the certificate. The presence of the flag doesn't prevent repudiation, rather it indicates that repudiation isn't likely to survive reasonable scrutiny.So in this specific case, the CA is giving the user the option of a certificate that does or does not include the non-repudiation element. If you want to assert to those verifying the signature that you can't easily deny it was you who signed it (the USB token is the key enabler here), use the non-repudiation certificate. Otherwise, use the certificate marked for digital signatures. (Depending on the other attributes in the certificate, you may or may not be able to sign documents with either or both certificates.)
