Kubernetes-cockroachdb-nfs单机版部署
背景
简单的在kubernetes上基于nfs动态存储部署cockroachdb
一、环境准备，安装kubernetes集群
当前节点base、master、node1、node2、node3
kubernetes安装不在此赘述，请参考参考资料进行安装
kubernetes安装nfs不在此赘述，请参考参考资料进行安装
参考资料
/kubernetes/1-Kubernetes基于Centos7构建基础环境（一）
/kubernetes/2-Kubernetes基于Centos7构建基础环境（二）
/kubernetes/3-Kubernetes基于Centos7构建基础环境（三）
4-Kubernetes-基于Centos7安装面板及监控（四）
/kubernetes/nfs/1-kubernetes-nfs动态存储部署
| 集群名称 | 集群域名 | 说明 |
|---|---|---|
| base | base.xincan.cn | 部署harbor、nfs等服务 |
| master | master.xincan.cn | kubernetes主节点，做污点容忍，排除业务资源，nfs客户端等 |
| node1 | node1.xincan.cn | kubernetes从节点，nfs客户端等 |
| node2 | node2.xincan.cn | kubernetes从节点，nfs客户端等 |
| node3 | node3.xincan.cn | kubernetes从节点，nfs客户端等 |
二、总体流程：
- 当前cockroachdb的版本为v21.1.4
- cockroachdb/cockroach:v21.1.4
- cockroachdb/cockroach-k8s-request-cert:0.4
- 在Kubernetes集群的主节点上创建cockroachdb文件夹，用于存放下载资源，和Kuberntes文件编排，文件目录如下；
- git文件夹下的文件是参考文件
[root@master cockroachdb]# tree
├── 1-cockroachdb-namespace.yaml
├── 2-cockroachdb-sa.yaml
├── 3-cockroachdb-provisioner.yaml
├── 4-cockroachdb-storage.yaml
├── 5-cockroachdb-service.yaml
├── 6-cockroachdb-poddisruptionbudget.yaml
├── 7-sentinel-statefuset.yaml
├── client
├── 1-cluster-secure.yaml
├── init
├── 1-cluster-init-secure.yaml
└── git
└── cockroach
└── cloud
└── kubernetes
├── client-secure.yaml
├── cluster-init-secure.yaml
├── cluster-init.yaml
├── cockroachdb-statefulset-secure.yaml
├── cockroachdb-statefulset.yaml
├── demo.sh
├── example-app-secure.yaml
├── example-app.yaml
├── fluentd-configmap.yml
└── README.md
- 在cockroachdb文件夹下创建git，然后进入git，通过git获取cockroachdb应用；
- 镜像制作；
- 资源创建；
- 效果展示；
三、下载cockroachdb
- git下载cockroach
[root@master git]# https://hub.fastgit.org/cockroachdb/cockroach.git
- 下载镜像
- 在base机器上下载镜像
- 下载镜像：docker pull cockroachdb/cockroach:v21.1.4
- 下载镜像：docker pull cockroachdb/cockroach-k8s-request-cert:0.4
- 将镜像打包到Harbor私服
[root@base ~]# docker pull cockroachdb/cockroach:v21.1.4
[root@base ~]# docker tag cockroachdb/cockroach:v21.1.4 base.hatech.com.cn/cockroachdb/cockroach:v21.1.4
[root@base ~]# docker push base.hatech.com.cn/cockroachdb/cockroach:v21.1.4
[root@base ~]# docker pull cockroachdb/cockroach-k8s-request-cert:0.4
[root@base ~]# docker tag cockroachdb/cockroach-k8s-request-cert:0.4 base.hatech.com.cn/cockroachdb/cockroach-k8s-request-cert:0.4
[root@base ~]# docker push base.hatech.com.cn/cockroachdb/cockroach-k8s-request-cert:0.4
四、资源创建
参照cockroachdb/git/cockroach/cloud/kubernetes下的编排文件，进行改动
- 创建cockroachdb的命名空间Namespace，名称为：cockroachdb
- 将所有的资源挂载到此命名空间下
[root@master cockroachdb]# vim 1-cockroachdb-namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cockroachdb
labels:
app: cockroachdb
- 创建cockroachdb权限配置ServiceAccount、ClusterRole、ClusterRoleBinding、Role、RoleBinding，名称为：cockroachdb
[root@master cockroachdb]# vim 2-cockroachdb-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- create
- get
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cockroachdb
subjects:
- kind: ServiceAccount
name: cockroachdb
namespace: cockroachdb
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cockroachdb
subjects:
- kind: ServiceAccount
name: cockroachdb
namespace: cockroachdb
- 创建NFS资源Deployment,名称为：cockroachdb-nfs-provisionerr
- mountPath: /persistentvolumes 该文件夹是nfs-client-provisioner镜像运行之后容器内部固定的文件夹它会mount到/hatech/nfs/data/xincan/cockroachdb nfs服务器cockroachdb 文件夹下
- value: 192.168.1.80 该地址是对应NFS服务器地址
- path: /hatech/nfs/data/xincan/cockroachdb path地址路径则是NFS服务器自己创建挂载点的文件路径，也就是后续数据库存储的位置
[root@master cockroachdb]# vim 3-cockroachdb-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cockroachdb-nfs-provisioner
labels:
app: cockroachdb-nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: cockroachdb
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: cockroachdb-nfs-provisioner
template:
metadata:
labels:
app: cockroachdb-nfs-provisioner
spec:
serviceAccountName: cockroachdb
containers:
- name: cockroachdb-nfs-provisioner
image: base.hatech.com.cn/library/nfs-client-provisioner:v1.5.2
volumeMounts:
- name: cockroachdb-nfs-client-root
# 该文件夹是nfs-client-provisioner运行之后容器内部固定的文件夹它会mount到/nfs/data/nginx nfs服务器nginx文件夹下
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.1.80
- name: NFS_PATH
value: /hatech/nfs/data/xincan/cockroachdb
volumes:
- name: cockroachdb-nfs-client-root
nfs:
server: 192.168.1.80
path: /hatech/nfs/data/xincan/cockroachdb
-
创建cockroachdb主节点StorageClass，名称为：cockroachdb-nfs-storage
- 后续pv、pvc通过Deployment动态创建
[root@master cockroachdb]# vim 4-cockroachdb-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cockroachdb-nfs-storage
namespace: cockroachdb
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
- 创建cockroachdb服务外暴露访问资源Service，名称为：cockroachdb-service，cockroachdb-public,cockroachdb
- mysql-service：对外暴露服务
[root@master cockroachdb]# vim 5-cockroachdb-service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: cockroachdb
name: cockroachdb-service
labels:
app: cockroachdb
spec:
type: NodePort
ports:
- port: 26257
targetPort: 26257
nodePort: 31181
name: grpc
- port: 8080
targetPort: 8080
nodePort: 31180
name: http
selector:
app: cockroachdb
---
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
namespace: cockroachdb
name: cockroachdb-public
labels:
app: cockroachdb
spec:
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: grpc
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: http
selector:
app: cockroachdb
---
apiVersion: v1
kind: Service
metadata:
# This service only exists to create DNS entries for each pod in the stateful
# set such that they can resolve each other's IP addresses. It does not
# create a load-balanced ClusterIP and should not be used directly by clients
# in most circumstances.
name: cockroachdb
namespace: cockroachdb
labels:
app: cockroachdb
annotations:
# Use this annotation in addition to the actual publishNotReadyAddresses
# field below because the annotation will stop being respected soon but the
# field is broken in some versions of Kubernetes:
# https://github.com/kubernetes/kubernetes/issues/58662
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
# Enable automatic monitoring of all instances when Prometheus is running in the cluster.
prometheus.io/scrape: "true"
prometheus.io/path: "_status/vars"
prometheus.io/port: "8080"
spec:
ports:
- port: 26257
targetPort: 26257
name: grpc
- port: 8080
targetPort: 8080
name: http
# We want all pods in the StatefulSet to have their addresses published for
# the sake of the other CockroachDB pods even before they're ready, since they
# have to be able to talk to each other in order to become ready.
publishNotReadyAddresses: true
clusterIP: None
selector:
app: cockroachdb
- 创建cockroachdb主节点PodDisruptionBudget，名称为：cockroachdb-budget
[root@master cockroachdb]# vim 6-cockroachdb-poddisruptionbudget.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: cockroachdb-budget
namespace: cockroachdb
labels:
app: cockroachdb
spec:
selector:
matchLabels:
app: cockroachdb
maxUnavailable: 1
- 创建cockroachdb无头服务StatefulSet，名称为：根据metadata.name+“-”+实例序号，序号从0开始,如我的最后结果是：cockroachdb-0，cockroachdb-1、cockroachdb-2
- replicas: 3 最终会生成3分cockroachdb实例
[root@master cockroachdb]# vim 7-cockroachdb-statefuset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cockroachdb
namespace: cockroachdb
spec:
serviceName: "cockroachdb"
replicas: 3
selector:
matchLabels:
app: cockroachdb
template:
metadata:
labels:
app: cockroachdb
spec:
serviceAccountName: cockroachdb
# Init containers are run only once in the lifetime of a pod, before
# it's started up for the first time. It has to exit successfully
# before the pod's main containers are allowed to start.
initContainers:
# The init-certs container sends a certificate signing request to the
# kubernetes cluster.
# You can see pending requests using: kubectl get csr
# CSRs can be approved using: kubectl certificate approve <csr name>
#
# All addresses used to contact a node must be specified in the --addresses arg.
#
# In addition to the node certificate and key, the init-certs entrypoint will symlink
# the cluster CA to the certs directory.
- name: init-certs
image: base.hatech.com.cn/cockroachdb/cockroach-k8s-request-cert:0.4
imagePullPolicy: IfNotPresent
command:
- "/bin/ash"
- "-ecx"
- "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=node -addresses=localhost,127.0.0.1,$(hostname -f),$(hostname -f|cut -f 1-2 -d '.'),cockroachdb-public,cockroachdb-public.$(hostname -f|cut -f 3- -d '.'),cockroachdb-public.$(hostname -f|cut -f 3-4 -d '.'),cockroachdb-public.$(hostname -f|cut -f 3 -d '.') -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: certs
mountPath: /cockroach-certs
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cockroachdb
topologyKey: kubernetes.io/hostname
containers:
- name: cockroachdb
image: base.hatech.com.cn/cockroachdb/cockroach:v21.1.4
imagePullPolicy: IfNotPresent
# TODO: Change these to appropriate values for the hardware that you're running. You can see
# the resources that can be allocated on each of your Kubernetes nodes by running:
# kubectl describe nodes
# Note that requests and limits should have identical values.
resources:
requests:
cpu: "2"
memory: "8Gi"
limits:
cpu: "2"
memory: "8Gi"
ports:
- containerPort: 26257
name: grpc
- containerPort: 8080
name: http
# We recommend that you do not configure a liveness probe on a production environment, as this can impact the availability of production databases.
# livenessProbe:
# httpGet:
# path: "/health"
# port: http
# scheme: HTTPS
# initialDelaySeconds: 30
# periodSeconds: 5
readinessProbe:
httpGet:
path: "/health?ready=1"
port: http
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2
volumeMounts:
- name: datadir
mountPath: /cockroach/cockroach-data
- name: certs
mountPath: /cockroach/cockroach-certs
env:
- name: COCKROACH_CHANNEL
value: kubernetes-secure
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
divisor: "1"
- name: MEMORY_LIMIT_MIB
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: "1Mi"
command:
- "/bin/bash"
- "-ecx"
# The use of qualified `hostname -f` is crucial:
# Other nodes aren't able to look up the unqualified hostname.
# Memory caches are set as a fraction of the pod's memory limit.
- exec
/cockroach/cockroach
start
--logtostderr
--certs-dir /cockroach/cockroach-certs
--advertise-host $(hostname -f)
--http-addr 0.0.0.0
--join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb
--cache $(expr $MEMORY_LIMIT_MIB / 4)MiB
--max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
# No pre-stop hook is required, a SIGTERM plus some time is all that's
# needed for graceful shutdown of a node.
terminationGracePeriodSeconds: 60
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
- name: certs
emptyDir: {}
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
storageClassName: cockroachdb-nfs-storage
resources:
requests:
storage: 10Gi
- 创建cockroachdb资源
[root@master cockroachdb]# kubectl apply -f ../cockroachdb
- 查看部署的Pod
- 此时一直是Init状态，initController等待批准授权
[root@master cockroachdb]# kubectl -n cockroachdb get pod
NAME READY STATUS RESTARTS AGE
cockroachdb-0 0/1 Init:0/1 0 55s
cockroachdb-1 0/1 Init:0/1 0 55s
cockroachdb-2 0/1 Init:0/1 0 55s
cockroachdb-nfs-provisioner-5767c7c4f5-rnsnv 1/1 Running 0 56s
五、 手动批准授权
- 查看CSR
- 查看certificatesigningrequests.certificates.k8s.io
- 可以简写成csr
[root@master cockroachdb]# kubectl -n cockroachdb get certificatesigningrequests.certificates.k8s.io
NAME AGE SIGNERNAME REQUESTOR CONDITION
cockroachdb.node.cockroachdb-0 57s kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Pending
cockroachdb.node.cockroachdb-1 55s kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Pending
cockroachdb.node.cockroachdb-2 57s kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Pending
- init-certs容器发送CSR批准使用
[root@master cockroachdb]# kubectl -n cockroachdb certificate approve cockroachdb.node.cockroachdb-{0,1,2}
certificatesigningrequest.certificates.k8s.io/cockroachdb.node.cockroachdb-0 approved
certificatesigningrequest.certificates.k8s.io/cockroachdb.node.cockroachdb-1 approved
certificatesigningrequest.certificates.k8s.io/cockroachdb.node.cockroachdb-2 approved
- 查看Pod为Running状态，但还没有达到Ready
[root@master cockroachdb]# kubectl -n cockroachdb get pod
NAME READY STATUS RESTARTS AGE
cockroachdb-0 0/1 Running 0 6m21s
cockroachdb-1 0/1 Running 0 6m21s
cockroachdb-2 0/1 Running 0 6m21s
cockroachdb-nfs-provisioner-5767c7c4f5-rnsnv 1/1 Running 0 6m22s
六、初始化集群
- 在init文件夹下创建初始化Job
- 创建cockroachdb的Job，名称为：cluster-init-secure
[root@master init]# vim 1-cluster-init-secure.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: cluster-init-secure
namespace: cockroachdb
labels:
app: cockroachdb
spec:
template:
spec:
serviceAccountName: cockroachdb
initContainers:
# The init-certs container sends a certificate signing request to the
# kubernetes cluster.
# You can see pending requests using: kubectl get csr
# CSRs can be approved using: kubectl certificate approve <csr name>
#
# In addition to the client certificate and key, the init-certs entrypoint will symlink
# the cluster CA to the certs directory.
- name: init-certs
image: base.hatech.com.cn/cockroachdb/cockroach-k8s-request-cert:0.4
imagePullPolicy: IfNotPresent
command:
- "/bin/ash"
- "-ecx"
- "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs
containers:
- name: cluster-init
image: base.hatech.com.cn/cockroachdb/cockroach:v21.1.4
imagePullPolicy: IfNotPresent
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs
command:
- "/cockroach/cockroach"
- "init"
- "--certs-dir=/cockroach-certs"
- "--host=cockroachdb-0.cockroachdb"
restartPolicy: OnFailure
volumes:
- name: client-t-certs
emptyDir: {}
- 执行创建
- 查询Pod处在Init状态，应为需要手动批准授权
[root@master init]# kubectl apply -f 1-cluster-init-secure.yaml
job.batch/cluster-init-secure created
[root@master init]# kubectl -n cockroachdb get pod
NAME READY STATUS RESTARTS AGE
cluster-init-secure-2zbvl 0/1 Init:0/1 0 12s
cockroachdb-0 0/1 Running 0 14m
cockroachdb-1 0/1 Running 0 14m
cockroachdb-2 0/1 Running 0 14m
cockroachdb-nfs-provisioner-5767c7c4f5-rnsnv 1/1 Running 0 14m
- 查看CSR并对其批准授权
- 查看cockroachdb.client.root，处在Pending状态
- 对其进行批准授权
[root@master init]# kubectl -n cockroachdb get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
cockroachdb.client.root 112s kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Pending
cockroachdb.node.cockroachdb-0 15m kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Approved,Issued
cockroachdb.node.cockroachdb-1 15m kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Approved,Issued
cockroachdb.node.cockroachdb-2 15m kubernetes.io/legacy-unknown system:serviceaccount:cockroachdb:cockroachdb Approved,Issued
[root@master init]# kubectl -n cockroachdb certificate approve cockroachdb.client.root
certificatesigningrequest.certificates.k8s.io/cockroachdb.client.root approved
- 在次查看Pod
- 初始化集群Job执行完成，状态为Completed
- 发现cockroachdb集群Pod正常运行，状态为Running
- 至此cockroachdb安装完成
[root@master init]# kubectl -n cockroachdb get pod
NAME READY STATUS RESTARTS AGE
cluster-init-secure-2zbvl 0/1 Completed 0 3m11s
cockroachdb-0 1/1 Running 0 17m
cockroachdb-1 1/1 Running 0 17m
cockroachdb-2 1/1 Running 0 17m
cockroachdb-nfs-provisioner-5767c7c4f5-rnsnv 1/1 Running 0 17m
七、命令方式创建客户端链接
- 进入client文件夹下，创建1-client-secure.yaml资源
- 创建cockroachdb的Pod，名称为：cockroachdb-client-secure
[root@master client]# cat 1-client-secure.yaml
apiVersion: v1
kind: Pod
metadata:
name: cockroachdb-client-secure
namespace: cockroachdb
labels:
app: cockroachdb-client
spec:
serviceAccountName: cockroachdb
initContainers:
# The init-certs container sends a certificate signing request to the
# kubernetes cluster.
# You can see pending requests using: kubectl get csr
# CSRs can be approved using: kubectl certificate approve <csr name>
#
# In addition to the client certificate and key, the init-certs entrypoint will symlink
# the cluster CA to the certs directory.
- name: init-certs
image: base.hatech.com.cn/cockroachdb/cockroach-k8s-request-cert:0.4
imagePullPolicy: IfNotPresent
command:
- "/bin/ash"
- "-ecx"
- "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs
containers:
- name: cockroachdb-client
image: base.hatech.com.cn/cockroachdb/cockroach:v21.1.4
imagePullPolicy: IfNotPresent
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs
# Keep a pod open indefinitely so kubectl exec can be used to get a shell to it
# and run cockroach client commands, such as cockroach sql, cockroach node status, etc.
command:
- sleep
- "2147483648" # 2^31
# This pod isn't doing anything important, so don't bother waiting to terminate it.
terminationGracePeriodSeconds: 0
volumes:
- name: client-certs
emptyDir: {}
- 创建资源，并查看创建情况
- 命令方式客户端cockroachdb-client-secure创建成功
[root@master client]# kubectl apply -f 1-client-secure.yaml
pod/cockroachdb-client-secure created
[root@master client]# kubectl -n cockroachdb get pod
NAME READY STATUS RESTARTS AGE
cluster-init-secure-2zbvl 0/1 Completed 0 8m34s
cockroachdb-0 1/1 Running 0 22m
cockroachdb-1 1/1 Running 0 22m
cockroachdb-2 1/1 Running 0 22m
cockroachdb-client-secure 1/1 Running 0 6s
cockroachdb-nfs-provisioner-5767c7c4f5-rnsnv 1/1 Running 0 22m
八、设置用户登录信息
- 通过命令行客户端cockroachdb-client-secure的Pod，进入其容器，连上cockroachdb
[root@master client]# kubectl exec -n cockroachdb -it cockroachdb-client-secure -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
Defaulted container "cockroachdb-client" out of: cockroachdb-client, init-certs (init)
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.1.4 (x86_64-unknown-linux-gnu, built 2021/06/29 19:06:32, go1.15.11) (same version as client)
# Cluster ID: 6f8c7e65-653e-4fcb-b8c7-0270ff09200b
#
# Enter \? for a brief introduction.
#
root@cockroachdb-public:26257/defaultdb>
- 创建用户名密码
- 进入cockroachdb-client-secure的额Pod链接数据库
- 用户名称：xincan
- 用户密码：xincan-0818
[root@master client]# kubectl exec -n cockroachdb -it cockroachdb-client-secure -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
Defaulted container "cockroachdb-client" out of: cockroachdb-client, init-certs (init)
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.1.4 (x86_64-unknown-linux-gnu, built 2021/06/29 19:06:32, go1.15.11) (same version as client)
# Cluster ID: 6f8c7e65-653e-4fcb-b8c7-0270ff09200b
#
# Enter \? for a brief introduction.
#
root@cockroachdb-public:26257/defaultdb> CREATE USER xincan WITH PASSWORD 'xincan-0818';
CREATE ROLE
Time: 150ms total (execution 149ms / network 1ms)
root@cockroachdb-public:26257/defaultdb>
- 创建数据库，并将数据库挂载到用户下
root@cockroachdb-public:26257/defaultdb> CREATE DATABASE xincan;
CREATE DATABASE
Time: 56ms total (execution 55ms / network 1ms)
root@cockroachdb-public:26257/defaultdb> GRANT ALL ON DATABASE xincan TO xincan;
GRANT
Time: 279ms total (execution 73ms / network 206ms)
root@cockroachdb-public:26257/defaultdb>
九、界面访问
- 查看Service暴露端口信息
- 找到NodePort对应的SVC
- 根据IP地址加端口（31190）号进行登录
[root@master cockroachdb]# kubectl -n cockroachdb get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 14h
cockroachdb-public ClusterIP 10.101.239.130 <none> 26257/TCP,8080/TCP 14h
cockroachdb-service NodePort 10.106.168.34 <none> 26257:31191/TCP,8080:31190/TCP 14h
- 如图
- 进入登录页面
- 输入用户名：xincan， 密码：xincan-0818
十、密钥创建
- 通过客户端Pod进入容器
[root@master ~]# kubectl exec -n cockroachdb -it cockroachdb-client-secure -- /bin/sh
Defaulted container "cockroachdb-client" out of: cockroachdb-client, init-certs (init)
sh-4.4#
- 创建证书存放文件夹
certs：您将在此目录中生成 CA 证书以及所有节点和客户端证书和密钥，然后将一些文件上传到您的节点。xincan-certs：您将在此目录中生成 CA 密钥，然后在生成节点和客户端证书时引用该密钥。之后，您将妥善保管密钥；您不会将其上传到您的节点。
sh-4.4# mkdir certs xincan-certs
sh-4.4# ls
bin boot certs cockroach cockroach-certs dev etc home lib lib64 licenses lost+found media mnt opt proc root run sbin srv sys tmp usr var xincan-certs
- 生成 CA 证书和密钥
sh-4.4# cockroach cert create-ca --certs-dir=certs --ca-key=xincan-certs/ca.key
- 证书目录查看
sh-4.4# ls -l certs
total 4
-rw-r--r-- 1 root root 1151 Jul 9 05:47 ca.crt
sh-4.4# ls -l xincan-certs/
total 4
-rw------- 1 root root 1675 Jul 9 05:47 ca.key
sh-4.4#
- 为客户端创建证书密钥对
- 生成证书注意
sh-4.4# cockroach cert create-client xincan --certs-dir=certs --ca-key=xincan-certs/ca.key --also-generate-pkcs8-key
sh-4.4# ls -l certs/
total 12
-rw-r--r-- 1 root root 1151 Jul 9 05:47 ca.crt
-rw-r--r-- 1 root root 1147 Jul 9 06:03 client.xincan.crt
-rw------- 1 root root 1679 Jul 9 06:03 client.xincan.key
-rw------- 1 root root 1219 Jul 9 06:03 client.xincan.key.pk8
sh-4.4#
十一、SpringBoot程序对接
- pom.xml增加postgresql工具依赖
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
</dependency>
- yml配置链接
- 不安全链接
- 安全链接：增加 &sslrootcert=/certs/ca.crt&sslkey=/certs/client.xincan.key.pk8&sslcert=/certs/client.xincan.crt
spring:
application:
name: xincan-cockroachdb
profiles:
active: dev
cloud:
nacos:
server-addr: ${address.nacos}
config:
server-addr: ${spring.cloud.nacos.server-addr}
namespace: xincan-${spring.profiles.active}-0001
group: ${spring.profiles.active}_group
file-extension: yaml
datasource:
hikari:
driver-class-name: org.postgresql.Driver
jdbc-url: jdbc:postgresql://192.168.1.81:31191/xincan?useSSL=true&sslmode=require&sslrootcert=certs/ca.crt&sslkey=certs/client.xincan.key.pk8&sslcert=certs/client.xincan.crt
username: xincan
password: xincan-0818
connection-timeout: 30000
minimum-idle: 5
maximum-pool-size: 10
idle-timeout: 30000
max-lifetime: 30000
auto-commit: true
connection-test-query: SELECT 1
- 应用程序打包成jar之后，需要将certs文件夹copy到jar包相同目录下，不然程序找不到认证文件！！！
十二：结束语
至此集群版Kubernetes部署cockroachdb完成
