10-Kubernetes-elasticsearch-nfs集群部署

百里长风
503 阅读7分钟

Kubernetes-elasticsearch-nfs集群部署

背景

根据网络上的文章进行部署elasticsearch的时候发现了很多问题,现整理部署步骤,供大家学习

一、环境准备，安装kubernetes集群

​ 当前节点base、master、node1、node2、node3

​ kubernetes安装不在此赘述，请参考参考资料进行安装

​ kubernetes安装nfs不在此赘述，请参考参考资料进行安装

​ 参考资料

​ /kubernetes/1-Kubernetes基于Centos7构建基础环境（一）

​ /kubernetes/2-Kubernetes基于Centos7构建基础环境（二）

​ /kubernetes/3-Kubernetes基于Centos7构建基础环境（三）

​ 4-Kubernetes-基于Centos7安装面板及监控（四）

​ /kubernetes/nfs/1-kubernetes-nfs动态存储部署

集群名称集群域名说明
basebase.xincan.cn部署harbor、nfs等服务
mastermaster.xincan.cnkubernetes主节点，做污点容忍，排除业务资源，nfs客户端等
node1node1.xincan.cnkubernetes从节点，nfs客户端等
node2node2.xincan.cnkubernetes从节点，nfs客户端等
node3node3.xincan.cnkubernetes从节点，nfs客户端等

二、总体流程

  1. 下载elasticsearch的版本为7.4.2
  2. 原始镜像修改（另行创建文件夹处理镜像）,文件目录如下
[root@master /]# tree

└── elasticsearch-7  
   ├── Dockerfile
   └── run.sh
  1. 在Kubernetes集群的主节点上创建elasticsearch文件夹，用于存放Kuberntes文件编排，文件目录如下；
[root@master efk]# tree
.
├── 0-efk-namespace.yaml
├── 1-filebeat-kibana-rbac.yaml
├── elasticsearch
│   ├── 1-elasticsearch-rbac.yaml
│   ├── master
│   │   ├── 1-elasticsearch-master-service.yaml
│   │   └── 2-elasticsearch-master-statefulset.yaml
│   └── node
│       ├── 1-elasticsearch-node-storageclass.yaml
│       ├── 2-elasticsearch-node-nfs-provisioner.yaml
│       ├── 3-elasticsearch-node-service.yaml
│       └── 4-elasticsearch-node-statefulset.yaml
├── filebeat
│   ├── 1-filebeat-configmap.yaml
│   └── 2-filebeat-daemonset.yaml
└── kibana
    ├── 1-kibana-configmap.yaml
    ├── 2-kibana-deployment.yaml
    └── 3-kibana-service.yaml
  1. 镜像制作；
  2. 资源创建；
  3. 效果展示；

三、镜像处理

  1. 下载镜像
[root@master /]# docker pull elasticsearch:7.4.2
  1. 镜像tag
[root@master /]# docker tag elasticsearch:7.4.2 base.xincan.cn/library/elasticsearch:7.4.2
  1. 镜像提交到Harbor
[root@master /]# docker push base.xincan.cn/library/elasticsearch:7.4.2

四、镜像改造

  1. 为什么要修改elasticsearch镜像？

    • 因为在部署elasticsearch的时候,建议配置memlock:true,这个要求系统必须配置ulimit.经测试kubernetes并不支持配置系统的ulimit,通过 podStart 和 initContainer 都是无法生效。
    • 所以需要修改镜像,然其在容器内自动执行. 操作步骤如下.最终生成一个新的镜base.xincan.cn/library/elasticsearch:7.4.2-ulimit
    • 将base.xincan.cn/library/elasticsearch:7.4.2-ulimit 提交到Harbor，后续用此镜像

  2. 创建Dockerfile

[root@storage elasticsearch-7]# cat Dockerfile

FROM elasticsearch:7.4.2
MAINTAINER jiangxincan alittlexincan@163.com
COPY run.sh /
RUN chmod 775 /run.sh
CMD ["/run.sh"]

[root@storage elasticsearch-7]#
  1. 创建启动脚本内容
[root@storage elasticsearch-7]# cat run.sh


#!/bin/bash

# 设置memlock无限制
ulimit -l unlimited
exec su elasticsearch /usr/local/bin/docker-entrypoint.sh

[root@storage elasticsearch-7]#
  1. 构建镜像
[root@storage elasticsearch-7]# docker build --tag base.xincan.cn/library/elasticsearch:7.4.2-ulimit
  1. 推送镜像
[root@storage elasticsearch-7]# docker push base.xincan.cn/library/elasticsearch:7.4.2-ulimit

五，公共资源创建

  1. 创建efk的命名空间Namespace，名称为：efk
    • 将所有的资源挂载到此命名空间下
[root@master efk]# cat 0-efk-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/product: xincan
    
[root@master efk]#
  1. 创建filebeat、kibana权限配置ServiceAccount、ClusterRole、ClusterRoleBinding
[root@master efk]# cat 1-filebeat-kibana-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fk-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan

---


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fk-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
subjects:
- kind: ServiceAccount
  name: fk-rbac
  namespace: efk
roleRef:
  kind: ClusterRole
  name: fk-rbac
  apiGroup: rbac.authorization.k8s.io

---


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fk-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list

[root@master efk]#

六、创建Elasticsearch访问权限

[root@master elasticsearch]# cat 1-elasticsearch-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
---


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elasticsearch-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","watch","list", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]

---


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elasticsearch-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
subjects:
  - kind: ServiceAccount
    name: elasticsearch-rbac
    namespace: efk
roleRef:
  kind: ClusterRole
  name: elasticsearch-rbac
  apiGroup: rbac.authorization.k8s.io

---


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elasticsearch-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---


apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: elasticsearch-rbac
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
subjects:
  - kind: ServiceAccount
    name: elasticsearch-rbac
    namespace: efk
roleRef:
  kind: Role
  name: elasticsearch-rbac
  apiGroup: rbac.authorization.k8s.io
  
[root@master elasticsearch]#

七、创建elasticsearch主节点资源

  1. 创建elasticsearch服务访问资源Service，名称为：elasticsearch-master-service
[root@master master]# cat 1-elasticsearch-master-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master-service
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/role: master
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: elasticsearch

spec:
  selector:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/role: master
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: elasticsearch
  ports:
  - name: data
    protocol: TCP
    port: 9300
    targetPort: 9300
[root@master master]#
  1. 创建elasticsearch-master主节点无头服务StatefulSet，名称为：根据metadata.name+“-”+实例序号，序号从0开始,如我的最后结果是：elasticsearch-master-0，elasticsearch-master-1、elasticsearch-master-2
    • replicas: 3 最终会生成3分elasticsearch实例
[root@master master]# cat 2-elasticsearch-master-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/role: master
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: elasticsearch
spec:
  replicas: 3
  serviceName: elasticsearch-master-service
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/role: master
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/app: elasticsearch
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/type: plugins
        xincan.kubernetes.io/role: master
        xincan.kubernetes.io/product: xincan
        xincan.kubernetes.io/app: elasticsearch
    spec:
      serviceAccountName: elasticsearch
      restartPolicy: Always
      containers:
        - name: elasticsearch-master
          image: b.cn/library/elasticsearch:7.4.2-ulimit
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true  #获取root权限,这样才能进行初始化命令执行.
          lifecycle:
            postStart: #初始化命令,配置系统参数
              exec:
                command:
                - /bin/bash
                - -c
                - sysctl -w vm.max_map_count=262144; ulimit -l unlimited;
          ports:  #开放端口一个是集群端口,一个是数据端口
            - name: cluster
              containerPort: 9200
              protocol: TCP
            - name: data
              containerPort: 9300
              protocol: TCP
          env: #环境变量,非容器下在配置文件配置的,这里对应配置为环境变量就可以了
            - name: cluster.name
              value: "es_cluster"
            - name: bootstrap.memory_lock
              value: "true"
            - name: node.master
              value: "true"
            - name: node.data
              value: "false"
            - name: discovery.seed_hosts
              value: "elasticsearch-master-service"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
            - name: node.ingest
              value: "false"
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"
     # tolerations: #使其可以运行在k8s主节点上
     # - effect: NoSchedule
     #   key: node-role.kubernetes.io/master
[root@master master]#

八、创建elasticsearch子节点资源

  1. 创建elasticsearch子节点StorageClass，名称为：efk-nfs-storage
    • 后续pv、pvc通过Deployment动态创建
[root@master node]# cat 1-elasticsearch-node-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efk-nfs-storage
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
provisioner: nfs.xincan.efk/elasticsearch # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
  
[root@master node]#
  1. 创建NFS资源Deployment,名称为：elasticsearch-nfs-client-provisionerr
    • mountPath: /persistentvolumes 该文件夹是nfs-client-provisioner镜像运行之后容器内部固定的文件夹它会mount到/hatech/nfs/data/xincan/elasticsearch-multiple nfs服务器elasticsearch-multiple 文件夹下
    • value: 192.168.1.80 该地址是对应NFS服务器地址
    • path: /hatech/nfs/data/xincan/elasticsearch-multiple path地址路径则是NFS服务器自己创建挂载点的文件路径，也就是后续数据库存储的位置
[root@master node]# cat 2-elasticsearch-node-nfs-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch-node-nfs-client-provisioner
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/product: xincan
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/type: plugins
        xincan.kubernetes.io/product: xincan
    spec:
      serviceAccountName: elasticsearch-rbac
      containers:
        - name: elasticsearch-node-nfs-client-provisioner
          image: base.hatech.com.cn/library/nfs-client-provisioner:v1.5.2
          volumeMounts:
            - name: nfs-client-root
              # 该文件夹是nfs-client-provisioner运行之后容器内部固定的文件夹它会mount到/nfs/data/nginx nfs服务器nginx文件夹下
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs.xincan.efk/elasticsearch
            - name: NFS_SERVER
              value: 192.168.1.80
            - name: NFS_PATH
              value: /hatech/nfs/data/xincan/elasticsearch-multiple
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.80
            path: /hatech/nfs/data/xincan/elasticsearch-multiple
            
[root@master node]#
  1. 创建elasticsearch服务外暴露访问资源Service，名称为：elasticsearch-node-service
[root@master node]# cat 3-elasticsearch-node-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-node-service
  namespace: efk
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/role: node
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: elasticsearch
spec:
  type: NodePort
  selector:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/role: node
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: elasticsearch
  ports:
  - name: node
    protocol: TCP
    port: 9200
    targetPort: 9200
    nodePort: 31180
    
[root@master node]#
  1. 创建elasticsearch-node无头服务StatefulSet，名称为：根据metadata.name+“-”+实例序号，序号从0开始,如我的最后结果是：elasticsearch-node-0，elasticsearch-node-1、elasticsearch-node-2
  • replicas: 3 最终会生成3分cockroachdb实例
[root@master node]# cat 4-elasticsearch-node-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: elasticsearch-node
    namespace: efk
    labels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/role: node
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/app: elasticsearch
spec:
  replicas: 3
  serviceName: elasticsearch-node-service
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/role: node
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/app: elasticsearch
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/type: plugins
        xincan.kubernetes.io/role: node
        xincan.kubernetes.io/product: xincan
        xincan.kubernetes.io/app: elasticsearch
    spec:
      serviceAccountName: elasticsearch-rbac
      restartPolicy: Always
      containers:
        - name: elasticsearch-node
          image: base.hatech.com.cn/library/elasticsearch:7.4.2-ulimit
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          lifecycle:
            postStart:
              exec:
                command: ["/bin/bash", "-c", "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;"]
          ports:
            - name: node
              protocol: TCP
              containerPort: 9200
            - name: data
              protocol: TCP
              containerPort: 9300
          env:
            - name: cluster.name
              value: "es_cluster"
            - name: "bootstrap.memory_lock"
              value: "true"
            - name: node.master
              value: "false"
            - name: node.data
              value: "true"
            - name: discovery.seed_hosts
              value: "elasticsearch-master-service"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"
            - name: node.ingest
              value: "true"
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"
          volumeMounts:
            - name: esdata
              mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: esdata
      annotations:
        volume.beta.kubernetes.io/storage-class: efk-nfs-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi
          
[root@master node]#

九、创建elasticsearch资源

  1. 创建主节点

  2. 创建子节点

# 创建主节点
[root@master efk]# kubectl apply -f elasticsearch/master/
service/elasticsearch-master-service created
statefulset.apps/elasticsearch-master created


# 创建子节点
[root@master efk]# kubectl apply -f elasticsearch/node
storageclass.storage.k8s.io/efk-nfs-storage created
deployment.apps/elasticsearch-nfs-client-provisioner created
service/elasticsearch-node-service created
statefulset.apps/elasticsearch-node created

[root@master efk]#
  1. 查看部署的资源
[root@master efk]# kubectl -n efk get all
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0                                  1/1     Running   0          2m50s
pod/elasticsearch-master-1                                  1/1     Running   0          2m47s
pod/elasticsearch-master-2                                  1/1     Running   0          2m45s
pod/elasticsearch-nfs-client-provisioner-565fc47b64-x8rg6   1/1     Running   0          2m43s
pod/elasticsearch-node-0                                    1/1     Running   0          2m43s
pod/elasticsearch-node-1                                    1/1     Running   0          2m36s
pod/elasticsearch-node-2                                    1/1     Running   0          2m32s

NAME                                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/elasticsearch-master-service   ClusterIP   10.103.3.141    <none>        9300/TCP         2m50s
service/elasticsearch-node-service     NodePort    10.108.240.63   <none>        9200:31180/TCP   2m43s

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/elasticsearch-nfs-client-provisioner   1/1     1            1           2m43s

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/elasticsearch-nfs-client-provisioner-565fc47b64   1         1         1       2m43s

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   3/3     2m50s
statefulset.apps/elasticsearch-node     3/3     2m43s
[root@master efk]#
  1. 检验安装是否成功
    • 此种情况不要心急，耐心等待一会儿
[root@master efk]# curl -XGET "http://10.108.240.63:9200/_cluster/health?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

5：如下标识安装成功

[root@master efk]# curl -XGET "http://10.110.13.94:9200/_cluster/health?pretty"
{
  "cluster_name" : "es_cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

十：结束语

至此集群版Kubernetes部署elasticsearch完成

avatar
文章
阅读
粉丝