13-Kubernetes-skywaling-nfs集群部署

百里长风
397 阅读6分钟

Kubernetes-skywaling-nfs单机版部署

背景

简单的在kubernetes上基于nfs动态存储部署skywaling

一、环境准备，安装kubernetes集群

​ 当前节点base、master、node1、node2、node3

​ kubernetes安装不在此赘述，请参考参考资料进行安装

​ kubernetes安装nfs不在此赘述，请参考参考资料进行安装

​ 参考资料

​ /kubernetes/1-Kubernetes基于Centos7构建基础环境（一）

​ /kubernetes/2-Kubernetes基于Centos7构建基础环境（二）

​ /kubernetes/3-Kubernetes基于Centos7构建基础环境（三）

​ 4-Kubernetes-基于Centos7安装面板及监控（四）

​ /kubernetes/nfs/6-kubernetes-nfs动态存储部署

​ /kubernetes/nfs/1-kubernetes-nfs动态存储部署

集群名称集群域名说明
basebase.xincan.cn部署harbor、nfs等服务
mastermaster.xincan.cnkubernetes主节点，做污点容忍，排除业务资源，nfs客户端等
node1node1.xincan.cnkubernetes从节点，nfs客户端等
node2node2.xincan.cnkubernetes从节点，nfs客户端等
node3node3.xincan.cnkubernetes从节点，nfs客户端等

二、总体流程：

  1. 原始镜像下载

    • apache/skywalking-oap-server:8.6.0-es7
    • apache/skywalking-ui:8.6.0

  2. 将原始镜像下载后进行push到私有Harbor仓库

  3. sidecar工具包制作

  4. 在Kubernetes集群的主节点上创建skywalking文件夹，用于存放下载资源，和Kuberntes文件编排，文件目录如下；

[root@master skywalling]# tree
.
├── 0-skywalking-namespace.yaml
├── 1-skywalling-connfigmap.yaml
├── 2-skywalling-deployment.yaml
├── 3-skywalling-service.yaml
├── 4-skywalling-ui-deployment.yaml
└── 5-skywalling-ui-service.yaml
  1. 镜像制作；
  2. 资源创建；
  3. 访问展示；
  4. Java对接；
  5. 结果展示；

三、镜像下载处理

  1. 镜像下载
[root@master /]# docker pull apache/skywalking-oap-server:8.6.0-es7
[root@master /]# docker pull apache/skywalking-ui:8.6.0
  1. 镜像Tag
[root@master /]# docker tag apache/skywalking-oap-server:8.6.0-es7 base.xincan.cn/library/skywalking-oap-server:8.6.0-es7
[root@master /]# docker tag apache/skywalking-ui:8.6.0 base.xincan.cn/library/apache/skywalking-ui:8.6.0
  1. 镜像上传
[root@master /]# docker push base.xincan.cn/library/skywalking-oap-server:8.6.0-es7
[root@master /]# docker push base.xincan.cn/library/apache/skywalking-ui:8.6.0

四、资源创建

  1. 创建skywalling的命名空间Namespace，名称为：skywalling
    • 将所有的资源挂载到此命名空间下
[root@master skywalling]# cat 0-skywalking-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
[root@master skywalling]#
  1. 创建skywalling配置资源ConfigMap,名称为：skywalking-config
    • 此处用nacos作为skywalking的注册中心，地址为：nacos-service.xincan:8848
    • 此处用elasticsearch作为数据存储介质，地址为：elasticsearch-node-service.efk:9200
[root@master skywalling]# cat 1-skywalling-connfigmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: skywalking-config
  namespace: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
data:
  CLUSTER: 'nacos'
  CLUSTER_NACOS_HOST_PORT: 'nacos-service.xincan:8848'
  STORAGE: 'elasticsearch7'
  STORAGE_ES_CLUSTER_NODES: 'elasticsearch-node-service.efk:9200'
  #ES_USER: '用户名'
  #ES_PASSWORD: '密码'
  CORE_GRPC_PORT: '11800'
  CORE_REST_PORT: '12800'
[root@master skywalling]#
  1. 创建skywalking服务资源Deployement，名称为：skywalking-oap
    • replicas: 3 最终会生成3分skywalking实例
[root@master skywalling]# cat 2-skywalling-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: skywalking-oap
  namespace: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
spec:
  replicas: 3
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/app: skywalking
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/type: plugins
        xincan.kubernetes.io/product: xincan
        xincan.kubernetes.io/app: skywalking
    spec:
      containers:
      - name: skywalking-oap
        image: base.xincan.cn/library/skywalking-oap-server:8.6.0-es7
        imagePullPolicy: IfNotPresent
        envFrom:
        - prefix: SW_
          configMapRef:
            name: skywalking-config
        ports:
        - containerPort: 12800
          name: http
          protocol: TCP
        - containerPort: 11800
          name: grpc
          protocol: TCP
        resources:
          limits:
            cpu: '2'
            memory: 2Gi
          requests:
            cpu: '1'
            memory: 2Gi
        volumeMounts:
        - name: volume-localtime
          mountPath: /etc/localtime
      volumes:
        - name: volume-localtime
          hostPath:
            path: /etc/localtime
            type: ''
[root@master skywalling]#
  1. 创建skywalling服务外暴露访问资源Service，名称为：skywalking-oap-service
[root@master skywalling]# cat 3-skywalling-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: skywalking-oap-service
  namespace: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
spec:
  selector:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
  type: ClusterIP
  ports:
    - name: http
      port: 12800
      protocol: TCP
      targetPort: 12800
    - name: grpc
      port: 11800
      protocol: TCP
      targetPort: 11800
[root@master skywalling]#
  1. 创建skywalking客户端服务资源Deployement，名称为：skywalking-ui
[root@master skywalling]# cat 4-skywalling-ui-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: skywalking-ui
  namespace: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
spec:
  replicas: 1
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/type: plugins
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/app: skywalking
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/type: plugins
        xincan.kubernetes.io/product: xincan
        xincan.kubernetes.io/app: skywalking
    spec:
      containers:
      - name: skywalking-ui
        image: base.xincan.cn/library/skywalking-ui:8.6.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: '2'
            memory: 1Gi
          requests:
            cpu: '1'
            memory: 1Gi
        env:
        - name: SW_OAP_ADDRESS
          value: "skywalking-oap-service:12800"
        volumeMounts:
        - name: volume-localtime
          mountPath: /etc/localtime
      volumes:
      - name: volume-localtime
        hostPath:
          path: /etc/localtime
          type: ''
[root@master skywalling]#
  1. 创建skywalling客户端服务外暴露访问资源Service，名称为：skywalking-ui-service
[root@master skywalling]# cat 5-skywalling-ui-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: skywalking-ui-service
  namespace: skywalking
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
spec:
  selector:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/type: plugins
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/app: skywalking
  type: NodePort
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
    nodePort: 31200
[root@master skywalling]#
  1. 创建skywalking资源
[root@master skywalking]# kubectl apply -f .
  1. 查看部署情况
[root@master skywalling]# kubectl -n skywalking get all
NAME                                 READY   STATUS    RESTARTS   AGE
pod/skywalking-oap-cb6ff6f9-6flkl    1/1     Running   0          18h
pod/skywalking-oap-cb6ff6f9-nslrg    1/1     Running   0          18h
pod/skywalking-oap-cb6ff6f9-r26jk    1/1     Running   0          18h
pod/skywalking-ui-5fd8d446c9-bhrr9   1/1     Running   0          18h

NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
service/skywalking-oap-service   ClusterIP   10.98.122.226   <none>        12800/TCP,11800/TCP   18h
service/skywalking-ui-service    NodePort    10.100.214.91   <none>        8080:31200/TCP        18h

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/skywalking-oap   3/3     3            3           18h
deployment.apps/skywalking-ui    1/1     1            1           18h

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/skywalking-oap-cb6ff6f9    3         3         3       18h
replicaset.apps/skywalking-ui-5fd8d446c9   1         1         1       18h
[root@master skywalling]#

五、 访问验证

  1. 根据地址访问：192.168.1.10:31200

image-20210805132647217

六、Java程序对接

  1. Agent制作
  2. Dockerfile创建，并发布到Harbor，目录结构如下：
[root@master skywalking-java-apm]# tree
.
├── apache-skywalking-apm-8.6.0.tar.gz
├── Dockerfile
  1. Agent Dockerfile文件编排
[root@storage skywalking-java-apm]# cat Dockerfile

FROM alpine:3.8
LABEL maintainer = "alittlexincan@163.com"
ENV SKYWALKING_VERSION=8.6.0
# ADD http://mirrors.tuna.tsinghua.edu.cn/apache/skywalking/${SKYWALKING_VERSION}/apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz /
COPY apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz /
RUN tar -zxvf /apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz && \
    mv apache-skywalking-apm-bin skywalking && \
    mv /skywalking/agent/optional-plugins/apm-trace-ignore-plugin* /skywalking/agent/plugins/ && \
    echo -e "\n# Ignore Path" >> /skywalking/agent/config/agent.config && \
    echo "# see https://github.com/apache/skywalking/blob/v8.1.0/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md" >> /skywalking/agent/config/agent.config && \
    echo 'trace.ignore_path=${SW_IGNORE_PATH:/health}' >> /skywalking/agent/config/agent.config
    
[root@storage skywalking-java-apm]#
  1. 镜像打包
    • 当前目里打包，不能忘了最后的.
[root@storage skywalking-java-apm]# docker build -t base.xincan.cn/xincan/skywalking-java-apm-agent:8.6.0 .
  1. java应用程序编排
    • 以sidecar方式进行文件编排
    • 以xincan-goods为例子, 其中deployment编排如下
[root@master xincan-goods]# cat 3-xincan-goods-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xincan-goods
  namespace: xincan
  labels:
    xincan.kubernetes.io/company: xincan.cn
    xincan.kubernetes.io/version: 0.0.1
    xincan.kubernetes.io/product: xincan
    xincan.kubernetes.io/type: service
    xincan.kubernetes.io/app: xincan-goods
spec:
  replicas: 1
  selector:
    matchLabels:
      xincan.kubernetes.io/company: xincan.cn
      xincan.kubernetes.io/version: 0.0.1
      xincan.kubernetes.io/product: xincan
      xincan.kubernetes.io/type: service
      xincan.kubernetes.io/app: xincan-goods
  template:
    metadata:
      labels:
        xincan.kubernetes.io/company: xincan.cn
        xincan.kubernetes.io/version: 0.0.1
        xincan.kubernetes.io/product: xincan
        xincan.kubernetes.io/type: service
        xincan.kubernetes.io/app: xincan-goods
    spec:
      #构建初始化镜像(通过初始化镜像的方式集成SkyWalking Agent)
      initContainers:
        - name: init-skywalking-agent
          image: base.xincan.cn/xincan/skywalking-java-apm-agent:8.6.0
          imagePullPolicy: IfNotPresent
          command:
            - 'sh'
            - '-c'
            - 'set -ex;mkdir -p /vmskywalking/agent;cp -r /skywalking/agent/* /vmskywalking/agent;'
          volumeMounts:
            - name: skywalking-agent
              mountPath: /vmskywalking/agent
      containers:
        - name: xincan-goods
          image: base.xincan.cn/xincan/xincan-goods:0.0.1-SNAPSHOT
          imagePullPolicy: Always
          ports:
            - name: client
              containerPort: 8200
            - name: api
              containerPort: 8719
          env:
            #这里通过JAVA_TOOL_OPTIONS，而不是JAVA_OPTS可以实现不通过将agent命令加入到java应用jvm参数而实现agent的集成
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/opt/skywalking/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: xincan-goods
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              # FQDN: servicename.namespacename.svc.cluster.local
              value: skywalking-oap-service.skywalking:11800
            - name: SERVER_PORT
              value: "8200"
            - name: SPRING_PROFILES_ACTIVE
              value: dev
            - name: NACOS_ADDR
              valueFrom:
                configMapKeyRef:
                  name: xincan-config
                  key: nacos.addr
          resources:
            limits:
              cpu: 0.5
              memory: 2Gi
            requests:
              cpu: 0.5
              memory: 1Gi
          volumeMounts:
            - name: skywalking-agent
              mountPath: /opt/skywalking/agent
            - name: xincan-storage
              mountPath: /log
      volumes:
        - name: skywalking-agent
          emptyDir: {}
        - name: xincan-storage
          persistentVolumeClaim:
            claimName: xincan-goods-pvc
[root@master xincan-goods]#
  1. 部署xincan-goods应用，结果如下
[root@master xincan-goods]# kubectl -n xincan get pod,svc
NAME                                                READY   STATUS    RESTARTS   AGE
pod/mysql-0                                         2/2     Running   0          8d
pod/mysql-1                                         2/2     Running   1          8d
pod/mysql-2                                         2/2     Running   1          8d
pod/mysql-nfs-client-provisioner-7c6dc94477-2xbp6   1/1     Running   0          8d
pod/nacos-76dc4fd7f5-8k8hz                          1/1     Running   0          7d23h
pod/redis-server-6c954459b6-mj4tx                   1/1     Running   0          7d21h
pod/sentinel-dashboard-77d4cb96d4-vb9kx             1/1     Running   0          7d17h
pod/xincan-goods-566b9cd57f-2p58t                   1/1     Running   0          3h15m

NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
service/mysql-cluster                ClusterIP   10.102.155.220   <none>        3306/TCP                         8d
service/mysql-master                 NodePort    10.105.100.135   <none>        3306:31110/TCP                   8d
service/mysql-read-1                 NodePort    10.98.99.38      <none>        3306:31120/TCP                   8d
service/mysql-read-2                 NodePort    10.107.85.138    <none>        3306:31130/TCP                   8d
service/nacos-service                NodePort    10.110.164.35    <none>        8848:31150/TCP                   7d23h
service/redis                        NodePort    10.107.7.83      <none>        6379:31140/TCP,16379:31141/TCP   7d21h
service/sentinel-dashboard-service   NodePort    10.105.19.132    <none>        8858:31160/TCP,8719:31170/TCP    7d17h
service/xincan-goods-service         NodePort    10.96.220.214    <none>        8200:31220/TCP,8719:31222/TCP    3h15m
[root@master xincan-goods]#

七、访问测试

  1. 访问xincan-goods服务
    • 地址为：192.168.1.10:31220

image-20210805134458817

  1. skywalking服务访问
    • 地址为：192.168.1.10:31200

image-20210805134624051

image-20210805134700987

image-20210805134713738

image-20210805134743587

八：结束语

至此集群版Kubernetes部署skywalking完成

avatar
文章
阅读
粉丝