【K8s-ControllerStudy-简易记录】Operator-2 （极具参考意义项目）

ShadowYD
2,619 阅读7分钟

0. 项目源码 (下载参考)

github.com/Shadow-linu…

1.CodeGenerator

github.com/kubernetes/…

1.1 安装及生产

  • 常用的几个生成功能 代码生成务必在 Linux 上执行

    • deepcopy-gen CustomResources 实现runtime.Object 接口对应的 DeepCopy 方法
    • client-gen clientsets 相关代码
    • informer-gen 创建 informers( watch 对应 CR变化产生的事件)
    • lister-gen 对 GET/List 请求提供只读的缓存层

  • 需明确以下操作

    • go 命令可直接使用，版本>= 1.15
    • export GOPATH=/opt/gopath GOPATH的设置
    • export GOPROXY=https://goproxy.cn GOPROXY的设置

  • 项目代码结构

    # tree /opt/operator-db/
/opt/operator-db/
├── go.mod
├── go.sum
├── main.go
└── pkg
    └── apis
        └── dbconfig  //最终要生成的类型目录
            └── v1
                ├── doc.go
                ├── register.go
                ├── types.go
                
# cat go.mod
module github.com/shenyisyn/dbcore

go 1.16

require (
        k8s.io/apimachinery v0.20.2

)

  • 在$GOPATH 中下载 code-generator

    # mkdir -p $GOPATH/src
# cd $GOPATH/src
# go get -u k8s.io/code-generator@0.20.15
# mv code-generator@0.20.15 code-generator
# cd  code-generator
# chmod a+x generate-groups.sh
# ls ./code-generator/ -l
总用量 92
dr-xr-xr-x 14 root root  4096 1月  22 15:54 cmd
-r--r--r--  1 root root   148 1月  22 15:54 code-of-conduct.md
-r--r--r--  1 root root   750 1月  22 15:54 CONTRIBUTING.md
dr-xr-xr-x  2 root root  4096 1月  22 15:54 examples
dr-xr-xr-x  6 root root  4096 1月  22 15:54 _examples
-r-xr-xr-x  1 root root  3979 1月  22 15:54 generate-groups.sh
-r-xr-xr-x  1 root root  5834 1月  22 15:54 generate-internal-groups.sh
-r--r--r--  1 root root   651 1月  22 15:54 go.mod
-r--r--r--  1 root root 13536 1月  22 15:54 go.sum
dr-xr-xr-x  2 root root  4096 1月  22 15:54 hack
-r--r--r--  1 root root 11358 1月  22 15:54 LICENSE
-r--r--r--  1 root root   184 1月  22 15:54 OWNERS
dr-xr-xr-x  4 root root  4096 1月  22 15:54 pkg
-r--r--r--  1 root root  1440 1月  22 15:54 README.md
-r--r--r--  1 root root   550 1月  22 15:54 SECURITY_CONTACTS
dr-xr-xr-x  3 root root  4096 1月  22 15:54 third_party
-r--r--r--  1 root root  1247 1月  22 15:54 tools.go
// 保障起见下载好需要用的库
# go mod download

  • 在 $GOPATH 中创建同样的项目文件

    # cd $GOPATH/src
# mkdir -p $GOPATH/src/github.com/shenyisyn/dbcore
// copy 项目到GOPATH下
# cp -av /opt/operator-db/* $GOPATH/src/github.com/shenyisyn/dbcore/
# cd $GOPATH/src/github.com/shenyisyn/dbcore/
// 执行生成，正常下是慢慢生成的
# $GOPATH/src/k8s.io/code-generator/generate-groups.sh all github.com/shenyisyn/dbcore/pkg/client github.com/shenyisyn/dbcore/pkg/apis dbconfig:v1
Generating deepcopy funcs
Generating clientset for dbconfig:v1 at github.com/shenyisyn/dbcore/pkg/client/clientset
Generating listers for dbconfig:v1 at github.com/shenyisyn/dbcore/pkg/client/listers
Generating informers for dbconfig:v1 at github.com/shenyisyn/dbcore/pkg/client/informers

// 生成成功后便会出现 zz_generated.deepcopy.go
# ls $GOPATH/src/github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1/
doc.go  register.go  types.go  zz_generated.deepcopy.go
// 生成成功
# ls $GOPATH/src/github.com/shenyisyn/dbcore/pkg/client/
clientset  informers  listers

2. CRD

  • vim /opt/operator-db/crd.yaml

    apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需与下面的 spec 字段匹配，并且格式为 '<名称的复数形式>.<组名>'
  name: dbconfigs.api.jtthink.com
spec:
  # 分组名，在REST API中也会用到的，格式是: /apis/分组名/CRD版本
  group: api.jtthink.com
  # 范围是属于namespace的 ,可以是 Namespaced 或 Cluster
  scope: Namespaced
  # 列举此 CustomResourceDefinition 所支持的版本
  versions:
    - name: v1
      # 是否有效
      served: true
      storage: true
      # 额外打印或暴露字段
      additionalPrinterColumns:
#        - jsonPath: .status.readyReplicas
#          name: Ready
#          type: string
        - jsonPath: .status.readyReplicas
          name: Ready
          type: string
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          properties:
            # 状态字段字段
            status:
              type: object
              properties:
                readyReplicas:
                  type: string
                replicas:
                  type: integer
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  # 字段合法性检测
                  # https://kubernetes.io/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation
                  # https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.0.md#schemaObject
                  minimum: 1
                  maximum: 10
                dsn:
                  type: string
              # 字段合法性检测
              required:
                - replicas
                - dsn
      # https://kubernetes.io/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#subresources
      subresources:
        # 当启用了 status 子资源时，对应定制资源的 /status 子资源会被暴露出来。
        status: {}
        # 当启用了 scale 子资源时，定制资源的 /scale 子资源就被暴露出来。 针对 /scale 所发送的对象是 autoscaling/v1.Scale。
        # 可以使用 kubectl scale 控制资源的副本数量
        # kubectl scale --replicas=3 dc/my-dbconfig
        scale:
          # specReplicasPath 定义定制资源中对应 scale.spec.replicas 的 JSON 路径
          specReplicasPath: .spec.replicas
          # statusReplicasPath 定义定制资源中对应 scale.status.replicas 的 JSON 路径
          statusReplicasPath: .status.replicas

  names:
    # 复数名
    plural: dbconfigs
    singular: dbconfig
    kind: DbConfig
    listKind: DbConfigList
    shortNames:
      - dc


    # kubectl create -f crd/crd.yaml
# kubectl get crd |grep dbconfig

  • vim /opt/operator-db/test_crd.yaml

    kind: DbConfig
apiVersion: api.jtthink.com/v1
metadata:
  name: dbcore-my-dbconfig
spec:
  replicas: 1
  dsn: mysql dsn

3. 开发Controller

3.1 配置连接

  • /opt/opearotr-db/k8sConfig/initConfig.go 
    package k8sConfig

import (
        "k8s.io/client-go/rest"
        "k8s.io/client-go/tools/clientcmd"
        "log"
        "os"
)

const (
        NSFile = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
)

//POD里  体内
func K8sRestConfigInPod() *rest.Config {

        config, err := rest.InClusterConfig()
        if err != nil {
                log.Fatal(err)
        }
        return config
}

func K8sRestConfig() *rest.Config {

        if os.Getenv("release") == "1" { //自定义环境
                log.Println("run in cluster")
                return K8sRestConfigInPod()
        }

        log.Println("run outside cluster")
        config, err := clientcmd.BuildConfigFromFlags("", "/Users/liangyedong/.kube/config")
        if err != nil {
                log.Fatal(err)
        }

        config.Insecure = false
        return config
}

3.2 Builders (核心代码)

  • /opt/opearotr-db/pkg/builders/deploy_tpl.go

    package builders

const deployTpl = `
apiVersion: apps/v1
kind: Deployment
metadata:   
  name: {{ .Name }}
  namespace: {{ .Namespace}}
spec:
  selector:
    matchLabels:
      app: {{ .Namespace}}-{{ .Name }}
  replicas: 1
  template:
    metadata:
      labels:
        app: {{ .Namespace}}-{{ .Name }}
        version: v1
    spec:
      containers:
        - name: {{ .Namespace}}-{{ .Name }}-container
          image: docker.io/shenyisyn/dbcore:v1
          imagePullPolicy: IfNotPresent
          ports:
             - containerPort: 8081
             - containerPort: 8090
`

  • /opt/opearotr-db/pkg/builders/deploy_builder.go

    package builders

import (
        "bytes"
        "context"
        "fmt"
        configv1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
        v1 "k8s.io/api/apps/v1"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/apimachinery/pkg/util/yaml"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "text/template"
)



type DeployBuilder struct {
        client.Client
        config *configv1.DbConfig
        deploy    *v1.Deployment
}

//目前软件的 命名规则
func deployName(name string ) string {
        return name + "-deployment"
}

func NewDeployBuilder(config *configv1.DbConfig, c client.Client) (*DeployBuilder, error) {
        deploy := &v1.Deployment{}
        err := c.Get(context.Background(), types.NamespacedName{
                Name: deployName(config.Name), Namespace: config.Namespace,
        }, deploy)
        if err != nil {

                deploy.Name, deploy.Namespace = deployName(config.Name), config.Namespace
                tpl, err := template.New("deploy").Parse(deployTpl)
                if err != nil {
                        return nil, err
                }

                var doc bytes.Buffer
                err = tpl.Execute(&doc, deploy)
                if err != nil {
                        return nil, err
                }

                err = yaml.Unmarshal(doc.Bytes(), deploy)
                if err != nil {
                        return nil, err
                }

        }
        return &DeployBuilder{deploy: deploy, Client: c, config: config}, nil

}

func (this *DeployBuilder) apply() *DeployBuilder {
        *this.deploy.Spec.Replicas = int32(this.config.Spec.Replicas)
        return this
}

// 设置属主，用作资源关联，删除时可连带删除
func (this *DeployBuilder) setOwner() *DeployBuilder {
        this.deploy.OwnerReferences = append(this.deploy.OwnerReferences,
                metav1.OwnerReference{
                        APIVersion: this.config.APIVersion,
                        Kind: this.config.Kind,
                        Name: this.config.Name,
                        UID: this.config.UID,
                },
                )
        return this
}

func (this *DeployBuilder) Build(ctx context.Context) error {
        // 如果没有creationTime 是nil，则创建deploy
        if this.deploy.CreationTimestamp.IsZero() {
                this.apply().setOwner()
                if err := this.Create(ctx, this.deploy); err != nil {
                        return err
                }
        } else {
                // 合并又有对象
                patch := client.MergeFrom(this.deploy.DeepCopy())
                fmt.Printf("=== Patch: %+v \n", patch)
                this.apply()
                if err := this.Patch(ctx, this.deploy, patch); err != nil {
                        return err
                }
                // 查看状态
                // 获取ready replicas
                readyReplicas := this.deploy.Status.ReadyReplicas
                this.config.Status.ReadyReplicas = fmt.Sprintf("%d/%d",readyReplicas, this.config.Spec.Replicas)
                this.config.Status.Replicas = *this.deploy.Spec.Replicas
                if err := this.Status().Update(ctx, this.config); err != nil {
                        return err
                }


        }
        return nil
}

3.3 controllers

  • /opt/opearotr-db/pkg/controllers/DbConfigController.go

    package controllers

import (
        "context"
        "fmt"
        v1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
        "github.com/shenyisyn/dbcore/pkg/builders"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/client-go/util/workqueue"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/event"
        "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

const (
        ResourceKind            = "DbConfig"
        ResourceApiGroupVersion = "api.jtthink.com/v1"
)

type DbConfigController struct {
        client.Client
}

func NewDbConfigController() *DbConfigController {
        return &DbConfigController{}
}

// 捕获更新事件
func (r *DbConfigController) OnUpdate(updateEvent event.UpdateEvent, wq workqueue.RateLimitingInterface) {
        for _, ref := range updateEvent.ObjectNew.GetOwnerReferences() {
                if ref.Kind == ResourceKind && ref.APIVersion == ResourceApiGroupVersion {
                        wq.Add(reconcile.Request{
                                types.NamespacedName{
                                        Namespace: updateEvent.ObjectNew.GetNamespace(),
                                        Name:      ref.Name,
                                },
                        })
                }
        }
}

// 捕获删除事件
func (r *DbConfigController) OnDelete(event event.DeleteEvent, wq workqueue.RateLimitingInterface) {

        for _, ref := range event.Object.GetOwnerReferences() {
                if ref.Kind == ResourceKind && ref.APIVersion == ResourceApiGroupVersion {
                        wq.Add(reconcile.Request{
                                NamespacedName: types.NamespacedName{
                                        Namespace: event.Object.GetNamespace(),
                                        Name:      event.Object.GetName(),
                                },
                        })
                }
        }

}

func (r *DbConfigController) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
        config := &v1.DbConfig{}
        err := r.Get(ctx, req.NamespacedName, config)
        if err != nil {
                return reconcile.Result{}, err
        }

        bd, err := builders.NewDeployBuilder(config, r.Client)
        if err != nil {
                return reconcile.Result{}, err
        }

        if err = bd.Build(ctx); err != nil {
                return reconcile.Result{}, err
        }

        fmt.Println(config)

        return reconcile.Result{}, err
}

func (r *DbConfigController) InjectClient(c client.Client) error {
        r.Client = c
        return nil
}

3.4 Init Controller Manger

  • /opt/opearotr-db/k8sConfig/initRuntime.go
package k8sConfig

import (
	"fmt"
	v1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
	"github.com/shenyisyn/dbcore/pkg/controllers"
	appv1 "k8s.io/api/apps/v1"
	"log"
	"os"
	"sigs.k8s.io/controller-runtime/pkg/builder"
	"sigs.k8s.io/controller-runtime/pkg/handler"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	"sigs.k8s.io/controller-runtime/pkg/manager"
	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
	"sigs.k8s.io/controller-runtime/pkg/source"
)

func InitManager()  {

	logf.SetLogger(zap.New())
	mgr, err := manager.New(K8sRestConfig(), manager.Options{
		Logger: logf.Log.WithName("dbcore"),
	})
	if err != nil {
		log.Fatal(fmt.Sprintf("unable to setup manager, err: %s", err.Error()))
	}

	if err = v1.SchemeBuilder.AddToScheme(mgr.GetScheme()); err != nil {
		mgr.GetLogger().Error(err, "unable add scheme")
	}
	dbconfigController:=controllers.NewDbConfigController()
	if err = builder.ControllerManagedBy(mgr).
		For(&v1.DbConfig{}).
		Watches(&source.Kind{Type: &appv1.Deployment{}},
			handler.Funcs{
				UpdateFunc: dbconfigController.OnUpdate,
				DeleteFunc: dbconfigController.OnDelete,
			},
			).
		Complete(controllers.NewDbConfigController()); err != nil {
			mgr.GetLogger().Error(err, "unable to create manager.")
			os.Exit(1)
	}

	if err = mgr.Start(signals.SetupSignalHandler()); err != nil {
		mgr.GetLogger().Error(err, "unable to start manager")
		os.Exit(1)
	}


}

3.5 启动Controller

  • /opt/opearotr-db/main.go
package main

import "github.com/shenyisyn/dbcore/k8sConfig"

func main()  {
	//k8sCfg := k8sConfig.K8sRestConfig()
	//client, err := clientv1.NewForConfig(k8sCfg)
	//
	//if err != nil {
	//	log.Fatal(err)
	//}
	//
	//dcList, _ := client.DbConfigs("default").List(context.Background(), metav1.ListOptions{})
	//fmt.Println(dcList)

	// 初始化并启动manager
	k8sConfig.InitManager()

}

3.6 当前阶段功能演示

// 创建dbconfig
# kubectl apply -f crd/test-crd.yaml
// 获取 dbconfig 资源，可响应test-crd.yaml 中的replicas
# kubectl get dc 
NAME                 READY   AGE
dbcore-my-dbconfig   2/2     51s
// 查看启动的deploy 对应的pod, 可联级删除资源，deploy资源删除后可重建
kubectl get deploy

4. ConfigMap 的使用

4.1 ConfigMap创建与映射

  • vim /opt/operator-db/pkg/builders/configmap_builder.go

    package builders

import (
        "bytes"
        "context"
        configv1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
        corev1 "k8s.io/api/core/v1"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/types"
        "log"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "text/template"

        //metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type ConfigMapBuilder struct {
        cm     *corev1.ConfigMap
        config *configv1.DbConfig
        client.Client
        DataKey string
}

func NewConfigMapBuilder(config *configv1.DbConfig, client client.Client) (*ConfigMapBuilder, error) {
        cm := &corev1.ConfigMap{}
        err := client.Get(context.Background(),
                types.NamespacedName{
                Namespace: config.Name, Name: deployName(config.Name),
        },cm)
        // 没取到则拼接一个空的data
        if err != nil {
                cm.Name, cm.Namespace = deployName(config.Name), config.Namespace
                cm.Data = make(map[string]string)
        }

        return &ConfigMapBuilder{cm: cm, config: config, Client: client}, nil
}

// 设置连带关系
func (this *ConfigMapBuilder) setOwner() *ConfigMapBuilder  {
        this.cm.OwnerReferences = append(this.cm.OwnerReferences,
                metav1.OwnerReference{
                        APIVersion: this.config.APIVersion,
                        Kind: this.config.Kind,
                        Name: this.config.Name,
                        UID: this.config.UID,
                })
        return this
}

const configMapKey = "app.yml"

//把configmap里面的 key=app.yml的内容 取出变成md5,
func (this *ConfigMapBuilder) parseKey() *ConfigMapBuilder  {
        if appData, ok := this.cm.Data[configMapKey]; ok {
                this.DataKey = Md5(appData)
                return this
        }
        this.DataKey = ""
        return this
}

func (this *ConfigMapBuilder) apply() *ConfigMapBuilder  {
        // 解析模版并赋值
        tpl, err := template.New(configMapKey).Delims("[[", "]]").Parse(cmtpl)
        if err != nil {
                log.Println(err)
                return this
        }

        var tplBuffer bytes.Buffer
        if err = tpl.Execute(&tplBuffer, this.config.Spec); err != nil {
                log.Println(err)
                return this
        }

        this.cm.Data[configMapKey] = tplBuffer.String()
        return this
}

func (this *ConfigMapBuilder) Build(ctx context.Context) error {
        if this.cm.CreationTimestamp.IsZero() {
                this.apply().setOwner().parseKey()
                if err := this.Create(ctx, this.cm); err != nil {
                        return err
                }
        } else {
                patch := client.MergeFrom(this.cm.DeepCopy())
                this.apply().parseKey()
                if err := this.Patch(ctx, this.cm, patch); err != nil {
                        return err
                }
        }

        return nil
}

  • vim /opt/dbcore-db/pkg/builders/deploy_tpl.go

    package builders

const cmtpl=`
  dbConfig:
    dsn: "[[ .Dsn ]]"
    maxOpenConn: [[ .MaxOpenConn ]]
    maxLifeTime: [[ .MaxLifeTime ]]
    maxIdleConn: [[ .MaxIdleConn ]]
  appConfig:
    rpcPort: 8081
    httpPort: 8090
  apis:
    - name: test
      sql: "select * from test"
`

// md5 设置在template 的 annotation中，所以当annotation发生变化时，所有的pod 信息都会进行更新为你声明的内容 ，从而更新了/app/app.yml
const deployTpl = `
apiVersion: apps/v1
kind: Deployment
metadata:   
  name: dbcore-{{ .Name }}
  namespace: {{ .Namespace}}
spec:
  selector:
    matchLabels:
      app: dbcore-{{ .Namespace}}-{{ .Name }}
  replicas: 1
  template:
    metadata:
      annotations:
        dbcore.config/md5: ''
      labels:
        app: dbcore-{{ .Namespace}}-{{ .Name }}
        version: v1
    spec:
      containers:
        - name: dbcore-{{ .Namespace}}-{{ .Name }}-container
          image: docker.io/shenyisyn/dbcore:v1
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: configdata
              mountPath: /app/app.yml
              subPath: app.yml
          ports:
            - containerPort: 8081
            - containerPort: 8090
      volumes:
        - name: configdata
          configMap:
            defaultMode: 0644
            name: dbcore-{{ .Name }}
`

  • vim /opt/dbcore-db/pkg/buidlers/deploy_builder.go, 把configmap buider 组合进来进行调用；

    package builders

import (
        "bytes"
        "context"
        "fmt"
        configv1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
        v1 "k8s.io/api/apps/v1"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/apimachinery/pkg/util/yaml"
        "log"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "text/template"
)

type DeployBuilder struct {
        client.Client
        config    *configv1.DbConfig
        cmBuilder *ConfigMapBuilder
        deploy    *v1.Deployment
}

//目前软件的 命名规则
func deployName(name string) string {
        return "dbcore-" + name
}

func NewDeployBuilder(config *configv1.DbConfig, c client.Client) (*DeployBuilder, error) {
        deploy := &v1.Deployment{}
        err := c.Get(context.Background(), types.NamespacedName{
                Name: deployName(config.Name), Namespace: config.Namespace,
        }, deploy)
        if err != nil {

                deploy.Name, deploy.Namespace = config.Name, config.Namespace
                tpl, err := template.New("deploy").Parse(deployTpl)
                if err != nil {
                        return nil, err
                }

                var doc bytes.Buffer
                err = tpl.Execute(&doc, deploy)
                if err != nil {
                        return nil, err
                }

                err = yaml.Unmarshal(doc.Bytes(), deploy)
                if err != nil {
                        return nil, err
                }

        }

        cmBuilder, err := NewConfigMapBuilder(config, c)
        if err != nil {
                log.Println("cm error:", err)
                return nil, err
        }

        return &DeployBuilder{
                deploy: deploy,
                Client: c, config: config,
                cmBuilder: cmBuilder}, nil

}

func (this *DeployBuilder) apply() *DeployBuilder {
        *this.deploy.Spec.Replicas = int32(this.config.Spec.Replicas)
        return this
}

const CMAnnotation = "dbcore.config/md5"

func (this *DeployBuilder) setCMAnnotation(configStr string) {
        this.deploy.Spec.Template.Annotations[CMAnnotation] = configStr
}

// 设置属主，用作资源关联，删除时可连带删除
func (this *DeployBuilder) setOwner() *DeployBuilder {
        this.deploy.OwnerReferences = append(this.deploy.OwnerReferences,
                metav1.OwnerReference{
                        APIVersion: this.config.APIVersion,
                        Kind:       this.config.Kind,
                        Name:       this.config.Name,
                        UID:        this.config.UID,
                },
        )
        return this
}

func (this *DeployBuilder) Build(ctx context.Context) error {
        // 如果没有creationTime 是nil，则创建deploy
        if this.deploy.CreationTimestamp.IsZero() {

                this.apply().setOwner()

                if err := this.cmBuilder.Build(ctx); err != nil {
                        log.Printf("Build ConfigMap Error: %+v \n", err)
                }

                //设置 config md5
                this.setCMAnnotation(this.cmBuilder.DataKey)

                if err := this.Create(ctx, this.deploy); err != nil {
                        log.Printf("Create Deployment Error: %+v \n", err)
                        return err
                }
        } else {
                if err := this.cmBuilder.Build(ctx); err != nil {
                        log.Printf("Build ConfigMap Error: %+v \n", err)
                }
                // 合并原有对象
                patch := client.MergeFrom(this.deploy.DeepCopy())
                fmt.Printf("=== Patch: %+v \n", patch)
                this.apply()
                //设置 config md5
                this.setCMAnnotation(this.cmBuilder.DataKey)
                if err := this.Patch(ctx, this.deploy, patch); err != nil {
                        log.Printf("Update Deployment Error: %+v \n", err)
                }

        }
        // 查看状态
        // 获取ready replicas
        readyReplicas := this.deploy.Status.ReadyReplicas
        this.config.Status.ReadyReplicas = fmt.Sprintf("%d/%d", readyReplicas, this.config.Spec.Replicas)
        this.config.Status.Replicas = *this.deploy.Spec.Replicas
        if err := this.Status().Update(ctx, this.config); err != nil {
                return err
        }

        return nil
}

5. Dashboard

5.1 定义Dashboard

package dashboard

import (
	"context"
	"github.com/gin-gonic/gin"
	v1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
	"io/ioutil"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/apimachinery/pkg/util/yaml"
	"k8s.io/client-go/rest"
	metricclient "k8s.io/metrics/pkg/client/clientset/versioned"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func isChildDeploy(cfg *v1.DbConfig, objectRef corev1.ObjectReference, c client.Client) bool {
	deploy := &appv1.Deployment{}
	err := c.Get(context.Background(), types.NamespacedName{
		Name: "dbcore-" + cfg.Name, Namespace: cfg.Namespace,
	}, deploy)
	if err != nil {
		return false
	}
	if objectRef.UID == deploy.UID {
		return true
	}
	return false
}

type AdminUi struct {
	r *gin.Engine
	c client.Client
	config *rest.Config
}

func NewAdminUi(c client.Client, config *rest.Config) *AdminUi  {

	r := gin.New()
	r.Use(errorHandler())
	r.GET("/", func(context *gin.Context) {
		context.JSON(200, gin.H{"message": "ok"})
	})

	return &AdminUi{r: r, c: c, config: config}
}

func (this *AdminUi) tops ()  {
	mc, err := metricclient.NewForConfig(this.config)
	Error(err)
	this.r.GET("/tops", func(c *gin.Context) {
		list, err := mc.MetricsV1beta1().NodeMetricses().List(c, metav1.ListOptions{})
		Error(err)
		c.JSON(200, list.Items)
	})
}

func (this *AdminUi) usage ()  {
	mc, err := metricclient.NewForConfig(this.config)
	Error(err)
	this.r.GET("/top-usage", func(c *gin.Context) {
		nodeName := c.Query("name")
		nm, err := mc.MetricsV1beta1().NodeMetricses().Get(c, nodeName, metav1.GetOptions{})
		Error(err)
		c.JSON(200, gin.H{"usage": nm.Usage})
	})
}

func (this *AdminUi) events()   {
	this.r.GET("/events/:ns/:name", func(c *gin.Context) {
		var ns, name =c.Param("ns"),c.Param("name")
		cfg := &v1.DbConfig{}
		Error(this.c.Get(c, types.NamespacedName{
			Namespace: ns, Name: name,
		}, cfg))

		eList := &corev1.EventList{}
		Error(this.c.List(c, eList, &client.ListOptions{}))
		retEvents := []corev1.Event{}
		for _, e := range eList.Items {
			//这是匹配 自定义资源 对应的 event
			//&& e.InvolvedObject.UID==cfg.UID
			if e.InvolvedObject.UID == cfg.UID && e.InvolvedObject.Name == cfg.Name {
				retEvents = append(retEvents, e)
				continue
			}
			//代表判断，当前资源是否是dbconfig 创建出来的 deployment
			if isChildDeploy(cfg, e.InvolvedObject, this.c) {
				retEvents = append(retEvents, e)
			}

		}
		c.JSON(200, retEvents)
	})
}

// 创建资源
func (this *AdminUi) postResource ()  {
	this.r.POST("/resource", func(c *gin.Context) {
		b, err := ioutil.ReadAll(c.Request.Body)
		Error(err)
		cfg := &v1.DbConfig{}
		Error(yaml.Unmarshal(b, cfg))
		if cfg.Namespace == "" {
			cfg.Namespace = "default"
		}
		Error(this.c.Create(c, cfg))
		c.JSON(200, gin.H{"message": "success"})
	})
}

// 删除资源
func (this *AdminUi) deleteResource ()  {
	this.r.DELETE("/resource/:ns/:name", func(c *gin.Context) {
		cfg := &v1.DbConfig{}
		Error(this.c.Get(c, types.NamespacedName{
			Name: c.Param("name"),
			Namespace: c.Param("ns"),
		}, cfg))
		Error(this.c.Delete(c, cfg))
		c.JSON(200, gin.H{"message": "OK"})
	})
}

func (this *AdminUi) total ()  {
	this.r.GET("/top-total", func(c *gin.Context) {
		nodeName := c.Query("name")
		nodeObj := &corev1.Node{}
		err := this.c.Get(c, types.NamespacedName{
			Name: nodeName,
		}, nodeObj)
		Error(err)
		c.JSON(200, gin.H{
			"cpu": float64(nodeObj.Status.Capacity.Cpu().MilliValue()),
			"mem": nodeObj.Status.Capacity.Name(corev1.ResourceMemory, resource.DecimalSI),
		})
	})
}

func (this *AdminUi) Start(ctx context.Context) error  {
	this.r.GET("/configs", func(c *gin.Context) {
		list := &v1.DbConfigList{}
		Error(this.c.List(ctx, list, &client.ListOptions{}))
		c.JSON(200, list.Items)
	})
	this.tops()
	this.usage()
	this.total()
	this.postResource()
	this.deleteResource()
	this.events()
	return this.r.Run(":9003")
}

6. 所有功能补充

  • event recorder;
  • 通过 mgr 启动 dashboard;
  • 开启控制器选主;
  • 加入自定义指标；
  • 定义 metrics address;
package k8sConfig

import (
	"fmt"
	v1 "github.com/shenyisyn/dbcore/pkg/apis/dbconfig/v1"
	"github.com/shenyisyn/dbcore/pkg/controllers"
	"github.com/shenyisyn/dbcore/pkg/dashboard"
	"github.com/shenyisyn/dbcore/pkg/mymetrics"
	appv1 "k8s.io/api/apps/v1"
	"log"
	"os"
	"sigs.k8s.io/controller-runtime/pkg/builder"
	"sigs.k8s.io/controller-runtime/pkg/handler"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
	"sigs.k8s.io/controller-runtime/pkg/manager"
	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
	"sigs.k8s.io/controller-runtime/pkg/metrics"
	"sigs.k8s.io/controller-runtime/pkg/source"
)

func InitManager()  {

	logf.SetLogger(zap.New())
	mgr, err := manager.New(K8sRestConfig(), manager.Options{
		Logger: logf.Log.WithName("dbcore"),
		// 开启控制器选主
		LeaderElection: true,
		LeaderElectionID: "dbcore-lock",
		LeaderElectionNamespace: "default",
                // 定义metrics address
		MetricsBindAddress: ":8082",
	})

	//加入自定义指标
	metrics.Registry.MustRegister(mymetrics.MyReconcileTotal)

	if err != nil {
		log.Fatal(fmt.Sprintf("unable to setup manager, err: %s", err.Error()))
	}

	if err = v1.SchemeBuilder.AddToScheme(mgr.GetScheme()); err != nil {
		mgr.GetLogger().Error(err, "unable add scheme")
	}
	// 获取event recorder
	dbconfigController:=controllers.NewDbConfigController(mgr.GetEventRecorderFor("dbconfig"))
	if err = builder.ControllerManagedBy(mgr).
		For(&v1.DbConfig{}).
		Watches(&source.Kind{Type: &appv1.Deployment{}},
			handler.Funcs{
				UpdateFunc: dbconfigController.OnUpdate,
				DeleteFunc: dbconfigController.OnDelete,
			},
			).
		Complete(dbconfigController); err != nil {
			mgr.GetLogger().Error(err, "unable to create manager.")
			os.Exit(1)
	}

	// 启动管理器，只要实现Start接口皆可以通过mgr 启动
	if err = mgr.Add(dashboard.NewAdminUi(mgr.GetClient(), K8sRestConfig())); err != nil {
		mgr.GetLogger().Error(err, "unable to start dashboard")
		os.Exit(1)
	}

	if err = mgr.Start(signals.SetupSignalHandler()); err != nil {
		mgr.GetLogger().Error(err, "unable to start manager")
		os.Exit(1)
	}
}
avatar
文章
阅读
粉丝