1. 环境

Linux: centos-7\5.4.225-1.el7.elrepo.x86_64
Git: git version 2.41.0
Go: go version go1.20.5 linux/amd64
Docker: Docker version 24.0.2, build cb74dfc
Kubectl: Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", 
GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1

1.1 安装Controller-gen

controller-gen： （不用单独安装）

git clone https://github.com/kubernetes-sigs/controller-tools
cd controller-tools
go install ./cmd/controller-gen/ # 将controller-gen制作可执行文件并放到/usr/local/bin/go/bin/controller-gen
或者
go build ./cmd/controller-go/main.go #直接生成可执行文件

1.2 安装code-generator

Code-generator: （不用单独安装）

git clone https://github.com/kubernetes/code-generator.git
cd code-generator
git checkout 0.28.2
ls cmd/
go install ./cmd/{client-gen,deepcopy-gen,informer-gen,lister-gen} # 安装client-gen,deepcopy-gen,informer-gen,lister-gen到/usr/local/go/bin下
# 或者 build可执行文件
go build ./cmd/client-gen/main.go client-gen 
mv client-gen /usr/bin/

1.3 安装kubebuilder

Kubebuilder官方文档

# download kubebuilder and install locally.
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

2. code-generator介绍

k8s.io/client-go提供了对k8s原生资源的informer和clientset等，但对于自定义资源的操作则相对低效，需要使用rest api和dynamic client来操作，并自己实现反序列化等功能。 code-generator提供了以下工具用于为k8s中的资源生成相关代码，可以更加方便的操作自定义资源：

• deepcopy-gen：生成深度拷贝对象方法使用方法 在文件中添加注释 //+k8s:deepcopy-gen=package 为单个类型添加自动生成//+k8s:deepcopy-gen=true 为单个类型关闭自动生成//+k8s:deepcopy-gen=false
• client-gen：为资源生成标准的操作方法(get;listl;watch;create;update;patch;delete)在pkg/apis/GROUP/{GROUP}/GROUP/{VERSION}/types.go中使用。使用// +genclient标记对应类型生成的客户端，如果与该类型相关联的资源不是命名空间范围的，则需要附加 //+genclient:nonNamespaced标记
• // +genclient-生成默认的客户端动作函数(create,update,delete,get,list,update,patch,watch以及是否生成updateStatus取决于.Status字段是否存在)
• // +genclient:nonNamespaced-所有动作函数都是在没有名称空间的情况下生成
• // +genclient:oonlyVerbs=create.get-指定的动作函数被生成
• // +跟client：skipVerbswatch-生成watch以外的所有的动作函数
• // +genclient:noStatus-即使.Status字段存在也不生成udaoteStatus动作函数
• informer-gen：生成informer，提供事件机制(AddFunc, UpdateFunc, DeleteFunc)来响应kubernetes的event
• Lister-gen: 为get何list方法提供只读缓存层
• conversion-gen是用于自动生成在内部和外部类型之间转换的函数的工具一般的转换代码生成任务设计三套程序包：使用方法 一套包含内部类型的程序包 一套包含外部类型的程序包 单个目标程序包（生成的转换函数所在的位置，以及开发人员授权的转换功能所在的位置）。包含内部类型的包在kubernetes的常规代码生成框架中扮演着称为peer package的角色 标记转换内部软件包// +k8s:conversion-gen= 标记转换外部软件包//+k8s:conversion-gen-external-types= 标记不转换对应注释或结构 //+k8s:conversion-gen=false
• Default-gen 用于生产Defaulter函数 为包含字段的所有类型创建defaulters, // +k8s:defaulter-gen= 所有都生成//+k8s:defaulter-gen=true|false code-generator整合了这些gen,使用脚本generate-groups.sh和generate-internal-groups.sh可以为自定义资源生产相关代码。 tangxusc.github.io/blog/2019/0…

3.开始一个项目

3.1 使用kubebuilder创建crd资源

mkdir -p ~/projects/guestbook
cd ~/projects/guestbook
kubebuilder init --domain my.domain --repo my.domain/guestbook
kubebuilder create api --group webapp --version v1 --kind Guestbook
make manifests

3.2 使用code-generator生成clientset、informer、lister

// +build tools
package toolsimport _ "k8s.io/code-generator"

3.2.1 在hack下新增脚本tools.go

// +build tools
package toolsimport _ "k8s.io/code-generator"

3.2.2 新增脚本update-codegen.sh

    vim ./hack/update-codegen.sh #!/usr/bin/env bash 
    #表示有报错即退出 跟set -e含义一样 
    set -o errexit #执行脚本的时候，如果遇到不存在的变量，Bash 默认忽略它 ,跟 set -u含义一样 
    set -o nounset # 只要一个子命令失败，整个管道命令就失败，脚本就会终止执行 
    set -o pipefail #kubebuilder项目的MODULE
    MODULE=my.domain/example 
    #api包 
    APIS_PKG=api 
    
    #代码生出输出，生成Resource时指定的group一样
    OUTPUT_PKG=generated/example 
    
    # group-version such as cronjob:v1 
    GROUP=example 
    VERSION=v1 
    GROUP_VERSION=$GROUP:$VERSION SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)} 
    # kubebuilder2.3.2版本生成的api目录结构code-generator无法直接使用 
    rm -rf "${APIS_PKG}/${GROUP}" && mkdir -p "${APIS_PKG}/${GROUP}" && cp -r "${APIS_PKG}/${VERSION}/" "${APIS_PKG}/${GROUP}" 
    # generate the code with: 
    # --output-base because this script should also be able to run inside the vendor dir of 
    # k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir 
    # instead of the $GOPATH directly. For normal projects this can be dropped. 
    #client,informer,lister(注意: code-generator 生成的deepcopy不适配 kubebuilder 所生成的api) bash "${CODEGEN_PKG}"/generate-groups.sh "client,informer,lister" \ ${MODULE}/${OUTPUT_PKG} ${MODULE}/${APIS_PKG} \ ${GROUP_VERSION} \ --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt 
    # --output-base "${SCRIPT_ROOT}" 
    # --output-base "${SCRIPT_ROOT}/../../.."

注意：

1. kubebuilder2.3.2版本生成的api目录结构code-generator无法直接使用，需要在sh脚本中进行处理。
2. 修改脚本执行参数可以选择生成的代码，如：“client,informer,lister”。注意无需再次生成deepcopy： code-generator 生成的deepcopy不适配 kubebuilder 所生成的api。

3.3 下载code-generator

    # 查看go.mod内，client-go的版本号
    K8S_VERSION=v0.26.1
    go get k8s.io/code-generator@$K8S_VERSION
    go mod vendor
    # 检查go.mod下k8s.io/client-go、k8s.io/apimachinery版本号保持一致
    chmod +x vendor/k8s.io/code-generator/generate-groups.sh

3.4 生成代码

修改*_types.go文件，添加tag

    // +genclient
    // +kubebuilder:object:root=true
    // Guestbook is the Schema for the guestbooks API
    type Guestbook struct {
    }

新建apis/webapp/v1/doc.go，注意// +groupName=``webapp.my.domain：（group.domain）

// +groupName=webapp.example.com
package v1

新建apis/webapp/v1/register.go，code generator生成的代码需要用到它： group和version与前面使用kubebuilder create时保持一致

package v1
  
import (
        "k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects.
var SchemeGroupVersion = schema.GroupVersion{Group: "webapp", Version: "v1"}

func Resource(resource string) schema.GroupResource {
        return SchemeGroupVersion.WithResource(resource).GroupResource()
}

执行hack/update-codegen.sh

chmod +x hack/update-codegen.sh
hack/update-codegen.sh

4. 服务部署

4.1 将crd注册到集群中

kubectl apply -f config/crd/bases/webapp.my.domain_guestbooks.yaml

4.2 开发controller

4.2.1 修改crd

自定义crd的字段以及输出方式: 生成新的crd文件

make && make manifests

4.2.2 在reconcile实现代码逻辑

在cmd/main.go下SetupWithManager进入：

4.2.3 可以修改GuestbookReconciler结构，使用缓存，优化请求

// GuestbookReconciler reconciles a Guestbook object
type GuestbookReconciler struct {
   client.Client
   Scheme *runtime.Scheme
   Cache  cache.Cache
}

// main.go文件下
if err = (&controller.GuestbookReconciler{
   Client: mgr.GetClient(),
   Scheme: mgr.GetScheme(),
   Cache:  mgr.GetCache(),
}).SetupWithManager(mgr); err != nil {
   setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
   os.Exit(1)
}

4.2.4 修改QPS、Burst(controller和apiserver的并发请求)

在cmd/main.go文件下，默认的QPS=20.0, Burst=30。如下为具体的查看参数方式

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
   Scheme:                 scheme,
   MetricsBindAddress:     metricsAddr,
   Port:                   9443,
   HealthProbeBindAddress: probeAddr,
   LeaderElection:         enableLeaderElection,
   LeaderElectionID:       "ecaf1259.my.domain",
   // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
   // when the Manager ends. This requires the binary to immediately end when the
   // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
   // speeds up voluntary leader transitions as the new leader don't have to wait
   // LeaseDuration time first.
   //
   // In the default scaffold provided, the program ends immediately after
   // the manager stops, so would be fine to enable this option. However,
   // if you are doing or is intended to do any operation such as perform cleanups
   // after the manager stops then its usage might be unsafe.
   // LeaderElectionReleaseOnCancel: true,
})

进入到ctrl.GetConfigOrDie()：

GetConfigOrDie = config.GetConfigOrDie

进入到config.GetConfigOrDie

// GetConfigOrDie creates a *rest.Config for talking to a Kubernetes apiserver.
// If --kubeconfig is set, will use the kubeconfig file at that location.  Otherwise will assume running
// in cluster and use the cluster provided kubeconfig.
//
// Will log an error and exit if there is an error creating the rest.Config.
func GetConfigOrDie() *rest.Config {
   config, err := GetConfig()
   if err != nil {
      log.Error(err, "unable to get kubeconfig")
      os.Exit(1)
   }
   return config
}

进入GetConfig()

func GetConfig() (*rest.Config, error) {
   return GetConfigWithContext("")
}

进入到GetConfigWithContext("")，可以看到默认的QPS和Burst，该取值影响开发的controller与apiserver的并发量

func GetConfigWithContext(context string) (*rest.Config, error) {
   cfg, err := loadConfig(context)
   if err != nil {
      return nil, err
   }

   if cfg.QPS == 0.0 {
      cfg.QPS = 20.0
      cfg.Burst = 30.0
   }

   return cfg, nil
}

修改方式如下：

// get default config
config := ctrl.GetConfigOrDie()

// edit qps and burst
config.QPS = float32(Qps)
config.Burst = Burst

mgr, err := ctrl.NewManager(config, ctrl.Options{
   Scheme:                 scheme,
   MetricsBindAddress:     metricsAddr,
   Port:                   9443,
   HealthProbeBindAddress: probeAddr,
   LeaderElection:         enableLeaderElection,
   LeaderElectionID:       "ecaf1259.my.domain",
   // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
   // when the Manager ends. This requires the binary to immediately end when the
   // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
   // speeds up voluntary leader transitions as the new leader don't have to wait
   // LeaseDuration time first.
   //
   // In the default scaffold provided, the program ends immediately after
   // the manager stops, so would be fine to enable this option. However,
   // if you are doing or is intended to do any operation such as perform cleanups
   // after the manager stops then its usage might be unsafe.
   // LeaderElectionReleaseOnCancel: true,
})

4.3 本地调试

在cmd/main.go文件下，通过如下方式连接到k8s集群：

config, err := clientcmd.BuildConfigFromFlags("", "D:/config_file/k8s-071905_1.yaml")
if err != nil {
   os.Exit(1)
}

完整的main.go文件：

/*
Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package main

import (
   "flag"
   "k8s.io/client-go/tools/clientcmd"
   "os"

   // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
   // to ensure that exec-entrypoint and run can make use of them.
   _ "k8s.io/client-go/plugin/pkg/client/auth"

   "k8s.io/apimachinery/pkg/runtime"
   utilruntime "k8s.io/apimachinery/pkg/util/runtime"
   clientgoscheme "k8s.io/client-go/kubernetes/scheme"
   ctrl "sigs.k8s.io/controller-runtime"
   "sigs.k8s.io/controller-runtime/pkg/healthz"
   "sigs.k8s.io/controller-runtime/pkg/log/zap"

   webappv1 "my.domain/guestbook/api/v1"
   "my.domain/guestbook/internal/controller"
   //+kubebuilder:scaffold:imports
)

var (
   scheme   = runtime.NewScheme()
   setupLog = ctrl.Log.WithName("setup")
)

func init() {
   utilruntime.Must(clientgoscheme.AddToScheme(scheme))

   utilruntime.Must(webappv1.AddToScheme(scheme))
   //+kubebuilder:scaffold:scheme
}

func main() {
   var metricsAddr string
   var enableLeaderElection bool
   var probeAddr string
   var Qps float64
   var Burst int
   flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
   flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
   flag.BoolVar(&enableLeaderElection, "leader-elect", false,
      "Enable leader election for controller manager. "+
         "Enabling this will ensure there is only one active controller manager.")
   flag.Float64Var(&Qps, "Qps", float64(50), "qps")
   flag.IntVar(&Burst, "Burst", 100, "Burst")
   opts := zap.Options{
      Development: true,
   }
   opts.BindFlags(flag.CommandLine)
   flag.Parse()

   ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

   // get default config
   //config := ctrl.GetConfigOrDie()

   //get config from local-file
   config, err := clientcmd.BuildConfigFromFlags("", "D:/config_file/k8s-071905_1.yaml")
   if err != nil {
      os.Exit(1)
   }

   // edit qps and burst
   config.QPS = float32(Qps)
   config.Burst = Burst

   mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
      Scheme:                 scheme,
      MetricsBindAddress:     metricsAddr,
      Port:                   9443,
      HealthProbeBindAddress: probeAddr,
      LeaderElection:         enableLeaderElection,
      LeaderElectionID:       "ecaf1259.my.domain",
      // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
      // when the Manager ends. This requires the binary to immediately end when the
      // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
      // speeds up voluntary leader transitions as the new leader don't have to wait
      // LeaseDuration time first.
      //
      // In the default scaffold provided, the program ends immediately after
      // the manager stops, so would be fine to enable this option. However,
      // if you are doing or is intended to do any operation such as perform cleanups
      // after the manager stops then its usage might be unsafe.
      // LeaderElectionReleaseOnCancel: true,
   })
   if err != nil {
      setupLog.Error(err, "unable to start manager")
      os.Exit(1)
   }

   if err = (&controller.GuestbookReconciler{
      Client: mgr.GetClient(),
      Scheme: mgr.GetScheme(),
   }).SetupWithManager(mgr); err != nil {
      setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
      os.Exit(1)
   }
   //+kubebuilder:scaffold:builder

   if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
      setupLog.Error(err, "unable to set up health check")
      os.Exit(1)
   }
   if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
      setupLog.Error(err, "unable to set up ready check")
      os.Exit(1)
   }

   setupLog.Info("starting manager")
   if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
      setupLog.Error(err, "problem running manager")
      os.Exit(1)
   }
}

参考资料：

混合kubebuilder与code Generator编写CRD

github.com/kubernetes/…

使用code-generator生成crd的clientset、informer、listers

kubebuilder 进阶使用教程