kube-controller-manager walk-around(2): leader-election

·  阅读 770

leader-election in kube-controller-manager

kube-controller-manager中的leader-election框架如下面的代码所示

    id, err := os.Hostname()
	if err != nil {
		return err
	}
    
    // add a uniquifier so that two processes on the same host don't accidentally both become active
	id = id + "_" + string(uuid.NewUUID())
	rl, err := resourcelock.New(c.ComponentConfig.Generic.LeaderElection.ResourceLock,
		"kube-system",
		"kube-controller-manager",
		c.LeaderElectionClient.CoreV1(),
		c.LeaderElectionClient.CoordinationV1(),
		resourcelock.ResourceLockConfig{
			Identity:      id,
			EventRecorder: c.EventRecorder,
		})
	if err != nil {
		klog.Fatalf("error creating lock: %v", err)
	}

	leaderelection.RunOrDie(context.TODO(), leaderelection.LeaderElectionConfig{
		Lock:          rl,
		LeaseDuration: c.ComponentConfig.Generic.LeaderElection.LeaseDuration.Duration,
		RenewDeadline: c.ComponentConfig.Generic.LeaderElection.RenewDeadline.Duration,
		RetryPeriod:   c.ComponentConfig.Generic.LeaderElection.RetryPeriod.Duration,
		Callbacks: leaderelection.LeaderCallbacks{
			OnStartedLeading: run,
			OnStoppedLeading: func() {
				klog.Fatalf("leaderelection lost")
			},
		},
		WatchDog: electionChecker,
		Name:     "kube-controller-manager",
	})
复制代码

首先,一个controller-manager实例的id是其hostname+一串UUID,之后生成一个资源锁,其Identity是这个id。资源锁有三种类型:

  • EndpointsResourceLock
  • ConfigMapsResourceLock
  • LeasesResourceLock

之后通过RunOrDie开始选主。运行或者死亡的意思就是一旦获得leader的权利,就运行run,一旦失去leader的地位,就由fatalf杀死自己。看一下RunOrDie的实现

func RunOrDie(ctx context.Context, lec LeaderElectionConfig) {
	le, err := NewLeaderElector(lec)
	if err != nil {
		panic(err)
	}
	if lec.WatchDog != nil {
		lec.WatchDog.SetLeaderElection(le)
	}
	le.Run(ctx)
}
复制代码

NewLeaderElector中新建了一个leaderElector对象,并调用其Run()方法。

	le := LeaderElector{
		config:  lec,
		clock:   clock.RealClock{},
		metrics: globalMetricsFactory.newLeaderMetrics(),
	}
	if lec.WatchDog != nil {
		lec.WatchDog.SetLeaderElection(le)
	}
	le.Run(ctx)
复制代码

metrics: globalMetricsFactory.newLeaderMetrics()这句中,globalMetricsFactory是一个noopMetricsProvider,就是一个空结构体,它的newLeaderMetrics()方法同样返回一个空结构体noMetrics{}。之后le.metrics.leaderOff(le.config.Name)其实上调用的是func (noopMetric) Off(name string) {},即啥都没干。

WatchDog在上一层传入的是electionChecker,是一个HealthzAdaptor类型,它的作用是

HealthzAdaptor associates the /healthz endpoint with the LeaderElection object. It helps deal with the /healthz endpoint being set up prior to the LeaderElection. This contains the code needed to act as an adaptor between the leader election code the health check code. It allows us to provide health status about the leader election. Most specifically about if the leader has failed to renew without exiting the process. In that case we should report not healthy and rely on the kubelet to take down the process.

这段话的含义是:electionChecker在LeaderElector的基础上加入了一个timeout,因为leaderElector可能假死,在这种特殊的情况下,可以触发timeout的超时。所以这里如果选项里开启了leader election,watchDog是不为空。

// SetLeaderElection ties a leader election object to a HealthzAdaptor
func (l *HealthzAdaptor) SetLeaderElection(le *LeaderElector) {
	l.pointerLock.Lock()
	defer l.pointerLock.Unlock()
	l.le = le
}
复制代码

所以会把这个le的信息赋值给healthzAdaptor。之后是最关键的le.Run(ctx)部分

// Run starts the leader election loop
func (le *LeaderElector) Run(ctx context.Context) {
	defer func() {
		runtime.HandleCrash()
		le.config.Callbacks.OnStoppedLeading()
	}()
	if !le.acquire(ctx) {
		return // ctx signalled done
	}
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()
	go le.config.Callbacks.OnStartedLeading(ctx)
	le.renew(ctx)
}
复制代码

尝试进行acquire(ctx)操作,这里的context是一个TODO的context,所以暂时不用管它。如果acquire成功,建立一个子context,开启一个go routine运行OnStartedLeading(ctx),并执行renew(ctx)操作。如果acquire失败,将直接return,触发defer,运行OnStoppedLeading()OnStoppedLeading()将终止这个进程,直接粗暴的fatal了:klog.Fatalf("leaderelection lost")

先看一下acquire()

// acquire loops calling tryAcquireOrRenew and returns true immediately when tryAcquireOrRenew succeeds.
// Returns false if ctx signals done.
func (le *LeaderElector) acquire(ctx context.Context) bool {
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()
	succeeded := false
	desc := le.config.Lock.Describe()
	klog.Infof("attempting to acquire leader lease  %v...", desc)
	wait.JitterUntil(func() {
		succeeded = le.tryAcquireOrRenew()
		le.maybeReportTransition()
		if !succeeded {
			klog.V(4).Infof("failed to acquire lease %v", desc)
			return
		}
		le.config.Lock.RecordEvent("became leader")
		le.metrics.leaderOn(le.config.Name)
		klog.Infof("successfully acquired lease %v", desc)
		cancel()
	}, le.config.RetryPeriod, JitterFactor, true, ctx.Done())
	return succeeded
}
复制代码

尝试le.tryAcquireOrRenew(),它分为以下几步

  1. 通过le.config.Lock.Get()获取已经存在的record。这个record包含identity,即记录当前谁是leader。get方法的话,假设当前是endpointLock,它的get方法其实就是从endpoints资源查找endpoint,只有这个endpoint的annotation中有control-plane.alpha.kubernetes.io/leader这个key,才返回一个record,否则均返回空,并返回错误类型。
  2. 检查错误,如果没有错误,说明之前选主过了。如果有错,看错误类型是否为notFound,如果不是not found,说明可能是endpoint不存在或别的错误。如果是not found,说明没有选主过,调用create方法,如果没有返回错误,该函数返回true,标志着该实例成为leader。
  3. 如果选主完成过了,需要检查record的时间戳是否expired。如果自己不是leader,且未超时,则返回false。在剩下的情况里,如果自己是leader,尝试update时间戳;如果不是leader,说明时间戳超时了,其他实例将尝试成为leader。

如果成为了leader,除了调用onStarting()回调函数外,还将进行le.renew(ctx)。renew操作也将阻塞,不停的尝试调用tryAcquireOrRenew()。当renew函数终止的时候,要么context超时了,要么acquier失败了,任何情况下它都将终止,并调用le.release()。renew结束之后,将触发defer cancel(),之后触发fatalf终止进程。

分类:
阅读
标签:
收藏成功!
已添加到「」, 点击更改