K8S客户端工具:LeaderElector之ResourceLock

1,463 阅读3分钟

LeaderElector是什么

在多副本的后端服务部署架构中,有时我们只能有一个副本来处理任务,其他副本只作为热备,这时就需要通过选举决定出主从,只有主才能处理任务,这时我们可以使用LeaderElector来帮助我么实现选主

选主模型

选主模型并不陌生,比较有代表性的就是K8S的一大基础组件etcd的raft算法,通过选主的模型来实现多副本数据强一致性

这里LeaderElector使用的比较简单,我们先从LeaderElectionRecord的字段来了解

// LeaderElectionRecord is the record that is stored in the leader election annotation.
// This information should be used for observational purposes only and could be replaced
// with a random string (e.g. UUID) with only slight modification of this code.
// TODO(mikedanese): this should potentially be versioned
type LeaderElectionRecord struct {
   // HolderIdentity is the ID that owns the lease. If empty, no one owns this lease and
   // all callers may acquire. Versions of this library prior to Kubernetes 1.14 will not
   // attempt to acquire leases with empty identities and will wait for the full lease
   // interval to expire before attempting to reacquire. This value is set to empty when
   // a client voluntarily steps down.
   HolderIdentity       string      `json:"holderIdentity"`
   LeaseDurationSeconds int         `json:"leaseDurationSeconds"`
   AcquireTime          metav1.Time `json:"acquireTime"`
   RenewTime            metav1.Time `json:"renewTime"`
   LeaderTransitions    int         `json:"leaderTransitions"`
}
  • HolderIdentity:主实例的ID
  • LeaseDurationSeconds:任期
  • AcquireTime:初次继任时间
  • RenewTime:连任或继任时间
  • LeaderTransitions:第几次选举 从上面的名称中,很容易的能联想出选举与继任的过程:
  • 在无主时,开启选举,选出新主
  • 在有主时,主不断连任来延续自己的任期
  • 在主无法继任超出任期后,其他跟随者开启新的一轮选举,选主新主

ResourceLock

// Interface offers a common interface for locking on arbitrary
// resources used in leader election.  The Interface is used
// to hide the details on specific implementations in order to allow
// them to change over time.  This interface is strictly for use
// by the leaderelection code.
type Interface interface {
   // Get returns the LeaderElectionRecord
   Get(ctx context.Context) (*LeaderElectionRecord, []byte, error)

   // Create attempts to create a LeaderElectionRecord
   Create(ctx context.Context, ler LeaderElectionRecord) error

   // Update will update and existing LeaderElectionRecord
   Update(ctx context.Context, ler LeaderElectionRecord) error

   // RecordEvent is used to record events
   RecordEvent(string)

   // Identity will return the locks Identity
   Identity() string

   // Describe is used to convert details on current resource lock
   // into a string
   Describe() string
}

上面提到的选主过程中,从开启选举到选出新主是通过ResourceLock实现的,也就是谁抢到了锁,谁就是主。

由于使用锁的客户端是不同进程的服务,所以就需要使用分布式锁,常见的分布式锁可以通过数据库、redis实现,这里因为是在k8s的环境中,所以使用k8s的api-server通过操作Resource资源来实现。

k8s的api-server是并发安全的,在并发创建、更新时,只会有其中一个并发是成功的,当并发创建时,其他并发会返回AlreadyExists错误;在并发更新时,会携带上次获取Resource资源的ResourceVersion,当率先被处理的并发线程被处理成功后,ResourceVersion会自动+1,其他并发线程再被处理时,由于ResourceVersion低于当前版本,会被拒绝更新

LeaderElector

了解了选举模型与ResourceLock后,LeaderElector的功能也就很清晰了

  1. 程序刚启动时,没有LeaderElectionRecord记录,多个跟随者并发创建,创建成功的会被选为主
  2. 跟随者在下一轮try时,会获取到已经创建的LeaderElectionRecord记录,检测记录中的继任时间与任期,由于并没有过期,所以不会做什么
  3. 领导者在下一轮try时,会更新自己的继任时间,避免过期
  4. 当领导者无法继任,跟随者检测到已经过期时,会更新LeaderElectionRecord,将领导者改为自己,成功更新的跟随者会获得这轮选举的胜利,成为新的领导者

参考

concurrency-control-and-consistency

leaderelection