概述
在 sarama 中, Broker 结构负责与远程 kafka broker 的实际通信, 本节专注于 sarama 如何实现请求的发送与响应的回收
统一的 API 实现
在 broker.go 中为 Broker 实现了不同的方法, 这些方法对应着 Kafka 的 API, 如 GetMetadata
func (b *Broker) GetMetadata(request *MetadataRequest) (*MetadataResponse, error) {
response := new(MetadataResponse)
err := b.sendAndReceive(request, response)
if err != nil {
return nil, err
}
return response, nil
}
这些方法的流程都是一致的, 传入相应的 request, 调用 sendAndReceive 返回相应的 response
sendAndReceive
func (b *Broker) sendAndReceive(req protocolBody, res protocolBody) error {
responseHeaderVersion := int16(-1)
if res != nil {
responseHeaderVersion = res.headerVersion()
}
promise, err := b.send(req, res != nil, responseHeaderVersion)
if err != nil {
return err
}
if promise == nil {
return nil
}
select {
case buf := <-promise.packets:
return versionedDecode(buf, res, req.version())
case err = <-promise.errors:
return err
}
}
sendAndReceive 方法调用 send 方法生成了一个 promise, 之后从 promise.packets 中读取响应体解析到 res 中, 或者从 promise.errors 中读取到 error 返回失败
type versionedDecoder interface {
decode(pd packetDecoder, version int16) error
}
func versionedDecode(buf []byte, in versionedDecoder, version int16) error {
if buf == nil {
return nil
}
helper := realDecoder{raw: buf}
err := in.decode(&helper, version)
if err != nil {
return err
}
if helper.off != len(buf) {
return PacketDecodingError{"invalid length"}
}
return nil
}
各种 response 结构都实现了 versionedDecoder 接口, 用于从响应体中解析 response 结构, 在 versionedDecode 中, 便调用了不同响应体实现的 decode 方法解析响应体
send
func (b *Broker) send(rb protocolBody, promiseResponse bool, responseHeaderVersion int16) (*responsePromise, error) {
...
req := &request{correlationID: b.correlationID, clientID: b.conf.ClientID, body: rb}
buf, err := encode(req, b.conf.MetricRegistry)
if err != nil {
return nil, err
}
requestTime := time.Now()
// Will be decremented in responseReceiver (except error or request with NoResponse)
b.addRequestInFlightMetrics(1)
bytes, err := b.write(buf)
b.updateOutgoingCommunicationMetrics(bytes)
if err != nil {
b.addRequestInFlightMetrics(-1)
return nil, err
}
b.correlationID++
if !promiseResponse {
// Record request latency without the response
b.updateRequestLatencyAndInFlightMetrics(time.Since(requestTime))
return nil, nil
}
promise := responsePromise{requestTime, req.correlationID, responseHeaderVersion, make(chan []byte), make(chan error)}
b.responses <- promise
return &promise, nil
}
send 方法核心流程在于将请求写入与远程 Kafka broker 的连接中, 生成一个 promise 写入 broker.responses, 写入之后将 promise 返回, 而上述的 sendAndReceive 方法正是从返回的 promise 中读取响应或错误, 由此也可以得知, broker.responses 的 buffer 数决定了一个 broker in-flight 的请求数
responseReceiver
func (b *Broker) responseReceiver() {
var dead error
for response := range b.responses {
if dead != nil {
// This was previously incremented in send() and
// we are not calling updateIncomingCommunicationMetrics()
b.addRequestInFlightMetrics(-1)
response.errors <- dead
continue
}
headerLength := getHeaderLength(response.headerVersion)
header := make([]byte, headerLength)
bytesReadHeader, err := b.readFull(header)
requestLatency := time.Since(response.requestTime)
if err != nil {
b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
dead = err
response.errors <- err
continue
}
decodedHeader := responseHeader{}
err = versionedDecode(header, &decodedHeader, response.headerVersion)
if err != nil {
b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
dead = err
response.errors <- err
continue
}
if decodedHeader.correlationID != response.correlationID {
b.updateIncomingCommunicationMetrics(bytesReadHeader, requestLatency)
// TODO if decoded ID < cur ID, discard until we catch up
// TODO if decoded ID > cur ID, save it so when cur ID catches up we have a response
dead = PacketDecodingError{fmt.Sprintf("correlation ID didn't match, wanted %d, got %d", response.correlationID, decodedHeader.correlationID)}
response.errors <- dead
continue
}
buf := make([]byte, decodedHeader.length-int32(headerLength)+4)
bytesReadBody, err := b.readFull(buf)
b.updateIncomingCommunicationMetrics(bytesReadHeader+bytesReadBody, requestLatency)
if err != nil {
dead = err
response.errors <- err
continue
}
response.packets <- buf
}
close(b.done)
}
broker.responses 的读取与处理则实现在 responseReceiver 中, 它被实现为一个 background goroutine, responseReceiver goroutine 不断地从 broker.responses 中读取 promise, 从 promise 中获取响应头的长度读取响应头, 在从响应头中读取响应体的长度读取响应体, 将响应体写入 promise.packets 中, 这样 sendAndReceive 方法就可以从 promise 中读取响应体了
总结
在 sarama 处理与 broker 的通信过程中, send 方法与 responseReceiver goroutine 是核心逻辑, send 方法将请求写入连接, 构造 promise 返回给上层 API 以供读取响应或判断错误, 同时将 promise 传入 broker.reponses 中, 而 responseReceiver goroutine 从 channel 中读取 promise, 从连接中读取响应数据并写入 promise 的 channel 中, 这样上层的 API 就可以读取到响应了
这里利用了 TCP 有序性的保证和 go channel FIFO 的性质, 先写入连接的请求会先被 broker 处理, 响应也会先返回, 同样, 先写入的请求对应的 promise 也是先写入 broker.responses 中被 responseReceiver goroutine 先处理的, 读取到了对应的响应
