一、什么是粘包拆包？
1、TCP的粘包拆包
TCP是个"流"协议，所谓流，就是没有界限的一串数据。TCP底层并不了解上层业务数据的具体含义，它会根据TCP缓冲区的实际情况进行包的划分，所以在业务上认为，一个完整的包可能会被TCP拆分成多个包进行发送，也有可能把多个小的包封装成一个大的数据包发送，这就是所谓的TCP粘包和拆包问题
2、TCP粘包拆包的问题如何解决？
由于底层的TCP无法理解上层的业务数据，所以在底层是无法保证数据包不被拆分和重组的，这个问题只能通过上层的应用协议栈设计来解决，根据业界的主流协议的解决方案，可以归纳如下
- 消息定长，例如每个报文的大小为固定长度200字节，如果不够，空位补空格
- 在包尾增加回车换行符进行分割，例如FTP协议；利用分割符
- 将消息分为消息头和消息体，消息头中包含表示消息总长度(或者消息体长度)的字段，通常设计思路为消息头的第一个字段使用int32来表示消息的总长度；例如http协议
- 更复杂的应用层协议
二、netty的粘包拆包
1、DelimiterBasedFrameDecoder
- 消息用指定分隔符进行分割来解决粘包拆包问题
- 在使用时需要将该decoder加入到pipeline中，并且需要放到自定义处理数据handler的前面，因为他是用来解析数据的，例如
b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class)
// 指定连接队列大小
.option(ChannelOption.SO_BACKLOG, 128)
//KeepAlive
.childOption(ChannelOption.SO_KEEPALIVE, true)
//Handler
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel channel) throws Exception {
// DelimiterBasedFrameDecoder以分隔符来进行拆包粘包
channel.pipeline().addLast(new DelimiterBasedFrameDecoder(1024, Unpooled.copiedBuffer("$".getBytes())))
.addLast(new IdleStateHandler(5, 5, 10)).addLast(new NettyServerHandler());
}
});
1.1 ByteToMessageDecoder
- 用来将字节消息转成字符串
- 核心逻辑是在chandleRead中，用来进行粘包拆包的处理
1.1.1 channelRead
- 先进行粘包处理，将上次没处理完的消息和本次新接受的消息进行合并处理
- 调用callDecode对信息进行分隔符处理
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// 只有ByteBuf对象才会对输入信息做处理
if (msg instanceof ByteBuf) {
// 存放分割信息的对象列表
CodecOutputList out = CodecOutputList.newInstance();
try {
// first表示是否是首次，这里的目的是为了处理某一次消息未发现分隔符的情况，也就是粘包的处理
first = cumulation == null;
cumulation = cumulator.cumulate(ctx.alloc(),
first ? Unpooled.EMPTY_BUFFER : cumulation, (ByteBuf) msg);
// 对输入信息进行decode
callDecode(ctx, cumulation, out);
} catch (DecoderException e) {
throw e;
} catch (Exception e) {
throw new DecoderException(e);
} finally {
try {
if (cumulation != null && !cumulation.isReadable()) {
numReads = 0;
cumulation.release();
cumulation = null;
} else if (++numReads >= discardAfterReads) {
// We did enough reads already try to discard some bytes so we not risk to see a OOME.
// See https://github.com/netty/netty/issues/4275
numReads = 0;
discardSomeReadBytes();
}
int size = out.size();
firedChannelRead |= out.insertSinceRecycled();
fireChannelRead(ctx, out, size);
} finally {
out.recycle();
}
}
} else {
ctx.fireChannelRead(msg);
}
}
1.1.2 callDecode
- 循环处理多分割符情况
- 调用decodeRemovalReentryProtection方法处理分隔符
protected void callDecode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) {
try {
// 循环处理，可能存在多分隔符的情况，比如$为分隔符，消息为1$2$3$
while (in.isReadable()) {
int outSize = out.size();
if (outSize > 0) {
fireChannelRead(ctx, out, outSize);
out.clear();
// Check if this handler was removed before continuing with decoding.
// If it was removed, it is not safe to continue to operate on the buffer.
//
// See:
// - https://github.com/netty/netty/issues/4635
if (ctx.isRemoved()) {
break;
}
outSize = 0;
}
int oldInputLength = in.readableBytes();
// 真正处理decode的方法
decodeRemovalReentryProtection(ctx, in, out);
// Check if this handler was removed before continuing the loop.
// If it was removed, it is not safe to continue to operate on the buffer.
//
// See https://github.com/netty/netty/issues/1664
if (ctx.isRemoved()) {
break;
}
if (outSize == out.size()) {
if (oldInputLength == in.readableBytes()) {
break;
} else {
continue;
}
}
if (oldInputLength == in.readableBytes()) {
throw new DecoderException(
StringUtil.simpleClassName(getClass()) +
".decode() did not read anything but decoded a message.");
}
if (isSingleDecode()) {
break;
}
}
} catch (DecoderException e) {
throw e;
} catch (Exception cause) {
throw new DecoderException(cause);
}
}
1.1.3 decodeRemovalReentryProtection
- 设置当前状态为调用子类decode（STATE_CALLING_CHILD_DECODE）
- 调用子类decode方法来进行处理
final void decodeRemovalReentryProtection(ChannelHandlerContext ctx, ByteBuf in, List<Object> out)
throws Exception {
decodeState = STATE_CALLING_CHILD_DECODE;
try {
// 调用模版方法来进行decode
decode(ctx, in, out);
} finally {
boolean removePending = decodeState == STATE_HANDLER_REMOVED_PENDING;
decodeState = STATE_INIT;
if (removePending) {
fireChannelRead(ctx, out, out.size());
out.clear();
handlerRemoved(ctx);
}
}
}
1.2 DelimiterBasedFrameDecoder
1.2.1 decode
- 如果用户设置的分隔符是换行符（\n 和\r\n），那么就会使用LineBasedFrameDecoder来进行decode（这部分的初始化在构造器中）
- 遍历所有分割副（可能会有
34&）,比如这里会找到$
- 如果找到了分隔符，那么会将分隔符前面的数据slice出来并添加到out列表中，例如这里的12
- 如果没找到分隔符，他会将这单消息保存起来和后面的消息进行合并处理，直到找到分隔符再触发后面的channelRead
- 处理过程中会有一些最大frame的判断，该字段在初始化时会设置，超过该大小就会丢弃消息
@Override
protected final void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
Object decoded = decode(ctx, in);
// 将解析出来的数据加入到返回列表中
if (decoded != null) {
out.add(decoded);
}
}
/**
* Create a frame out of the {@link ByteBuf} and return it.
*
* @param ctx the {@link ChannelHandlerContext} which this {@link ByteToMessageDecoder} belongs to
* @param buffer the {@link ByteBuf} from which to read data
* @return frame the {@link ByteBuf} which represent the frame or {@code null} if no frame could
* be created.
*/
protected Object decode(ChannelHandlerContext ctx, ByteBuf buffer) throws Exception {
// 判断是不是以换行符分割，时则使用换行处理器来进行处理，这个详见下面对lineBasedDecoder的解析
if (lineBasedDecoder != null) {
return lineBasedDecoder.decode(ctx, buffer);
}
// Try all delimiters and choose the delimiter which yields the shortest frame.
// 找到最近的分隔符
int minFrameLength = Integer.MAX_VALUE;
ByteBuf minDelim = null;
for (ByteBuf delim: delimiters) {
int frameLength = indexOf(buffer, delim);
if (frameLength >= 0 && frameLength < minFrameLength) {
minFrameLength = frameLength;
minDelim = delim;
}
}
if (minDelim != null) {
int minDelimLength = minDelim.capacity();
ByteBuf frame;
if (discardingTooLongFrame) {
// We've just finished discarding a very large frame.
// Go back to the initial state.
discardingTooLongFrame = false;
buffer.skipBytes(minFrameLength + minDelimLength);
int tooLongFrameLength = this.tooLongFrameLength;
this.tooLongFrameLength = 0;
if (!failFast) {
fail(tooLongFrameLength);
}
return null;
}
if (minFrameLength > maxFrameLength) {
// Discard read frame.
buffer.skipBytes(minFrameLength + minDelimLength);
fail(minFrameLength);
return null;
}
// 找到分隔符后将数据slice出来，并添加到out列表中
if (stripDelimiter) {
frame = buffer.readRetainedSlice(minFrameLength);
buffer.skipBytes(minDelimLength);
} else {
frame = buffer.readRetainedSlice(minFrameLength + minDelimLength);
}
return frame;
} else {
if (!discardingTooLongFrame) {
if (buffer.readableBytes() > maxFrameLength) {
// Discard the content of the buffer until a delimiter is found.
tooLongFrameLength = buffer.readableBytes();
buffer.skipBytes(buffer.readableBytes());
discardingTooLongFrame = true;
if (failFast) {
fail(tooLongFrameLength);
}
}
} else {
// Still discarding the buffer since a delimiter is not found.
tooLongFrameLength += buffer.readableBytes();
buffer.skipBytes(buffer.readableBytes());
}
return null;
}
}
2、LineBasedFrameDecoder
- 消息以换行符为分隔符来解决粘包拆包问题
- 原理和DelimiterBasedFrameDecoder差不多，这里不做详细解析
3、FixedLengthFrameDecoder
- 消息固定长度字节来解决粘包拆包问题
- 原理和DelimiterBasedFrameDecoder差不多，这里不做详细解析
4、LengthFieldBasedFrameDecoder
- 消息增加length字段来解决粘包拆包问题
- 这个实现非常简单，就是需要初始化LengthFieldBasedFrameDecoder时需要指定消息体中对应length字段对应的offset和length，这样就可以在根据lenght来进行消息处理
