前言

`ProtoBuf`

protocol buffers 是一种语言无关、平台无关、可扩展的序列化结构数据的方法，它可用于（数据）通信协议、数据存储等。Protocol Buffers 是一种灵活，高效，自动化机制的结构数据序列化方法－可类比 XML，但是比 XML 更小（3 ~ 10倍）、更快（20 ~ 100倍）、更为简单。你可以定义数据的结构，然后使用特殊生成的源代码轻松的在各种数据流中使用各种语言进行编写和读取结构数据。你甚至可以更新数据结构，而不破坏由旧数据结构编译的已部署程序。

简单来讲， ProtoBuf 是结构数据序列化[1] 方法，可简单类比于 XML[2]，其具有以下特点：

语言无关、平台无关。即 ProtoBuf 支持 Java、C++、Python 等多种语言，支持多个平台
高效。即比 XML 更小（3 ~ 10倍）、更快（20 ~ 100倍）、更为简单
扩展性、兼容性好。你可以更新数据结构，而不影响和破坏原有的旧程序

`protostuff`

protostuff是一个基于protobuf实现的序列化方法，它较于protobuf最明显的好处是，在几乎不损耗性能的情况下做到了不用我们写.proto文件来实现序列化

一、压测工具

内含压测工具和压测结果

github.com/eishay/jvm-…

二、对比测试

测试来看，ProtoStuff + compress 方案会是比较好的选择

json 结构 ProtoStuff + compress 为原有方案( Gson+Snappy )时间的 30% 空间的 77%

xml 结构 ProtoStuff + compress 为原有方案( Gson+Snappy )时间的 33% 空间的 84%

三方测试 www.cnblogs.com/xiaoMzjm/p/…

2.1 工具代码

import com.google.common.collect.Maps;
import io.protostuff.LinkedBuffer;
import io.protostuff.ProtostuffIOUtil;
import io.protostuff.Schema;
import io.protostuff.runtime.RuntimeSchema;
 
import java.util.Map;
 
/**
 * pb 序列化工具
 *
 */
public class ProtoStuffSerializerUtils {
 
    private static Map<Class<?>, Schema<?>> schemaCacheMap = Maps.newConcurrentMap();
 
    /**
     * 序列化
     * @param object 序列化对象
     * @param <T> 序列化对象类型
     * @return 序列化后的数据
     */
    @SuppressWarnings("unchecked")
    public static <T> byte[] serialize(T object) {
        Class<T> cls = (Class<T>) object.getClass();
        LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE * 2 * 20);
        byte[] bytes;
        try {
            Schema<T> schema = getSchema(cls);
            bytes = serializeInner(object, schema, buffer);
        } catch (Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        } finally {
            buffer.clear();
        }
        return bytes;
    }
 
    /**
     * 反序列化
     * @param data 序列化后的数据
     * @param cls 序列化对象类型
     * @param <T> 序列化对象泛型
     * @return 反序列化后的对象
     */
    public static <T> T deserialize(byte[] data, Class<T> cls) {
        T deserialize;
        try {
            Schema<T> schema = getSchema(cls);
            deserialize = deserializeInner(data, schema.newMessage(), schema);
        } catch (Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        }
        return deserialize;
    }
 
    /**
     * 序列化
     *
     * @param src
     * @param schema
     * @param buffer
     * @param <T>
     * @return
     */
    private static <T> byte[] serializeInner(final T src, final Schema<T> schema, final LinkedBuffer buffer) {
        return ProtostuffIOUtil.toByteArray(src, schema, buffer);
    }
 
    /**
     * 反序列化
     *
     * @param data
     * @param result
     * @param schema
     * @param <T>
     * @return
     */
    private static <T> T deserializeInner(final byte[] data, final T result, final Schema<T> schema) {
        ProtostuffIOUtil.mergeFrom(data, result, schema);
        return result;
    }
 
    /**
     * 获取 schema
     *
     * @param cls 数据类
     * @param <T> 泛型
     * @return schema
     */
    @SuppressWarnings("unchecked")
    private static <T> Schema<T> getSchema(Class<T> cls) {
        // 可以考虑启动时预热两个常用的对象
        Schema<T> schemaCache = (Schema<T>) schemaCacheMap.get(cls);
        if (null != schemaCache) {
            return schemaCache;
        } else {
            RuntimeSchema<T> schemaNew = RuntimeSchema.createFrom(cls);
            schemaCacheMap.put(cls, schemaNew);
            return schemaNew;
        }
    }
}

2.2 测试代码

package utils;
 
import com.jd.finance.jrpayorder.rest.vo.QueryOrderResponse;
import com.jd.finance.jrpaypassword.common.utils.GsonUtils;
import com.jd.finance.jrpaypassword.domain.vo.OrderPinCacheVo;
import com.jd.finance.jrpaypassword.rest.vo.OrderPinCheckPwdCacheRequest;
import com.jd.finance.jrpaypassword.utils.CompressUtils;
import com.jd.jr.mall.order.service.utils.ConverterUtils;
import com.jdd.mall.order.utils.MallOrderXmlUtils;
import com.jdd.mall.order.utils.ProtoStuffSerializerUtils;
import org.apache.commons.lang.StringUtils;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class JvmSerializerTest {
    private static final Logger LOGGER = LoggerFactory.getLogger(JvmSerializerTest.class);
 
    @Test
    public void testProtoBuff() throws Exception {
        int count = 20000;
        testJson(count);
        testXml(count);
        LOGGER.error("-------------------------");
        testJson(count);
        testXml(count);
    }
 
    private void testJson(int count) throws Exception {
        long size = 0;
        OrderPinCheckPwdCacheRequest request = MallOrderXmlUtils.genOrderPinCheckPwdCacheReq("1", "testApp2016#$%", StringUtils.EMPTY);
        OrderPinCacheVo orderPinCacheVo = ConverterUtils.writeRequest2XmlVo(request);
        QueryOrderResponse orderResponse = new QueryOrderResponse();
        orderResponse = ConverterUtils.xmlVo2Json(orderPinCacheVo, orderResponse);
        long start = System.nanoTime();
        for (int i = 0; i < count; i++) {
            byte[] serialize = ProtoStuffSerializerUtils.serialize(orderResponse);
            byte[] compress = CompressUtils.compressByte(serialize);
            size = compress.length;
            byte[] decompressByte = CompressUtils.decompressByte(compress);
            QueryOrderResponse deserialize = ProtoStuffSerializerUtils.deserialize(decompressByte, QueryOrderResponse.class);
        }
        long end = System.nanoTime();
        long use = end - start;
        LOGGER.info("pb j time(ns):{}, count:{}, size:{}", use, count, size);
        start = System.nanoTime();
        for (int i = 0; i < count; i++) {
            String value = GsonUtils.toJson(orderResponse);
            byte[] compress = CompressUtils.compress(value, CompressUtils.CPMPRESS_TYPE_SNAPPY);
            size = compress.length;
            String decompress = CompressUtils.decompress(compress);
            QueryOrderResponse deserialize = GsonUtils.fromJson(decompress, QueryOrderResponse.class);
        }
        end = System.nanoTime();
        use = end - start;
        LOGGER.info("gc j time(ns):{}, count:{}, size:{}", use, count, size);
    }
 
    private void testXml(int count) throws Exception {
        long size = 0;
        OrderPinCheckPwdCacheRequest request = MallOrderXmlUtils.genOrderPinCheckPwdCacheReq("1", "testApp2016#$%", StringUtils.EMPTY);
        OrderPinCacheVo orderPinCacheVo = ConverterUtils.writeRequest2XmlVo(request);
        long start = System.nanoTime();
        for (int i = 0; i < count; i++) {
            byte[] serialize = ProtoStuffSerializerUtils.serialize(orderPinCacheVo);
            byte[] compress = CompressUtils.compressByte(serialize);
            size = compress.length;
            byte[] decompressByte = CompressUtils.decompressByte(compress);
            OrderPinCacheVo deserialize = ProtoStuffSerializerUtils.deserialize(decompressByte, OrderPinCacheVo.class);
        }
        long end = System.nanoTime();
        long use = end - start;
        LOGGER.info("pb x time(ns):{}, count:{}, size:{}", use, count, size);
        start = System.nanoTime();
        for (int i = 0; i < count; i++) {
            String value = GsonUtils.toJson(orderPinCacheVo);
            byte[] compress = CompressUtils.compress(value, CompressUtils.CPMPRESS_TYPE_SNAPPY);
            size = compress.length;
            String decompress = CompressUtils.decompress(compress);
            OrderPinCacheVo deserialize = GsonUtils.fromJson(decompress, OrderPinCacheVo.class);
        }
        end = System.nanoTime();
        use = end - start;
        LOGGER.info("gc x time(ns):{}, count:{}, size:{}", use, count, size);
    }
}

2.3 测试数据

// LinkedBuffer.DEFAULT_BUFFER_SIZE * 2 * 20
20-11-22.00:37:05.965 [main            ] INFO  JvmSerializerTest      - pb j time(ns):580574613,    count:20000, size:4222
20-11-22.00:37:09.719 [main            ] INFO  JvmSerializerTest      - gc j time(ns):3753833917,   count:20000, size:3597
time gc/pb = 6.46 size gc/pb = 0.85

20-11-22.00:37:10.393 [main            ] INFO  JvmSerializerTest      - pb x time(ns):673694103,    count:20000, size:13919
20-11-22.00:37:18.991 [main            ] INFO  JvmSerializerTest      - gc x time(ns):8597234849,   count:20000, size:6769
time gc/pb = 12.76 size gc/pb = 0.48

// LinkedBuffer.DEFAULT_BUFFER_SIZE * 2 * 20 pb add snappy compress
20-11-22.00:57:11.856 [main            ] INFO  JvmSerializerTest      - pb j time(ns):1131198655, count:20000, size:2755
20-11-22.00:57:15.618 [main            ] INFO  JvmSerializerTest      - gc j time(ns):3761834247, count:20000, size:3597
time gc/pbc = 3.32 size gc/pbc = 1.30
time pbc/pb = 1.94 size pbc/pb = 0.65

20-11-22.00:57:18.455 [main            ] INFO  JvmSerializerTest      - pb x time(ns):2835673303, count:20000, size:5668
20-11-22.00:57:27.005 [main            ] INFO  JvmSerializerTest      - gc x time(ns):8550253723, count:20000, size:6769
time gc/pbc = 3.01 size gc/pbc = 1.19
time pbc/pb = 4.2 size pbc/pb = 0.40

序列化[1]：将结构数据或对象转换成能够被存储和传输（例如网络传输）的格式，同时应当要保证这个序列化结果在之后（可能在另一个计算环境中）能够被重建回原来的结构数据或对象。更为详尽的介绍可参阅维基百科。

类比于 XML[2]：这里主要指在数据通信和数据存储应用场景中序列化方面的类比，但个人认为 XML 作为一种扩展标记语言和 ProtoBuf 还是有着本质区别的。

【序列化】什么序列化居然能够提效30%以上｜ 8月更文挑战

前言

ProtoBuf

protostuff