6、亿级流量系统架构之如何在上万并发场景下设计可扩展架构(上)?
7、亿级流量系统架构之如何在上万并发场景下设计可扩展架构(中)?
8、亿级流量系统架构之如何在上万并发场景下设计可扩展架构(下)?
老规矩!我们首先看一下这个复杂的系统架构演进到当前阶段,整体的架构图是什么样子的。
笔者再次友情提醒,如果各位小伙伴对下面这个复杂的架构图还有什么不理解的地方,一定要先回看之前的文章,因为系列文必须对上下文有清晰的理解和认识。
接着文本我们来聊聊一个核心系统每天承载百亿流量的背景下,应该如何来保证复杂系统中的数据一致性?
二、什么是数据一致性?
简单来说,在一个复杂的系统中一定会对一些数据做出非常复杂的处理,而且可能是多个不同的子系统,甚至是多个服务。
对一个数据按照一定的顺序依次做出复杂的业务逻辑的执行,最终可能就会生产出一份宝贵的系统核心数据,落地到存储里去,比如说在数据库里存储。
给大家来一张手绘彩图,感受下这个现场的氛围:
从上图中我们就可以看到,多个系统如何对一个数据依次进行处理,最终拿到一份核心数据,并落地到存储里去。
那么在这个过程中,就可能会产生所谓的数据不一致的问题。
什么意思呢?给大家举一个最简单的例子,我们本来期望数据的变化过程是:数据1 -> 数据2 -> 数据3 -> 数据4。
那么最后落地到数据库里的应该是数据4,对不对?
结果呢?不知道为啥,经过上面那个复杂的分布式系统中的各个子系统,或者是各个服务的协作处理,最后居然搞出来一个数据87。
搞了半天,搞了一个跟数据4风马牛不相及的一个东西,最后落地到了数据库里。
然后啊,这套系统的最终用户,可能通过前台的界面看到了一个莫名其妙的数据87。
这就尴尬了,用户明显会觉得这个数据有错误,就会反馈给公司的客服,此时就会上报bug到工程师团队,大家就开始吭哧吭哧的找问题。
上面说的这个场景,其实就是一种数据不一致的问题,也是我们接下来几篇文章要讨论的一个问题。
实际上,在任何一个大规模分布式系统里,都会存在类似的问题。无论是电商,O2O,还是本文举例的数据平台系统,都一样。
三、一个数据计算链路的梳理
那么既然已经明确了问题,接下来就来看看在数据平台这个系统里,到底是什么问题可能会导致一个最终落地存储的数据的异常呢?
要明白这个问题,咱们先回过头看看,在之前提过的数据平台这个项目里,一个最终落地的数据的计算链路是什么样的?
大家看看下面的图:
如上图所示,其实从最简单的一个角度来说,这个数据计算的链路大概也就是上面的那个样子。
- 首先,通过MySQL binlog采集中间件获取到数据,转发给数据接入层。
- 然后,数据接入层会把原始数据落地到kv存储里去
- 接着,是实时计算平台会从kv存储里提取数据进行计算
- 最后,会将计算结果写入到数据库+缓存的集群里。数据查询平台会从数据库 + 缓存的集群里提取数据,提供用户来进行查询
看起来很简单,对吧?
但是哪怕是这个系统里,数据计算链路,也绝对不是这么简单的。
如果大家看过之前的系列文章的话,就应该知道,这个系统为了支撑高并发、高可用、高性能等场景,引入了大量的复杂机制。
所以实际上一条原始数据进入到系统,一直到最后落地到存储里,计算链路还会包含下面的东西:
- 接入层的限流处理
- 实时计算层的失败重试
- 实时计算层的本地内存存储的降级机制
- 数据分片的聚合与计算,单条数据在这里可能会进入一个数据分片里
- 数据查询层的多级缓存机制
上面只不过是随便列举了几条。然而哪怕只是上述几条,都可以把一个数据的计算链路变得复杂很多倍了。
四、数据计算链路的bug
既然大家已经明白了,在一个复杂系统里,一份核心数据可能是经过一个极为复杂的计算链路的处理,中间百转千回,任何可能的情况都会发生。
那么就可以理解在大型分布式系统中,数据不一致的问题是如何产生的了。
其实原因非常的简单,说白了,就是数据计算链路的bug。
也就是说,在数据的计算过程中,某个子系统出现了bug,并没有按照我们预期的行为去处理,导致最终产出去的数据变得错误了。
那么,为什么会在数据计算链路中出现这种bug呢?
原因很简单,如果大家曾经参与过上百人协作的大型分布式系统,或者是主导过上百人协作开发的大型分布式系统的架构设计,应该对核心数据的异常和错误非常熟悉,并且会感到头疼不已。
大规模分布式系统中,动辄上百人协作开发。很可能某个子系统或者是某个服务的负责人,对数据的处理逻辑理解偏差了,代码里写了一个隐藏的bug。
而这个bug,轻易不会触发,并且在QA测试环境还没测出来,结果带着一颗定时炸弹,系统上线。
最后在线上某种特殊的场景下,触发了这个bug,导致最终的数据出现问题。
五、电商库存数据的不一致问题
接触过电商的同学,可能此时脑子里就可以快速的想到一个类似的经典场景:电商中的库存。
在大规模的电商系统中,库存数据绝对是核心中的核心。但是实际上,在一个分布式系统中,很多系统可能都会采用一定的逻辑来更新库存。
这就可能导致跟上述说的场景类似的问题,就是多个系统都更新库存,但就是某个系统对库存的更新出现了bug。
这可能是因为那个系统的负责人没理解到底应该如何更新库存,也或者是他更新的时候采用的逻辑,没有考虑到一些特殊情况。
这样导致的结果就是,系统里的库存和仓库中实际的库存,死活对不上。但就是不知道到底哪个环节出了问题,导致库存数据出错。
这个,其实就是一个典型的数据不一致的问题。
六、大型系统的数据不一致排查有多困难
当面对一个大型分布式系统时,如果你之前压根儿没考虑过数据不一致的问题,那么我敢打赌,当你负责的系统在线上被客服反馈有某个核心数据不一致的时候,你绝对会一脸蒙圈。
因为一个核心数据的处理,少则涉及几个系统的协作处理,多则涉及十个以上的系统的协作处理。
如果你没有留存任何日志、或者仅仅就是有部分日志,然后基本就只能所有人干瞪眼,大家大眼对小眼,都盯着自己的代码看。
大家根据一个数据最后的错误结果,比如数据87。10多个人对着自己的代码,反复的思考,冥思苦想。
然后每个人都在大脑中疯狂的模拟自己代码的运行,但是就是想不明白,为什么本来应该是数据4的,结果出来了一个数据87?
所以现实问题就是这样,这种数据不一致的问题,大概有以下几个痛点:
- 自己基本无法主动提前感知到数据问题,要被动等待用户发现,反馈给客服,这很可能导致你的产品被大量投诉,老板很生气,后果很严重。
- 即使客服告诉了你数据错了,但是你们没法还原现场,没有留存证据,基本就是一群工程师对着代码想象,猜测。
- 即使你解决了一次数据不一致的问题,但是以后也许还有下一次,这样搞下去,会导致团队里好几个能干的小伙儿时间都搭在这种破事儿上。
七、下篇预告
所以针对本文描述的大型分布式系统数据不一致的问题,下篇文章我们将给出:在百亿流量的场景下,一套复杂系统我们是如何构建整套核心数据保证方案的。
敬请期待:
- 亿级流量系统架构之如何保证百亿流量下的数据一致性(中)?
- 亿级流量系统架构之如何保证百亿流量下的数据一致性(下)?
end
如有收获,请帮忙转发,您的鼓励是作者最大的动力,谢谢!
一大波微服务、分布式、高并发、高可用的原创系列文章正在路上
欢迎扫描下方二维码,持续关注:
石杉的架构笔记(id:shishan100)
十余年BAT架构经验倾囊相授
**> **推荐阅读:**
>
> 1、[拜托!面试请不要再问我Spring Cloud底层原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be13b83f265da6116393fc7)
>
> 2、[【双11狂欢的背后】微服务注册中心如何承载大型系统的千万级访问?](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be3f8dcf265da613a5382ca)
>
> 3、[【性能优化之道】每秒上万并发下的Spring Cloud参数优化实战](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be83e166fb9a049a7115580)
>
> 4、[微服务架构如何保障双11狂欢下的99.99%高可用](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5be99a68e51d4511a8090440)
>
> 5、[兄弟,用大白话告诉你小白都能听懂的Hadoop架构原理](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5beaf02ce51d457e90196069)
>
> 6、[大规模集群下Hadoop NameNode如何承载每秒上千次的高并发访问](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bec278c5188253e64332c76)
>
> 7、【[性能优化的秘密】Hadoop如何将TB级大文件的上传性能优化上百倍](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Flink.juejin.im%2525252525252F%2525252525253Ftarget%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Flink.juejin.im%252525252525252F%252525252525253Ftarget%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fjuejin.im%25252525252525252Fpost%25252525252525252F5bed82a9e51d450f9461cfc7)
>
> [8、](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4)[拜托,面试请不要再问我TCC分布式事务的实现原理坑爹呀!](https://juejin.cn/post/6844903716089233416) [](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4)
>
> 9、[【坑爹呀!】最终一致性分布式事务如何保障实际生产中99.99%高可用?](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf2c6b6e51d456693549af4)
>
> 10、[拜托,面试请不要再问我Redis分布式锁的实现原理!](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf3f15851882526a643e207)
>
> **11、****[【眼前一亮!】看Hadoop底层算法如何优雅的将大规模集群性能提升10倍以上?](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Flink.juejin.im%25252525253Ftarget%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fjuejin.im%2525252525252Fpost%2525252525252F5bf5396f51882509a768067e)**
>
> **12、****[亿级流量系统架构之如何支撑百亿级数据的存储与计算](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Flink.juejin.im%252525252F%252525253Ftarget%252525253Dhttps%25252525253A%25252525252F%25252525252Fjuejin.im%25252525252Fpost%25252525252F5bfab59fe51d4551584c7bcf)**
>
> 13、[亿级流量系统架构之如何设计高容错分布式计算系统](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Flink.juejin.im%2525253Ftarget%2525253Dhttps%252525253A%252525252F%252525252Fjuejin.im%252525252Fpost%252525252F5bfbeeb9f265da61407e9679)
>
> 14、[亿级流量系统架构之如何设计承载百亿流量的高性能架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Flink.juejin.im%25253Ftarget%25253Dhttps%2525253A%2525252F%2525252Fjuejin.im%2525252Fpost%2525252F5bfd2df1e51d4574b133dd3a)
>
> 15、[亿级流量系统架构之如何设计每秒十万查询的高并发架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Flink.juejin.im%252F%253Ftarget%253Dhttps%25253A%25252F%25252Fjuejin.im%25252Fpost%25252F5bfe771251882509a7681b3a)
>
> 16、[亿级流量系统架构之如何设计全链路99.99%高可用架构](https://link.juejin.im/?target=https%3A%2F%2Flink.juejin.im%3Ftarget%3Dhttps%253A%252F%252Fjuejin.im%252Fpost%252F5bffab686fb9a04a102f0022)
>
> 17、[七张图彻底讲清楚ZooKeeper分布式锁的实现原理](https://link.juejin.im/?target=https%3A%2F%2Fjuejin.im%2Fpost%2F5c01532ef265da61362232ed)
>
> 18、[大白话聊聊Java并发面试问题之volatile到底是什么?](https://juejin.cn/post/6844903730303746061)
>
> 19、[大白话聊聊Java并发面试问题之Java 8如何优化CAS性能?](https://juejin.cn/post/6844903731234865160)
>
> 20、[大白话聊聊Java并发面试问题之谈谈你对AQS的理解?](https://juejin.cn/post/6844903732061159437)
>
> 21、[大白话聊聊Java并发面试问题之公平锁与非公平锁是啥?](https://juejin.cn/post/6844903732883226637)
>
> 22、[大白话聊聊Java并发面试问题之微服务注册中心的读写锁优化](https://juejin.cn/post/6844903734267510798)
>
> 23、[互联网公司的面试官是如何360°无死角考察候选人的?(上篇)](https://juejin.cn/post/6844903734930046989)
>
> 24、[互联网公司面试官是如何360°无死角考察候选人的?(下篇)](https://juejin.cn/post/6844903735655661581)
>
> 25、[Java进阶面试系列之一:哥们,你们的系统架构中为什么要引入消息中间件?](https://juejin.cn/post/6844903736444207117)
>
> 26、[【Java进阶面试系列之二】:哥们,那你说说系统架构引入消息中间件有什么缺点?](https://juejin.cn/post/6844903737123667975)
>
> 27、[【行走的Offer收割机】记一位朋友斩获BAT技术专家Offer的面试经历](https://juejin.cn/post/6844903741213130765)
>
> 28、[【Java进阶面试系列之三】哥们,消息中间件在你们项目里是如何落地的?](https://juejin.cn/post/6844903742114906125)
>
> 29、[【Java进阶面试系列之四】扎心!线上服务宕机时,如何保证数据100%不丢失?](https://juejin.cn/post/6844903742928601095)
>
> 30、[一次JVM FullGC的背后,竟隐藏着惊心动魄的线上生产事故!](https://juejin.cn/post/6844903743712935944)
>
> 31、[【高并发优化实践】10倍请求压力来袭,你的系统会被击垮吗?](https://juejin.cn/post/6844903744514031624)
>
> 32、[【Java进阶面试系列之五】消息中间件集群崩溃,如何保证百万生产数据不丢失?](https://juejin.cn/post/6844903746636349453)
>
> 33、[亿级流量系统架构之如何在上万并发场景下设计可扩展架构(上)?](https://juejin.cn/post/6844903747345219598)
>
> 34、[亿级流量系统架构之如何在上万并发场景下设计可扩展架构(中)?](https://juejin.cn/post/6844903747903029261)
>
> 35、[亿级流量系统架构之如何在上万并发场景下设计可扩展架构(下)?](https://juejin.cn/post/6844903748565745678)
>
> 36、[亿级流量架构第二弹:你的系统真的无懈可击吗?](https://juejin.cn/post/6844903752030240782)**