前言
数据库和缓存的一致性问题由来已久，我个人认为订阅binlog来实现缓存更新是比较靠谱且优雅的一个方案，所以写个小demo记录一下。
1.使用docker搭建环境
version: '2.2'
services:
mysql:
image: mysql:latest
container_name: canal_mysql
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=12345
networks:
- canal_example
redis:
image: redis:latest
container_name: canal_redis
ports:
- "6379:6379"
networks:
- canal_example
canal:
image: canal/canal-server:v1.1.5
container_name: canal_server
environment:
- canal.instance.master.address=canal_mysql:3306
- canal.instance.dbUsername=root
- canal.instance.dbPassword=12345
- canal.instance.connectionCharset=UTF-8
- canal.instance.tsdb.enable=true
- canal.instance.gtidon=false
ports:
- "11110:11110"
- "11111:11111"
- "11112:11112"
- "9100:9100"
networks:
- canal_example
networks:
canal_example:
driver: bridge
编写docker compose 文件，借助docker-compose一键搭建测试环境。
2.编写代码进行测试
- 创建spring项目，连接数据库与redis
@Data
@Entity
@Table(name = "book")
public class Book {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
private String name;
private String author;
private Integer price;
}
随便创建一个实体类，使用jpa自动映射到数据库中(docker启动数据库之后需要先创建一个库，在通过代码连接)。
- 编写代码实现有缓存则访问缓存，无缓存则访问数据库功能
@RestController
@AllArgsConstructor
@RequestMapping("book")
public class CanalController {
private BookRepository bookRepository;
private RedisTemplate<String, String> redisTemplate;
@GetMapping("count")
public Long queryBookCount() {
String bookCount = redisTemplate.opsForValue().get(Constant.REDIS_BOOK_COUNT_KEY);
if (bookCount != null) {
log.info("query book count hit cache");
return Long.parseLong(bookCount);
}
return countBook();
}
@PostMapping("add")
public String addBook(@RequestBody Book book) {
bookRepository.save(book);
return "success";
}
private synchronized long countBook() {
String bookCount = redisTemplate.opsForValue().get(Constant.REDIS_BOOK_COUNT_KEY);
if (bookCount != null) {
log.info("query book count hit cache");
return Long.parseLong(bookCount);
}
log.info("query book count miss cache,query database and set cache!");
long count = bookRepository.count();
redisTemplate.opsForValue().set(Constant.REDIS_BOOK_COUNT_KEY, String.valueOf(count));
return count;
}
}
- 使用canal订阅mysql的binlog日志，实现缓存更新逻辑
public class SimpleCanalClientTask implements Runnable {
private CanalConnector connector;
private RedisTemplate<String, String> redisTemplate;
public SimpleCanalClientTask(RedisTemplate<String, String> redisTemplate) {
this.redisTemplate = redisTemplate;
this.connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostIp(),
11111), "example", "", "");
}
@Override
public void run() {
int batchSize = 100;
try {
connector.connect();
connector.subscribe(".*");
connector.rollback();
while (true) {
Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
try {
//如果没有获取到数据，等待5秒继续
Thread.sleep(5000);
} catch (InterruptedException e) {
break;
}
} else {
handleEntry(message.getEntries());
}
connector.ack(batchId); // 提交确认
}
} finally {
connector.disconnect();
}
}
private void handleEntry(List<Entry> entries) {
for (Entry entry : entries) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
continue;
}
RowChange rowChange;
try {
rowChange = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
e);
}
EventType eventType = rowChange.getEventType();
String schema = entry.getHeader().getSchemaName();
String table = entry.getHeader().getTableName();
//如果获取到了指定表的更新操作，则删除缓存，让下一次查询直接查数据库
if (Constant.CONTROL_SCHEMA.equals(schema) && Constant.CONTROL_TABLE.equals(table)) {
if (eventType == EventType.DELETE || eventType == EventType.INSERT) {
log.info("receive change in book ,delete cache!");
redisTemplate.delete(Constant.REDIS_BOOK_COUNT_KEY);
}
}
}
}
}
接下来就是测试环节了，首先初始化的时候在表里放一些数据
INSERT INTO `canal`.`book` (`id`, `author`, `name`, `price`) VALUES (1, '吴承恩', '西游记', 15);
INSERT INTO `canal`.`book` (`id`, `author`, `name`, `price`) VALUES (2, '曹雪芹', '红楼梦', 12);
INSERT INTO `canal`.`book` (`id`, `author`, `name`, `price`) VALUES (3, '罗贯中', '三国演义', 17);
INSERT INTO `canal`.`book` (`id`, `author`, `name`, `price`) VALUES (4, '施耐庵', '水浒传', 13);
通过日志和打印出的sql可以判断请求是否查询了缓存
2022-02-28 10:45:23.745 INFO 2092 --- [nio-8080-exec-1] c.v.s.c.c.controller.CanalController : query book count miss cache,query database and set cache!
Hibernate: select count(*) as col_0_0_ from book book0_
2022-02-28 10:46:30.308 INFO 2092 --- [nio-8080-exec-3] c.v.s.c.c.controller.CanalController : query book count hit cache
接下来可以手动修改数据库信息，随便添加或删除一条数据
receive change in book ,delete cache!
可以看到日志中触发了删除缓存的逻辑，也就是订阅到了binlog。接下来再次查询的话则又会去查数据库了。这样通过订阅binlog，我们就可以控制缓存的更新逻辑。
