Redis从入门到入坑10——Redis Stream

灰色轨迹丶
541 阅读16分钟

Redis从入门到入坑10——Redis Stream

Redis Stream 简介(官网原文)

Introduction to Redis streams

A Redis stream is a data structure that acts like an append-only log but also implements several operations to overcome some of the limits of a typical append-only log. These include random access in O(1) time and complex consumption strategies, such as consumer groups. You can use streams to record and simultaneously syndicate events in real time. Examples of Redis stream use cases include:

  • Event sourcing (e.g., tracking user actions, clicks, etc.)
  • Sensor monitoring (e.g., readings from devices in the field)
  • Notifications (e.g., storing a record of each user's notifications in a separate stream)

Redis generates a unique ID for each stream entry. You can use these IDs to retrieve their associated entries later or to read and process all subsequent entries in the stream.

Redis streams support several trimming strategies (to prevent streams from growing unbounded) and more than one consumption strategy (see XREAD, XREADGROUP, and XRANGE).

Redis 流是一种数据结构，其作用类似于仅追加日志，但也实现了多个操作来克服典型仅追加日志的一些限制。其中包括O（1）时间的随机访问和复杂的消费策略，如消费者群体。 您可以使用流实时记录和同时联合事件。 Redis 流用例的示例包括：

  • 事件溯源（例如，跟踪用户操作、点击等）
  • 传感器监控（例如，现场设备的读数）
  • 通知（例如，将每个用户的通知记录存储在单独的流中）

Redis 为每个流条目生成一个唯一的 ID。 您可以在以后使用这些 ID 检索其关联的条目，或读取和处理流中的所有后续条目。

Redis 流支持多种修剪策略（以防止流无限增长）和多个消费策略（请参考 XREAD、XREADGROUP 和 XRANGE）。

Stream 的结构

一个消息链表，将所有加入的消息都串起来，每个消息都有一个唯一的 ID 和对应的内容

image.png

1Message Content消息内容
2Consumer group消费组，通过XGROUP CREATE 命令创建，同一个消费组可以有多个消费者
3Last_delivered_id游标，每个消费组会有个游标 last_delivered_id，任意一个消费者读取了消息都会使游标 last_delivered_id 往前移动。
4Consumer消费者，消费组中的消费者
5Pending_ids消费者会有一个状态变量，用于记录被当前消费已读取但未ack的消息Id，如果客户端没有ack，这个变量里面的消息ID会越来越多，一旦某个消息被ack它就开始减少。这个pending_ids变量在Redis官方被称之为 PEL(Pending Entries List)，记录了当前已经被客户端读取的消息，但是还没有 ack (Acknowledge character：确认字符），它用来确保客户端至少消费了消息一次，而不会在网络传输的中途丢失了没处理

Stream 常用命令

查看 Stream 数据类型下所有命令： help @Stream

127.0.0.1:6379> help @Stream

  XACK key group id [id ...]
  summary: Marks a pending message as correctly processed, effectively removing it from the pending entries list of the consumer group. Return value of the command is the number of messages successfully acknowledged, that is, the IDs we were actually able to resolve in the PEL.
  since: 5.0.0

  XADD key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold [LIMIT count]] *|id field value [field value ...]
  summary: Appends a new entry to a stream
  since: 5.0.0

  XAUTOCLAIM key group consumer min-idle-time start [COUNT count] [JUSTID]
  summary: Changes (or acquires) ownership of messages in a consumer group, as if the messages were delivered to the specified consumer.
  since: 6.2.0

  XCLAIM key group consumer min-idle-time id [id ...] [IDLE ms] [TIME unix-time-milliseconds] [RETRYCOUNT count] [FORCE] [JUSTID]
  summary: Changes (or acquires) ownership of a message in a consumer group, as if the message was delivered to the specified consumer.
  since: 5.0.0

  XDEL key id [id ...]
  summary: Removes the specified entries from the stream. Returns the number of items actually deleted, that may be different from the number of IDs passed in case certain IDs do not exist.
  since: 5.0.0

  XGROUP 
  summary: A container for consumer groups commands
  since: 5.0.0

  XGROUP CREATE key groupname id|$ [MKSTREAM] [ENTRIESREAD entries_read]
  summary: Create a consumer group.
  since: 5.0.0

  XGROUP CREATECONSUMER key groupname consumername
  summary: Create a consumer in a consumer group.
  since: 6.2.0

  XGROUP DELCONSUMER key groupname consumername
  summary: Delete a consumer from a consumer group.
  since: 5.0.0

  XGROUP DESTROY key groupname
  summary: Destroy a consumer group.
  since: 5.0.0

  XGROUP HELP 
  summary: Show helpful text about the different subcommands
  since: 5.0.0

  XGROUP SETID key groupname id|$ [ENTRIESREAD entries_read]
  summary: Set a consumer group to an arbitrary last delivered ID value.
  since: 5.0.0

  XINFO 
  summary: A container for stream introspection commands
  since: 5.0.0

  XINFO CONSUMERS key groupname
  summary: List the consumers in a consumer group
  since: 5.0.0

  XINFO GROUPS key
  summary: List the consumer groups of a stream
  since: 5.0.0

  XINFO HELP 
  summary: Show helpful text about the different subcommands
  since: 5.0.0

  XINFO STREAM key [FULL [COUNT count]]
  summary: Get information about a stream
  since: 5.0.0

  XLEN key
  summary: Return the number of entries in a stream
  since: 5.0.0

  XPENDING key group [[IDLE min-idle-time] start end count [consumer]]
  summary: Return information and entries from a stream consumer group pending entries list, that are messages fetched but never acknowledged.
  since: 5.0.0

  XRANGE key start end [COUNT count]
  summary: Return a range of elements in a stream, with IDs matching the specified IDs interval
  since: 5.0.0

  XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]
  summary: Return never seen elements in multiple streams, with IDs greater than the ones reported by the caller for each stream. Can block.
  since: 5.0.0

  XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] id [id ...]
  summary: Return new entries from a stream using a consumer group, or access the history of the pending entries for a given consumer. Can block.
  since: 5.0.0

  XREVRANGE key end start [COUNT count]
  summary: Return a range of elements in a stream, with IDs matching the specified IDs interval, in reverse order (from greater to smaller IDs) compared to XRANGE
  since: 5.0.0

  XSETID key last-id [ENTRIESADDED entries_added] [MAXDELETEDID max_deleted_entry_id]
  summary: An internal command for replicating stream values
  since: 5.0.0

  XTRIM key MAXLEN|MINID [=|~] threshold [LIMIT count]
  summary: Trims the stream to (approximately if '~' is passed) a certain size
  since: 5.0.0

队列相关指令

XADD key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold [LIMIT count]] *|id field value [field value ...]：添加消息到流的末尾

image.png

127.0.0.1:6379> xadd mystream * k1 v1
1690357936633-0  --- 不指定id时自动生成的ID
127.0.0.1:6379> xrange mystream - +
1690357936633-0
k1
v1
-- 指定最大消息数量为 1 后 再添加消息，会把原有的消息挤出队列
127.0.0.1:6379> xadd mystream maxlen = 1 * k2 v2
1690357985193-0
127.0.0.1:6379> xrange mystream - +
1690357985193-0
k2
v2

-- 指定id时添加消息
127.0.0.1:6379> xadd stream_id01 1690357985194-0 k3 v3
1690357985194-0
-- 添加的下一条消息ID必须要大于上一条的消息ID
127.0.0.1:6379> xadd stream_id01 1690357985195-0 k4 v4
1690357985195-0
-- 添加的消息id 不大于上一条的消息id会直接报错
127.0.0.1:6379> xadd stream_id01 1690357985195-0 k5 v5
ERR The ID specified in XADD is equal or smaller than the target stream top item


---精确限制流中只有一个消息(等号 "=" 可以省略)
127.0.0.1:6379> xadd limitStream maxlen = 1 * k1 v1
1690358338694-0
127.0.0.1:6379> xrange limitStream - +
1690358338694-0
k1
v1
127.0.0.1:6379> xadd limitStream maxlen = 1 * k2 v2
1690358364259-0

--- 查看流中只有一个消息，上一个消息被挤出了，
127.0.0.1:6379> xrange limitStream - +
1690358364259-0
k2
v2
--- 注意：若要限制流中只有一个消息，每次执行XADD命令时都要加上 maxlen 限制，否则会直接往流中增加消息时，本应移除的消息不会被移除
127.0.0.1:6379> xadd limitStream  * k3 v3
1690358407535-0
127.0.0.1:6379> xrange limitStream - +
1690358364259-0
k2
v2
1690358407535-0
k3
v3

--- 模糊限制(在选项**MAXLEN**和实际计数中间的参数`~`的意思是，用户不是真的需要精确的5个项目。 它可以多几个条目。通过使用这个参数，仅当我们移除整个节点的时候才执行修整)
127.0.0.1:6379> xadd m1 maxlen ~ 5 * k6 v6
1690360193405-0
127.0.0.1:6379> xadd m1 maxlen ~ 5 * k7 v7
1690360198605-0
127.0.0.1:6379> xlen m1
7

--- MINID 参数可以指定在添加消息时从流中某个位置截取后再添加，
127.0.0.1:6379> xadd m1 * k1 v1
1690358860914-0
127.0.0.1:6379> xadd m1 * k2 v2
1690358866327-0
127.0.0.1:6379> xadd m1 * k3 v3
1690358871309-0
127.0.0.1:6379> xadd m1 minid = 1690358871309-0 * k4 v4
1690358903866-0
127.0.0.1:6379> xlen m1
2
127.0.0.1:6379> xrange m1 - +
1690358871309-0
k3
v3
1690358903866-0
k4
v4

XRANGE key start end [COUNT count] :返回流中的元素范围，其id与指定的id间隔匹配

127.0.0.1:6379> xadd mystream * k1 v1 k2 v2
1690362852006-0
127.0.0.1:6379> xadd mystream * k3 v3
1690362859160-0
127.0.0.1:6379> xadd mystream * k4 v4
1690362863166-0
127.0.0.1:6379> xrange mystream - +
1690362852006-0
k1
v1
k2
v2
1690362859160-0
k3
v3
1690362863166-0
k4
v4
127.0.0.1:6379> xrange mystream - + count 2
1690362852006-0
k1
v1
k2
v2
1690362859160-0
k3
v3
127.0.0.1:6379> xrange mystream - + count 1
1690362852006-0
k1
v1
k2
v2

XREVRANGE key end start [COUNT count]:返回流中的元素范围，其id与指定的id间隔匹配，顺序与XRANGE相反(从大到小)

127.0.0.1:6379> xrevrange mystream + - 
1690362863166-0
k4
v4
1690362859160-0
k3
v3
1690362852006-0
k1
v1
k2
v2
127.0.0.1:6379> xrevrange mystream + - count 2
1690362863166-0
k4
v4
1690362859160-0
k3
v3

XDEL key id [id ...]: 删除流中指定ID的消息，并返回删除的消息数

127.0.0.1:6379> xrange mystream - +
1690362852006-0
k1
v1
k2
v2
1690362859160-0
k3
v3
1690362863166-0
k4
v4
127.0.0.1:6379> xdel mystream 1690362852006-0
1
127.0.0.1:6379> xrange mystream - +
1690362859160-0
k3
v3
1690362863166-0
k4
v4

XLEN key: 返回流中消息的个数

127.0.0.1:6379> xlen mystream
2

XTRIM key <MAXLEN | MINID> [= | ~] threshold [LIMIT count]：用于对流中数据的截取

127.0.0.1:6379> xrange mystream - +
1690362859160-0
k3
v3
1690362863166-0
k4
v4
127.0.0.1:6379> xtrim mystream maxlen 1 
1
127.0.0.1:6379> xrange mystream - +
1690362863166-0
k4
v4

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]：用于获取消息(阻塞/非阻塞)，只会返回大于指定ID的消息。COUNT :最多读取的消息条数；BLOCK:是否以阻塞的方式读取，millseconds：阻塞时间，如果为0表示永久阻塞

-- $代表特殊ID，表示以当前Stream已经存储的最大的ID作为最后一个ID，当前Stream中不存在大于当前最大ID的消息，因此此时返回nil 
-- 0-0代表从最小的ID开始获取Stream中的消息，当不指定count，将会返回Stream中的所有消息，注意也可以使用0（00/000也都是可以的……）
127.0.0.1:6379> xread count 2 streams mystream $
(nil)

消息消费相关命令

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]

COUNT:最多读取多少条消息 BLOCK:是否阻塞（默认不阻塞,如果millseconds设置为0 则表示永远阻塞）

非阻塞方式
-- $代表特殊ID，表示以当前Stream已经存储的最大的ID作为最后一个ID，当前Stream中不存在大于当前最大ID的消息，因此此时返回nil
127.0.0.1:6379> xread count 2 streams mystream $

-- 0-0代表从最小的ID开始获取Stream中的消息，当不指定count，将会返回Stream中的所有消息，注意也可以使用0（00/000也都是可以的……）
127.0.0.1:6379> xread count 2 streams mystream 0-0
mystream
1690362863166-0
k4
v4
1690365553826-0
k5
v5

阻塞方式

127.0.0.1:6379> xdel myblockstream 1690857521160-0
1
127.0.0.1:6379> xrange myblockstream - +

127.0.0.1:6379> xread count 2 block 0 streams myblockstream 0-0
阻塞中

----------------------------------------------
启动另一个客户端连接上redis 并添加一条消息
127.0.0.1:6379> xadd myblockstream * k1 v1
1690857680489-0

查看之前的客户端，返回添加的消息
127.0.0.1:6379> xread count 2 block 0 streams myblockstream 0-0
myblockstream
1690857680489-0
k1
v1
(37.28s)

XGROUP CREATE key groupname id|$ [MKSTREAM] [ENTRIESREAD entries_read]：用于创建消费者组

-- 如果key不存在则会报错

创建消费者组的时候必须指定 ID, ID 为 0 表示从头开始消费，为 $ 表示只消费新的消息，队尾新来
127.0.0.1:6379> xgroup create mystream groupA $
ERR The XGROUP subcommand requires the key to exist. Note that for CREATE you may want to use the MKSTREAM option to create an empty stream automatically

-- $表示从Stream尾部开始消费
-- 0表示从Stream头部开始消费
127.0.0.1:6379> xadd mystream * k1 v1
1690515315604-0
127.0.0.1:6379> xgroup create mystream groupA $
OK
127.0.0.1:6379> xgroup create mystream groupB 0
OK

XGROUP CREATECONSUMER key groupname consumername :给消费者组创建消费者

127.0.0.1:6379> xgroup createconsumer mystream groupA consumer1
1

XGROUP DELCONSUMER key groupname consumername ：删除消费者组的消费者

127.0.0.1:6379> xgroup delconsumer mystream groupA consumer1
0

XGROUP DESTROY key groupname ：干掉消费者组

127.0.0.1:6379> xgroup destroy mystream groupA
1
--- 干掉消费者组后再执行之前的消费者组命令会消费者组不存在
127.0.0.1:6379> xgroup delconsumer mystream groupA consumer1
NOGROUP No such consumer group 'groupA' for key name 'mystream'

XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] id [id ...]：消费者组读取流中的消息，且每条消息只能被同一个消费者组消费一次：

127.0.0.1:6379> xreadgroup group group2 consumer1 streams stream1 >
stream1
1690872386043-0
k1
v1
1690872391265-0
k2
v2
1690872395420-0
k3
v3
1690872400087-0
k4
v4
1690872405877-0
k5
v5

---stream中的消息—旦被消费组里的一个消费者读取了，就不能再被该消费组内的其他消费者读取了，即同一个消费组里的消费者不能消费同—条消息。刚才的XREADGROUP命令再执行—次，此时读到的就是空值
127.0.0.1:6379> xreadgroup group group2 consumer2 streams stream1 >
(nil)

XPENDING key group [[IDLE min-idle-time] start end count [consumer]]:查询每个消费组内所有的消费者已读取但未确认的消息

127.0.0.1:6379> xrange stream1 - +
1690860626675-0
k1
v1
1690860632663-0
k2
v2
1690860636787-0
k3
v3
1690860640839-0
k4
v4
1690860646072-0
k5
v5

127.0.0.1:6379> xreadgroup group group1 consumer1 streams stream1 >
stream1
1690860626675-0
k1
v1
1690860632663-0
k2
v2
1690860636787-0
k3
v3
1690860640839-0
k4
v4
1690860646072-0
k5
v5

127.0.0.1:6379> xpending stream1 group1 
5
1690860626675-0
1690860646072-0
consumer1
5

-- 输出该消费者组的待定消息总数、待定消息中最小和最大的ID，然后列出消费者组中至少有一个待定消息的每个消费者及其待定消息的数量
127.0.0.1:6379> xpending stream1 group1 
5
1690860626675-0
1690860646072-0
consumer1
5
-- 每个消息返回四个属性:消息的ID、获取消息且仍需确认消息的消费者的名称。我们称其为消息的当前所有者、自上次将此消息传递给此消费者以来所经过的毫秒数、此消息被传递的次数
127.0.0.1:6379> xpending stream1 group1 - + 10
1690860626675-0   ---消息的ID
consumer1         ---获取消息且仍需确认消息的消费者的名称（我们称其为消息的当前所有者）
530998            ---自上次将此消息传递给此消费者以来所经过的毫秒
1                 ---此消息被传递的次数
1690860632663-0
consumer1
530998
1
1690860636787-0
consumer1
530998
1
1690860640839-0
consumer1
530998
1
1690860646072-0
consumer1
530998
1

--- 第一种情况将返回整个组中空闲时间超过9秒的前10个(或更少)PEL记录

127.0.0.1:6379> XPENDING stream1 group1 IDLE 9000 - + 10
1690860626675-0
consumer1
9994106
1
1690860632663-0
consumer1
9994106
1
1690860636787-0
consumer1
9994106
1
1690860640839-0
consumer1
9994106
1
1690860646072-0
consumer1
9994106
1
127.0.0.1:6379> XPENDING stream1 group1 IDLE 9000 - + 1
1690860626675-0
consumer1
9999287
1

-- 只返回消费者 consumer3 的消息
127.0.0.1:6379> xpending stream1 group3 - + 10 consumer3
1690872386043-0
consumer3
32027
1
1690872391265-0
consumer3
32027
1
1690872395420-0
consumer3
32027
1
1690872400087-0
consumer3
32027
1
1690872405877-0
consumer3
32027
1

XACK key group id [id ...] ：从流消费者组的Pending Entries List (PEL)中删除一条或多条消息

127.0.0.1:6379> xack stream1 group1 1690872386043-0
1
127.0.0.1:6379> xpending stream1 group1 - + 10 
1690872391265-0
consumer1
1781076
1
1690872395420-0
consumer1
1781076
1
1690872400087-0
consumer1
1781076
1
1690872405877-0
consumer1
1781076
1

XINFO :用于打印 STREAM CONSUMER GROUP 相关信息

XINFO CONSUMERS key groupname

127.0.0.1:6379> xinfo consumers stream1 group1
name       --- 消费者的名称
consumer1
pending    --- 消费但未确认的消息数量
5
idle       ---最后一次交互（XREADGROUP、XCLAIM、XAUTOCLAIM）后经过的时间
809992

XINFO GROUPS key

127.0.0.1:6379> xinfo groups stream1
name
group1     --- 消费者组的名称
consumers  --- 消费者的数量
1
pending    --- 消费后但未确认的消息数量
5
last-delivered-id  --- 传递给消费者组中的消费者最后一个消息ID
1690872405877-0
entries-read      --- 传递给消费者组中的消费者最后一个消息的逻辑“读计数器”
5
lag        --- 流中仍在等待交付给组消费者的消息数，或者当该数字无法确定时为NULL。
0
name
group2
consumers
1
pending
5
last-delivered-id
1690872405877-0
entries-read
5
lag
0
name
group3
consumers
1
pending
5
last-delivered-id
1690872405877-0
entries-read
5
lag
0

XINFO STREAM key [FULL [COUNT count]]

127.0.0.1:6379> xinfo stream stream1 
length  --------------------流中消息的个数
5
radix-tree-keys ------------ 基础基数数据结构中的键数
1
radix-tree-nodes ----------- 底层基数数据结构中的节点数
2
last-generated-id ---------- 最近添加到流中的消息的ID
1690872405877-0
max-deleted-entry-id ------- 从流中删除的最大条目ID
0-0
entries-added -------------- 流中添加的消息数量
5
recorded-first-entry-id ---- 流中第一个消息的ID
1690872386043-0
groups -------------------- 为流定义的消费者组的数量
3
first-entry --------------- 流中第一个消息的ID
1690872386043-0
k1
v1
last-entry -------------- 流中最后一个消息的ID
1690872405877-0
k5
v5

应用场景

实现消息队列，它支持消息的持久化、支持自动生成全局唯一ID、支持ack确认消息的模式、支持消费组模式等，让消息队列更加的稳定和可靠

avatar
文章
阅读
粉丝