原文地址:nsq.io/components/…
组件
nsqd
nsqd
是接收消息,形成消息队列并将消息传递到客户端的守护进程。
它可以独立运行,但通常在具有nsqlookupd
实例的集群中配置(在这种情况下,它将声明主题和通道用以服务发现)。
它侦听两个TCP端口,一个侦听客户端,另一个侦听HTTP API。也可以选择在第三个端口上侦听HTTPS。
命令行参数选项
-auth-http-address value
<addr>:<port> to query auth server (may be given multiple times)
-broadcast-address string
address that will be registered with lookupd (defaults to the OS hostname) (default "yourhost.local")
-broadcast-http-port int
HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening to)
-broadcast-tcp-port int
TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening to)
-config string
path to config file
-data-path string
path to store disk-backed messages
-deflate
enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-https-address string
<addr>:<port> to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
lookupd TCP address (may be given multiple times)
-max-body-size int
maximum size of a single command body (default 5242880)
-max-bytes-per-file int
number of bytes per diskqueue file before rolling (default 104857600)
-max-channel-consumers int
maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)
-max-deflate-level int
max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
maximum client configurable duration of time between flushing to a client (default 30s)
-max-rdy-count int
maximum RDY count for a client (default 2500)
-max-req-timeout duration
maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
number of messages to keep in memory (per topic/channel) (default 10000)
-min-output-buffer-timeout duration
minimum client configurable duration of time between flushing to a client (default 25ms)
-msg-timeout duration
default duration to wait before auto-requeing a message (default 1m0s)
-node-id int
unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 248)
-output-buffer-timeout duration
default duration of time between flushing data to clients (default 250ms)
-snappy
enable snappy feature negotiation (client compression) (default true)
-statsd-address string
UDP <addr>:<port> of a statsd daemon for pushing stats
-statsd-interval duration
duration between pushing to statsd (default 1m0s)
-statsd-mem-stats
toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-statsd-udp-packet-size int
the size in bytes of statsd UDP packets (default 508)
-sync-every int
number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
duration of time per diskqueue fsync (default 2s)
-tcp-address string
<addr>:<port> to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
path to certificate file
-tls-client-auth-policy string
client certificate auth policy ('require' or 'require-verify')
-tls-key string
path to key file
-tls-min-version value
minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
path to certificate authority file
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
-worker-id
[deprecated] use --node-id
HTTP API
/ping
-是否存活/info
- 版本信息/stats
-详尽的运行时遥测/pub
-发布主题消息/mpub
-将多个消息发布到一个主题/config
-配置nsqd/debug/pprof
-pprof调试门户/debug/pprof/profile
-生成pprof CPU配置文件/debug/pprof/goroutine
-生成pprof goroutine配置文件/debug/pprof/heap
-生成pprof堆配置文件/debug/pprof/block
-生成pprof block配置文件/debug/pprof/threadcreate
-生成pprof OS线程配置文件
v1
命名空间(自nsqd
v0.2.29+
):
-
/topic/create
-创建一个新主题 -
/topic/delete
-删除主题 -
/topic/empty
-清空主题 -
/topic/pause
-暂停主题的消息流 -
/topic/unpause
-取消暂停的主题消息流 -
/channel/create
-建立新通道 -
/channel/delete
-删除通道 -
/channel/empty
-清空通道 -
/channel/pause
-暂停通道的消息流 -
/channel/unpause
-取消暂停的通道消息流 -
POST
/pub
发布一个消息:
Query Params:
topic - 要发布的主题 defer - 消息延迟发送的毫秒数(可选参数)
Body:
原始消息字节
Example:
$ curl -d "<message>" http://127.0.0.1:4151/pub?topic=name
-
POST
/mpub
一次往返发布多条消息
Query Params:
topic - 要发布的主题 binary - 布尔值 ('true' or 'false'),是否开启二进制模式
Body
\n 分割的原始消息字节
注意:默认情况下,
/mpub
期望用\n
来分隔消息,使用?binary=true
query参数来启用二进制模式,其中POST body应采用以下格式(发送的HTTP header中Content-Length
应为POST
body的总大小):[ 4-byte num messages ] [ 4-byte message #1 size ][ N-byte binary data ] ... (repeated <num_messages> times)
Example:
$ curl -d "<message>\n<message>\n<message>" http://127.0.0.1:4151/mpub?topic=name
-
POST
/topic/create
创建一个主题
Query Params:
topic - 要创建的主题
Example:
$ curl -X POST http://127.0.0.1:4151/topic/create?topic=name
-
POST
/topic/delete
删除一个现存的主题(以及所有的通道)
Query Params:
topic - 要删除的主题
Example:
$ curl -X POST http://127.0.0.1:4151/topic/delete?topic=name
-
POST
/channel/create
给现存的主题创建一个通道
Query Params:
topic - 现存的主题 channel - 要创建的通道
Example:
$ curl -X POST http://127.0.0.1:4151/channel/create?topic=name&channel=name
-
POST
/channel/delete
删除一个现有主题的一个通道
Query Params:
topic - 现存的主题 channel - 要删除的通道
Example:
$ curl -X POST http://127.0.0.1:4151/channel/delete?topic=name&channel=name
-
POST
/topic/empty
清空一个现有主题的所有队列消息(内存和磁盘中的)
Query Params:
topic - 要清空的现有主题
Example:
$ curl -X POST http://127.0.0.1:4151/topic/empty?topic=name
-
POST
/channel/empty
清空一个现有通道的所有队列消息(内存和磁盘中的)
Query Params:
topic - 现有主题 channel - 要清空的通道
Example:
$ curl -X POST http://127.0.0.1:4151/channel/empty?topic=name&channel=name
-
POST
/topic/pause
暂停一个主题所有通道的消息流(消息将以主题进行排队)
Query Params:
topic - 现有主题
Example:
$ curl -X POST http://127.0.0.1:4151/topic/pause?topic=name
-
POST
/topic/unpause
恢复现有已暂停主题的通道消息流
Query Params:
topic -现有主题
Example:
$ curl -X POST http://127.0.0.1:4151/topic/unpause?topic=name
-
POST
/channel/pause
暂停发往消费者的通道消息流(消息将以通道进行排队)
Query Params:
topic - 现有主题 channel - 要暂停的通道
Example:
$ curl -X POST http://127.0.0.1:4151/channel/pause?topic=name&channel=name
-
POST
/channel/unpause
恢复发往消费者的已暂停的通道消息流
Query Params:
topic - 现有主题 channel - 要恢复的通道
Example:
$ curl -X POST http://127.0.0.1:4151/channel/unpause?topic=name&channel=name
-
GET
/stats
返回内部统计
Query Params
format - (可选) `text` or `json` (default = `text`) topic - (可选) 筛选主题 channel - (可选) 筛选通道
Example:
$ curl http://127.0.0.1:4151/stats
-
GET
/ping
监控端点,应该返回
200 OK
。如果运行不健康,返回HTTP 500。**注意:**唯一的"unhealthy"状态是在发生溢出时未能将消息写入磁盘。
-
GET
/info
版本信息
-
GET
/debug/pprof
可用的调试端点索引页。
-
GET
/debug/pprof/profile
启动
pprof
CPU配置文件30秒钟,并通过请求返回输出注意:此端点未在
/debug/pprof
索引页面中列出,因为它会影响运行时性能。 -
GET
/debug/pprof/goroutine
返回所有运行中的
goroutines
的栈追踪。 -
GET
/debug/pprof/heap
返回堆和内存统计配置信息(顶部可以用作
pprof
内存配置信息) -
GET
/debug/pprof/block
返回一个
goroutine blocking
配置信息 -
GET
/debug/pprof/threadcreate
返回创建OS线程的
goroutine
栈跟踪 -
GET
/config/nsqlookupd_tcp_addresses
nsqlookupd TCP地址列表
Example:
$ curl http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses
-
PUT
/config/nsqlookupd_tcp_addresses
更新nsqlookupd TCP地址。
Body:
JSON array of TCP addresses.
Example:
$ curl -X PUT http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses \ -d '["127.0.0.1:4160", "127.0.0.2:4160"]'
调试和分析
nsqd
提供了一套可与Go的pprof 工具直接集成的性能分析端点。如果您安装了go工具套件,只需运行:
# memory profiling
$ go tool pprof http://localhost:4151/debug/pprof/heap
# cpu profiling
$ go tool pprof http://localhost:4151/debug/pprof/profile
TLS
当nsqd
配置了--tls-cert
与--tls-key
,客户端可以协商升级他们的TLS连接以增强安全性。
此外,您可以 使用--tls-required
要求客户端协商TLS(自nsqd
v0.2.28+
)。
您可以通过--tls-client-auth-policy
(require
或 require-verify
)配置nsqd
客户端证书策略:
require
-客户必须提供证书,否则被拒绝require-verify
-客户必须根据默认的CA或指定的链--tls-root-ca-file
提供有效的证书,否则将被拒绝
这可以用作客户端身份验证的一种形式(自nsqd
v0.2.28+
)。
如果要使用openssl生成无密码的自签名证书:
$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes
AUTH
注意: nsqd v0.2.29+
可用
要配置nsqd
为需要授权,您需要使用-auth-http-address=host:port
来指定一个符合Auth HTTP协议的Auth Server。
注意:当使用授权时,期望只有nsqd
TCP协议公开给外部客户端,而不是HTTP(S)端点。请参阅以下有关使用auth向客户端公开统计信息和查找的注释。
身份验证服务器必须接受以下HTTP请求:
/auth?remote_ip=...&tls=...&auth_secret=...
然后返回以下格式的响应:
{
"ttl": 3600,
"identity": "username",
"identity_url": "https://....",
"authorizations": [
{
"permissions": [
"subscribe",
"publish"
],
"topic": ".*",
"channels": [
".*"
]
}
]
}
请注意,主题和频道字符串必须是正则表达式nsqd
才能应用权限。nsqd
将在TTL持续时间内缓存响应,并在该时间间隔内重新请求授权。
预期在大多数情况下,TLS将使用授权来保障从客户端传递到nsqd
的敏感信息加密传递。nsqd
和Auth Server之间的通信应该通过受信任的网络进行(并且不会进行加密)。如果Auth Server仅基于远程IP信息进行身份验证选择,则客户端可以使用占位符字符串(如.
)来作为AUTH
命令主体(body
)。
一个Auth Server示例是pynsqauthd。
可以在nsqauthfilter中找到一个帮助服务器,该服务器向客户端公开了通过Auth Server权限过滤的nsqlookupd
和nsqd
/stats
数据。
使用命令行实用程序时,可以通过--reader-opt
标志使用授权。
$ nsq_tail ... -reader-opt="tls_v1,true" -reader-opt="auth_secret,$SECRET"
端到端处理延迟
您可以选择使用--e2e-processing-latency-percentile
标志配置nsqd
来收集和发起可配置百分比的端到端消息处理延迟。
使用*《数据流上有偏分位数的有效计算》*所述的概率百分位技术来计算值。我们使用 bmizerany的perks包,这个包实现了该算法。
为了使视图偏向于最近的处理行为,我们仅保留了过去N
分钟的分位数信息(可通过--e2e-processing-latency-window-time
进行配置)。在内部,我们每个通道维护两个分位数,每个分位数存储N/2
分钟的延迟数据。每隔一 N/2
分钟我们就会重置一个分位数(并开始向其中插入新数据)。由于分位数可以合并,因此会导致滚动窗口变大。
由于我们仅在通道层级收集数据,因此对于一个主题,我们汇总并合并了所有通道分位数。仅当数据在同一nsqd
实例上时才能使用此技术。但是,当跨nsqd
(例如通过nsqlookupd
)累积数据时,我们取每个nsqd
分位数的平均值。考虑到跨nsqd
的延迟分布,为了保持某些统计的准确性,除了平均值以外,我们还提供了最小/最大值。
注意:如果没有消费者连接,则尽管(很明显)排队消息的端到端处理时间在逐渐增加,但值仍然无法更新。这是因为对于来自客户端给定的消息,只有当nsqd
接收了FIN
信号时,才会计算端到端的指标。当消费者重新连接时,这些值将适当调整。
Statsd / Graphite集成
当使用--statsd-address
为 statsd(或如statsdemon之类的一个statsd端口)指定UDP <addr>:<port>
时,nsqd
将根据--statsd-interval
指定的间隔(重要:此间隔应 始终小于或等于statsd刷新到graphite的间隔)定期向statsd推送指标。启用此功能后,可以配置nsqadmin
直接显示graphite
图表。
我们建议使用以下graphite配置(但应根据您的可用资源和要求评估这些选择)。同样,要记住的重要一点是statsd的刷新间隔应小于或等于storage-schemas.conf
中的最小时间段, 并且nsqd
应配置刷新时间等于或小于--statsd-interval
指定的间隔。
# storage-schemas.conf
[nsq]
pattern = ^nsq\..*
retentions = 1m:1d,5m:30d,15m:1y
# storage-aggregation.conf
[default_nsq]
pattern = ^nsq\..*
xFilesFactor = 0.2
aggregationMethod = average
nsqd
实例将推送到如下statsd
路径:
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_bytes
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.clients [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.deferred_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.in_flight_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.requeue_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.timeout_count
# if --statsd-mem-stats is enabled
nsq.<nsqd_host>_<nsqd_port>.mem.heap_objects [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_idle_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_in_use_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_released_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_100 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_99 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_95 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.mem.next_gc_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_runs
# if --e2e-processing-latency-percentile is specified, for each percentile
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.e2e_processing_latency_<percent> [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.e2e_processing_latency_<percent> [gauge]
nsqlookupd
nsqlookupd
是管理拓扑信息的守护程序。客户端通过nsqlookupd
进行查询以发现特定主题的nsqd
生产者,同时nsqd
节点广播主题和通道信息。
有两个接口:一个nsqd
用于广播的TCP接口和一个客户端用于执行服务发现和管理操作的HTTP接口。
命令行参数选项
-broadcast-address string
address of this lookupd node, (default to the OS hostname) (default "yourhost.local")
-config string
path to config file
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqlookupd] ")
-tcp-address string
<addr>:<port> to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
HTTP 接口
GET
/lookup
返回一个主题生产者列表
Params:
topic - 要列出生产者列表的主题
GET
/topics
返回一个所有已知主题的列表
GET
/channels
返回一个主题的所有已知通道列表
Params:
topic - 要列出通道列表的主题
GET
/nodes
返回一个所有已知的nsqd
列表
POST
/topic/create
注册一个主题到nsqlookupd
Params:
topic - 主题名称
POST
/topic/delete
删除一个存在的主题
Params:
topic - 要删除的主题
POST
/channel/create
注册一个通道到nsqlookupd
Params:
topic - 主题名
channel - 通道名
POST
/channel/delete
删除一个主题的一个通道
Params:
topic - 现存的一个主题
channel - 要删除的一个已存在的通道
POST
/topic/tombstone
tombstone 不知如何翻译,保持原词
tombstones 一个主题的特定生产者。 详见删除和tombstones。
Params:
topic - 现存的主题
node - 要tombstone(通过<broadcast_address>:<http_port>进行鉴别)的生产者(nsqd)
GET
/ping
监控端点,应当返回OK
GET
/info
返回版本信息
删除和tombstones
当一个主题不再全局生成时,从集群中清除该信息是相对简单的操作。假设所有正在生成消息的应用程序都已关闭,则使用nsqlookupd
实例的/delete_topic
端点是完成操作所必需的(内部,它将标识相关的nsqd
生产者并在那些节点上执行适当的操作)。
删除全局通道的过程是相似的,唯一的区别是在nsqlookupd
实例上使用/delete_channel
端点,并且需要确保所有订阅了该通道的消费者均已关闭。
然而,当在节点的子集上不再生成主题时,情况将变得更加复杂。由于消费者查询nsqlookupd
以及连接到所有生产者的方式,您会进入竞争状态并尝试从集群中移除信息,而消费者发现该节点并进行重连接(从而推动更新,该节点上仍会生成主题)。在这些情况下,解决方案是使用"tombstones
"。在nsqlookupd
上下文中tombstone
是特定于生产者的,并且持续时间可通过--tombstone-lifetime
配置。在该窗口期间,生产者将不会在/lookup
查询中列出,允许节点删除主题,并将该信息传播到nsqlookupd
(然后删除tombstoned
生产者),并防止任何消费者重新发现该节点。
nsqadmin
nsqadmin
是一个Web UI,用于实时查看集群的聚合统计信息并执行各种管理任务。
命令行参数选项
-acl-http-header string
HTTP header to check for authenticated admin users (default "X-Forwarded-User")
-admin-user value
admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)
-allow-config-from-cidr string
A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-base-path string
URL base path (default "/")
-config string
path to config file
-graphite-url string
graphite HTTP address
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-http-client-tls-cert string
path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
path to key file for the HTTP client
-http-client-tls-root-ca-file string
path to CA file for the HTTP client
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
nsqd HTTP address (may be given multiple times)
-proxy-graphite
proxy HTTP requests to graphite
-statsd-counter-format string
The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
statsd / Graphite 集成
当使用nsqd --statsd-address=...
时,你可以指定nsqadmin --graphite-url=https://graphite.yourdomain.com
在nsqadmin
中启用graphite图表。如果使用不带前缀键的statsd
克隆(如statsdaemon),则还要指定 --use-statsd-prefix=false
。
管理通知(Admin Notifications)
如果设置了--notification-http-endpoint
标志,nsqadmin
在每次执行管理操作(例如暂停通道)时,都会向指定(完全限定)的端点发送POST请求。
请求的正文包含有关操作的信息(适当时将省略某些字段),如下所示:
{
"action": "unpause_channel",
"channel": "mouth",
"topic": "beer",
"timestamp": 1357683731,
"user": "df",
"user_agent": "Mozilla/5.0 (Macintosh; Iphone 8)",
"remote_ip": "1.2.3.4:5678",
"url": "http://nsqadmin.local/api/topics/api_requests/nsq_to_file",
"via": "localhost"
}
如果对nsqdadmin
的请求中包含HTTP基本身份认证的用户名,那么user
字段将被填充。例如在http密码认证或者在oauth2_proxy
之下运行时。
提示:您可以通过设置--notification-http-endpoint
来创建主题名称为admin_actions
的NSQ管理员操作通知数据流来指向nsqd
HTTP Publish API。例如,如果nsqd
运行在同一节点上,你可以使用--notification-http-endpoint="http://127.0.0.1:4151/put?topic=admin_actions"
有社区贡献的实用程序nsqadmin2slack
和nsqadmin2hipchat
可以分别将消息中继到HipChat和Slack。
捕获的动作有:
create_channel
create_topic
delete_channel
delete_topic
empty_channel
empty_topic
pause_channel
pause_topic
tombstone_topic_producer
unpause_channel
unpause_topic
指标
下述指标通过nsqadmin
在主题、通道和客户端连接上进行公开。
消息队列
Depth
: 当前内存和磁盘中的消息总数 (即待发送的消息"积压")In-Flight
:当前已发送但尚未完成(FIN
)、重新排队(REQ
)或超时的消息数。Deferred
: 当前已重新排队和显式推迟但尚无法传递的消息数。
统计
Requeued
: 由于超时或显式重新排队而导致消息被添加回队列的总次数。Timed Out
: 在配置的超时之前以及未收到来自客户端的响应之后,消息重新排队的总时间。Messages
:自节点启动以来收到的新消息总数。Rate
: 过去两个统计间隔内新消息的每秒速率(仅在启用了graphite
集成时可用)。Connections
: 当前连接的客户端数。
客户端连接
Client Host
:客户端ID(主机名)以及悬停(on-hover
)连接的远程地址。Protocol
: NSQ协议版本以及客户端用户代理。Attributes
: TLS和AUTH连接状态。NSQd Host
: 客户端连接的nsqd节点地址。In-flight
: 当前已发送到此客户端的等待响应的消息数。Ready Count
: 此连接上可发送的最大消息数。由客户端的max_in_flight
设置控制。Finished
: 客户端已完成的消息总数(FIN
)。Requeued
: 客户端已重新排队的消息总数(REQ
)。Messages
: 发送到客户端的消息总数。
utilities
这些是有助于通用功能和对数据流进行省察的实用程序。
nsq_stat
对指定主题/通道的所有生产者进行轮询/stats
,并显示汇总统计信息
---------------depth---------------+--------------metadata---------------
total mem disk inflt def | req t-o msgs clients
24660 24660 0 0 20 | 102688 0 132492418 1
25001 25001 0 0 20 | 102688 0 132493086 1
21132 21132 0 0 21 | 102688 0 132493729 1
命令行选项
-channel string
NSQ channel
-count value
number of reports
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-interval duration
duration of time between polling/printing output (default 2s)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-nsqd-http-address value
nsqd HTTP address (may be given multiple times)
-topic string
NSQ topic
-version
print version
nsq_tail
消费特定的主题/通道并写入到标准输出(本着tail(1)
的精神)
命令行选项
-channel string
NSQ channel
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-n int
total messages to show (will wait if starved)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-print-topic
print topic name where message was received
-topic value
NSQ topic (may be given multiple times)
-version
print version string
nsq_to_file
消费指定的主题/通道并写入到换行符分隔的文件中,可以选择滚动 和/或 压缩文件。
命令行选项
-channel string
nsq channel (default "nsq_to_file")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-datetime-format string
strftime compatible format for <DATETIME> in filename format (default "%Y-%m-%d_%H")
-filename-format string
output filename format (<TOPIC>, <HOST>, <PID>, <DATETIME>, <REV> are replaced. <REV> is increased when file already exists) (default "<TOPIC>.<HOST><REV>.<DATETIME>.log")
-gzip
gzip output files.
-gzip-level int
gzip compression level (1-9, 1=BestSpeed, 9=BestCompression) (default 6)
-host-identifier string
value to output in log filename in place of hostname. <SHORT_HOST> and <HOSTNAME> are valid replacement tokens
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-log-level string
set log verbosity: debug, info, warn, error, or fatal (default "info")
-log-prefix string
log message prefix (default "[nsq_to_file] ")
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-output-dir string
directory to write output files to (default "/tmp")
-rotate-interval duration
rotate the file every duration
-rotate-size rotate-size
rotate the file when it grows bigger than rotate-size bytes
-skip-empty-files
skip writing empty files
-sync-interval duration
sync file to disk every duration (default 30s)
-topic value
nsq topic (may be given multiple times)
-topic-pattern string
only log topics matching the following pattern
-topic-refresh duration
how frequently the topic list should be refreshed (default 1m0s)
-version
print version string
-work-dir string
directory for in-progress files before moving to output-dir
nsq_to_http
消费指定的主题/通道,并向指定的端点执行HTTP请求(GET / POST)。
命令行选项
-channel string
nsq channel (default "nsq_to_http")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-content-type string
the Content-Type used for POST requests (default "application/octet-stream")
-get value
HTTP address to make a GET request to. '%s' will be printf replaced with data (may be given multiple times)
-header value
Custom header for HTTP requests (may be given multiple times)
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 20s)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-mode string
the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-n int
number of concurrent publishers (default 100)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-post value
HTTP address to make a POST request to. data will be in the body (may be given multiple times)
-sample float
% of messages to publish (float b/w 0 -> 1) (default 1)
-status-every int
the # of requests between logging status (per handler), 0 disables (default 250)
-topic string
nsq topic
-version
print version string
nsq_to_nsq
消费特定的主题/通道,并通过TCP将消息重新发布到目标nsqd
。
命令行选项
-channel string
nsq channel (default "nsq_to_nsq")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-destination-nsqd-tcp-address value
destination nsqd TCP address (may be given multiple times)
-destination-topic string
use this destination topic for all consumed topics (default is consumed topic name)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-mode string
the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-producer-opt value
option to passthrough to nsq.Producer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-require-json-field string
for JSON messages: only pass messages that contain this field
-require-json-value string
for JSON messages: only pass messages in which the required field has this value
-status-every int
the # of requests between logging status (per destination), 0 disables (default 250)
-topic value
nsq topic (may be given multiple times)
-version
print version string
-whitelist-json-field value
for JSON messages: pass this field (may be given multiple times)
to_nsq
接收标准输入流并通过换行符进行分割(默认),通过TCP重新发布到目标nsqd
。
命令行选项
-delimiter string
character to split input from stdin (default "\n")
-nsqd-tcp-address value
destination nsqd TCP address (may be given multiple times)
-producer-opt value
option to passthrough to nsq.Producer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-rate int
Throttle messages to n/second. 0 to disable
-topic string
NSQ topic to publish to