原文地址:nsq.io/components/…
组件
nsqd
nsqd 是接收消息,形成消息队列并将消息传递到客户端的守护进程。
它可以独立运行,但通常在具有nsqlookupd 实例的集群中配置(在这种情况下,它将声明主题和通道用以服务发现)。
它侦听两个TCP端口,一个侦听客户端,另一个侦听HTTP API。也可以选择在第三个端口上侦听HTTPS。
命令行参数选项
-auth-http-address value
<addr>:<port> to query auth server (may be given multiple times)
-broadcast-address string
address that will be registered with lookupd (defaults to the OS hostname) (default "yourhost.local")
-broadcast-http-port int
HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening to)
-broadcast-tcp-port int
TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening to)
-config string
path to config file
-data-path string
path to store disk-backed messages
-deflate
enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-https-address string
<addr>:<port> to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
lookupd TCP address (may be given multiple times)
-max-body-size int
maximum size of a single command body (default 5242880)
-max-bytes-per-file int
number of bytes per diskqueue file before rolling (default 104857600)
-max-channel-consumers int
maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)
-max-deflate-level int
max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
maximum client configurable duration of time between flushing to a client (default 30s)
-max-rdy-count int
maximum RDY count for a client (default 2500)
-max-req-timeout duration
maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
number of messages to keep in memory (per topic/channel) (default 10000)
-min-output-buffer-timeout duration
minimum client configurable duration of time between flushing to a client (default 25ms)
-msg-timeout duration
default duration to wait before auto-requeing a message (default 1m0s)
-node-id int
unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 248)
-output-buffer-timeout duration
default duration of time between flushing data to clients (default 250ms)
-snappy
enable snappy feature negotiation (client compression) (default true)
-statsd-address string
UDP <addr>:<port> of a statsd daemon for pushing stats
-statsd-interval duration
duration between pushing to statsd (default 1m0s)
-statsd-mem-stats
toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-statsd-udp-packet-size int
the size in bytes of statsd UDP packets (default 508)
-sync-every int
number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
duration of time per diskqueue fsync (default 2s)
-tcp-address string
<addr>:<port> to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
path to certificate file
-tls-client-auth-policy string
client certificate auth policy ('require' or 'require-verify')
-tls-key string
path to key file
-tls-min-version value
minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
path to certificate authority file
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
-worker-id
[deprecated] use --node-id
HTTP API
/ping-是否存活/info- 版本信息/stats-详尽的运行时遥测/pub-发布主题消息/mpub-将多个消息发布到一个主题/config-配置nsqd/debug/pprof-pprof调试门户/debug/pprof/profile-生成pprof CPU配置文件/debug/pprof/goroutine-生成pprof goroutine配置文件/debug/pprof/heap-生成pprof堆配置文件/debug/pprof/block-生成pprof block配置文件/debug/pprof/threadcreate-生成pprof OS线程配置文件
v1命名空间(自nsqd v0.2.29+):
-
/topic/create-创建一个新主题 -
/topic/delete-删除主题 -
/topic/empty-清空主题 -
/topic/pause-暂停主题的消息流 -
/topic/unpause-取消暂停的主题消息流 -
/channel/create-建立新通道 -
/channel/delete-删除通道 -
/channel/empty-清空通道 -
/channel/pause-暂停通道的消息流 -
/channel/unpause-取消暂停的通道消息流 -
POST/pub发布一个消息:
Query Params:
topic - 要发布的主题 defer - 消息延迟发送的毫秒数(可选参数)Body:
原始消息字节Example:
$ curl -d "<message>" http://127.0.0.1:4151/pub?topic=name -
POST/mpub一次往返发布多条消息
Query Params:
topic - 要发布的主题 binary - 布尔值 ('true' or 'false'),是否开启二进制模式Body
\n 分割的原始消息字节注意:默认情况下,
/mpub期望用\n来分隔消息,使用?binary=truequery参数来启用二进制模式,其中POST body应采用以下格式(发送的HTTP header中Content-Length应为POSTbody的总大小):[ 4-byte num messages ] [ 4-byte message #1 size ][ N-byte binary data ] ... (repeated <num_messages> times)Example:
$ curl -d "<message>\n<message>\n<message>" http://127.0.0.1:4151/mpub?topic=name -
POST/topic/create创建一个主题
Query Params:
topic - 要创建的主题Example:
$ curl -X POST http://127.0.0.1:4151/topic/create?topic=name -
POST/topic/delete删除一个现存的主题(以及所有的通道)
Query Params:
topic - 要删除的主题Example:
$ curl -X POST http://127.0.0.1:4151/topic/delete?topic=name -
POST/channel/create给现存的主题创建一个通道
Query Params:
topic - 现存的主题 channel - 要创建的通道Example:
$ curl -X POST http://127.0.0.1:4151/channel/create?topic=name&channel=name -
POST/channel/delete删除一个现有主题的一个通道
Query Params:
topic - 现存的主题 channel - 要删除的通道Example:
$ curl -X POST http://127.0.0.1:4151/channel/delete?topic=name&channel=name -
POST/topic/empty清空一个现有主题的所有队列消息(内存和磁盘中的)
Query Params:
topic - 要清空的现有主题Example:
$ curl -X POST http://127.0.0.1:4151/topic/empty?topic=name -
POST/channel/empty清空一个现有通道的所有队列消息(内存和磁盘中的)
Query Params:
topic - 现有主题 channel - 要清空的通道Example:
$ curl -X POST http://127.0.0.1:4151/channel/empty?topic=name&channel=name -
POST/topic/pause暂停一个主题所有通道的消息流(消息将以主题进行排队)
Query Params:
topic - 现有主题Example:
$ curl -X POST http://127.0.0.1:4151/topic/pause?topic=name -
POST/topic/unpause恢复现有已暂停主题的通道消息流
Query Params:
topic -现有主题Example:
$ curl -X POST http://127.0.0.1:4151/topic/unpause?topic=name -
POST/channel/pause暂停发往消费者的通道消息流(消息将以通道进行排队)
Query Params:
topic - 现有主题 channel - 要暂停的通道Example:
$ curl -X POST http://127.0.0.1:4151/channel/pause?topic=name&channel=name -
POST/channel/unpause恢复发往消费者的已暂停的通道消息流
Query Params:
topic - 现有主题 channel - 要恢复的通道Example:
$ curl -X POST http://127.0.0.1:4151/channel/unpause?topic=name&channel=name -
GET/stats返回内部统计
Query Params
format - (可选) `text` or `json` (default = `text`) topic - (可选) 筛选主题 channel - (可选) 筛选通道Example:
$ curl http://127.0.0.1:4151/stats -
GET/ping监控端点,应该返回
200 OK。如果运行不健康,返回HTTP 500。**注意:**唯一的"unhealthy"状态是在发生溢出时未能将消息写入磁盘。
-
GET/info版本信息
-
GET/debug/pprof可用的调试端点索引页。
-
GET/debug/pprof/profile启动
pprofCPU配置文件30秒钟,并通过请求返回输出注意:此端点未在
/debug/pprof索引页面中列出,因为它会影响运行时性能。 -
GET/debug/pprof/goroutine返回所有运行中的
goroutines的栈追踪。 -
GET/debug/pprof/heap返回堆和内存统计配置信息(顶部可以用作
pprof内存配置信息) -
GET/debug/pprof/block返回一个
goroutine blocking配置信息 -
GET/debug/pprof/threadcreate返回创建OS线程的
goroutine栈跟踪 -
GET/config/nsqlookupd_tcp_addressesnsqlookupd TCP地址列表
Example:
$ curl http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses -
PUT/config/nsqlookupd_tcp_addresses更新nsqlookupd TCP地址。
Body:
JSON array of TCP addresses.Example:
$ curl -X PUT http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses \ -d '["127.0.0.1:4160", "127.0.0.2:4160"]'
调试和分析
nsqd提供了一套可与Go的pprof 工具直接集成的性能分析端点。如果您安装了go工具套件,只需运行:
# memory profiling
$ go tool pprof http://localhost:4151/debug/pprof/heap
# cpu profiling
$ go tool pprof http://localhost:4151/debug/pprof/profile
TLS
当nsqd配置了--tls-cert与--tls-key,客户端可以协商升级他们的TLS连接以增强安全性。
此外,您可以 使用--tls-required要求客户端协商TLS(自nsqd v0.2.28+)。
您可以通过--tls-client-auth-policy(require或 require-verify)配置nsqd客户端证书策略:
require-客户必须提供证书,否则被拒绝require-verify-客户必须根据默认的CA或指定的链--tls-root-ca-file提供有效的证书,否则将被拒绝
这可以用作客户端身份验证的一种形式(自nsqd v0.2.28+)。
如果要使用openssl生成无密码的自签名证书:
$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes
AUTH
注意: nsqd v0.2.29+可用
要配置nsqd为需要授权,您需要使用-auth-http-address=host:port 来指定一个符合Auth HTTP协议的Auth Server。
注意:当使用授权时,期望只有nsqdTCP协议公开给外部客户端,而不是HTTP(S)端点。请参阅以下有关使用auth向客户端公开统计信息和查找的注释。
身份验证服务器必须接受以下HTTP请求:
/auth?remote_ip=...&tls=...&auth_secret=...
然后返回以下格式的响应:
{
"ttl": 3600,
"identity": "username",
"identity_url": "https://....",
"authorizations": [
{
"permissions": [
"subscribe",
"publish"
],
"topic": ".*",
"channels": [
".*"
]
}
]
}
请注意,主题和频道字符串必须是正则表达式nsqd才能应用权限。nsqd将在TTL持续时间内缓存响应,并在该时间间隔内重新请求授权。
预期在大多数情况下,TLS将使用授权来保障从客户端传递到nsqd的敏感信息加密传递。nsqd和Auth Server之间的通信应该通过受信任的网络进行(并且不会进行加密)。如果Auth Server仅基于远程IP信息进行身份验证选择,则客户端可以使用占位符字符串(如.)来作为AUTH命令主体(body)。
一个Auth Server示例是pynsqauthd。
可以在nsqauthfilter中找到一个帮助服务器,该服务器向客户端公开了通过Auth Server权限过滤的nsqlookupd和nsqd /stats数据。
使用命令行实用程序时,可以通过--reader-opt标志使用授权。
$ nsq_tail ... -reader-opt="tls_v1,true" -reader-opt="auth_secret,$SECRET"
端到端处理延迟
您可以选择使用--e2e-processing-latency-percentile标志配置nsqd来收集和发起可配置百分比的端到端消息处理延迟。
使用*《数据流上有偏分位数的有效计算》*所述的概率百分位技术来计算值。我们使用 bmizerany的perks包,这个包实现了该算法。
为了使视图偏向于最近的处理行为,我们仅保留了过去N分钟的分位数信息(可通过--e2e-processing-latency-window-time进行配置)。在内部,我们每个通道维护两个分位数,每个分位数存储N/2分钟的延迟数据。每隔一 N/2分钟我们就会重置一个分位数(并开始向其中插入新数据)。由于分位数可以合并,因此会导致滚动窗口变大。
由于我们仅在通道层级收集数据,因此对于一个主题,我们汇总并合并了所有通道分位数。仅当数据在同一nsqd实例上时才能使用此技术。但是,当跨nsqd(例如通过nsqlookupd)累积数据时,我们取每个nsqd分位数的平均值。考虑到跨nsqd的延迟分布,为了保持某些统计的准确性,除了平均值以外,我们还提供了最小/最大值。
注意:如果没有消费者连接,则尽管(很明显)排队消息的端到端处理时间在逐渐增加,但值仍然无法更新。这是因为对于来自客户端给定的消息,只有当nsqd接收了FIN信号时,才会计算端到端的指标。当消费者重新连接时,这些值将适当调整。
Statsd / Graphite集成
当使用--statsd-address为 statsd(或如statsdemon之类的一个statsd端口)指定UDP <addr>:<port>时,nsqd将根据--statsd-interval指定的间隔(重要:此间隔应 始终小于或等于statsd刷新到graphite的间隔)定期向statsd推送指标。启用此功能后,可以配置nsqadmin直接显示graphite图表。
我们建议使用以下graphite配置(但应根据您的可用资源和要求评估这些选择)。同样,要记住的重要一点是statsd的刷新间隔应小于或等于storage-schemas.conf中的最小时间段, 并且nsqd应配置刷新时间等于或小于--statsd-interval指定的间隔。
# storage-schemas.conf
[nsq]
pattern = ^nsq\..*
retentions = 1m:1d,5m:30d,15m:1y
# storage-aggregation.conf
[default_nsq]
pattern = ^nsq\..*
xFilesFactor = 0.2
aggregationMethod = average
nsqd实例将推送到如下statsd路径:
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_bytes
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.clients [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.deferred_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.in_flight_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.requeue_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.timeout_count
# if --statsd-mem-stats is enabled
nsq.<nsqd_host>_<nsqd_port>.mem.heap_objects [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_idle_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_in_use_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_released_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_100 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_99 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_95 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.mem.next_gc_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_runs
# if --e2e-processing-latency-percentile is specified, for each percentile
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.e2e_processing_latency_<percent> [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.e2e_processing_latency_<percent> [gauge]
nsqlookupd
nsqlookupd是管理拓扑信息的守护程序。客户端通过nsqlookupd进行查询以发现特定主题的nsqd生产者,同时nsqd节点广播主题和通道信息。
有两个接口:一个nsqd用于广播的TCP接口和一个客户端用于执行服务发现和管理操作的HTTP接口。
命令行参数选项
-broadcast-address string
address of this lookupd node, (default to the OS hostname) (default "yourhost.local")
-config string
path to config file
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqlookupd] ")
-tcp-address string
<addr>:<port> to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
HTTP 接口
GET /lookup
返回一个主题生产者列表
Params:
topic - 要列出生产者列表的主题
GET /topics
返回一个所有已知主题的列表
GET /channels
返回一个主题的所有已知通道列表
Params:
topic - 要列出通道列表的主题
GET /nodes
返回一个所有已知的nsqd列表
POST /topic/create
注册一个主题到nsqlookupd
Params:
topic - 主题名称
POST /topic/delete
删除一个存在的主题
Params:
topic - 要删除的主题
POST /channel/create
注册一个通道到nsqlookupd
Params:
topic - 主题名
channel - 通道名
POST /channel/delete
删除一个主题的一个通道
Params:
topic - 现存的一个主题
channel - 要删除的一个已存在的通道
POST /topic/tombstone
tombstone 不知如何翻译,保持原词
tombstones 一个主题的特定生产者。 详见删除和tombstones。
Params:
topic - 现存的主题
node - 要tombstone(通过<broadcast_address>:<http_port>进行鉴别)的生产者(nsqd)
GET /ping
监控端点,应当返回OK
GET /info
返回版本信息
删除和tombstones
当一个主题不再全局生成时,从集群中清除该信息是相对简单的操作。假设所有正在生成消息的应用程序都已关闭,则使用nsqlookupd实例的/delete_topic端点是完成操作所必需的(内部,它将标识相关的nsqd生产者并在那些节点上执行适当的操作)。
删除全局通道的过程是相似的,唯一的区别是在nsqlookupd实例上使用/delete_channel端点,并且需要确保所有订阅了该通道的消费者均已关闭。
然而,当在节点的子集上不再生成主题时,情况将变得更加复杂。由于消费者查询nsqlookupd以及连接到所有生产者的方式,您会进入竞争状态并尝试从集群中移除信息,而消费者发现该节点并进行重连接(从而推动更新,该节点上仍会生成主题)。在这些情况下,解决方案是使用"tombstones"。在nsqlookupd上下文中tombstone是特定于生产者的,并且持续时间可通过--tombstone-lifetime配置。在该窗口期间,生产者将不会在/lookup查询中列出,允许节点删除主题,并将该信息传播到nsqlookupd(然后删除tombstoned生产者),并防止任何消费者重新发现该节点。
nsqadmin
nsqadmin 是一个Web UI,用于实时查看集群的聚合统计信息并执行各种管理任务。
命令行参数选项
-acl-http-header string
HTTP header to check for authenticated admin users (default "X-Forwarded-User")
-admin-user value
admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)
-allow-config-from-cidr string
A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-base-path string
URL base path (default "/")
-config string
path to config file
-graphite-url string
graphite HTTP address
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-http-client-tls-cert string
path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
path to key file for the HTTP client
-http-client-tls-root-ca-file string
path to CA file for the HTTP client
-log-level value
set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
nsqd HTTP address (may be given multiple times)
-proxy-graphite
proxy HTTP requests to graphite
-statsd-counter-format string
The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-verbose
[deprecated] has no effect, use --log-level
-version
print version string
statsd / Graphite 集成
当使用nsqd --statsd-address=... 时,你可以指定nsqadmin --graphite-url=https://graphite.yourdomain.com在nsqadmin中启用graphite图表。如果使用不带前缀键的statsd克隆(如statsdaemon),则还要指定 --use-statsd-prefix=false。
管理通知(Admin Notifications)
如果设置了--notification-http-endpoint标志,nsqadmin在每次执行管理操作(例如暂停通道)时,都会向指定(完全限定)的端点发送POST请求。
请求的正文包含有关操作的信息(适当时将省略某些字段),如下所示:
{
"action": "unpause_channel",
"channel": "mouth",
"topic": "beer",
"timestamp": 1357683731,
"user": "df",
"user_agent": "Mozilla/5.0 (Macintosh; Iphone 8)",
"remote_ip": "1.2.3.4:5678",
"url": "http://nsqadmin.local/api/topics/api_requests/nsq_to_file",
"via": "localhost"
}
如果对nsqdadmin的请求中包含HTTP基本身份认证的用户名,那么user字段将被填充。例如在http密码认证或者在oauth2_proxy之下运行时。
提示:您可以通过设置--notification-http-endpoint来创建主题名称为admin_actions的NSQ管理员操作通知数据流来指向nsqd HTTP Publish API。例如,如果nsqd运行在同一节点上,你可以使用--notification-http-endpoint="http://127.0.0.1:4151/put?topic=admin_actions"
有社区贡献的实用程序nsqadmin2slack和nsqadmin2hipchat可以分别将消息中继到HipChat和Slack。
捕获的动作有:
create_channelcreate_topicdelete_channeldelete_topicempty_channelempty_topicpause_channelpause_topictombstone_topic_producerunpause_channelunpause_topic
指标
下述指标通过nsqadmin在主题、通道和客户端连接上进行公开。
消息队列
Depth: 当前内存和磁盘中的消息总数 (即待发送的消息"积压")In-Flight:当前已发送但尚未完成(FIN)、重新排队(REQ)或超时的消息数。Deferred: 当前已重新排队和显式推迟但尚无法传递的消息数。
统计
Requeued: 由于超时或显式重新排队而导致消息被添加回队列的总次数。Timed Out: 在配置的超时之前以及未收到来自客户端的响应之后,消息重新排队的总时间。Messages:自节点启动以来收到的新消息总数。Rate: 过去两个统计间隔内新消息的每秒速率(仅在启用了graphite集成时可用)。Connections: 当前连接的客户端数。
客户端连接
Client Host:客户端ID(主机名)以及悬停(on-hover)连接的远程地址。Protocol: NSQ协议版本以及客户端用户代理。Attributes: TLS和AUTH连接状态。NSQd Host: 客户端连接的nsqd节点地址。In-flight: 当前已发送到此客户端的等待响应的消息数。Ready Count: 此连接上可发送的最大消息数。由客户端的max_in_flight设置控制。Finished: 客户端已完成的消息总数(FIN)。Requeued: 客户端已重新排队的消息总数(REQ)。Messages: 发送到客户端的消息总数。
utilities
这些是有助于通用功能和对数据流进行省察的实用程序。
nsq_stat
对指定主题/通道的所有生产者进行轮询/stats,并显示汇总统计信息
---------------depth---------------+--------------metadata---------------
total mem disk inflt def | req t-o msgs clients
24660 24660 0 0 20 | 102688 0 132492418 1
25001 25001 0 0 20 | 102688 0 132493086 1
21132 21132 0 0 21 | 102688 0 132493729 1
命令行选项
-channel string
NSQ channel
-count value
number of reports
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-interval duration
duration of time between polling/printing output (default 2s)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-nsqd-http-address value
nsqd HTTP address (may be given multiple times)
-topic string
NSQ topic
-version
print version
nsq_tail
消费特定的主题/通道并写入到标准输出(本着tail(1)的精神)
命令行选项
-channel string
NSQ channel
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-n int
total messages to show (will wait if starved)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-print-topic
print topic name where message was received
-topic value
NSQ topic (may be given multiple times)
-version
print version string
nsq_to_file
消费指定的主题/通道并写入到换行符分隔的文件中,可以选择滚动 和/或 压缩文件。
命令行选项
-channel string
nsq channel (default "nsq_to_file")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-datetime-format string
strftime compatible format for <DATETIME> in filename format (default "%Y-%m-%d_%H")
-filename-format string
output filename format (<TOPIC>, <HOST>, <PID>, <DATETIME>, <REV> are replaced. <REV> is increased when file already exists) (default "<TOPIC>.<HOST><REV>.<DATETIME>.log")
-gzip
gzip output files.
-gzip-level int
gzip compression level (1-9, 1=BestSpeed, 9=BestCompression) (default 6)
-host-identifier string
value to output in log filename in place of hostname. <SHORT_HOST> and <HOSTNAME> are valid replacement tokens
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 5s)
-log-level string
set log verbosity: debug, info, warn, error, or fatal (default "info")
-log-prefix string
log message prefix (default "[nsq_to_file] ")
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-output-dir string
directory to write output files to (default "/tmp")
-rotate-interval duration
rotate the file every duration
-rotate-size rotate-size
rotate the file when it grows bigger than rotate-size bytes
-skip-empty-files
skip writing empty files
-sync-interval duration
sync file to disk every duration (default 30s)
-topic value
nsq topic (may be given multiple times)
-topic-pattern string
only log topics matching the following pattern
-topic-refresh duration
how frequently the topic list should be refreshed (default 1m0s)
-version
print version string
-work-dir string
directory for in-progress files before moving to output-dir
nsq_to_http
消费指定的主题/通道,并向指定的端点执行HTTP请求(GET / POST)。
命令行选项
-channel string
nsq channel (default "nsq_to_http")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-content-type string
the Content-Type used for POST requests (default "application/octet-stream")
-get value
HTTP address to make a GET request to. '%s' will be printf replaced with data (may be given multiple times)
-header value
Custom header for HTTP requests (may be given multiple times)
-http-client-connect-timeout duration
timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
timeout for HTTP request (default 20s)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-mode string
the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-n int
number of concurrent publishers (default 100)
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-post value
HTTP address to make a POST request to. data will be in the body (may be given multiple times)
-sample float
% of messages to publish (float b/w 0 -> 1) (default 1)
-status-every int
the # of requests between logging status (per handler), 0 disables (default 250)
-topic string
nsq topic
-version
print version string
nsq_to_nsq
消费特定的主题/通道,并通过TCP将消息重新发布到目标nsqd。
命令行选项
-channel string
nsq channel (default "nsq_to_nsq")
-consumer-opt value
option to passthrough to nsq.Consumer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-destination-nsqd-tcp-address value
destination nsqd TCP address (may be given multiple times)
-destination-topic string
use this destination topic for all consumed topics (default is consumed topic name)
-lookupd-http-address value
lookupd HTTP address (may be given multiple times)
-max-in-flight int
max number of messages to allow in flight (default 200)
-mode string
the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-producer-opt value
option to passthrough to nsq.Producer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-require-json-field string
for JSON messages: only pass messages that contain this field
-require-json-value string
for JSON messages: only pass messages in which the required field has this value
-status-every int
the # of requests between logging status (per destination), 0 disables (default 250)
-topic value
nsq topic (may be given multiple times)
-version
print version string
-whitelist-json-field value
for JSON messages: pass this field (may be given multiple times)
to_nsq
接收标准输入流并通过换行符进行分割(默认),通过TCP重新发布到目标nsqd。
命令行选项
-delimiter string
character to split input from stdin (default "\n")
-nsqd-tcp-address value
destination nsqd TCP address (may be given multiple times)
-producer-opt value
option to passthrough to nsq.Producer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-rate int
Throttle messages to n/second. 0 to disable
-topic string
NSQ topic to publish to