NSQ之组件

1,401 阅读19分钟

原文地址:nsq.io/components/…

组件

nsqd

nsqd 是接收消息,形成消息队列并将消息传递到客户端的守护进程。

它可以独立运行,但通常在具有nsqlookupd 实例的集群中配置(在这种情况下,它将声明主题和通道用以服务发现)。

它侦听两个TCP端口,一个侦听客户端,另一个侦听HTTP API。也可以选择在第三个端口上侦听HTTPS。

命令行参数选项

-auth-http-address value
    <addr>:<port> to query auth server (may be given multiple times)
-broadcast-address string
    address that will be registered with lookupd (defaults to the OS hostname) (default "yourhost.local")
-broadcast-http-port int
    HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening to)
-broadcast-tcp-port int
    TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening to)
-config string
    path to config file
-data-path string
    path to store disk-backed messages
-deflate
    enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
    message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
    calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-https-address string
    <addr>:<port> to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
    lookupd TCP address (may be given multiple times)
-max-body-size int
    maximum size of a single command body (default 5242880)
-max-bytes-per-file int
    number of bytes per diskqueue file before rolling (default 104857600)
-max-channel-consumers int
    maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)
-max-deflate-level int
    max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
    maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
    maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
    maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
    maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
    maximum client configurable duration of time between flushing to a client (default 30s)
-max-rdy-count int
    maximum RDY count for a client (default 2500)
-max-req-timeout duration
    maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
    number of messages to keep in memory (per topic/channel) (default 10000)
-min-output-buffer-timeout duration
    minimum client configurable duration of time between flushing to a client (default 25ms)
-msg-timeout duration
    default duration to wait before auto-requeing a message (default 1m0s)
-node-id int
    unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 248)
-output-buffer-timeout duration
    default duration of time between flushing data to clients (default 250ms)
-snappy
    enable snappy feature negotiation (client compression) (default true)
-statsd-address string
    UDP <addr>:<port> of a statsd daemon for pushing stats
-statsd-interval duration
    duration between pushing to statsd (default 1m0s)
-statsd-mem-stats
    toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-statsd-udp-packet-size int
    the size in bytes of statsd UDP packets (default 508)
-sync-every int
    number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
    duration of time per diskqueue fsync (default 2s)
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
    path to certificate file
-tls-client-auth-policy string
    client certificate auth policy ('require' or 'require-verify')
-tls-key string
    path to key file
-tls-min-version value
    minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
    require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
    path to certificate authority file
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string
-worker-id
    [deprecated] use --node-id

HTTP API

v1命名空间(自nsqd v0.2.29+):

  • /topic/create -创建一个新主题

  • /topic/delete -删除主题

  • /topic/empty -清空主题

  • /topic/pause -暂停主题的消息流

  • /topic/unpause -取消暂停的主题消息流

  • /channel/create -建立新通道

  • /channel/delete -删除通道

  • /channel/empty -清空通道

  • /channel/pause -暂停通道的消息流

  • /channel/unpause -取消暂停的通道消息流

  • POST /pub

    发布一个消息:

    Query Params:

    topic - 要发布的主题
    defer - 消息延迟发送的毫秒数(可选参数)
    

    Body:

    原始消息字节
    

    Example:

    $ curl -d "<message>" http://127.0.0.1:4151/pub?topic=name
    
  • POST /mpub

    一次往返发布多条消息

    Query Params:

    topic - 要发布的主题
    binary - 布尔值 ('true' or 'false'),是否开启二进制模式
    

    Body

    \n 分割的原始消息字节
    

    注意:默认情况下,/mpub期望用\n来分隔消息,使用?binary=truequery参数来启用二进制模式,其中POST body应采用以下格式(发送的HTTP header中Content-Length应为POST body的总大小):

    [ 4-byte num messages ]
    [ 4-byte message #1 size ][ N-byte binary data ]
          ... (repeated <num_messages> times)
    

    Example:

    $ curl -d "<message>\n<message>\n<message>" http://127.0.0.1:4151/mpub?topic=name
    
  • POST /topic/create

    创建一个主题

    Query Params:

    topic - 要创建的主题
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/topic/create?topic=name
    
  • POST /topic/delete

    删除一个现存的主题(以及所有的通道)

    Query Params:

    topic - 要删除的主题
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/topic/delete?topic=name
    
  • POST /channel/create

    给现存的主题创建一个通道

    Query Params:

    topic - 现存的主题
    channel - 要创建的通道
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/channel/create?topic=name&channel=name
    
  • POST /channel/delete

    删除一个现有主题的一个通道

    Query Params:

    topic - 现存的主题
    channel - 要删除的通道
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/channel/delete?topic=name&channel=name
    
  • POST /topic/empty

    清空一个现有主题的所有队列消息(内存和磁盘中的)

    Query Params:

    topic - 要清空的现有主题
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/topic/empty?topic=name
    
  • POST /channel/empty

    清空一个现有通道的所有队列消息(内存和磁盘中的)

    Query Params:

    topic - 现有主题
    channel - 要清空的通道
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/channel/empty?topic=name&channel=name
    
  • POST /topic/pause

    暂停一个主题所有通道的消息流(消息将以主题进行排队)

    Query Params:

    topic - 现有主题
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/topic/pause?topic=name
    
  • POST /topic/unpause

    恢复现有已暂停主题的通道消息流

    Query Params:

    topic -现有主题
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/topic/unpause?topic=name
    
  • POST /channel/pause

    暂停发往消费者的通道消息流(消息将以通道进行排队)

    Query Params:

    topic - 现有主题
    channel - 要暂停的通道
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/channel/pause?topic=name&channel=name
    
  • POST /channel/unpause

    恢复发往消费者的已暂停的通道消息流

    Query Params:

    topic - 现有主题
    channel - 要恢复的通道
    

    Example:

    $ curl -X POST http://127.0.0.1:4151/channel/unpause?topic=name&channel=name
    
  • GET /stats

    返回内部统计

    Query Params

    format - (可选) `text` or `json` (default = `text`)
    topic - (可选) 筛选主题
    channel - (可选) 筛选通道
    

    Example:

    $ curl http://127.0.0.1:4151/stats
    
  • GET /ping

    监控端点,应该返回200 OK。如果运行不健康,返回HTTP 500。

    **注意:**唯一的"unhealthy"状态是在发生溢出时未能将消息写入磁盘。

  • GET /info

    版本信息

  • GET /debug/pprof

    可用的调试端点索引页。

  • GET /debug/pprof/profile

    启动pprofCPU配置文件30秒钟,并通过请求返回输出

    注意:此端点未在/debug/pprof索引页面中列出,因为它会影响运行时性能。

  • GET /debug/pprof/goroutine

    返回所有运行中的goroutines的栈追踪。

  • GET /debug/pprof/heap

    返回堆和内存统计配置信息(顶部可以用作pprof内存配置信息)

  • GET /debug/pprof/block

    返回一个goroutine blocking 配置信息

  • GET /debug/pprof/threadcreate

    返回创建OS线程的goroutine栈跟踪

  • GET /config/nsqlookupd_tcp_addresses

    nsqlookupd TCP地址列表

    Example:

    $ curl http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses
    
  • PUT /config/nsqlookupd_tcp_addresses

    更新nsqlookupd TCP地址。

    Body:

    JSON array of TCP addresses.
    

    Example:

    $ curl -X PUT http://127.0.0.1:4151/config/nsqlookupd_tcp_addresses \
        -d '["127.0.0.1:4160", "127.0.0.2:4160"]'
    

调试和分析

nsqd提供了一套可与Go的pprof 工具直接集成的性能分析端点。如果您安装了go工具套件,只需运行:

# memory profiling
$ go tool pprof http://localhost:4151/debug/pprof/heap

# cpu profiling
$ go tool pprof http://localhost:4151/debug/pprof/profile

TLS

nsqd配置了--tls-cert--tls-key,客户端可以协商升级他们的TLS连接以增强安全性。

此外,您可以 使用--tls-required要求客户端协商TLS(自nsqd v0.2.28+)。

您可以通过--tls-client-auth-policyrequirerequire-verify)配置nsqd客户端证书策略:

  • require -客户必须提供证书,否则被拒绝
  • require-verify-客户必须根据默认的CA或指定的链--tls-root-ca-file提供有效的证书,否则将被拒绝

这可以用作客户端身份验证的一种形式(自nsqd v0.2.28+)。

如果要使用openssl生成无密码的自签名证书:

$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes

AUTH

注意: nsqd v0.2.29+可用

要配置nsqd为需要授权,您需要使用-auth-http-address=host:port 来指定一个符合Auth HTTP协议的Auth Server。

注意:当使用授权时,期望只有nsqdTCP协议公开给外部客户端,而不是HTTP(S)端点。请参阅以下有关使用auth向客户端公开统计信息和查找的注释。

身份验证服务器必须接受以下HTTP请求:

/auth?remote_ip=...&tls=...&auth_secret=...

然后返回以下格式的响应:

{
  "ttl": 3600,
  "identity": "username",
  "identity_url": "https://....",
  "authorizations": [
    {
      "permissions": [
        "subscribe",
        "publish"
      ],
      "topic": ".*",
      "channels": [
        ".*"
      ]
    }
  ]
}

请注意,主题和频道字符串必须是正则表达式nsqd才能应用权限。nsqd将在TTL持续时间内缓存响应,并在该时间间隔内重新请求授权。

预期在大多数情况下,TLS将使用授权来保障从客户端传递到nsqd的敏感信息加密传递。nsqd和Auth Server之间的通信应该通过受信任的网络进行(并且不会进行加密)。如果Auth Server仅基于远程IP信息进行身份验证选择,则客户端可以使用占位符字符串(如.)来作为AUTH命令主体(body)。

一个Auth Server示例是pynsqauthd

可以在nsqauthfilter中找到一个帮助服务器,该服务器向客户端公开了通过Auth Server权限过滤的nsqlookupdnsqd /stats数据。

使用命令行实用程序时,可以通过--reader-opt标志使用授权。

$ nsq_tail ... -reader-opt="tls_v1,true" -reader-opt="auth_secret,$SECRET"

端到端处理延迟

您可以选择使用--e2e-processing-latency-percentile标志配置nsqd来收集和发起可配置百分比的端到端消息处理延迟。

使用*《数据流上有偏分位数的有效计算》*所述的概率百分位技术来计算值。我们使用 bmizeranyperks包,这个包实现了该算法。

为了使视图偏向于最近的处理行为,我们仅保留了过去N分钟的分位数信息(可通过--e2e-processing-latency-window-time进行配置)。在内部,我们每个通道维护两个分位数,每个分位数存储N/2分钟的延迟数据。每隔一 N/2分钟我们就会重置一个分位数(并开始向其中插入新数据)。由于分位数可以合并,因此会导致滚动窗口变大。

由于我们仅在通道层级收集数据,因此对于一个主题,我们汇总并合并了所有通道分位数。仅当数据在同一nsqd实例上时才能使用此技术。但是,当跨nsqd(例如通过nsqlookupd)累积数据时,我们取每个nsqd分位数的平均值。考虑到跨nsqd的延迟分布,为了保持某些统计的准确性,除了平均值以外,我们还提供了最小/最大值。

注意:如果没有消费者连接,则尽管(很明显)排队消息的端到端处理时间在逐渐增加,但值仍然无法更新。这是因为对于来自客户端给定的消息,只有当nsqd接收了FIN信号时,才会计算端到端的指标。当消费者重新连接时,这些值将适当调整。

Statsd / Graphite集成

当使用--statsd-addressstatsd(或如statsdemon之类的一个statsd端口)指定UDP <addr>:<port>时,nsqd将根据--statsd-interval指定的间隔(重要:此间隔应 始终小于或等于statsd刷新到graphite的间隔)定期向statsd推送指标。启用此功能后,可以配置nsqadmin直接显示graphite图表。

我们建议使用以下graphite配置(但应根据您的可用资源和要求评估这些选择)。同样,要记住的重要一点是statsd的刷新间隔应小于或等于storage-schemas.conf中的最小时间段, 并且nsqd应配置刷新时间等于或小于--statsd-interval指定的间隔。

# storage-schemas.conf
[nsq]
pattern = ^nsq\..*
retentions = 1m:1d,5m:30d,15m:1y

# storage-aggregation.conf
[default_nsq]
pattern = ^nsq\..*
xFilesFactor = 0.2
aggregationMethod = average

nsqd实例将推送到如下statsd路径:

nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.message_bytes
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.backend_depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.clients [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.deferred_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.depth [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.in_flight_count [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.message_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.requeue_count
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.timeout_count

# if --statsd-mem-stats is enabled
nsq.<nsqd_host>_<nsqd_port>.mem.heap_objects [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_idle_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_in_use_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.heap_released_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_100 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_99 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_pause_usec_95 [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.mem.next_gc_bytes [gauge]
nsq.<nsqd_host>_<nsqd_port>.mem.gc_runs

# if --e2e-processing-latency-percentile is specified, for each percentile
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.e2e_processing_latency_<percent> [gauge]
nsq.<nsqd_host>_<nsqd_port>.topic.<topic_name>.channel.<channel_name>.e2e_processing_latency_<percent> [gauge]

nsqlookupd

nsqlookupd是管理拓扑信息的守护程序。客户端通过nsqlookupd进行查询以发现特定主题的nsqd生产者,同时nsqd节点广播主题和通道信息。

有两个接口:一个nsqd用于广播的TCP接口和一个客户端用于执行服务发现和管理操作的HTTP接口。

命令行参数选项

-broadcast-address string
    address of this lookupd node, (default to the OS hostname) (default "yourhost.local")
-config string
    path to config file
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
    duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqlookupd] ")
-tcp-address string
    <addr>:<port> to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
    duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

HTTP 接口

GET /lookup

返回一个主题生产者列表

Params:

topic - 要列出生产者列表的主题

GET /topics

返回一个所有已知主题的列表

GET /channels

返回一个主题的所有已知通道列表

Params:

topic - 要列出通道列表的主题

GET /nodes

返回一个所有已知的nsqd列表

POST /topic/create

注册一个主题到nsqlookupd

Params:

topic - 主题名称

POST /topic/delete

删除一个存在的主题

Params:

topic - 要删除的主题

POST /channel/create

注册一个通道到nsqlookupd

Params:

topic - 主题名
channel - 通道名

POST /channel/delete

删除一个主题的一个通道

Params:

topic - 现存的一个主题
channel - 要删除的一个已存在的通道

POST /topic/tombstone

tombstone 不知如何翻译,保持原词

tombstones 一个主题的特定生产者。 详见删除和tombstones

Params:

topic - 现存的主题
node - 要tombstone(通过<broadcast_address>:<http_port>进行鉴别)的生产者(nsqd)

GET /ping

监控端点,应当返回OK

GET /info

返回版本信息

删除和tombstones

当一个主题不再全局生成时,从集群中清除该信息是相对简单的操作。假设所有正在生成消息的应用程序都已关闭,则使用nsqlookupd实例的/delete_topic端点是完成操作所必需的(内部,它将标识相关的nsqd生产者并在那些节点上执行适当的操作)。

删除全局通道的过程是相似的,唯一的区别是在nsqlookupd实例上使用/delete_channel端点,并且需要确保所有订阅了该通道的消费者均已关闭。

然而,当在节点的子集上不再生成主题时,情况将变得更加复杂。由于消费者查询nsqlookupd以及连接到所有生产者的方式,您会进入竞争状态并尝试从集群中移除信息,而消费者发现该节点并进行重连接(从而推动更新,该节点上仍会生成主题)。在这些情况下,解决方案是使用"tombstones"。在nsqlookupd上下文中tombstone是特定于生产者的,并且持续时间可通过--tombstone-lifetime配置。在该窗口期间,生产者将不会在/lookup查询中列出,允许节点删除主题,并将该信息传播到nsqlookupd(然后删除tombstoned生产者),并防止任何消费者重新发现该节点。

nsqadmin

nsqadmin 是一个Web UI,用于实时查看集群的聚合统计信息并执行各种管理任务。

命令行参数选项

-acl-http-header string
    HTTP header to check for authenticated admin users (default "X-Forwarded-User")
-admin-user value
    admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)
-allow-config-from-cidr string
    A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-base-path string
    URL base path (default "/")
-config string
    path to config file
-graphite-url string
    graphite HTTP address
-http-address string
    <addr>:<port> to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-http-client-tls-cert string
    path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
    configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
    path to key file for the HTTP client
-http-client-tls-root-ca-file string
    path to CA file for the HTTP client
-log-level value
    set log verbosity: debug, info, warn, error, or fatal (default INFO)
-log-prefix string
    log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
    HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
    nsqd HTTP address (may be given multiple times)
-proxy-graphite
    proxy HTTP requests to graphite
-statsd-counter-format string
    The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
    The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
    time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-verbose
    [deprecated] has no effect, use --log-level
-version
    print version string

statsd / Graphite 集成

当使用nsqd --statsd-address=... 时,你可以指定nsqadmin --graphite-url=https://graphite.yourdomain.comnsqadmin中启用graphite图表。如果使用不带前缀键的statsd克隆(如statsdaemon),则还要指定 --use-statsd-prefix=false

管理通知(Admin Notifications)

如果设置了--notification-http-endpoint标志,nsqadmin在每次执行管理操作(例如暂停通道)时,都会向指定(完全限定)的端点发送POST请求。

请求的正文包含有关操作的信息(适当时将省略某些字段),如下所示:

{
  "action": "unpause_channel",
  "channel": "mouth",
  "topic": "beer",
  "timestamp": 1357683731,
  "user": "df",
  "user_agent": "Mozilla/5.0 (Macintosh; Iphone 8)",
  "remote_ip": "1.2.3.4:5678",
  "url": "http://nsqadmin.local/api/topics/api_requests/nsq_to_file",
  "via": "localhost"
}

如果对nsqdadmin的请求中包含HTTP基本身份认证的用户名,那么user字段将被填充。例如在http密码认证或者在oauth2_proxy之下运行时。

提示:您可以通过设置--notification-http-endpoint来创建主题名称为admin_actions的NSQ管理员操作通知数据流来指向nsqd HTTP Publish API。例如,如果nsqd运行在同一节点上,你可以使用--notification-http-endpoint="http://127.0.0.1:4151/put?topic=admin_actions"

有社区贡献的实用程序nsqadmin2slacknsqadmin2hipchat可以分别将消息中继到HipChat和Slack。

捕获的动作有:

  • create_channel
  • create_topic
  • delete_channel
  • delete_topic
  • empty_channel
  • empty_topic
  • pause_channel
  • pause_topic
  • tombstone_topic_producer
  • unpause_channel
  • unpause_topic

指标

下述指标通过nsqadmin在主题、通道和客户端连接上进行公开。

消息队列

  • Depth: 当前内存和磁盘中的消息总数 (即待发送的消息"积压")
  • In-Flight:当前已发送但尚未完成(FIN)、重新排队(REQ)或超时的消息数。
  • Deferred: 当前已重新排队和显式推迟但尚无法传递的消息数。

统计

  • Requeued: 由于超时或显式重新排队而导致消息被添加回队列的总次数。
  • Timed Out: 在配置的超时之前以及未收到来自客户端的响应之后,消息重新排队的总时间。
  • Messages:自节点启动以来收到的新消息总数。
  • Rate: 过去两个统计间隔内新消息的每秒速率(仅在启用了graphite集成时可用)。
  • Connections: 当前连接的客户端数。

客户端连接

  • Client Host:客户端ID(主机名)以及悬停(on-hover)连接的远程地址。
  • Protocol: NSQ协议版本以及客户端用户代理。
  • Attributes: TLS和AUTH连接状态。
  • NSQd Host: 客户端连接的nsqd节点地址。
  • In-flight: 当前已发送到此客户端的等待响应的消息数。
  • Ready Count: 此连接上可发送的最大消息数。由客户端的max_in_flight设置控制。
  • Finished: 客户端已完成的消息总数(FIN)。
  • Requeued: 客户端已重新排队的消息总数(REQ)。
  • Messages: 发送到客户端的消息总数。

utilities

这些是有助于通用功能和对数据流进行省察的实用程序。

nsq_stat

对指定主题/通道的所有生产者进行轮询/stats,并显示汇总统计信息

---------------depth---------------+--------------metadata---------------
  total    mem    disk inflt   def |     req     t-o         msgs clients
  24660  24660       0     0    20 |  102688       0    132492418       1
  25001  25001       0     0    20 |  102688       0    132493086       1
  21132  21132       0     0    21 |  102688       0    132493729       1

命令行选项

-channel string
    NSQ channel
-count value
    number of reports
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-interval duration
    duration of time between polling/printing output (default 2s)
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-nsqd-http-address value
    nsqd HTTP address (may be given multiple times)
-topic string
    NSQ topic
-version
    print version

nsq_tail

消费特定的主题/通道并写入到标准输出(本着tail(1)的精神)

命令行选项

-channel string
    NSQ channel
-consumer-opt value
    option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-max-in-flight int
    max number of messages to allow in flight (default 200)
-n int
    total messages to show (will wait if starved)
-nsqd-tcp-address value
    nsqd TCP address (may be given multiple times)
-print-topic
    print topic name where message was received
-topic value
    NSQ topic (may be given multiple times)
-version
    print version string

nsq_to_file

消费指定的主题/通道并写入到换行符分隔的文件中,可以选择滚动 和/或 压缩文件。

命令行选项

-channel string
    nsq channel (default "nsq_to_file")
-consumer-opt value
    option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-datetime-format string
    strftime compatible format for <DATETIME> in filename format (default "%Y-%m-%d_%H")
-filename-format string
    output filename format (<TOPIC>, <HOST>, <PID>, <DATETIME>, <REV> are replaced. <REV> is increased when file already exists) (default "<TOPIC>.<HOST><REV>.<DATETIME>.log")
-gzip
    gzip output files.
-gzip-level int
    gzip compression level (1-9, 1=BestSpeed, 9=BestCompression) (default 6)
-host-identifier string
    value to output in log filename in place of hostname. <SHORT_HOST> and <HOSTNAME> are valid replacement tokens
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-log-level string
    set log verbosity: debug, info, warn, error, or fatal (default "info")
-log-prefix string
    log message prefix (default "[nsq_to_file] ")
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-max-in-flight int
    max number of messages to allow in flight (default 200)
-nsqd-tcp-address value
    nsqd TCP address (may be given multiple times)
-output-dir string
    directory to write output files to (default "/tmp")
-rotate-interval duration
    rotate the file every duration
-rotate-size rotate-size
    rotate the file when it grows bigger than rotate-size bytes
-skip-empty-files
    skip writing empty files
-sync-interval duration
    sync file to disk every duration (default 30s)
-topic value
    nsq topic (may be given multiple times)
-topic-pattern string
    only log topics matching the following pattern
-topic-refresh duration
    how frequently the topic list should be refreshed (default 1m0s)
-version
    print version string
-work-dir string
    directory for in-progress files before moving to output-dir

nsq_to_http

消费指定的主题/通道,并向指定的端点执行HTTP请求(GET / POST)。

命令行选项

-channel string
    nsq channel (default "nsq_to_http")
-consumer-opt value
    option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-content-type string
    the Content-Type used for POST requests (default "application/octet-stream")
-get value
    HTTP address to make a GET request to. '%s' will be printf replaced with data (may be given multiple times)
-header value
    Custom header for HTTP requests (may be given multiple times)
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 20s)
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-max-in-flight int
    max number of messages to allow in flight (default 200)
-mode string
    the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-n int
    number of concurrent publishers (default 100)
-nsqd-tcp-address value
    nsqd TCP address (may be given multiple times)
-post value
    HTTP address to make a POST request to.  data will be in the body (may be given multiple times)
-sample float
    % of messages to publish (float b/w 0 -> 1) (default 1)
-status-every int
    the # of requests between logging status (per handler), 0 disables (default 250)
-topic string
    nsq topic
-version
    print version string

nsq_to_nsq

消费特定的主题/通道,并通过TCP将消息重新发布到目标nsqd

命令行选项

-channel string
    nsq channel (default "nsq_to_nsq")
-consumer-opt value
    option to passthrough to nsq.Consumer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-destination-nsqd-tcp-address value
    destination nsqd TCP address (may be given multiple times)
-destination-topic string
    use this destination topic for all consumed topics (default is consumed topic name)
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-max-in-flight int
    max number of messages to allow in flight (default 200)
-mode string
    the upstream request mode options: round-robin, hostpool (default), epsilon-greedy (default "hostpool")
-nsqd-tcp-address value
    nsqd TCP address (may be given multiple times)
-producer-opt value
    option to passthrough to nsq.Producer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)
-require-json-field string
    for JSON messages: only pass messages that contain this field
-require-json-value string
    for JSON messages: only pass messages in which the required field has this value
-status-every int
    the # of requests between logging status (per destination), 0 disables (default 250)
-topic value
    nsq topic (may be given multiple times)
-version
    print version string
-whitelist-json-field value
    for JSON messages: pass this field (may be given multiple times)

to_nsq

接收标准输入流并通过换行符进行分割(默认),通过TCP重新发布到目标nsqd

命令行选项

-delimiter string
    character to split input from stdin (default "\n")
-nsqd-tcp-address value
    destination nsqd TCP address (may be given multiple times)
-producer-opt value
    option to passthrough to nsq.Producer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)
-rate int
    Throttle messages to n/second. 0 to disable
-topic string
    NSQ topic to publish to

Previous

NSQ之概览

Next

NSQ之客户端