K6简介
K6是Loki官方推荐的性能压测工具,支持对Loki进行读写性能压测。
K6编译
K6官方提供的二进制包缺少Loki相关的类库,无法对Loki进行性能压测,需自行下载xk6-loki进行编译。
K6基于Golang开发的,需先准备Golang环境。下载Golang离线安装包,解压并配置环境变量,这里推荐部署1.20以上版本。
Golang下载地址:All releases - The Go Programming Language (google.cn)
[root@node1 ~]# tar -xf go1.20.1.linux-amd64.tar.gz -C /usr/local/
[root@node1 ~]# tail -n 20 /etc/profile
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r "$i" ]; then
if [ "${-#*i}" != "$-" ]; then
. "$i"
else
. "$i" >/dev/null
fi
fi
done
export GOPROXY=https://goproxy.cn
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=/opt/go
export PATH=$PATH:$GOPATH/bin
unset i
unset -f pathmunge
[root@node1 ~]# source /etc/profile
[root@node1 ~]# go version
go version go1.20.1 linux/amd64
[root@node1 ~]#
安装编译需要的库
[root@node1 go]# go install go.k6.io/xk6/cmd/xk6@latest
go: downloading go.k6.io/xk6 v0.8.1
go: downloading github.com/Masterminds/semver/v3 v3.1.1
[root@node1 go]# ll /opt/go/
total 8
drwxr-xr-x 2 root root 4096 Feb 16 16:43 bin
drwxr-xr-x 4 root root 4096 Feb 16 16:42 pkg
下载编译xk6-loki
需下载xk6-loki源码到本地编译,源码github地址:github.com/grafana/xk6…
[root@node1 loki]# git clone https://github.com/grafana/xk6-loki
Cloning into 'xk6-loki'...
remote: Enumerating objects: 240, done.
remote: Counting objects: 100% (240/240), done.
remote: Compressing objects: 100% (92/92), done.
remote: Total 240 (delta 157), reused 182 (delta 143), pack-reused 0
Receiving objects: 100% (240/240), 200.06 KiB | 0 bytes/s, done.
Resolving deltas: 100% (157/157), done.
[root@node1 loki]# cd xk6-loki/
[root@node1 xk6-loki]# ll
total 380
-rw-r--r-- 1 root root 6411 Feb 16 16:45 batch.go
-rw-r--r-- 1 root root 1962 Feb 16 16:45 batch_test.go
-rw-r--r-- 1 root root 11495 Feb 16 16:45 client.go
drwxr-xr-x 2 root root 4096 Feb 16 16:45 examples
-rw-r--r-- 1 root root 956 Feb 16 16:45 go.mod
-rw-r--r-- 1 root root 305920 Feb 16 16:45 go.sum
-rw-r--r-- 1 root root 11357 Feb 16 16:45 LICENSE
-rw-r--r-- 1 root root 5402 Feb 16 16:45 loki_easyjson.go
-rw-r--r-- 1 root root 5557 Feb 16 16:45 loki.go
-rw-r--r-- 1 root root 630 Feb 16 16:45 Makefile
-rw-r--r-- 1 root root 1627 Feb 16 16:45 query.go
-rw-r--r-- 1 root root 12064 Feb 16 16:45 README.md
[root@node1 xk6-loki]# make k6
xk6 build master \
--replace "github.com/mingrammer/flog=github.com/chaudum/flog@v0.4.4-0.20220419113107-eb2f67f18b99" \
--replace "google.golang.org/grpc=google.golang.org/grpc@v1.45.0" \
--replace "github.com/hashicorp/consul=github.com/hashicorp/consul@v1.5.1" \
--replace "github.com/go-kit/log=github.com/dannykopping/go-kit-log@v0.2.2-0.20221002180827-5591c1641b6b" \
--with "github.com/grafana/xk6-loki=/root/loki/xk6-loki"
2023/02/16 16:45:41 [INFO] Temporary folder: /tmp/buildenv_2023-02-16-1645.936328856
2023/02/16 16:45:41 [INFO] Initializing Go module
2023/02/16 16:45:41 [INFO] exec (timeout=10s): /usr/local/go/bin/go mod init k6
go: creating new go.mod: module k6
2023/02/16 16:45:41 [INFO] Replace github.com/mingrammer/flog => github.com/chaudum/flog@v0.4.4-0.20220419113107-eb2f67f18b99
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace github.com/mingrammer/flog=github.com/chaudum/flog@v0.4.4-0.20220419113107-eb2f67f18b99
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: warning: "all" matched no packages
2023/02/16 16:45:41 [INFO] Replace google.golang.org/grpc => google.golang.org/grpc@v1.45.0
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace google.golang.org/grpc=google.golang.org/grpc@v1.45.0
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: warning: "all" matched no packages
2023/02/16 16:45:41 [INFO] Replace github.com/hashicorp/consul => github.com/hashicorp/consul@v1.5.1
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace github.com/hashicorp/consul=github.com/hashicorp/consul@v1.5.1
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: warning: "all" matched no packages
2023/02/16 16:45:41 [INFO] Replace github.com/go-kit/log => github.com/dannykopping/go-kit-log@v0.2.2-0.20221002180827-5591c1641b6b
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace github.com/go-kit/log=github.com/dannykopping/go-kit-log@v0.2.2-0.20221002180827-5591c1641b6b
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: warning: "all" matched no packages
2023/02/16 16:45:41 [INFO] Replace github.com/grafana/xk6-loki => /root/loki/xk6-loki
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace github.com/grafana/xk6-loki=/root/loki/xk6-loki
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: warning: "all" matched no packages
2023/02/16 16:45:41 [INFO] Pinning versions
2023/02/16 16:45:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: found github.com/grafana/xk6-loki in github.com/grafana/xk6-loki v0.0.0-00010101000000-000000000000
go: downloading github.com/brianvoe/gofakeit/v6 v6.9.0
go: downloading github.com/dop251/goja v0.0.0-20221003171542-5ea1285e6c91
go: downloading github.com/brianvoe/gofakeit v3.11.5+incompatible
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/golang/snappy v0.0.4
go: downloading github.com/grafana/loki v1.6.2-0.20221006221238-7979cfbe076d
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/chaudum/flog v0.4.4-0.20220419113107-eb2f67f18b99
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/prometheus/common v0.37.0
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading go.k6.io/k6 v0.41.0
go: downloading github.com/dlclark/regexp2 v1.7.0
go: downloading golang.org/x/text v0.3.7
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/opentracing/opentracing-go v1.2.0
go: downloading github.com/prometheus/prometheus v1.8.2-0.20211119115433-692a54649ed7
go: downloading google.golang.org/grpc v1.45.0
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/dustin/go-humanize v1.0.0
go: downloading github.com/dannykopping/go-kit-log v0.2.2-0.20221002180827-5591c1641b6b
go: downloading github.com/josharian/intern v1.0.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading bou.ke/monkey v1.0.1
go: downloading golang.org/x/sys v0.0.0-20220928140112-f11e5e49a4ec
go: downloading github.com/serenize/snaker v0.0.0-20201027110005-a7ad2135616e
go: downloading github.com/spf13/afero v1.6.0
go: downloading github.com/oxtoacart/bpool v0.0.0-20190530202638-03653db5a59c
go: downloading golang.org/x/time v0.0.0-20220922220347-f3bd1da661af
go: downloading gopkg.in/guregu/null.v3 v3.3.0
go: downloading github.com/mstoykov/envconfig v1.4.1-0.20220114105314-765c6d8c76f1
go: downloading github.com/Azure/go-ntlmssp v0.0.0-20220621081337-cb9428e4ac1e
go: downloading github.com/Soontao/goHttpDigestClient v0.0.0-20170320082612-6d28bb1415c5
go: downloading github.com/andybalholm/brotli v1.0.4
go: downloading github.com/klauspost/compress v1.15.11
go: downloading github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
go: downloading golang.org/x/net v0.0.0-20221002022538-bcab6841153b
go: downloading github.com/mccutchen/go-httpbin v1.1.2-0.20190116014521-c5cb2f4802fa
go: downloading github.com/mstoykov/atlas v0.0.0-20220808085829-90340e9998bd
go: downloading github.com/go-sourcemap/sourcemap v2.1.4-0.20211119122758-180fcef48034+incompatible
go: downloading go.uber.org/goleak v1.2.0
go: downloading github.com/grafana/dskit v0.0.0-20220928083349-b1b307db4f30
go: downloading github.com/prometheus/client_golang v1.13.0
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/weaveworks/common v0.0.0-20220916164235-da3fde973eec
go: downloading go.uber.org/atomic v1.9.0
go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading google.golang.org/genproto v0.0.0-20220815135757-37a418bb8959
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/google/go-cmp v0.5.8
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/go-logfmt/logfmt v0.5.1
go: downloading github.com/onsi/ginkgo v1.16.5
go: downloading github.com/onsi/gomega v1.18.1
go: downloading github.com/fatih/color v1.13.0
go: downloading golang.org/x/crypto v0.0.0-20220926161630-eccd6366d1be
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading github.com/dennwc/varint v1.0.0
go: downloading github.com/gogo/status v1.1.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/prometheus/procfs v0.8.0
go: downloading google.golang.org/protobuf v1.28.1
go: downloading github.com/stretchr/objx v0.4.0
go: downloading github.com/gogo/googleapis v1.4.0
go: downloading github.com/kr/pretty v0.3.0
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/weaveworks/promrus v1.2.0
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: downloading github.com/opentracing-contrib/go-grpc v0.0.0-20210225150812-73cb765af46e
go: downloading github.com/prometheus/exporter-toolkit v0.7.2-0.20220901134540-2434b08435da
go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible
go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible
go: downloading github.com/mattn/go-colorable v0.1.13
go: downloading github.com/mattn/go-isatty v0.0.16
go: downloading golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
go: downloading github.com/hashicorp/consul/api v1.14.0
go: downloading github.com/hashicorp/go-cleanhttp v0.5.2
go: downloading go.etcd.io/etcd/api/v3 v3.5.4
go: downloading github.com/hashicorp/consul v1.5.1
go: downloading go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489
go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.4
go: downloading go.etcd.io/etcd/client/v3 v3.5.4
go: downloading github.com/armon/go-metrics v0.4.0
go: downloading github.com/hashicorp/go-sockaddr v1.0.2
go: downloading github.com/hashicorp/memberlist v0.3.1
go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/rogpeppe/go-internal v1.9.0
go: downloading github.com/sercand/kuberesolver v2.4.0+incompatible
go: downloading github.com/felixge/httpsnoop v1.0.3
go: downloading github.com/opentracing-contrib/go-stdlib v1.0.0
go: downloading github.com/nxadm/tail v1.4.8
go: downloading github.com/oklog/ulid v1.3.1
go: downloading go.uber.org/zap v1.21.0
go: downloading github.com/hashicorp/go-immutable-radix v1.3.1
go: downloading github.com/pascaldekloe/goe v0.1.0
go: downloading github.com/hashicorp/go-hclog v0.16.2
go: downloading github.com/hashicorp/go-rootcerts v1.0.2
go: downloading github.com/hashicorp/serf v0.9.7
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/hashicorp/consul/sdk v0.10.0
go: downloading github.com/hashicorp/go-uuid v1.0.2
go: downloading github.com/google/btree v1.0.1
go: downloading github.com/hashicorp/go-msgpack v0.5.5
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
go: downloading github.com/miekg/dns v1.1.50
go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: downloading golang.org/x/oauth2 v0.0.0-20220808172628-8227340efae7
go: downloading github.com/HdrHistogram/hdrhistogram-go v1.1.2
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137
go: downloading github.com/prometheus/common/sigv4 v0.1.0
go: downloading golang.org/x/tools v0.1.12
go: downloading go.uber.org/multierr v1.8.0
go: downloading github.com/coreos/go-semver v0.3.0
go: downloading github.com/coreos/go-systemd/v22 v22.3.2
go: downloading github.com/hashicorp/golang-lru v0.5.4
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf
go: downloading github.com/hashicorp/errwrap v1.1.0
go: downloading github.com/jpillora/backoff v1.0.0
go: downloading github.com/fsnotify/fsnotify v1.5.4
go: downloading github.com/aws/aws-sdk-go v1.44.72
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4
go: downloading github.com/jmespath/go-jmespath v0.4.0
2023/02/16 16:46:57 [INFO] Writing main module: /tmp/buildenv_2023-02-16-1645.936328856/main.go
2023/02/16 16:46:57 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -require go.k6.io/k6@master
2023/02/16 16:46:57 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: downloading go.k6.io/k6 v0.42.1-0.20230213192334-bbfe357a39c4
go: downloading github.com/grafana/xk6-output-prometheus-remote v0.1.0
go: downloading github.com/spf13/cobra v1.4.0
go: downloading golang.org/x/term v0.0.0-20220919170432-7a66f970e087
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/tidwall/pretty v1.2.1
go: downloading github.com/grafana/xk6-redis v0.1.1
go: downloading github.com/grafana/xk6-timers v0.1.2
go: downloading github.com/grafana/xk6-websockets v0.1.6
go: downloading github.com/influxdata/influxdb1-client v0.0.0-20200827194710-b269163b24ab
go: downloading github.com/DataDog/datadog-go v3.2.0+incompatible
go: downloading go.buf.build/grpc/go/prometheus/prometheus v1.4.4
go: downloading github.com/mstoykov/k6-taskqueue-lib v0.1.0
go: downloading github.com/go-redis/redis/v8 v8.11.5
go: downloading github.com/grafana/xk6-browser v0.8.1-0.20230207135343-cfd6a83dfc42
go: downloading github.com/go-redis/redis v6.15.9+incompatible
go: downloading github.com/jhump/protoreflect v1.13.0
go: downloading github.com/PuerkitoBio/goquery v1.8.0
go: downloading github.com/tidwall/gjson v1.14.3
go: downloading go.buf.build/grpc/go/gogo/protobuf v1.4.9
go: downloading github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f
go: downloading github.com/tidwall/match v1.1.1
go: downloading github.com/andybalholm/cascadia v1.3.1
go: downloading github.com/chromedp/cdproto v0.0.0-20221023212508-67ada9507fb2
go: downloading github.com/chromedp/sysutil v1.0.0
go: downloading github.com/prometheus/client_golang v1.14.1-0.20221122130035-8b6e68085b10
go: downloading github.com/prometheus/client_model v0.3.0
go: downloading github.com/dop251/goja v0.0.0-20230128084908-78b980256d04
go: downloading golang.org/x/sync v0.0.0-20220819030929-7fc1605a5dde
2023/02/16 16:47:00 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
2023/02/16 16:47:01 [INFO] Build environment ready
2023/02/16 16:47:01 [INFO] Building k6
2023/02/16 16:47:01 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
2023/02/16 16:47:01 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /root/loki/xk6-loki/k6 -ldflags=-w -s -trimpath
2023/02/16 16:49:54 [INFO] Build complete: ./k6
2023/02/16 16:49:54 [INFO] Cleaning up temporary folder: /tmp/buildenv_2023-02-16-1645.936328856
xk6 has now produced a new k6 binary which may be different than the command on your system path!
Be sure to run './k6 run <SCRIPT_NAME>' from the '/root/loki/xk6-loki' directory.
[root@node1 xk6-loki]#
[root@node1 xk6-loki]# ll
total 43520
-rw-r--r-- 1 root root 6411 Feb 16 16:45 batch.go
-rw-r--r-- 1 root root 1962 Feb 16 16:45 batch_test.go
-rw-r--r-- 1 root root 11495 Feb 16 16:45 client.go
drwxr-xr-x 2 root root 4096 Feb 16 16:45 examples
-rw-r--r-- 1 root root 956 Feb 16 16:45 go.mod
-rw-r--r-- 1 root root 305920 Feb 16 16:45 go.sum
-rwxr-xr-x 1 root root 44175360 Feb 16 16:49 k6
-rw-r--r-- 1 root root 11357 Feb 16 16:45 LICENSE
-rw-r--r-- 1 root root 5402 Feb 16 16:45 loki_easyjson.go
-rw-r--r-- 1 root root 5557 Feb 16 16:45 loki.go
-rw-r--r-- 1 root root 630 Feb 16 16:45 Makefile
-rw-r--r-- 1 root root 1627 Feb 16 16:45 query.go
-rw-r--r-- 1 root root 12064 Feb 16 16:45 README.md
[root@node1 xk6-loki]#
性能压测
参考官方的test.js,对loki进行写入性能压测。
test.js内容参考如下,注意需修改loki的URL为实际的Loki服务端IP。
import loki from 'k6/x/loki';
const timeout = 5000; // ms
const conf = loki.Config("http://localhost:3100", timeout);
const client = loki.Client(conf);
export default () => {
client.pushParameterized(2, 512*1024, 1024*1024);
};
执行k6 run test.js,执行写入性能压测。
[root@ops-test-01 k6]# k6 run test.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
WARN[0000] Running in multi-tenant-mode. Each VU has its own X-Scope-OrgID
execution: local
script: test_bigdata.js
output: engine
scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
* default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)
WARN[0000] Running in multi-tenant-mode. Each VU has its own X-Scope-OrgID
WARN[0000] Running in multi-tenant-mode. Each VU has its own X-Scope-OrgID
data_received....................: 64 B 771 B/s
data_sent........................: 1.0 MB 12 MB/s
http_req_blocked.................: avg=808.73µs min=808.73µs med=808.73µs max=808.73µs p(90)=808.73µs p(95)=808.73µs
http_req_connecting..............: avg=715.61µs min=715.61µs med=715.61µs max=715.61µs p(90)=715.61µs p(95)=715.61µs
http_req_duration................: avg=48.33ms min=48.33ms med=48.33ms max=48.33ms p(90)=48.33ms p(95)=48.33ms
{ expected_response:true }.....: avg=48.33ms min=48.33ms med=48.33ms max=48.33ms p(90)=48.33ms p(95)=48.33ms
http_req_failed..................: 0.00% ✓ 0 ✗ 1
http_req_receiving...............: avg=32.7µs min=32.7µs med=32.7µs max=32.7µs p(90)=32.7µs p(95)=32.7µs
http_req_sending.................: avg=5.09ms min=5.09ms med=5.09ms max=5.09ms p(90)=5.09ms p(95)=5.09ms
http_req_tls_handshaking.........: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting.................: avg=43.2ms min=43.2ms med=43.2ms max=43.2ms p(90)=43.2ms p(95)=43.2ms
http_reqs........................: 1 12.043615/s
iteration_duration...............: avg=82.79ms min=82.79ms med=82.79ms max=82.79ms p(90)=82.79ms p(95)=82.79ms
iterations.......................: 1 12.043615/s
loki_client_lines................: 5320 64072.034309/s
loki_client_uncompressed_bytes...: 884 kB 11 MB/s
running (00m00.1s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m00.1s/10m0s 1/1 iters, 1 per VU
loki_client_lines和loki_client_uncompressed_bytes即为关键的性能指标,依次为每秒写入条数和每秒写入数据量大小