在本次部署过程中，我们将使用 Helm 来简化 Redis 集群的部署，并利用 Kubernetes（K8s）直接部署 Predixy 作为代理层（后续我写了一个predxiy的chart，实现helm部署predixy了，感兴趣可以看下自己编写predixy chart，实现helm快速搭建Redis集群（redis-cluster+predixy）），以提高集群的可访问性和负载均衡能力。具体架构如下：
一.安装Redis集群
参考：redis 20.6.1 · bitnami/bitnami
1 添加仓库
helm repo add bitnami https://charts.bitnami.com/bitnami
2 修改values.yaml文件
#查询 Redis 资源
helm repo update
helm search repo redis
helm pull bitnami/redis-cluster --version 11.0.0
tar -xvf redis-cluster-11.0.0.tgz
cd redis-cluster
也可以直接从git上下载values.yaml文件charts/bitnami/redis at main · bitnami/charts
在 values.yaml 文件中，主要需要修改以下字段：
storageClass：指定存储类password：设置 Redis 密码
global:
imageRegistry: ""
## E.g.
## imagePullSecrets:
## - myRegistryKeySecretName
##
imagePullSecrets: []
defaultStorageClass: ""
storageClass: "local-path"
redis:
password: "*********"
3 安装chart
#创建命名空间
kubectl create ns redis-cluster
#安装Redis集群
helm -n redis-cluster install redis-cluster redis-cluster -f values.yaml
二 安装代理
为什么需要代理？以及代理的选型可以参考小米这篇文档
小米Redis的K8s容器化部署实践小米的Redis使用规模很大，现在有数万个实例，并且每天有百万亿次的访问频率，支撑了 - 掘金
我也安装了一下redis-cluster-proxy，发现存在一个问题，感兴趣可以看看
ERR Protocol error: invalid bulk length · Issue #94 · RedisLabs/redis-cluster-proxy
redis报错：(error) ERR Protocol error: invalid bulk length-CSDN博客
废话不多说了，直接安装predixy作为我们redis集群的代理。
1.构建predixy镜像
参考：predixy/Dockerfile at master · haandol/predixy
FROM ubuntu:18.04
LABEL maintainer="haandol <ldg55d@gmail.com>"
ENV VERSION 1.0.5
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y build-essential wget unzip \
&& wget --no-check-certificate https://github.com/joyieldInc/predixy/archive/${VERSION}.zip -O predixy-${VERSION}.zip \
&& unzip predixy-${VERSION}.zip \
&& cd predixy-${VERSION} \
&& make \
&& mv src/predixy /usr/local/bin \
&& mkdir -p /etc/predixy/conf
ADD conf /etc/predixy/conf
CMD ["/usr/local/bin/predixy", "/etc/predixy/conf/predixy.conf"]
也试了一下7.0.1版本，编译有点问题。
#构建镜像
docker build . -t prodixy:v1.0.5
#推送镜像
docker tag prodixy:v1.0.5 */*/prodixy:v1.0.5
docker push */*/prodixy:v1.0.5
2.部署predixy
这里无需多言，就是部署pvc、deployment、services。
(1)创建ConfigMap
kind: ConfigMap
apiVersion: v1
metadata:
name: predixy-config
namespace: redis-cluster
data:
predixy.conf: >
################################### GENERAL
####################################
## Predixy configuration file example
## Specify a name for this predixy service
## redis command INFO can get this
Name Predixy-DefaultNS
## Specify listen address, support IPV4, IPV6, Unix socket
## Examples:
# Bind 127.0.0.1:7617
# Bind 0.0.0.0:7617
# Bind /tmp/predixy
## Default is 0.0.0.0:7617
Bind 0.0.0.0:7617
## Worker threads
WorkerThreads 4
## Memory limit, 0 means unlimited
## Examples:
# MaxMemory 100M
# MaxMemory 1G
# MaxMemory 0
## MaxMemory can change online by CONFIG SET MaxMemory xxx
## Default is 0
# MaxMemory 0
## Close the connection after a client is idle for N seconds (0 to disable)
## ClientTimeout can change online by CONFIG SET ClientTimeout N
## Default is 0 为0时表示禁止该功能，不主动断开客户端连接
ClientTimeout 0
## IO buffer size
## Default is 4096
# BufSize 4096
################################### LOG
########################################
## Log file path
## Unspecify will log to stdout
## Default is Unspecified
# Log /data/predixy.log
## LogRotate support
## 1d rotate log every day
## nh rotate log every n hours 1 <= n <= 24
## nm rotate log every n minutes 1 <= n <= 1440
## nG rotate log evenry nG bytes
## nM rotate log evenry nM bytes
## time rotate and size rotate can combine eg 1h 2G, means 1h or 2G roate a
time
## Examples:
# LogRotate 1d 2G
# LogRotate 1d
LogRotate 1d
## Default is disable LogRotate
## In multi-threads, worker thread log need lock,
## AllowMissLog can reduce lock time for improve performance
## AllowMissLog can change online by CONFIG SET AllowMissLog true|false
## Default is true
# AllowMissLog false
## LogLevelSample, output a log every N
## all level sample can change online by CONFIG SET LogXXXSample N
LogVerbSample 0
LogDebugSample 0
LogInfoSample 100
LogNoticeSample 1
LogWarnSample 1
LogErrorSample 1
################################### AUTHORITY
##################################
# Include auth.conf
Authority {
Auth "***" {
Mode read
}
Auth "***" {
Mode write
}
Auth "***" {
Mode admin
}
}
################################### SERVERS
####################################
#Include cluster.conf
# Include sentinel.conf
# Include try.conf
###############################################################################
#这个clusterserverpool也可以放到cluster.conf文件中，看自已的需求
ClusterServerPool {
Password ****
MasterReadPriority 60
StaticSlaveReadPriority 50
DynamicSlaveReadPriority 60
RefreshInterval 1
ServerTimeout 1
ServerFailureLimit 10
ServerRetryTimeout 1
KeepAlive 120
Servers {
+ redis-cluster-0.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
+ redis-cluster-1.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
+ redis-cluster-2.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
+ redis-cluster-3.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
+ redis-cluster-4.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
+ redis-cluster-5.redis-cluster-headless.redis-cluster.svc.cluster.local:6379
}
}
################################### DATACENTER
#################################
## LocalDC specify current machine dc
# LocalDC bj
## see dc.conf
# Include dc.conf
################################### COMMAND
####################################
## Custom command define, see command.conf
#Include command.conf
################################### LATENCY
####################################
## Latency monitor define, see latency.conf
#Include latency.conf
在 ConfigMap 文件中，主要需要修改以下字段：
Authority：redis集群的密码ClusterServerPool.password：proxy的密码ClusterServerPool.Servers：redis集群的地址
可以看到我们部署完redis-cluster之后是有一个Headless Service
所以proxy可以直接利用Headless Service与集群pod通信。为什么使用Headless Service呢？
因为 Redis 集群中每个节点的状态不同，且节点之间需要直接通信，因此必须使用 Headless Service。
(2)创建PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: predixy-pvc-nfs
namespace: redis-cluster
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: nfs-storage
(3)创建Deployment
kind: Deployment
apiVersion: apps/v1
metadata:
name: predixy
namespace: redis-cluster
labels:
app: predixy
version: v1
spec:
replicas: 2
selector:
matchLabels:
app: predixy
template:
metadata:
creationTimestamp: null
labels:
app: predixy
version: v1
annotations:
cni.projectcalico.org/ipv4pools: '["default-ipv4-ippool"]'
kubesphere.io/creator: admin
kubesphere.io/imagepullsecrets: '{}'
kubesphere.io/restartedAt: '2023-11-06T08:25:17.537Z'
logging.kubesphere.io/logsidecar-config: '{}'
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
- name: volume-jc9s47
configMap:
name: predixy-config
items:
- key: predixy.conf
path: predixy.conf
defaultMode: 420
containers:
- name: predixy
image: '**/**/prodixy:v1.0.5'
command:
- predixy
args:
- /etc/predixy/predixy.conf
ports:
- name: http-7617
containerPort: 7617
protocol: TCP
resources: {}
volumeMounts:
- name: host-time
readOnly: true
mountPath: /etc/localtime
- name: volume-jc9s47
readOnly: true
mountPath: /etc/predixy/predixy.conf
subPath: predixy.conf
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
(4)创建Service
kind: Service
apiVersion: v1
metadata:
name: predixy
namespace: redis-cluster
spec:
ports:
- name: http-7617
protocol: TCP
port: 7617
targetPort: 7617
sessionAffinity: None
type: NodePort
selector:
app: predixy
我这里为了方便测试直接nodePort了，直接用客户端工具测一下就可以，这里就不演示了。
后记
感觉这镜像有点大，毕竟基础镜像用的ubuntu，想用alpine做基础镜像来着，小菜鸡没构建成功，所以只能用其它镜像做基础镜像了，最后参考这位大佬写的文件predixy/Dockerfile at master · shilazi/predixy，自己简单写了一个，镜像大小从原来的从362MB->91.8MB，还是有进步的。
FROM debian:bullseye as builder
ENV VERSION 1.0.5
RUN apt update && apt install -y build-essential wget unzip \
&& wget --no-check-certificate https://github.com/joyieldInc/predixy/archive/${VERSION}.zip -O predixy-${VERSION}.zip \
&& unzip predixy-${VERSION}.zip && mv predixy-${VERSION} predixy &&cd predixy && make
# ---------- 8< ----------
FROM bitnami/minideb:bullseye
COPY --from=builder predixy/src/predixy /usr/bin/predixy
VOLUME ["/etc/predixy", "/var/log/predixy"]
CMD ["/usr/local/bin/predixy", "/etc/predixy/conf/predixy.conf"]
