Redis Cluster容器化需要解决以下问题：
1. Redis Cluster数据持久化
-> 可以通过StatefulSet + PVC来做持久化存储Redis配置和集群信息
2. Redis Cluster自愈(重建或重启Pod)
-> 在Redis启动命令中指定cluster-announce-ip pod-ip
3. Redis Cluster对Kubernetes内部提供服务
-> 通过Service进行服务发布即可
4. Redis Cluster对Kubernetes外部提供服务
-> 因Redis Cluster请求时，会自动重定向，通过NodePort对外发布Redis服务，但是在K8S外部也无法连接Redis Pod IP，因此需要通过redis-cluster-proxy来实现代理访问。
5. Redis Cluster创建集群方式
-> 因当前版本（6.0.6）redis-cli --cluster create 命令不支持hostname或dns，因此需要获取Pod IP后手动执行Redis Cluster创建集群。
Redis Cluster架构
Redis Cluster 最少需要3个Master节点，每个Master节点可以有一个或多个Slave节点。
在K8S中部署Redis Cluster
ConfigMap保存redis.conf配置文件
---
# Redis Config
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
namespace: demo
data:
redis.conf: |
bind 0.0.0.0 # 绑定IP
port 6379 # 绑定端口
maxmemory 4GB # 限制内存大小
protected-mode yes # 启用保护模式
masterauth P@ssw0rd # 启用密码认证
requirepass P@ssw0rd # 启用密码认证
dir /var/lib/redis # 配置redis目录
logfile /var/log/redis.log
cluster-enabled yes # 启用Cluster
cluster-config-file nodes.conf
cluster-migration-barrier 1
cluster-require-full-coverage noStatefulSet保存Redis数据和Cluster状态
---
# Redis Service
apiVersion: v1
kind: Service
metadata:
name: redis-cluster
namespace: demo
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
---
# Redis StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
namespace: demo
spec:
replicas: 6 # 部署6个节点，3个Master 3个Slave
selector:
matchLabels:
app: redis
serviceName: redis
template:
metadata:
labels:
app: redis
spec:
imagePullSecrets:
- name: harbor
containers:
- name: redis
image: redis:latest
imagePullPolicy: IfNotPresent
command: ["redis-server"]
args:
- /etc/redis/redis.conf
- --cluster-announce-ip # 用于重启Pod后向集群更新自身IP
- "$(MY_POD_IP)"
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- name: redis-6379
containerPort: 6379
protocol: TCP
- name: redis-16379
containerPort: 16379
protocol: TCP
resources:
requests: # 请求的资源
cpu: "1"
memory: "4096Mi"
limits: # 限制的资源
cpu: "1" # 1代表1核
memory: "4096Mi"
volumeMounts:
- name: redis-data
mountPath: /var/lib/redis/
- name: redis-conf
mountPath: /etc/redis/
volumes: # 挂载配置文件
- name: redis-conf
configMap:
name: redis-cluster
volumeClaimTemplates: # 挂载持久化存储
- metadata:
name: redis-data
namespace: demo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: vsan手动K8S中创建Redis Cluster
1. 获取Pod IP
获取6个Redis Pod IP地址，因为当前版本创建redis cluster只能使用IP地址，不支持域名方式。
# kubectl get pod -n demo -o wide | grep redis
redis-0 1/1 Running 0 2m12s 10.42.0.192 k8s-test-node02 <none> <none>
redis-1 1/1 Running 0 119s 10.42.2.196 k8s-test-node01 <none> <none>
redis-2 1/1 Running 0 96s 10.42.1.51 k8s-test-node03 <none> <none>
redis-3 1/1 Running 0 88s 10.42.0.193 k8s-test-node02 <none> <none>
redis-4 1/1 Running 0 77s 10.42.2.197 k8s-test-node01 <none> <none>
redis-5 1/1 Running 0 57s 10.42.1.52 k8s-test-node03 <none> <none>2. 执行创建命令
在任意节点上执行以下命令创建Redis Cluster
redis-cli --cluster create 10.42.0.188:6379 10.42.1.47:6379 10.42.2.192:6379 10.42.0.189:6379 10.42.2.193:6379 10.42.1.48:6379 --cluster-replicas 1 -a 'P@ssw0rd'
root@redis-0:/data# redis-cli --cluster create 10.42.0.188:6379 10.42.1.47:6379 10.42.2.192:6379 10.42.0.189:6379 10.42.2.193:6379 10.42.1.48:6379 --cluster-replicas 1 -a 'P@ssw0rd'
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.42.2.193:6379 to 10.42.0.188:6379
Adding replica 10.42.1.48:6379 to 10.42.1.47:6379
Adding replica 10.42.0.189:6379 to 10.42.2.192:6379
M: dae13b0bbc2674078851cce1953ba6d59caf4536 10.42.0.188:6379
slots:[0-5460] (5461 slots) master
M: 365d43c32353607fdb21bcd286ee6706014b7ede 10.42.1.47:6379
slots:[5461-10922] (5462 slots) master
M: e5210fb13d2dac4ca03d6a8165944a90f21c4373 10.42.2.192:6379
slots:[10923-16383] (5461 slots) master
S: e71993737dd8a69819917178f1c4dd479f40a431 10.42.0.189:6379
replicates e5210fb13d2dac4ca03d6a8165944a90f21c4373
S: f43e356d038c4af1730b8bf9f8807a1ac9c16957 10.42.2.193:6379
replicates dae13b0bbc2674078851cce1953ba6d59caf4536
S: 080e184652ee18d3c262e39787b5cfd84d9558e9 10.42.1.48:6379
replicates 365d43c32353607fdb21bcd286ee6706014b7ede
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 10.42.0.188:6379)
M: dae13b0bbc2674078851cce1953ba6d59caf4536 10.42.0.188:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 365d43c32353607fdb21bcd286ee6706014b7ede 10.42.1.47:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: f43e356d038c4af1730b8bf9f8807a1ac9c16957 10.42.2.193:6379
slots: (0 slots) slave
replicates dae13b0bbc2674078851cce1953ba6d59caf4536
S: e71993737dd8a69819917178f1c4dd479f40a431 10.42.0.189:6379
slots: (0 slots) slave
replicates e5210fb13d2dac4ca03d6a8165944a90f21c4373
M: e5210fb13d2dac4ca03d6a8165944a90f21c4373 10.42.2.192:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 080e184652ee18d3c262e39787b5cfd84d9558e9 10.42.1.48:6379
slots: (0 slots) slave
replicates 365d43c32353607fdb21bcd286ee6706014b7ede
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@redis-0:/data#部署Redis-Cluster-Proxy提供代理
如果需要在Kubernetes集群外部访问Redis Cluster则需要部署redis-cluster-proxy。
目前Redis官方还没有将redis-cluster-proxy打包为docker-iamge，因此需要手动打包docker-image。
找一台CentOS7，然后执行以下操作：
1. 安装GCC
因为Centos7自带的GCC版本为4.8.x，而编译redis-cluster-proxy要求gcc为4.9+
# Install GCC
yum install centos-release-scl
yum install devtoolset-8-gcc devtoolset-8-gcc-c++
scl enable devtoolset-8 -- bash2. 编译安装
# Make Install
git clone https://github.com/artix75/redis-cluster-proxy
cd redis-cluster-proxy
make PREFIX=/usr/local/redis_cluster_proxy install3. docker image打包
在目录/usr/local/redis_cluster_proxy/bin/ 下复制redis-cluster-proxy到当前目录，然后创建dockerfile 内容如下：
FROM centos:7
WORKDIR /data
ADD redis-cluster-proxy /usr/local/bin/
EXPOSE 6379执行打包操作
docker build . -t redis-cluster-proxy:v14. 在K8S中运行redis-cluster-proxy
redis-cluster-proxy configmap
---
# Redis-Proxy Config
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-proxy
namespace: demo
data:
proxy.conf: |
cluster redis-cluster:6379 # 配置为Redis Cluster Service
bind 0.0.0.0
port 6379 # redis-cluster-proxy 对外暴露端口
threads 8 # 线程数量
daemonize no
enable-cross-slot yes
auth P@ssw0rd # 配置Redis Cluster 认证密码
log-level errorredis-cluster-proxy Deployment
---
# Redis-Proxy NodePort
apiVersion: v1
kind: Service
metadata:
name: redis-proxy
namespace: demo
spec:
type: NodePort # 对K8S外部提供服务
ports:
- name: redis-proxy
nodePort: 30001 # 对外提供的端口
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis-proxy
---
# Redis-Proxy Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-proxy
namespace: demo
spec:
replicas: 1
selector:
matchLabels:
app: redis-proxy
template:
metadata:
labels:
app: redis-proxy
spec:
imagePullSecrets:
- name: harbor
containers:
- name: redis-proxy
image: redis-cluster-proxy:v1
imagePullPolicy: Always
command: ["redis-cluster-proxy"]
args:
- -c
- /data/proxy.conf # 指定启动配置文件
ports:
- name: redis-6379
containerPort: 6379
protocol: TCP
volumeMounts:
- name: redis-proxy-conf
mountPath: /data/
volumes: # 挂载proxy配置文件
- name: redis-proxy-conf
configMap:
name: redis-proxy