Istio + k8s 安装 Redis 集群

一、安装 NFS 存储

创建 NFS 存储主要是为了给 Redis 提供稳定的后端存储，当 Redis 的 Pod 重启或迁移后，依然能获得原先的数据。这里，我们先要创建 NFS，然后通过使用 PV 为 Redis 挂载一个远程的 NFS 路径。

1.1 安装 NFS

yum -y install nfs-utils
yum -y install rpcbind

1.2 新增 /etc/exports 文件，设置共享路径

cat << EOF > /etc/exports
/data/redis/cluster0 *(rw,sync,no_root_squash)
/data/redis/cluster1 *(rw,sync,no_root_squash)
/data/redis/cluster2 *(rw,sync,no_root_squash)
/data/redis/cluster3 *(rw,sync,no_root_squash)
/data/redis/cluster4 *(rw,sync,no_root_squash)
/data/redis/cluster5 *(rw,sync,no_root_squash)
EOF

1.3 创建相应目录

mkdir -p /data/redis/cluster{0..5}

1.4 启动 NFS 和 rpcbind 服务

systemctl restart rpcbind
systemctl restart nfs
systemctl enable nfs
exportfs -v

1.5 客户端

yum -y install nfs-utils

1.6 查看存储共享

showmount -e 192.168.20.100

二、配置 Redis 集群

2.1 创建命名空间

namespace.yaml 内容如下：

apiVersion: v1
kind: Namespace
metadata:
  name: scut-tech

命令如下：

kubectl apply -f namespace.yaml

2.2 创建 PV

每一个 Redis Pod 都需要一个独立的 PV 来存储自己的数据，因此可以创建一个 pv.yaml 文件，包含6个PV（具体内容请查看 pv.yaml）。

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv0
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster0

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv2
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster4
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  nfs:
    server: 192.168.20.100
    path: /data/redis/cluster5

命令如下：

kubectl create -f pv.yaml

2.3 创建 Configmap

2.3.1 创建 redis.conf

cat << EOF > redis.conf
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
EOF

2.3.2 根据 redis.conf 创建 Configmap

kubectl create configmap redis-conf -n scut-tech --from-file=redis.conf

2.3.3 查看创建的 Configmap

kubectl describe cm redis-conf -n scut-tech

三、创建 Redis 集群

3.1 创建 redis.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: scut-tech
  labels:
    app: redis
spec:
  selector:
    app: redis
    appCluster: redis-cluster
  ports:
  - name: redis
    port: 6379
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: redis-access
  namespace: scut-tech
  labels:
    app: redis
spec:
  selector:
    app: redis
    appCluster: redis-cluster
  ports:
  - name: redis-access
    protocol: TCP
    port: 6379
    targetPort: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: scut-tech
spec:
  serviceName: redis
  replicas: 6
  selector:
    matchLabels:
      app: redis
      appCluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: redis:latest
        command:
        - "redis-server"
        args:
        - "/etc/redis/redis.conf"
        - "--protected-mode"
        - "no"
        - "--cluster-announce-ip"
        - "$(POD_IP)"
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        resources:
          requests:
            cpu: "1000m"
            memory: "1Gi"
        ports:
        - containerPort: 6379
          name: redis
          protocol: TCP
        - containerPort: 16379
          name: cluster
          protocol: TCP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis
        - name: data
          mountPath: /var/lib/redis
      volumes:
      - name: conf
        configMap:
          name: redis-conf
          items:
          - key: redis.conf
            path: redis.conf
  volumeClaimTemplates:
  - metadata:
      name: data
      namespace: scut-tech
    spec:
      accessModes: ["ReadWriteMany"]
      resources:
        requests:
          storage: 2Gi

命令如下：

kubectl apply -f redis.yaml

3.2 初始化集群

查看集群节点状态：

kubectl get pods -n scut-tech

所有节点（redis-0 ~ redis-5）的 STATUS 都是 Running 状态后，再执行如下命令初始化集群：

kubectl exec -it redis-0 -n scut-tech -- redis-cli --cluster create `kubectl get pods -n scut-tech -l app=redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 '|awk '{print $1" "$2" "$3" "$4" "$5" "$6" "}'` --cluster-replicas 1

3.3 验证集群部署

kubectl exec -it redis-0 -n scut-tech -- redis-cli cluster info 
for x in $(seq 0 5); do echo "redis-$x"; kubectl exec redis-$x -n cluster-redis -- redis-cli role; echo; done

3.4 查看pod对应的DNS域名

for x in $(seq 0 5); do kubectl exec redis-$x -n scut-tech -- hostname -f; done

四、问题解答

4.1 redis 集群出错怎样处理？

4.1.1 删除集群

kubectl delete -f redis.yaml

4.1.2删除 nfs 配置

rm -rf /data/redis/cluster*/**

4.1.3 重启集群

kubectl apply -f redis.yaml

4.1.4 重新初始化集群

查看集群状态，所有的 redis 节点都正常启动后，再执行如下命令初始化集群：

kubectl exec -it redis-0 -n scut-tech -- redis-cli --cluster create `kubectl get pods -n scut-tech -l app=redis -o jsonpath='{range.items[*]}{.status.podIP}:6379 '|awk '{print $1" "$2" "$3" "$4" "$5" "$6" "}'` --cluster-replicas 1