Redis: Add sentinel-aware proxy

[ghmer]

Name and Version

redis 0.16.0

What is the problem this feature will solve?

when using the sentinel architecture, the redis client must be sentinel-aware. Otherwise, chances are that the plain redis client connects to one of the replicas, eventually resulting in write-attempts to a read-only replica.

As this client might be included in a piece of software we cannot modify ourselves, using a high availability architecture absurdly causes more outages then relying on the standalone architecture.

Describe the solution you'd like?

I imagine a proxy that can interact with sentinel to always be aware of the current master. A plain redis client then can connect to the proxy instance, effectively leveraging the high availability architecture without having to modify the code.

This could be achieved in these steps:

1. have an initcontainer performing DNS lookups to discover all Redis Sentinel pods via the headless service, then iterate over those to establish a connection with the first responsive Sentinel instance using redis-cli PING commands.
2. Once connected to a Sentinel, query it to auto-discover the master name using SENTINEL masters. Retrieve the complete Redis topology including current master IP/port and replica information via SENTINEL master and SENTINEL replicas commands.
3. Generate a complete HAProxy configuration file with TCP health checks that authenticate with Redis, verify connectivity via PING, and crucially check role:master to ensure traffic only routes to the current master. Dynamically adds all StatefulSet pods (redis-0, redis-1, redis-2, ...) as backend servers.
4. HAProxy's built-in health checks continuously execute the TCP check sequence on all backend servers. Only the server responding with role:master passes the health check, ensuring automatic failover when Sentinel promotes a new master during failures.
5. Alternatively, HAProxy provides an agent-check feature. This could query sentinel in real-time to determine the current master.

What alternatives have you considered?

Buying new software that utilizes sentinel... But management did not like this idea.

Additional informations

I created a proof-of-concept that seems to work, but only tested on a local kubernetes cluster with little to none restrictions. I.E. installing "socat" is probably not a great idea. Nevertheless, maybe you get some inspiration.

Thanks for this awesome helm chart!

redis-proxy-configmap.yaml

redis-proxy-deployment.yaml

Affected Helm charts

redis

[nfarhadian]

Maybe a better approach is creating a separate pod (or sidecar) which periodically checks for Redis master using sentinel and adds a label on master pod. Then we can have a service with that label selector which always selects master pod.
This way we don't need a proxy.

[ckalaka]

FWIF we solved it like this (it's ansible, the jinja has nothing todo with the redis helm chart itself):

architecture: replication
persistence:
  storageClass: {{ local_storage_class }}
auth:
  enabled: false # can also be true
  sentinel: false # if set to true will probably require changing the redis-cli invocation
sentinel:
  enabled: true
metrics:
  enabled: {{ helm_prometheus_installed | default(false) }}
  serviceMonitor:
    enabled: {{ helm_prometheus_installed | default(false) }}
nodeSelector:
  {{ local_storage_selector }}: "true"

### important part starts here ###
serviceAccount:
  create: true
extraObjects:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: redis-master-svc
      namespace: "{{ namespace }}"
    rules:
      - apiGroups: [""]
        resources: ["pods", "services"]
        verbs: ["get", "list"]
      - apiGroups: [""]
        resources: ["pods/exec"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["patch"]
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: redis-master-svc
      namespace: "{{ namespace }}"
    roleRef:
      kind: Role
      name: redis-master-svc
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: redis
        namespace: "{{ namespace }}"
  - apiVersion: v1
    kind: Service
    metadata:
      name: redis-master
      namespace: "{{ namespace }}"
    spec:
      ports:
      - name: redis
        port: 6379
        protocol: TCP
        targetPort: redis
      selector:
        statefulset.kubernetes.io/pod-name: not-yet-decided
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: redis-master-svc
      namespace: "{{ namespace }}"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: redis-master-svc
      template:
        metadata:
          labels:
            app: redis-master-svc
        spec:
          serviceAccountName: redis
          automountServiceAccountToken: true
          containers:
          - name: redis-master-svc
            image: alpine/kubectl:{{ kubectl_version }}
            command: ["/bin/sh", "-c"]
            args: 
              - |
                oldmaster="<none>"
                while true
                do
                  pod=$(kubectl get pods --selector="app.kubernetes.io/instance=redis" --selector="app.kubernetes.io/name=redis" -o name | shuf | head -n1)
                  [ -z "$pod" ] && echo "did not get any pods" && sleep 10 && continue
                  master=$(kubectl exec redis-0 -c sentinel -- redis-cli --raw -p 26379 sentinel GET-MASTER-ADDR-BY-NAME mymaster | head -n1 | grep -oE '^[^.]+')
                  [ -z "$master" ] && echo "did not get the master" && sleep 10 && continue
                  [ "$master" == "$oldmaster" ] && sleep 60 && continue
                  echo "new master: '$master' (was: $oldmaster)"
                  ! kubectl patch service redis-master -p '{"spec": {"selector": {"statefulset.kubernetes.io/pod-name": "'$master'"}}}' && echo "failed to update service" && sleep 10 && continue
                  oldmaster=$master
                  sleep 60
                done

This also assumes that redis is installed as "redis"

[zOnlyKroks]

Mind testing #703 for me?