Enabling Replication

Adding replication to your ClickHouse® cluster

There are two ways to add replication support to your ClickHouse cluster: ClickHouse Keeper and Zookeeper. Here are the installation instructions for both:

When you install the operator, it creates four custom resource definitions (CRDs). We’ve already worked with the ClickHouseInstallation, but we also have a ClickHouseKeeperInstallation (abbreviation chk) that makes it easy to install ClickHouse Keeper.

Copy and paste the following into keeper.yaml:

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: clickhouse-keeper
spec:
  configuration:
    clusters:
      - name: "chk01"
        layout:
          replicasCount: 3

  defaults:
    templates:
      # Templates are specified as default for all clusters
      podTemplate: default
      dataVolumeClaimTemplate: default

  templates:
    podTemplates:
      - name: default
        metadata:
          labels:
            app: clickhouse-keeper
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: "clickhouse/clickhouse-keeper:latest"
              resources:
                requests:
                  memory: "256M"
                  cpu: "1"
                limits:
                  memory: "4Gi"
                  cpu: "2"
          securityContext:
            fsGroup: 101

    volumeClaimTemplates:
      - name: default
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

Now use kubectl apply to create the ClickHouseKeeperInstallation:

kubectl apply -f keeper.yaml -n operator

This creates the chk:

clickhousekeeperinstallation.clickhouse-keeper.altinity.com/clickhouse-keeper created

You’ll need to wait until state of the chk is Completed:

kubectl get chk -o wide -n operator

It will take a minute, but you’ll see this at some point:

NAME                VERSION   CLUSTERS   SHARDS   HOSTS   TASKID   STATUS      HOSTS-UNCHANGED   HOSTS-UPDATED   HOSTS-ADDED   HOSTS-COMPLETED   HOSTS-DELETED   HOSTS-DELETE   ENDPOINT                                              AGE     SUSPEND
clickhouse-keeper   0.25.1    1          1        3                Completed                                                                                                    keeper-clickhouse-keeper.operator.svc.cluster.local   3m47s

We’ll reference the chk resource in a YAML file to enable replication in our ClickHouse cluster.

Adding a replica to our cluster

The spec for a ClickHouse Installation includes a zookeeper parameter. (Yes, we’re using ClickHouse Keeper, but the zookeeper parameter works for both.) Copy this text and save it in the file manifest03.yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: cluster01
spec:
  templates:
    podTemplates:
      - name: clickhouse-pod-template
        spec:
          containers:
            - name: clickhouse
              image: altinity/clickhouse-server:24.8.14.10501.altinitystable
              volumeMounts:
                - name: clickhouse-storage
                  mountPath: /var/lib/clickhouse
    volumeClaimTemplates:
      - name: clickhouse-storage
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
          storageClassName: standard
  configuration:
    zookeeper:
      nodes:
        - host: keeper-clickhouse-keeper.operator.svc.cluster.local
          port: 2181
    clusters:
      - name: cluster01
        layout:
          shardsCount: 1
          replicasCount: 2
        templates:
          podTemplate: clickhouse-pod-template

Notice that we’re increasing the number of replicas (replicasCount) from the manifest02.yaml file on the Adding persistent storage to your cluster page.

We’re updating the definition of our chi object to point to a chk object. To get the object’s endpoint, use this command:

kubectl get chk -n operator -o custom-columns="NAME:.metadata.name,ENDPOINT:.status.endpoint"

You’ll see something like this:

NAME                ENDPOINT
clickhouse-keeper   keeper-clickhouse-keeper.operator.svc.cluster.local

Take a look at manifest03.yaml. Make sure the endpoint of the chk object is the value of the configuration\zookeeper\nodes\host field. Also be sure the storageClass is set correctly for your cloud provider, then apply the new configuration file:

kubectl apply -f manifest03.yaml -n operator

You’ll get immediate feedback:

clickhouseinstallation.clickhouse.altinity.com/cluster01 configured

Check the status of your updated chi:

kubectl get chi -o wide -n operator

Wait until the status of the chi is Completed. Here we can see that we have one cluster, one shard, and two hosts.

NAME        VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      HOSTS-COMPLETED   HOSTS-UPDATED   HOSTS-ADDED   HOSTS-DELETED   ENDPOINT                                          AGE   SUSPEND
cluster01   0.25.1    1          1        2       94041e87-8689-4c68-8e87-f8cd9165c6a9   Completed                                                                   clickhouse-cluster01.operator.svc.cluster.local   33m

If we log into any of the two hosts in our cluster and look at the system.clusters table, we can show the updated results and that we have a total of two hosts for cluster01 - one each of the two replicas.

kubectl exec -it chi-cluster01-cluster01-0-1-0 -n operator -- clickhouse-client

We’ll select a few parameters that we care about:

SELECT
    cluster,
    shard_num,
    replica_num,
    host_name,
    host_address
FROM system.clusters
WHERE cluster = 'cluster01'

   ┌─cluster───┬─shard_num─┬─replica_num─┬─host_name───────────────────┬─host_address─┐
1. │ cluster01 │         1 │           1 │ chi-cluster01-cluster01-0-0 │ 10.244.2.4   │
2. │ cluster01 │         1 │           2 │ chi-cluster01-cluster01-0-1 │ 127.0.0.1    │
   └───────────┴───────────┴─────────────┴─────────────────────────────┴──────────────┘

2 rows in set. Elapsed: 0.001 sec.

👉 Type exit to end the clickhouse-client session.

Now that we’ve got ClickHouse Keeper in place, we’re ready to create a table that uses the ReplicatedMergeTree engine, and that engine will keep all our replicas synchronized. So let’s move on….

Zookeeper is easy to install and enable. (Although we should say that if you’re using the operator on minikube, it may be difficult to get Zookeeper’s PersistentVolumeClaims configured correctly. Proceed with caution. Or use ClickHouse Keeper instead.)

We’ll use kubectl apply to create a three-node Zookeeper deployment:

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/release-0.25.1/deploy/zookeeper/zookeeper-manually/quick-start-persistent-volume/zookeeper-3-nodes.yaml -n operator

This creates some new resources:

service/zookeeper created
service/zookeepers created
poddisruptionbudget.policy/zookeeper-pod-disruption-budget created
statefulset.apps/zookeeper created

Now make sure things are running. Run this command:

kubectl get all -n operator

Wait until you see READY 1/1 and Running for the three Zookeeper pods and READY 3/3 for its statefulset:

NAME                                       READY   STATUS    RESTARTS   AGE
pod/chi-cluster01-cluster01-0-0-0          1/1     Running   0          3m59s
pod/clickhouse-operator-6f655c97bd-crvhl   2/2     Running   0          9m55s
pod/zookeeper-0                            1/1     Running   0          2m15s
pod/zookeeper-1                            1/1     Running   0          92s
pod/zookeeper-2                            1/1     Running   0          49s

NAME                                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/chi-cluster01-cluster01-0-0   ClusterIP   None           <none>        9000/TCP,8123/TCP,9009/TCP   8m41s
service/clickhouse-cluster01          ClusterIP   None           <none>        8123/TCP,9000/TCP            8m29s
service/clickhouse-operator-metrics   ClusterIP   10.0.176.159   <none>        8888/TCP,9999/TCP            9m55s
service/zookeeper                     ClusterIP   10.0.47.117    <none>        2181/TCP,7000/TCP            2m15s
service/zookeepers                    ClusterIP   None           <none>        2888/TCP,3888/TCP            2m15s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clickhouse-operator   1/1     1            1           9m55s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/clickhouse-operator-6f655c97bd   1         1         1       9m55s

NAME                                           READY   AGE
statefulset.apps/chi-cluster01-cluster01-0-0   1/1     5m30s
statefulset.apps/zookeeper                     3/3     2m15s

Now we’ve got a running Zookeeper, so we can tell our ClickHouse cluster how to connect to it…and add a second replica.

Adding a replica to our cluster

The spec for a ClickHouseInstallation includes a zookeeper parameter. Copy the following text and save it as manifest03.yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: cluster01
spec:
  templates:
    podTemplates:
      - name: clickhouse-pod-template
        spec:
          containers:
            - name: clickhouse
              image: altinity/clickhouse-server:24.8.14.10501.altinitystable
              volumeMounts:
                - name: clickhouse-storage
                  mountPath: /var/lib/clickhouse
    volumeClaimTemplates:
      - name: clickhouse-storage
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
          storageClassName: standard
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper.operator.svc.cluster.local
          port: 2181
    clusters:
      - name: cluster01
        layout:
          shardsCount: 1
          replicasCount: 2
        templates:
          podTemplate: clickhouse-pod-template

Notice that we’re increasing the number of replicas from the manifest02.yaml file on the Adding persistent storage to your cluster page. The hostname for Zookeeper is the name of the service (zookeeper), its namespace (operator), followed by .svc.cluster.local.

Be sure you’re using the correct storageClassName for your Kubernetes provider. With that done, apply the new configuration file:

kubectl apply -f manifest03.yaml -n operator

You’ll get immediate feedback:

clickhouseinstallation.clickhouse.altinity.com/cluster01 configured

It will likely take a minute or two, but you should see your updated chi:

kubectl get chi -o wide -n operator

Wait until the status of the chi is Completed. Here we can see that we have one cluster, one shard, and two hosts.

NAME        VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      HOSTS-COMPLETED   HOSTS-UPDATED   HOSTS-ADDED   HOSTS-DELETED   ENDPOINT                                          AGE   SUSPEND
cluster01   0.25.1    1          1        2       21dfe453-2967-446e-933b-63880c3d0014   Completed                                                                   clickhouse-cluster01.operator.svc.cluster.local   17m

(Adding the -o wide option shows more data.) Taking a look at our new layout, we now have three services in our namespace:

kubectl get service -n operator

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
chi-cluster01-cluster01-0-0   ClusterIP   None           <none>        9000/TCP,8123/TCP,9009/TCP   20m
chi-cluster01-cluster01-0-1   ClusterIP   None           <none>        9000/TCP,8123/TCP,9009/TCP   6m45s
clickhouse-cluster01          ClusterIP   None           <none>        8123/TCP,9000/TCP            19m
clickhouse-operator-metrics   ClusterIP   10.0.176.159   <none>        8888/TCP,9999/TCP            21m
zookeeper                     ClusterIP   10.0.47.117    <none>        2181/TCP,7000/TCP            13m
zookeepers                    ClusterIP   None           <none>        2888/TCP,3888/TCP            13m

If we log into either of the hosts in our cluster (chi-cluster01-cluster01-0-0 or chi-cluster01-cluster01-0-1) and look at the system.clusters table, we can show the updated results and that we have a total of two hosts for cluster01 - one for each of the two replicas.

kubectl exec -it chi-cluster01-cluster01-0-1-0 -n operator -- clickhouse-client

We’ll select a few parameters that we care about:

SELECT
    cluster,
    shard_num,
    replica_num,
    host_name,
    host_address
FROM system.clusters
WHERE cluster = 'cluster01'

   ┌─cluster───┬─shard_num─┬─replica_num─┬─host_name───────────────────┬─host_address─┐
1. │ cluster01 │         1 │           1 │ chi-cluster01-cluster01-0-0 │ 10.244.2.4   │
2. │ cluster01 │         1 │           2 │ chi-cluster01-cluster01-0-1 │ 127.0.0.1    │
   └───────────┴───────────┴─────────────┴─────────────────────────────┴──────────────┘

2 rows in set. Elapsed: 0.003 sec.

👉 Type exit to end the clickhouse-client session.

Now that we’ve got Zookeeper in place, we’re ready to work with our replicas. We’ll set up tables with the ReplicatedMergeTree engine, and that engine will keep all our replicas synchronized. So let’s move on….

👉 Next: Working with replication