Zookeeper and Replicas

Install Zookeeper and Replicas

Now we’ve seen how to setup a basic cluster and upgrade it. Time to step up our game and setup our cluster with Zookeeper, and then add persistent storage to it.

The clickhouse-operator does not install or manage Zookeeper. Zookeeper must be provided and managed externally. The samples below are examples on establishing Zookeeper to provide replication support. For more information running and configuring Zookeeper, see the Apache Zookeeper site.

This step can not be skipped - your Zookeeper instance must have been set up externally from your ClickHouse clusters. Whether your Zookeeper installation is hosted by other Docker Images or separate servers is up to you.

Install Zookeeper

Once we start replicating clusters, we need Zookeeper to manage them. Create a new file sample03.yaml and populate it with the following:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
        nodes:
        - host: zookeeper-0.zookeepers.test
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 2
          replicasCount: 2

Notice that we’re increasing the number of replicas from the sample02.yaml file in the First Clusters - No Storage tutorial.

We’ll set up a minimal Zookeeper connecting cluster by applying our new configuration file:

>> kubectl apply -f sample03.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 created

Verify it with the following:

>> kubectl -n test get chi -o wide
NAME      VERSION   CLUSTERS   SHARDS   HOSTS   STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT
demo-01   0.13.0    1          2        4       Completed   2         2       0         0        clickhouse-demo-01.test.svc.cluster.local

>> kubectl get service -n test
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
chi-demo-01-demo-01-0-0   ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      20m
chi-demo-01-demo-01-0-1   ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      6m37s
chi-demo-01-demo-01-1-0   ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      17m
chi-demo-01-demo-01-1-1   ClusterIP      None            <none>        8123/TCP,9000/TCP,9009/TCP      6m4s
clickhouse-demo-01        LoadBalancer   10.100.122.58   localhost     8123:31461/TCP,9000:31965/TCP   20m

If we log into our cluster and show the clusters, we can show the updated results and that we have a total of 4 replicas of demo-01 - two shards for each node with two replicas.

chi-demo-01-demo-01-0-1-0.chi-demo-01-demo-01-0-1.test.svc.cluster.local :) select * from system.clusters;

SELECT *
FROM system.clusters

Query id: 66019924-545e-4ae3-aa11-2b0ed4f203ba

┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐
 all-replicated                                        1             1            1  chi-demo-01-demo-01-0-0  10.1.0.36     9000         0  default                               0                        0 
 all-replicated                                        1             1            2  chi-demo-01-demo-01-0-1  127.0.0.1     9000         1  default                               0                        0 
 all-replicated                                        1             1            3  chi-demo-01-demo-01-1-0  10.1.0.38     9000         0  default                               0                        0 
 all-replicated                                        1             1            4  chi-demo-01-demo-01-1-1  10.1.0.39     9000         0  default                               0                        0 
 all-sharded                                           1             1            1  chi-demo-01-demo-01-0-0  10.1.0.36     9000         0  default                               0                        0 
 all-sharded                                           2             1            1  chi-demo-01-demo-01-0-1  127.0.0.1     9000         1  default                               0                        0 
 all-sharded                                           3             1            1  chi-demo-01-demo-01-1-0  10.1.0.38     9000         0  default                               0                        0 
 all-sharded                                           4             1            1  chi-demo-01-demo-01-1-1  10.1.0.39     9000         0  default                               0                        0 
 demo-01                                               1             1            1  chi-demo-01-demo-01-0-0  10.1.0.36     9000         0  default                               0                        0 
 demo-01                                               1             1            2  chi-demo-01-demo-01-0-1  127.0.0.1     9000         1  default                               0                        0 
 demo-01                                               2             1            1  chi-demo-01-demo-01-1-0  10.1.0.38     9000         0  default                               0                        0 
 demo-01                                               2             1            2  chi-demo-01-demo-01-1-1  10.1.0.39     9000         0  default                               0                        0 
 test_cluster_two_shards                               1             1            1  127.0.0.1                127.0.0.1     9000         1  default                               0                        0 
 test_cluster_two_shards                               2             1            1  127.0.0.2                127.0.0.2     9000         0  default                               0                        0 
 test_cluster_two_shards_internal_replication          1             1            1  127.0.0.1                127.0.0.1     9000         1  default                               0                        0 
 test_cluster_two_shards_internal_replication          2             1            1  127.0.0.2                127.0.0.2     9000         0  default                               0                        0 
 test_cluster_two_shards_localhost                     1             1            1  localhost                127.0.0.1     9000         1  default                               0                        0 
 test_cluster_two_shards_localhost                     2             1            1  localhost                127.0.0.1     9000         1  default                               0                        0 
 test_shard_localhost                                  1             1            1  localhost                127.0.0.1     9000         1  default                               0                        0 
 test_shard_localhost_secure                           1             1            1  localhost                127.0.0.1     9440         0  default                               0                        0 
 test_unavailable_shard                                1             1            1  localhost                127.0.0.1     9000         1  default                               0                        0 
 test_unavailable_shard                                2             1            1  localhost                127.0.0.1        1         0  default                               0                        0 
└──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘

Distributed Tables

We have our clusters going - let’s test it out with some distributed tables so we can see the replication in action.

Login to your ClickHouse cluster and enter the following SQL statement:

CREATE TABLE test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')

Once our table is created, perform a select * from test command. We’ll see nothing because we didn’t give it any data, but that’s all right.

select * from test;

SELECT *
FROM test

Query id: cd727b6b-c9f8-4de1-abb5-9bcb42c7f05d

┌─dummy─┐
     0 
└───────┘
┌─dummy─┐
     0 
└───────┘

Now let’s test out our results coming in. Run the following command - this tells us just what shard is returning the results. It may take a few times, but you’ll start to notice the host name changes each time you run the command select hostName() from test:

select hostName() from test;

SELECT hostName()
FROM test

Query id: cb5a740d-1973-4138-8c08-94f978025671

┌─hostName()────────────────┐
 chi-demo-01-demo-01-0-1-0 
└───────────────────────────┘
┌─hostName()────────────────┐
 chi-demo-01-demo-01-1-0-0 
└───────────────────────────┘

2 rows in set. Elapsed: 0.006 sec.

chi-demo-01-demo-01-0-1-0.chi-demo-01-demo-01-0-1.test.svc.cluster.local :) select hostName() from test;

SELECT hostName()
FROM test

Query id: 0028e8a4-0787-480e-ab6b-22ab24f27512

┌─hostName()────────────────┐
 chi-demo-01-demo-01-0-1-0 
└───────────────────────────┘
┌─hostName()────────────────┐
 chi-demo-01-demo-01-1-1-0  <--- That's different than before!
└───────────────────────────┘

2 rows in set. Elapsed: 0.005 sec.

This is showing us that the query is being distributed across different shards. The good news is you can change your configuration files to change the shards and replication however suits your needs.

One issue though: there’s no persistent storage. If these clusters stop running, your data vanishes. Next instruction will be on how to add persistent storage to your ClickHouse clusters running on Kubernetes. In fact, we can test by creating a new configuration file called sample04.yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
        nodes:
        - host:  zookeeper-0.zookeepers.test
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 1
          replicasCount: 1

Make sure you’re exited out of your ClickHouse cluster, then install our configuration file:

>> kubectl apply -f sample04.yaml -n test
>> kubectl -n test get chi -o wide
NAME      VERSION   CLUSTERS   SHARDS   HOSTS   STATUS       UPDATED   ADDED   DELETED   DELETE   ENDPOINT
demo-01   0.13.0    1          1        1       InProgress   1         0       0         4        clickhouse-demo-01.test.svc.cluster.local
>> kubectl -n test get chi -o wide
NAME      VERSION   CLUSTERS   SHARDS   HOSTS   STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT
demo-01   0.13.0    1          2        4       Completed   0         2       0         0        clickhouse-demo-01.test.svc.cluster.local

Notice that during the update that four pods were deleted, and then two new ones added.

When your clusters are settled down and back down to just 1 shard with 1 replication, log back into your ClickHouse database and select from table test:

select * from test;

SELECT *
FROM test

Query id: 9c87a6d9-9feb-4102-97f0-37dfa20d470b


0 rows in set. Elapsed: 0.007 sec.


Received exception from server (version 21.1.2):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table default.test doesn't exist.

No persistent storage means any time your clusters are changed over, everything you’ve done is gone. The next article will cover how to correct that by adding storage volumes to your cluster.