Zookeeper and Replicas
Now we’ve seen how to setup a basic cluster and upgrade it. Time to step up our game and setup our cluster with Zookeeper, and then add persistent storage to it.
The clickhouse-operator does not install or manage Zookeeper. Zookeeper must be provided and managed externally. The samples below are examples on establishing Zookeeper to provide replication support. For more information running and configuring Zookeeper, see the Apache Zookeeper site.
This step can not be skipped - your Zookeeper instance must have been set up externally from your ClickHouse clusters. Whether your Zookeeper installation is hosted by other Docker Images or separate servers is up to you.
Install Zookeeper
Kubernetes Zookeeper Deployment
A simple method of installing a single Zookeeper node is provided from the clickhouse-operator deployment samples. These provide samples deployments of Grafana, Prometheus, Zookeeper and other applications.
See the clickhouse-operator deployment directory for a full list of sample scripts and Kubernetes deployment files.
The instructions below will create a new Kubernetes namespace zoo1ns, and create a Zookeeper node in that namespace. Kubernetes nodes will refer to that Zookeeper node by the hostname zookeeper.zoo1ns within the created Kubernetes networks.
To deploy a single Zookeeper node in Kubernetes from the clickhouse-operator Github repository:
-
Download the
clickhouse-operatorGithub repository, either withgit clone https://github.com/Altinity/clickhouse-operator.gitor by selecting Code->Download Zip from the clickhouse-operator Github repository. -
From a terminal, navigate to the
deploy/zookeeperdirectory and run the following:quick-start-volume-emptyDir/zookeeper-1-node-create.sh -
Verify the Zookeeper node is running in Kubernetes:
NAME READY STATUS RESTARTS AGE pod/zookeeper-0 1/1 Running 0 6m42s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/zookeeper ClusterIP 10.109.185.191 <none> 2181/TCP,7000/TCP 6m42s service/zookeepers ClusterIP None <none> 2888/TCP,3888/TCP 6m42s NAME READY AGE statefulset.apps/zookeeper 1/1 6m42s -
Kubernetes nodes will be able to refer to the Zookeeper node by the hostname
zookeeper.zoo1ns.
Configure Kubernetes with Zookeeper
Once we start replicating clusters, we need Zookeeper to manage them. Create a new file sample03.yaml and populate it with the following:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 2
replicasCount: 2
Notice that we’re increasing the number of replicas from the sample02.yaml file in the First Clusters - No Storage tutorial.
We’ll set up a minimal Zookeeper connecting cluster by applying our new configuration file:
>> kubectl apply -f sample03.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 created
Verify it with the following:
>> kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.13.0 1 2 4 Completed 2 2 0 0 clickhouse-demo-01.test.svc.cluster.local
>> kubectl get service -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-01-demo-01-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 20m
chi-demo-01-demo-01-0-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 6m37s
chi-demo-01-demo-01-1-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 17m
chi-demo-01-demo-01-1-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 6m4s
clickhouse-demo-01 LoadBalancer 10.100.122.58 localhost 8123:31461/TCP,9000:31965/TCP 20m
If we log into our cluster and show the clusters, we can show the updated results and that we have a total of 4 replicas of demo-01 - two shards for each node with two replicas.
chi-demo-01-demo-01-0-1-0.chi-demo-01-demo-01-0-1.test.svc.cluster.local :) select * from system.clusters;
SELECT *
FROM system.clusters
Query id: 66019924-545e-4ae3-aa11-2b0ed4f203ba
┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐
│ all-replicated │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 10.1.0.36 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 2 │ chi-demo-01-demo-01-0-1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 3 │ chi-demo-01-demo-01-1-0 │ 10.1.0.38 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 4 │ chi-demo-01-demo-01-1-1 │ 10.1.0.39 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ all-sharded │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 10.1.0.36 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ all-sharded │ 2 │ 1 │ 1 │ chi-demo-01-demo-01-0-1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ all-sharded │ 3 │ 1 │ 1 │ chi-demo-01-demo-01-1-0 │ 10.1.0.38 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ all-sharded │ 4 │ 1 │ 1 │ chi-demo-01-demo-01-1-1 │ 10.1.0.39 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 10.1.0.36 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 2 │ chi-demo-01-demo-01-0-1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ demo-01 │ 2 │ 1 │ 1 │ chi-demo-01-demo-01-1-0 │ 10.1.0.38 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ demo-01 │ 2 │ 1 │ 2 │ chi-demo-01-demo-01-1-1 │ 10.1.0.39 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │ 0 │ 0 │
│ test_unavailable_shard │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │
│ test_unavailable_shard │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 1 │ 0 │ default │ │ 0 │ 0 │
└──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘
Distributed Tables
We have our clusters going - let’s test it out with some distributed tables so we can see the replication in action.
Login to your ClickHouse cluster and enter the following SQL statement:
CREATE TABLE test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')
Once our table is created, perform a select * from test command. We’ll see nothing because we didn’t give it any data, but that’s all right.
select * from test;
SELECT *
FROM test
Query id: cd727b6b-c9f8-4de1-abb5-9bcb42c7f05d
┌─dummy─┐
│ 0 │
└───────┘
┌─dummy─┐
│ 0 │
└───────┘
Now let’s test out our results coming in. Run the following command - this tells us just what shard is returning the results. It may take a few times, but you’ll start to notice the host name changes each time you run the command select hostName() from test:
select hostName() from test;
SELECT hostName()
FROM test
Query id: cb5a740d-1973-4138-8c08-94f978025671
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-1-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-0-0 │
└───────────────────────────┘
2 rows in set. Elapsed: 0.006 sec.
chi-demo-01-demo-01-0-1-0.chi-demo-01-demo-01-0-1.test.svc.cluster.local :) select hostName() from test;
SELECT hostName()
FROM test
Query id: 0028e8a4-0787-480e-ab6b-22ab24f27512
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-1-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-1-0 │ <--- That's different than before!
└───────────────────────────┘
2 rows in set. Elapsed: 0.005 sec.
This is showing us that the query is being distributed across different shards. The good news is you can change your configuration files to change the shards and replication however suits your needs.
One issue though: there’s no persistent storage. If these clusters stop running, your data vanishes. Next instruction will be on how to add persistent storage to your ClickHouse clusters running on Kubernetes. In fact, we can test by creating a new configuration file called sample04.yaml:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 1
replicasCount: 1
Make sure you’re exited out of your ClickHouse cluster, then install our configuration file:
>> kubectl apply -f sample04.yaml -n test
>> kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.13.0 1 1 1 InProgress 1 0 0 4 clickhouse-demo-01.test.svc.cluster.local
>> kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.13.0 1 2 4 Completed 0 2 0 0 clickhouse-demo-01.test.svc.cluster.local
Notice that during the update that four pods were deleted, and then two new ones added.
When your clusters are settled down and back down to just 1 shard with 1 replication, log back into your ClickHouse database and select from table test:
select * from test;
SELECT *
FROM test
Query id: 9c87a6d9-9feb-4102-97f0-37dfa20d470b
0 rows in set. Elapsed: 0.007 sec.
Received exception from server (version 21.1.2):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table default.test doesn't exist.
No persistent storage means any time your clusters are changed over, everything you’ve done is gone. The next article will cover how to correct that by adding storage volumes to your cluster.
Feedback
Was this page helpful?
Glad to hear it!
Sorry to hear that. We'll track this issue and see how we can improve.