This is the multi-page printable view of this section. Click here to print.
If you’re running the Altinity Kubernetes Operator for the first time, or just want to get it up and running as quickly as possible, the Quick Start Guide is for you.
Requirements:
clickhouse-client. Full instructions for installing ClickHouse can be found on the ClickHouse Installation page.The Altinity Kubernetes Operator can be installed in just a few minutes with a single command into your existing Kubernetes environment.
For those who need a more customized installation or want to build the Altinity Kubernetes Operator themselves can do so through the Operator Installation Guide.
Before starting, make sure you have the following installed:
To install the Altinity Kubernetes Operator into your existing Kubernetes environment, run the following or download the Altinity Kubernetes Operator install file to modify it as best fits your needs. For more information custom Altinity Kubernetes Operator settings that can be applied, see the Operator Guide.
We recommend that when installing the Altinity Kubernetes Operator, specify the version to be installed. This insures maximum compatibility with applications and established Kubernetes environments running your ClickHouse clusters. For more information on installing other versions of the Altinity Kubernetes Operator, see the specific Version Installation Guide.
The most current version is 0.18.1:
kubectl apply -f https://github.com/Altinity/clickhouse-operator/raw/0.18.1/deploy/operator/clickhouse-operator-install-bundle.yaml
Never delete the operator or run the following command while there are live ClickHouse clusters managed by the operator:
kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
The command will hang due to the live clusters. If you then re-install the operator, those clusters will be deleted and the operator will not work correctly.
See Altinity/clickhouse-operator#830 for more details.
Output similar to the following will be displayed on a successful installation. For more information on the resources created in the installation, see [Altinity Kubernetes Operator Resources]({<ref “operatorresources” >})
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrole.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created
To verify that the installation was successful, run the following.
On a successful installation you’ll be able to see the clickhouse-operator pod
under the NAME column.
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
clickhouse-operator-77b54889b4-8sgt5 2/2 Running 0 5s
coredns-78fcd69978-nthp2 1/1 Running 2 (101m ago) 23d
etcd-minikube 1/1 Running 2 (101m ago) 23d
kube-apiserver-minikube 1/1 Running 2 (101m ago) 23d
kube-controller-manager-minikube 1/1 Running 2 (101m ago) 23d
kube-proxy-lsggn 1/1 Running 2 (101m ago) 23d
kube-scheduler-minikube 1/1 Running 2 (101m ago) 23d
storage-provisioner 1/1 Running 5 (100m ago) 23d
If you followed the Quick Installation guide, then you have the Altinity Kubernetes Operator for Kubernetes installed. Let’s give it something to work with.
For our examples, we’ll be setting up our own Kubernetes namespace test.
All of the examples will be installed into that namespace so we can track
how the cluster is modified with new updates.
Create the namespace with the following kubectl command:
kubectl create namespace test
namespace/test created
Just to make sure we’re in a clean environment, let’s check for any resources in our namespace:
kubectl get all -n test
No resources found in test namespace.
We’ll start with the simplest cluster: one shard, one replica. This template and others are available on the Altinity Kubernetes Operator Github example site, and contains the following:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
clusters:
- name: "demo-01"
layout:
shardsCount: 1
replicasCount: 1
Save this as sample01.yaml and launch it with the following:
kubectl apply -n test -f sample01.yaml
clickhouseinstallation.clickhouse.altinity.com/demo-01 created
Verify that the new cluster is running. When the status is
Running then it’s complete.
kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.18.1 1 1 1 6d1d2c3d-90e5-4110-81ab-8863b0d1ac47 Completed 1 clickhouse-demo-01.test.svc.cluster.local
To retrieve the IP information use the get service option:
kubectl get service -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-01-demo-01-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 2s
clickhouse-demo-01 LoadBalancer 10.111.27.86 <pending> 8123:31126/TCP,9000:32460/TCP 19s
So we can see our pods is running, and that we have the" load balancer for the cluster.
Let’s talk to our cluster and run some simple ClickHouse queries.
We can hop in directly through Kubernetes and run the clickhouse-client
that’s part of the image with the following command:
kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- clickhouse-client
ClickHouse client version 20.12.4.5 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.12.4 revision 54442.
chi-demo-01-demo-01-0-0-0.chi-demo-01-demo-01-0-0.test.svc.cluster.local :)
From within ClickHouse, we can check out the current clusters:
SELECT * FROM system.clusters
┌─cluster─────────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 2 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 3 │ 127.0.0.3 │ 127.0.0.3 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 1 │ 0 │ default │ │ 0 │ 0 │ 0 │
└─────────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘
Exit out of your cluster:
chi-demo-01-demo-01-0-0-0.chi-demo-01-demo-01-0-0.test.svc.cluster.local :) exit
Bye.
You can also use a remote client such as clickhouse-client to
connect to your cluster through the LoadBalancer.
The default username and password is set by the
clickhouse-operator-install.yaml file. These values can be altered
by changing the chUsername and chPassword ClickHouse
Credentials section:
clickhouse_operatorclickhouse_operator_password# ClickHouse credentials (username, password and port) to be used
# by operator to connect to ClickHouse instances for:
# 1. Metrics requests
# 2. Schema maintenance
# 3. DROP DNS CACHE
# User with such credentials can be specified in additional ClickHouse
# .xml config files,
# located in `chUsersConfigsPath` folder
chUsername: clickhouse_operator
chPassword: clickhouse_operator_password
chPort: 8123
In either case, the command to connect to your new cluster will
resemble the following, replacing {LoadBalancer hostname} with
the name or IP address or your LoadBalancer, then providing
the proper password. In our examples so far, that’s been localhost.
From there, just make your ClickHouse SQL queries as you please - but remember that this particular cluster has no persistent storage. If the cluster is modified in any way, any databases or tables created will be wiped clean.
Well that’s great - we have a cluster running. Granted, it’s really small and doesn’t do much, but what if we want to upgrade it?
Sure - we can do that any time we want.
Take your sample01.yaml and save it as sample02.yaml.
Let’s update it to give us two shards running with one replica:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
clusters:
- name: "demo-01"
layout:
shardsCount: 2
replicasCount: 1
Save your YAML file, and apply it. We’ve defined the name
in the metadata, so it knows exactly what cluster to update.
kubectl apply -n test -f sample02.yaml
clickhouseinstallation.clickhouse.altinity.com/demo-01 configured
Verify that the cluster is running - this may take a few minutes depending on your system:
kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.18.1 1 2 2 80102179-4aa5-4e8f-826c-1ca7a1e0f7b9 Completed 1 clickhouse-demo-01.test.svc.cluster.local
kubectl get service -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-01-demo-01-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 26s
chi-demo-01-demo-01-1-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 3s
clickhouse-demo-01 LoadBalancer 10.111.27.86 <pending> 8123:31126/TCP,9000:32460/TCP 43s
Once again, we can reach right into our cluster with
clickhouse-client and look at the clusters.
clickhouse-client --host localhost --user=clickhouse_operator --password=clickhouse_operator_password
ClickHouse client version 20.12.4.5 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.12.4 revision 54442.
chi-demo-01-demo-01-1-0-0.chi-demo-01-demo-01-1-0.test.svc.cluster.local :)
SELECT * FROM system.clusters
┌─cluster─────────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 2 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_one_shard_three_replicas_localhost │ 1 │ 1 │ 3 │ 127.0.0.3 │ 127.0.0.3 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 1 │ 0 │ default │ │ 0 │ 0 │ 0 │
└─────────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘
So far, so good. We can create some basic clusters. If we want to do more, we’ll have to move ahead with replication and zookeeper in the next section.
kubectl create namespace test
namespace/test created
Now we’ve seen how to setup a basic cluster and upgrade it. Time to step up our game and setup our cluster with Zookeeper, and then add persistent storage to it.
The Altinity Kubernetes Operator does not install or manage Zookeeper. Zookeeper must be provided and managed externally. The samples below are examples on establishing Zookeeper to provide replication support. For more information running and configuring Zookeeper, see the Apache Zookeeper site.
This step can not be skipped - your Zookeeper instance must have been set up externally from your ClickHouse clusters. Whether your Zookeeper installation is hosted by other Docker Images or separate servers is up to you.
A simple method of installing a single Zookeeper node is provided from the Altinity Kubernetes Operator deployment samples. These provide samples deployments of Grafana, Prometheus, Zookeeper and other applications.
See the Altinity Kubernetes Operator deployment directory for a full list of sample scripts and Kubernetes deployment files.
The instructions below will create a new Kubernetes namespace zoo1ns,
and create a Zookeeper node in that namespace.
Kubernetes nodes will refer to that Zookeeper node by the hostname
zookeeper.zoo1ns within the created Kubernetes networks.
To deploy a single Zookeeper node in Kubernetes from the Altinity Kubernetes Operator Github repository:
Download the Altinity Kubernetes Operator Github repository, either with
git clone https://github.com/Altinity/clickhouse-operator.git or by selecting Code->Download Zip from the
Altinity Kubernetes Operator GitHub repository
.
From a terminal, navigate to the deploy/zookeeper directory
and run the following:
cd clickhouse-operator/deploy/zookeeper
./quick-start-volume-emptyDir/zookeeper-1-node-create.sh
namespace/zoo1ns created
service/zookeeper created
service/zookeepers created
[33;1mWarning:[0m policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/zookeeper-pod-disruption-budget created
statefulset.apps/zookeeper created
kubectl get all --namespace zoo1ns
NAME READY STATUS RESTARTS AGE
pod/zookeeper-0 0/1 Running 0 2s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zookeeper ClusterIP 10.98.191.251 <none> 2181/TCP,7000/TCP 3s
service/zookeepers ClusterIP None <none> 2888/TCP,3888/TCP 3s
NAME READY AGE
statefulset.apps/zookeeper 0/1 3s
zookeeper.zoo1ns.Once we start replicating clusters, we need Zookeeper to manage them.
Create a new file sample03.yaml and populate it with the following:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 2
replicasCount: 2
templates:
podTemplate: clickhouse-stable
templates:
podTemplates:
- name: clickhouse-stable
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:21.8.10.1.altinitystable
Notice that we’re increasing the number of replicas from the
sample02.yaml file in the
[First Clusters - No Storage]({<ref “quickcluster”>}) tutorial.
We’ll set up a minimal Zookeeper connecting cluster by applying our new configuration file:
kubectl apply -f sample03.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 created
Verify it with the following:
kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.18.1 1 2 4 3c2cc45a-3d38-4e51-9eb5-f84cf4fcd778 Completed 4 clickhouse-demo-01.test.svc.cluster.local
kubectl get service -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-01-demo-01-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 110s
chi-demo-01-demo-01-0-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 93s
chi-demo-01-demo-01-1-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 48s
chi-demo-01-demo-01-1-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 17s
clickhouse-demo-01 LoadBalancer 10.103.31.139 <pending> 8123:30149/TCP,9000:30600/TCP 2m7s
If we log into our cluster and show the clusters, we can show
the updated results and that we have a total of 4 replicas
of demo-01 - two shards for each node with two replicas.
SELECT * FROM system.clusters
┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 2 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 3 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ all-replicated │ 1 │ 1 │ 4 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 2 │ 1 │ 1 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 3 │ 1 │ 1 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ all-sharded │ 4 │ 1 │ 1 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 1 │ 1 │ 2 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 2 │ 1 │ 1 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ demo-01 │ 2 │ 1 │ 2 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_internal_replication │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_cluster_two_shards_localhost │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │
│ test_unavailable_shard │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 1 │ 0 │ default │ │ 0 │ 0 │ 0 │
└──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘
We have our clusters going - let’s test it out with some distributed tables so we can see the replication in action.
Login to your ClickHouse cluster and enter the following SQL statement:
CREATE TABLE test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')
Once our table is created, perform a SELECT * FROM test command.
We’ll see nothing because we didn’t give it any data,
but that’s all right.
SELECT * FROM test
┌─dummy─┐
│ 0 │
└───────┘
┌─dummy─┐
│ 0 │
└───────┘
Now let’s test out our results coming in.
Run the following command - this tells us just what shard is
returning the results. It may take a few times, but you’ll
start to notice the host name changes each time you run the
command SELECT hostName() FROM test:
SELECT hostName() FROM test
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-0-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-0-0 │
└───────────────────────────┘
SELECT hostName() FROM test
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-0-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-1-0 │
└───────────────────────────┘
This is showing us that the query is being distributed across different shards. The good news is you can change your configuration files to change the shards and replication however suits your needs.
One issue though: there’s no persistent storage.
If these clusters stop running, your data vanishes.
Next instruction will be on how to add persistent storage
to your ClickHouse clusters running on Kubernetes.
In fact, we can test by creating a new configuration
file called sample04.yaml:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 1
replicasCount: 1
templates:
podTemplate: clickhouse-stable
templates:
podTemplates:
- name: clickhouse-stable
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:21.8.10.1.altinitystable
Make sure you’re exited out of your ClickHouse cluster, then install our configuration file:
kubectl apply -f sample04.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 configured
Notice that during the update that four pods were deleted, and then two new ones added.
When your clusters are settled down and back down to just 1 shard with 1 replication, log back into your ClickHouse database and select from table test:
SELECT * FROM test
Received exception from server (version 21.8.10):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table default.test doesn't exist.
command terminated with exit code 60
No persistent storage means any time your clusters are changed over, everything you’ve done is gone. The next article will cover how to correct that by adding storage volumes to your cluster.
kubectl create namespace test
namespace/test created
We’ve shown how to create ClickHouse clusters in Kubernetes, how to add zookeeper so we can create replicas of clusters. Now we’re going to show how to set persistent storage so you can change your cluster configurations without losing your hard work.
The examples here are built from the Altinity Kubernetes Operator examples, simplified down for our demonstrations.
latest tag. We highly encourage organizations to install a specific version of Altinity Stable builds to maximize compatibility. For information on the latest Altinity Stable Docker images, see the Altinity Stable for ClickHouse Docker page.
Create a new file called sample05.yaml with the following:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 2
replicasCount: 2
templates:
podTemplate: clickhouse-stable
volumeClaimTemplate: storage-vc-template
templates:
podTemplates:
- name: clickhouse-stable
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:21.8.10.1.altinitystable
volumeClaimTemplates:
- name: storage-vc-template
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Those who have followed the previous examples will recognize the clusters being created, but there are some new additions:
volumeClaimTemplate: This is setting up storage, and we’re
specifying the class as default. For full details on the different
storage classes see the
kubectl Storage Class documentationstorage: We’re going to give our cluster 1 Gigabyte of storage,
enough for our sample systems. If you need more space that
can be upgraded by changing these settings.podTemplate: Here we’ll specify what our pod types are going to be.
We’ll use the latest version of the ClickHouse containers,
but other versions can be specified to best it your needs.
For more information, see the
[ClickHouse on Kubernetes Operator Guide]({<ref “kubernetesoperatorguide”>}).Save your new configuration file and install it.
If you’ve been following this guide and already have the
namespace test operating, this will update it:
kubectl apply -f sample05.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 created
Verify it completes with get all for this namespace,
and you should have similar results:
kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.18.1 1 2 4 37d25436-c7e0-4f20-9d1c-73c50be0de88 Completed 4 clickhouse-demo-01.test.svc.cluster.local
kubectl get service -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chi-demo-01-demo-01-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 116s
chi-demo-01-demo-01-0-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 95s
chi-demo-01-demo-01-1-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 49s
chi-demo-01-demo-01-1-1 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 17s
clickhouse-demo-01 LoadBalancer 10.111.43.11 <pending> 8123:31462/TCP,9000:31735/TCP 2m13s
Everything is running, let’s verify that our storage is working. We’re going to exec into our cluster with a bash prompt on one of the pods created:
kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 32G 29G 1.5G 96% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 32G 29G 1.5G 96% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 7.7G 12K 7.7G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /proc/scsi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
And we can see we have about 1 Gigabyte of storage allocated into our cluster.
Let’s add some data to it. Nothing major, just to show that we can store information, then change the configuration and the data stays.
Exit out of your cluster and launch clickhouse-client on your LoadBalancer.
We’re going to create a database, then create a table in the database,
then show both.
SHOW DATABASES
┌─name────┐
│ default │
│ system │
└─────────┘
CREATE DATABASE teststorage
CREATE TABLE teststorage.test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')
SHOW DATABASES
┌─name────────┐
│ default │
│ system │
│ teststorage │
└─────────────┘
SELECT * FROM teststorage.test
┌─dummy─┐
│ 0 │
└───────┘
┌─dummy─┐
│ 0 │
└───────┘
If you followed the instructions from [Zookeeper and Replicas]({<ref “quickzookeeper” >}), note at the end when we updated the configuration of our sample cluster that all of the tables and data we made were deleted. Let’s recreate that experiment now with a new configuration.
Create a new file called sample06.yaml. We’re going to reduce
the shards and replicas to 1:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "demo-01"
layout:
shardsCount: 1
replicasCount: 1
templates:
podTemplate: clickhouse-stable
volumeClaimTemplate: storage-vc-template
templates:
podTemplates:
- name: clickhouse-stable
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:21.8.10.1.altinitystable
volumeClaimTemplates:
- name: storage-vc-template
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Update the cluster with the following:
kubectl apply -f sample06.yaml -n test
clickhouseinstallation.clickhouse.altinity.com/demo-01 configured
Wait until the configuration is done and all of the pods are spun down, then launch a bash prompt on one of the pods and check the storage available:
kubectl -n test get chi -o wide
NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT
demo-01 0.18.1 1 1 1 595d0709-b93d-48ee-a728-b8fbf179522b Completed 4 clickhouse-demo-01.test.svc.cluster.local
kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 32G 29G 1.5G 96% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 32G 29G 1.5G 96% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 7.7G 12K 7.7G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /proc/scsi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
Storage is still there. We can test if our databases are still available by logging into clickhouse:
SHOW DATABASES
┌─name────────┐
│ default │
│ system │
│ teststorage │
└─────────────┘
SELECT * FROM teststorage.test
┌─dummy─┐
│ 0 │
└───────┘
All of our databases and tables are there.
There are different ways of allocating storage - for data, for logging, multiple data volumes for your cluster nodes, but this will get you started in running your own Kubernetes cluster running ClickHouse in your favorite environment.
To remove the Altinity Kubernetes Operator, both the Altinity Kubernetes Operator and the components in its installed namespace will have to be removed. The proper command is to uses the same clickhouse-operator-install-bundle.yaml file that was used to install the Altinity Kubernetes Operator. For more details, see how to install and verify the Altinity Kubernetes Operator.
The following instructions are based on the standard installation instructions. For users who perform a custom installation, note that the any custom namespaces that the user wants to remove will have to be deleted separate from the Altinity Kubernetes Operator deletion.
For example, if the custom namespace operator-test is created, then it would be removed with the command kubectl delete namespaces operator-test.
Never delete the operator or run the following command while there are live ClickHouse clusters managed by the operator:
kubectl delete --namespace "kube-system" -f https://github.com/Altinity/clickhouse-operator/raw/0.18.1/deploy/operator/clickhouse-operator-install-bundle.yaml
The command will hang due to the live clusters. If you then re-install the operator, those clusters will be deleted and the operator will not work correctly.
See Altinity/clickhouse-operator#830 for more details.
To remove the Altinity Kubernetes Operator from your Kubernetes environment from a standard install:
Verify the Altinity Kubernetes Operator is in the kube-system namespace. The Altinity Kubernetes Operator and other pods will be displayed:
NAME READY STATUS RESTARTS AGE
clickhouse-operator-77b54889b4-mlzqc 2/2 Running 0 5s
coredns-78fcd69978-nthp2 1/1 Running 2 (107m ago) 23d
etcd-minikube 1/1 Running 2 (107m ago) 23d
kube-apiserver-minikube 1/1 Running 2 (107m ago) 23d
kube-controller-manager-minikube 1/1 Running 2 (107m ago) 23d
kube-proxy-lsggn 1/1 Running 2 (107m ago) 23d
kube-scheduler-minikube 1/1 Running 2 (107m ago) 23d
storage-provisioner 1/1 Running 5 (106m ago) 23d
Issue the kubectl delete command using the same YAML file used to install the Altinity Kubernetes Operator. By default the Altinity Kubernetes Operator is installed in the namespace kube-system. If this was installed into a custom namespace, verify that it installed in the uninstall command. In this example, we specified an installation of the Altinity Kubernetes Operator version 0.18.1 into the default kube-system namespace. This produces output similar to the following:
kubectl delete -f https://github.com/Altinity/clickhouse-operator/raw/0.18.1/deploy/operator/clickhouse-operator-install-bundle.yaml
customresourcedefinition.apiextensions.k8s.io "clickhouseinstallations.clickhouse.altinity.com" deleted
customresourcedefinition.apiextensions.k8s.io "clickhouseinstallationtemplates.clickhouse.altinity.com" deleted
customresourcedefinition.apiextensions.k8s.io "clickhouseoperatorconfigurations.clickhouse.altinity.com" deleted
serviceaccount "clickhouse-operator" deleted
clusterrole.rbac.authorization.k8s.io "clickhouse-operator-kube-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "clickhouse-operator-kube-system" deleted
configmap "etc-clickhouse-operator-files" deleted
configmap "etc-clickhouse-operator-confd-files" deleted
configmap "etc-clickhouse-operator-configd-files" deleted
configmap "etc-clickhouse-operator-templatesd-files" deleted
configmap "etc-clickhouse-operator-usersd-files" deleted
deployment.apps "clickhouse-operator" deleted
service "clickhouse-operator-metrics" deleted
To verify the Altinity Kubernetes Operator has been removed, use the kubectl get namespaces command:
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcd69978-nthp2 1/1 Running 2 (107m ago) 23d
etcd-minikube 1/1 Running 2 (107m ago) 23d
kube-apiserver-minikube 1/1 Running 2 (107m ago) 23d
kube-controller-manager-minikube 1/1 Running 2 (107m ago) 23d
kube-proxy-lsggn 1/1 Running 2 (107m ago) 23d
kube-scheduler-minikube 1/1 Running 2 (107m ago) 23d
storage-provisioner 1/1 Running 5 (106m ago) 23d