This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Altinity Kubernetes Operator Quick Start Guide

Become familiar with the Altinity Kubernetes Operator in the fewest steps.

1: How to install the Altinity ClickHouse-Operator to your Kubernetes environment
2: First Clusters
3: Zookeeper and Replicas
4: Persistent Storage
5: Uninstall

If you’re running the Altinity Kubernetes Operator for the first time, or just want to get it up and running as quickly as possible, the Quick Start Guide is for you.

Requirements:

An operating system running Kubernetes and Docker, or a service providing support for them such as AWS.
A ClickHouse remote client such as clickhouse-client. Full instructions for installing ClickHouse can be found on the ClickHouse Installation page.

1 - How to install the Altinity ClickHouse-Operator to your Kubernetes environment

How to install and verify the Altinity Kubernetes Operator

28 July 2023 · Read time 4 min

Introduction - Altinity ClickHouse-Operator

This page provides instructions to deploy the Altinity Kubernetes Operator to your Kubernetes environment.

Prerequisites

The following items are required:

Kubernetes version 1.15.11 or later
Kubernetes Install Tools - kubectl is required. The other tools listed on this page (kind, minikube, and kubeadm) are optional
clickhouse-operator-install-bundle.yaml - this is the latest (master branch) of the Altinity ClickHouse deployment YAML file

For Other Altinity deployment YAML file versions

To find other versions of the deployment YAML file, visit the file in our GitHub repo and select another branch from the GitHub branch menu.

Deployment Instructions

This example shows how to deploy version 0.21.2 of clickhouse-operator-install-bundle.yaml from the Altinity GitHub repository.

NOTE: Altinity recommends that you deploy a specific version, rather than using the latest clickhouse-operator YAML file from the master branch.

Installation Commands

To install a version 0.21.2 of the Altinity Kubernetes Operator to your existing Kubernetes environment, run the following command:

kubectl apply -f https://github.com/Altinity/clickhouse-operator/raw/0.21.2/deploy/operator/clickhouse-operator-install-bundle.yaml

The URL will be different if you’re using another version of the file.

Alternatively, to deploy your own version of the YAML file, download and modify the latest Altinity Kubernetes Operator YAML file and run the following command:

kubectl apply -f clickhouse-operator-install-bundle.yaml

Successful Installation

The following example response shows the result of a successful installation.

customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrole.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created

Installation Verification

To verify that the installation was successful, run the following command:

kubectl get deployment.apps -n kube-system

If the operator installed successfully, you’ll see clickhouse-operator in the output:

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
clickhouse-operator   1/1     1            1           80s
coredns               1/1     1            1           102d

More Information The following section provides more information on the resources created in the installation.

Altinity Kubernetes Operator Resources Details

Customization options

To customize Altinity Kubernetes Operator settings see:

Operator Guide

Altinity recommends that you install a specific version of the ClickHouse-operator version that you know will work with your Kubernetes environment, rather than use the latest build from the GitHub master branch.

For details on installing other versions of the Altinity Kubernetes Operator see:

Specific Version Installation Guide

Deleting a deployment

This section covers how to delete a deployment.

IMPORTANT NOTICE

Never delete the operator or run the following command while there are live ClickHouse clusters managed by the operator. The command will hang due to the live clusters. If you then re-install the operator, those clusters will be deleted and the operator will not work correctly.

To delete the operator, use this command with the URL you used when you installd it:

kubectl delete -f https://github.com/Altinity/clickhouse-operator/raw/0.21.2/deploy/operator/clickhouse-operator-install-bundle.yaml

If you used a customized version of the operator’s YAML file, use that filename instead:

kubectl delete -f clickhouse-operator-install-bundle.yaml

More information

Deleting the operator while clusters are running can cause them to be deleted without warning

To delete a deployment using the latest clickhouse-operator YAML file:

kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

To delete a deployment using your local clickhouse-operator YAML file:

kubectl delete -f clickhouse-operator-install-bundle.yaml

2 - First Clusters

Create your first ClickHouse Cluster

If you followed the Quick Installation guide, then you have the Altinity Kubernetes Operator for Kubernetes installed. Let’s give it something to work with.

Create Your Namespace

For our examples, we’ll be setting up our own Kubernetes namespace test. All of the examples will be installed into that namespace so we can track how the cluster is modified with new updates.

Create the namespace with the following kubectl command:

kubectl create namespace test

namespace/test created

Just to make sure we’re in a clean environment, let’s check for any resources in our namespace:

kubectl get all -n test

No resources found in test namespace.

The First Cluster

We’ll start with the simplest cluster: one shard, one replica. This template and others are available on the Altinity Kubernetes Operator Github example site, and contains the following:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 1
          replicasCount: 1

Save this as sample01.yaml and launch it with the following:

kubectl apply -n test -f sample01.yaml

clickhouseinstallation.clickhouse.altinity.com/demo-01 created

Verify that the new cluster is running. The operator defined a Custom Resource type called a ClickHouseInstallation and defined the short name chi for it. That makes it easy to see all of the ClickHouse Installations:

kubectl -n test get chi

When everything is ready, its status is Completed.

NAME      CLUSTERS   HOSTS   STATUS      HOSTS-COMPLETED   AGE
demo-01   1          1       Completed                     2m38s

To retrieve the IP information use the get service option:

kubectl get service -n test

NAME                      TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                         AGE
chi-demo-01-demo-01-0-0   ClusterIP      None          <none>        9000/TCP,8123/TCP,9009/TCP      9h
clickhouse-demo-01        LoadBalancer   10.100.9.83   <pending>     8123:30366/TCP,9000:31021/TCP   9h

So we can see our ClickHouse installation is running, and that we have the load balancer for the cluster.

Connect To Your Cluster With Exec

Let’s talk to our cluster and run some simple ClickHouse queries.

We can hop in directly through Kubernetes and run the clickhouse-client that’s part of the image with the following command:

kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- clickhouse-client

ClickHouse client version 20.12.4.5 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.12.4 revision 54442.

chi-demo-01-demo-01-0-0-0.chi-demo-01-demo-01-0-0.test.svc.cluster.local :)

From within ClickHouse, we can check out the current clusters:

SELECT * FROM system.clusters

┌─cluster─────────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated                                  │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                     │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                         │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           2 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           3 │ 127.0.0.3               │ 127.0.0.3    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                         │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                         │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication    │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication    │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost               │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost               │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost                            │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost_secure                     │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9440 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                          │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                          │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │    1 │        0 │ default │                  │            0 │               0 │                       0 │
└─────────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘

Exit out of your cluster:

chi-demo-01-demo-01-0-0-0.chi-demo-01-demo-01-0-0.test.svc.cluster.local :) exit
Bye.

Connect to Your Cluster with Remote Client

You can also use a remote client such as clickhouse-client to connect to your cluster through the LoadBalancer.

The default username and password is set by the clickhouse-operator-install.yaml file. These values can be altered by changing the chUsername and chPassword ClickHouse Credentials section:
- Default Username: clickhouse_operator
- Default Password: clickhouse_operator_password

# ClickHouse credentials (username, password and port) to be used
# by operator to connect to ClickHouse instances for:
# 1. Metrics requests
# 2. Schema maintenance
# 3. DROP DNS CACHE
# User with such credentials can be specified in additional ClickHouse
# .xml config files,
# located in `chUsersConfigsPath` folder
chUsername: clickhouse_operator
chPassword: clickhouse_operator_password
chPort: 8123

In either case, the command to connect to your new cluster will resemble the following, replacing {LoadBalancer hostname} with the name or IP address or your LoadBalancer, then providing the proper password. In our examples so far, that’s been localhost.

From there, just make your ClickHouse SQL queries as you please - but remember that this particular cluster has no persistent storage. If the cluster is modified in any way, any databases or tables created will be wiped clean.

Update Your First Cluster To 2 Shards

Well that’s great - we have a cluster running. Granted, it’s really small and doesn’t do much, but what if we want to upgrade it?

Sure - we can do that any time we want.

Take your sample01.yaml and save it as sample02.yaml.

Let’s update it to give us two shards running with one replica:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 2
          replicasCount: 1

Save your YAML file, and apply it. We’ve defined the name in the metadata, so it knows exactly what cluster to update.

kubectl apply -n test -f sample02.yaml

clickhouseinstallation.clickhouse.altinity.com/demo-01 configured

Verify that the cluster is running - this may take a few minutes depending on your system:

kubectl -n test get chi -o wide

NAME      VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT
demo-01   0.18.1    1          2        2       80102179-4aa5-4e8f-826c-1ca7a1e0f7b9   Completed             1                          clickhouse-demo-01.test.svc.cluster.local

kubectl get service -n test

NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
chi-demo-01-demo-01-0-0   ClusterIP      None           <none>        8123/TCP,9000/TCP,9009/TCP      26s
chi-demo-01-demo-01-1-0   ClusterIP      None           <none>        8123/TCP,9000/TCP,9009/TCP      3s
clickhouse-demo-01        LoadBalancer   10.111.27.86   <pending>     8123:31126/TCP,9000:32460/TCP   43s

Once again, we can reach right into our cluster with clickhouse-client and look at the clusters.

clickhouse-client --host localhost --user=clickhouse_operator --password=clickhouse_operator_password

ClickHouse client version 20.12.4.5 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.12.4 revision 54442.

chi-demo-01-demo-01-1-0-0.chi-demo-01-demo-01-1-0.test.svc.cluster.local :)

SELECT * FROM system.clusters

┌─cluster─────────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated                                  │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                     │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                         │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           2 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_one_shard_three_replicas_localhost │         1 │            1 │           3 │ 127.0.0.3               │ 127.0.0.3    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                         │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                         │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication    │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication    │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost               │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost               │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost                            │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost_secure                     │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9440 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                          │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                          │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │    1 │        0 │ default │                  │            0 │               0 │                       0 │
└─────────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘

So far, so good. We can create some basic clusters. If we want to do more, we’ll have to move ahead with replication and zookeeper in the next section.

3 - Zookeeper and Replicas

Install Zookeeper and Replicas

kubectl create namespace test

namespace/test created

Now we’ve seen how to setup a basic cluster and upgrade it. Time to step up our game and setup our cluster with Zookeeper, and then add persistent storage to it.

The Altinity Kubernetes Operator does not install or manage Zookeeper. Zookeeper must be provided and managed externally. The samples below are examples on establishing Zookeeper to provide replication support. For more information running and configuring Zookeeper, see the Apache Zookeeper site.

This step can not be skipped - your Zookeeper instance must have been set up externally from your ClickHouse clusters. Whether your Zookeeper installation is hosted by other Docker Images or separate servers is up to you.

Install Zookeeper

Kubernetes Zookeeper Deployment

A simple method of installing a single Zookeeper node is provided from the Altinity Kubernetes Operator deployment samples. These provide samples deployments of Grafana, Prometheus, Zookeeper and other applications.

See the Altinity Kubernetes Operator deployment directory for a full list of sample scripts and Kubernetes deployment files.

The instructions below will create a new Kubernetes namespace zoo1ns, and create a Zookeeper node in that namespace. Kubernetes nodes will refer to that Zookeeper node by the hostname zookeeper.zoo1ns within the created Kubernetes networks.

To deploy a single Zookeeper node in Kubernetes from the Altinity Kubernetes Operator Github repository:

Download the Altinity Kubernetes Operator Github repository, either with git clone https://github.com/Altinity/clickhouse-operator.git or by selecting Code->Download Zip from the Altinity Kubernetes Operator GitHub repository .
From a terminal, navigate to the deploy/zookeeper directory and run the following:

cd clickhouse-operator/deploy/zookeeper

./quick-start-volume-emptyDir/zookeeper-1-node-create.sh

namespace/zoo1ns created
service/zookeeper created
service/zookeepers created
[33;1mWarning:[0m policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/zookeeper-pod-disruption-budget created
statefulset.apps/zookeeper created

Verify the Zookeeper node is running in Kubernetes:

kubectl get all --namespace zoo1ns

NAME              READY   STATUS    RESTARTS   AGE
pod/zookeeper-0   0/1     Running   0          2s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
service/zookeeper    ClusterIP   10.100.31.86   <none>        2181/TCP,7000/TCP   2s
service/zookeepers   ClusterIP   None           <none>        2888/TCP,3888/TCP   2s

NAME                         READY   AGE
statefulset.apps/zookeeper   0/1     2s

Kubernetes nodes will be able to refer to the Zookeeper node by the hostname zookeeper.zoo1ns.

Configure Kubernetes with Zookeeper

Once we start replicating clusters, we need Zookeeper to manage them. Create a new file sample03.yaml and populate it with the following:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper.zoo1ns
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 2
          replicasCount: 2
        templates:
          podTemplate: clickhouse-stable
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
            - name: clickhouse
              image: altinity/clickhouse-server:21.8.10.1.altinitystable

Notice that we’re increasing the number of replicas from the sample02.yaml file in the [First Clusters - No Storage]({<ref “quickcluster”>}) tutorial.

We’ll set up a minimal Zookeeper connecting cluster by applying our new configuration file:

kubectl apply -f sample03.yaml -n test

clickhouseinstallation.clickhouse.altinity.com/demo-01 created

Verify it with the following:

kubectl -n test get chi -o wide

NAME      VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT                                    AGE
demo-01   0.18.3    1          2        4       5ec69e86-7e4d-4b8b-877f-f298f26161b2   Completed             4                          clickhouse-demo-01.test.svc.cluster.local   102s

kubectl get service -n test

NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
chi-demo-01-demo-01-0-0   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      85s
chi-demo-01-demo-01-0-1   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      68s
chi-demo-01-demo-01-1-0   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      47s
chi-demo-01-demo-01-1-1   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      16s
clickhouse-demo-01        LoadBalancer   10.104.157.249   <pending>     8123:32543/TCP,9000:30797/TCP   101s

If we log into our cluster and show the clusters, we can show the updated results and that we have a total of 4 replicas of demo-01 - two shards for each node with two replicas.

SELECT * FROM system.clusters

┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
│ all-replicated                               │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ all-replicated                               │         1 │            1 │           2 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ all-replicated                               │         1 │            1 │           3 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ all-replicated                               │         1 │            1 │           4 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                  │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                  │         2 │            1 │           1 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                  │         3 │            1 │           1 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ all-sharded                                  │         4 │            1 │           1 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                      │         1 │            1 │           1 │ chi-demo-01-demo-01-0-0 │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                      │         1 │            1 │           2 │ chi-demo-01-demo-01-0-1 │ 172.17.0.6   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                      │         2 │            1 │           1 │ chi-demo-01-demo-01-1-0 │ 172.17.0.7   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ demo-01                                      │         2 │            1 │           2 │ chi-demo-01-demo-01-1-1 │ 172.17.0.8   │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                      │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards                      │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication │         1 │            1 │           1 │ 127.0.0.1               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_internal_replication │         2 │            1 │           1 │ 127.0.0.2               │ 127.0.0.2    │ 9000 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost            │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_cluster_two_shards_localhost            │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost                         │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_shard_localhost_secure                  │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9440 │        0 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                       │         1 │            1 │           1 │ localhost               │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │               0 │                       0 │
│ test_unavailable_shard                       │         2 │            1 │           1 │ localhost               │ 127.0.0.1    │    1 │        0 │ default │                  │            0 │               0 │                       0 │
└──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘

Distributed Tables

We have our clusters going - let’s test it out with some distributed tables so we can see the replication in action.

CREATE TABLE test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')

Once our table is created, perform a SELECT * FROM test command. We’ll see nothing because we didn’t give it any data, but that’s all right.

SELECT * FROM test

┌─dummy─┐
│     0 │
└───────┘
┌─dummy─┐
│     0 │
└───────┘

Now let’s test out our results coming in. Run the following command - this tells us just what shard is returning the results. It may take a few times, but you’ll start to notice the host name changes each time you run the command SELECT hostName() FROM test:

SELECT hostName() FROM test

┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-0-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-1-0 │
└───────────────────────────┘

SELECT hostName() FROM test

┌─hostName()────────────────┐
│ chi-demo-01-demo-01-0-0-0 │
└───────────────────────────┘
┌─hostName()────────────────┐
│ chi-demo-01-demo-01-1-0-0 │
└───────────────────────────┘

This is showing us that the query is being distributed across different shards. The good news is you can change your configuration files to change the shards and replication however suits your needs.

One issue though: there’s no persistent storage. If these clusters stop running, your data vanishes. Next instruction will be on how to add persistent storage to your ClickHouse clusters running on Kubernetes. In fact, we can test by creating a new configuration file called sample04.yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper.zoo1ns
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 1
          replicasCount: 1
        templates:
          podTemplate: clickhouse-stable
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
            - name: clickhouse
              image: altinity/clickhouse-server:21.8.10.1.altinitystable

Make sure you’re exited out of your ClickHouse cluster, then install our configuration file:

kubectl apply -f sample04.yaml -n test

clickhouseinstallation.clickhouse.altinity.com/demo-01 configured

Notice that during the update that four pods were deleted, and then two new ones added.

When your clusters are settled down and back down to just 1 shard with 1 replication, log back into your ClickHouse database and select from table test:

SELECT * FROM test

Received exception from server (version 21.8.10):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Table default.test doesn't exist. 
command terminated with exit code 60

No persistent storage means any time your clusters are changed over, everything you’ve done is gone. The next article will cover how to correct that by adding storage volumes to your cluster.

4 - Persistent Storage

How to set up persistent storage for your ClickHouse Kubernetes cluster.

kubectl create namespace test

namespace/test created

We’ve shown how to create ClickHouse clusters in Kubernetes, how to add zookeeper so we can create replicas of clusters. Now we’re going to show how to set persistent storage so you can change your cluster configurations without losing your hard work.

The examples here are built from the Altinity Kubernetes Operator examples, simplified down for our demonstrations.

IMPORTANT NOTE

The Altinity Stable builds for ClickHouse do not use the latest tag. We highly encourage organizations to install a specific version of Altinity Stable builds to maximize compatibility. For information on the latest Altinity Stable Docker images, see the Altinity Stable for ClickHouse Docker page.

Create a new file called sample05.yaml with the following:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
        nodes:
        - host: zookeeper.zoo1ns
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 2
          replicasCount: 2
        templates:
          podTemplate: clickhouse-stable
          volumeClaimTemplate: storage-vc-template
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
          - name: clickhouse
            image: altinity/clickhouse-server:21.8.10.1.altinitystable
    volumeClaimTemplates:
      - name: storage-vc-template
        spec:
          storageClassName: standard
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

Those who have followed the previous examples will recognize the clusters being created, but there are some new additions:

volumeClaimTemplate: This is setting up storage, and we’re specifying the class as default. For full details on the different storage classes see the kubectl Storage Class documentation
storage: We’re going to give our cluster 1 Gigabyte of storage, enough for our sample systems. If you need more space that can be upgraded by changing these settings.
podTemplate: Here we’ll specify what our pod types are going to be. We’ll use the latest version of the ClickHouse containers, but other versions can be specified to best it your needs. For more information, see the [ClickHouse on Kubernetes Operator Guide]({<ref “kubernetesoperatorguide”>}).

Save your new configuration file and install it. If you’ve been following this guide and already have the namespace test operating, this will update it:

kubectl apply -f sample05.yaml -n test

clickhouseinstallation.clickhouse.altinity.com/demo-01 created

Verify it completes with get all for this namespace, and you should have similar results:

kubectl -n test get chi -o wide

NAME      VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT                                    AGE
demo-01   0.18.3    1          2        4       57ec3f87-9950-4e5e-9b26-13680f66331d   Completed             4                          clickhouse-demo-01.test.svc.cluster.local   108s

kubectl get service -n test

NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
chi-demo-01-demo-01-0-0   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      81s
chi-demo-01-demo-01-0-1   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      63s
chi-demo-01-demo-01-1-0   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      45s
chi-demo-01-demo-01-1-1   ClusterIP      None             <none>        8123/TCP,9000/TCP,9009/TCP      8s
clickhouse-demo-01        LoadBalancer   10.104.236.138   <pending>     8123:31281/TCP,9000:30052/TCP   98s

Testing Persistent Storage

Everything is running, let’s verify that our storage is working. We’re going to exec into our cluster with a bash prompt on one of the pods created:

kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- df -h

Filesystem      Size  Used Avail Use% Mounted on
overlay          32G   26G  4.0G  87% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2        32G   26G  4.0G  87% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           7.7G   12K  7.7G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /proc/scsi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

And we can see we have about 1 Gigabyte of storage allocated into our cluster.

Let’s add some data to it. Nothing major, just to show that we can store information, then change the configuration and the data stays.

Exit out of your cluster and launch clickhouse-client on your LoadBalancer. We’re going to create a database, then create a table in the database, then show both.

SHOW DATABASES

┌─name────┐
│ default │
│ system  │
└─────────┘

CREATE DATABASE teststorage

CREATE TABLE teststorage.test AS system.one ENGINE = Distributed('demo-01', 'system', 'one')

SHOW DATABASES

┌─name────────┐
│ default     │
│ system      │
│ teststorage │
└─────────────┘

SELECT * FROM teststorage.test

┌─dummy─┐
│     0 │
└───────┘
┌─dummy─┐
│     0 │
└───────┘

If you followed the instructions from [Zookeeper and Replicas]({<ref “quickzookeeper” >}), note at the end when we updated the configuration of our sample cluster that all of the tables and data we made were deleted. Let’s recreate that experiment now with a new configuration.

Create a new file called sample06.yaml. We’re going to reduce the shards and replicas to 1:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "demo-01"
spec:
  configuration:
    zookeeper:
        nodes:
        - host: zookeeper.zoo1ns
          port: 2181
    clusters:
      - name: "demo-01"
        layout:
          shardsCount: 1
          replicasCount: 1
        templates:
          podTemplate: clickhouse-stable
          volumeClaimTemplate: storage-vc-template
  templates:
    podTemplates:
      - name: clickhouse-stable
        spec:
          containers:
          - name: clickhouse
            image: altinity/clickhouse-server:21.8.10.1.altinitystable
    volumeClaimTemplates:
      - name: storage-vc-template
        spec:
          storageClassName: standard
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

Update the cluster with the following:

kubectl apply -f sample06.yaml -n test

clickhouseinstallation.clickhouse.altinity.com/demo-01 configured

Wait until the configuration is done and all of the pods are spun down, then launch a bash prompt on one of the pods and check the storage available:

kubectl -n test get chi -o wide

NAME      VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT                                    AGE
demo-01   0.18.3    1          1        1       776c1a82-44e1-4c2e-97a7-34cef629e698   Completed                               4        clickhouse-demo-01.test.svc.cluster.local   2m56s

kubectl -n test exec -it chi-demo-01-demo-01-0-0-0 -- df -h

Filesystem      Size  Used Avail Use% Mounted on
overlay          32G   26G  4.0G  87% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2        32G   26G  4.0G  87% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           7.7G   12K  7.7G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /proc/scsi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

Storage is still there. We can test if our databases are still available by logging into clickhouse:

SHOW DATABASES

┌─name────────┐
│ default     │
│ system      │
│ teststorage │
└─────────────┘

SELECT * FROM teststorage.test

┌─dummy─┐
│     0 │
└───────┘

All of our databases and tables are there.

There are different ways of allocating storage - for data, for logging, multiple data volumes for your cluster nodes, but this will get you started in running your own Kubernetes cluster running ClickHouse in your favorite environment.

5 - Uninstall

How to uninstall the Altinity Kubernetes Operator and its namespace

To remove the Altinity Kubernetes Operator, both the Altinity Kubernetes Operator and the components in its installed namespace will have to be removed. The proper command is to uses the same clickhouse-operator-install-bundle.yaml file that was used to install the Altinity Kubernetes Operator. For more details, see how to install and verify the Altinity Kubernetes Operator.

The following instructions are based on the standard installation instructions. For users who perform a custom installation, note that the any custom namespaces that the user wants to remove will have to be deleted separate from the Altinity Kubernetes Operator deletion.

For example, if the custom namespace operator-test is created, then it would be removed with the command kubectl delete namespaces operator-test.

IMPORTANT NOTICE 1

Never delete the operator or run the following command while there are live ClickHouse clusters managed by the operator:

kubectl delete --namespace "kube-system" -f https://github.com/Altinity/clickhouse-operator/raw/0.18.3/deploy/operator/clickhouse-operator-install-bundle.yaml

The command will hang due to the live clusters. If you then re-install the operator, those clusters will be deleted and the operator will not work correctly.

See Altinity/clickhouse-operator#830 for more details.

IMPORTANT NOTICE 2

Please follow the instructions below. The uninstall command is geared to properly remove the Altinity Kubernetes Operator and its namespace. Deleting the namespace without properly removing the Altinity Kubernetes Operator can cause it to hang.

Instructions

To remove the Altinity Kubernetes Operator from your Kubernetes environment from a standard install:

Verify the Altinity Kubernetes Operator is in the kube-system namespace. The Altinity Kubernetes Operator and other pods will be displayed:

NAME                                   READY   STATUS    RESTARTS      AGE
clickhouse-operator-857c69ffc6-2frgl   2/2     Running   0             5s
coredns-78fcd69978-nthp2               1/1     Running   4 (23h ago)   51d
etcd-minikube                          1/1     Running   4 (23h ago)   51d
kube-apiserver-minikube                1/1     Running   4 (23h ago)   51d
kube-controller-manager-minikube       1/1     Running   4 (23h ago)   51d
kube-proxy-lsggn                       1/1     Running   4 (23h ago)   51d
kube-scheduler-minikube                1/1     Running   4 (23h ago)   51d
storage-provisioner                    1/1     Running   9 (23h ago)   51d

Issue the kubectl delete command using the same YAML file used to install the Altinity Kubernetes Operator. By default the Altinity Kubernetes Operator is installed in the namespace kube-system. If this was installed into a custom namespace, verify that it installed in the uninstall command. In this example, we specified an installation of the Altinity Kubernetes Operator version 0.18.3 into the default kube-system namespace. This produces output similar to the following:

kubectl delete -f https://github.com/Altinity/clickhouse-operator/raw/0.18.3/deploy/operator/clickhouse-operator-install-bundle.yaml

customresourcedefinition.apiextensions.k8s.io "clickhouseinstallations.clickhouse.altinity.com" deleted
customresourcedefinition.apiextensions.k8s.io "clickhouseinstallationtemplates.clickhouse.altinity.com" deleted
customresourcedefinition.apiextensions.k8s.io "clickhouseoperatorconfigurations.clickhouse.altinity.com" deleted
serviceaccount "clickhouse-operator" deleted
clusterrole.rbac.authorization.k8s.io "clickhouse-operator-kube-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "clickhouse-operator-kube-system" deleted
configmap "etc-clickhouse-operator-files" deleted
configmap "etc-clickhouse-operator-confd-files" deleted
configmap "etc-clickhouse-operator-configd-files" deleted
configmap "etc-clickhouse-operator-templatesd-files" deleted
configmap "etc-clickhouse-operator-usersd-files" deleted
deployment.apps "clickhouse-operator" deleted
service "clickhouse-operator-metrics" deleted

To verify the Altinity Kubernetes Operator has been removed, use the kubectl get namespaces command:

kubectl get pods --namespace kube-system

NAME                               READY   STATUS    RESTARTS      AGE
coredns-78fcd69978-nthp2           1/1     Running   4 (23h ago)   51d
etcd-minikube                      1/1     Running   4 (23h ago)   51d
kube-apiserver-minikube            1/1     Running   4 (23h ago)   51d
kube-controller-manager-minikube   1/1     Running   4 (23h ago)   51d
kube-proxy-lsggn                   1/1     Running   4 (23h ago)   51d
kube-scheduler-minikube            1/1     Running   4 (23h ago)   51d
storage-provisioner                1/1     Running   9 (23h ago)   51d