Altinity Kubernetes Operator for ClickHouse® on GKE
Organizations can host their Altinity Kubernetes Operator for ClickHouse® on the Google Kubernetes Engine (GKE). This can be done either through Altinity.Cloud or through a separate installation on GKE.
To setup a basic environment for the Altinity Kubernetes Operator for ClickHouse, use the following steps. The steps below use the current free Google Cloud services to set up a minimally viable Kubernetes with ClickHouse environment.
Prerequisites
- Register a Google Cloud Account: https://cloud.google.com/.
- Create a Google Cloud project: https://cloud.google.com/resource-manager/docs/creating-managing-projects
- Install
gcloudand rungcloud initorgcloud init --consoleto set up your environment: https://cloud.google.com/sdk/docs/install - Enable the Google Compute Engine: https://cloud.google.com/endpoints/docs/openapi/enable-api
- Enable GKE on your project: https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com.
- Select a default Computer Engine zone.
- Select a default Compute Engine region.
- Install kubectl on your local system. For sample instructions, see the Minikube on Linux installation instructions.
Altinity Kubernetes Operator for ClickHouse Installation instructions for GKE
Installing the Altinity Kubernetes Operator in GKE has the following main steps:
Create the Network
The first step in setting up the Altinity Kubernetes Operator for ClickHouse in GKE is creating the network. The complete details can be found on the Google Cloud documentation site regarding the gcloud compute networks create command. The following command will create a network called kubernetes-1 that will work for our minimal Altinity Kubernetes Operator cluster. Note that this network will not be available to external networks unless additional steps are made. Consult the Google Cloud documentation site for more details.
-
See a list of current networks available. In this example, there are no networks setup in this project:
gcloud compute networks list NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4 default AUTO REGIONAL -
Create the network in your Google Cloud project:
gcloud compute networks create kubernetes-1 --bgp-routing-mode regional --subnet-mode custom Created [https://www.googleapis.com/compute/v1/projects/betadocumentation/global/networks/kubernetes-1]. NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4 kubernetes-1 CUSTOM REGIONAL Instances on this network will not be reachable until firewall rules are created. As an example, you can allow all internal traffic between instances as well as SSH, RDP, and ICMP by running: $ gcloud compute firewall-rules create <FIREWALL_NAME> --network kubernetes-1 --allow tcp,udp,icmp --source-ranges <IP_RANGE> $ gcloud compute firewall-rules create <FIREWALL_NAME> --network kubernetes-1 --allow tcp:22,tcp:3389,icmp -
Verify its creation:
gcloud compute networks list NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4 default AUTO REGIONAL kubernetes-1 CUSTOM REGIONAL
Create the Cluster
Now that the network has been created, we can set up our cluster. The following cluster will be one using the ec2-micro machine type - the is still in the free model, and gives just enough power to run our basic cluster. The cluster will be called cluster-1, but you can replace it whatever name you feel appropriate. It uses the kubernetes-1 network specified earlier and creates a new subnet for the cluster under k-subnet-1.
To create and launch the cluster:
-
Verify the existing clusters with the
gcloudcommand. For this example there are no pre-existing clusters.gcloud container clusters list -
From the command line, issue the following
gcloudcommand to create the cluster:gcloud container clusters create cluster-1 --region us-west1 --node-locations us-west1-a --machine-type e2-micro --network kubernetes-1 --create-subnetwork name=k-subnet-1 --enable-ip-alias & -
Use the
clusters listcommand to verify when the cluster is available for use:gcloud container clusters list Created [https://container.googleapis.com/v1/projects/betadocumentation/zones/us-west1/clusters/cluster-1]. To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1/cluster-1?project=betadocumentation kubeconfig entry generated for cluster-1. NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS cluster-1 us-west1 1.21.6-gke.1500 35.233.231.36 e2-micro 1.21.6-gke.1500 3 RUNNING NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS cluster-1 us-west1 1.21.6-gke.1500 35.233.231.36 e2-micro 1.21.6-gke.1500 3 RUNNING [1]+ Done gcloud container clusters create cluster-1 --region us-west1 --node-locations us-west1-a --machine-type e2-micro --network kubernetes-1 --create-subnetwork name=k-subnet-1 --enable-ip-alias
Get Cluster Credentials
Importing the cluster credentials into your kubectl environment will allow you to issue commands directly to the cluster on Google Cloud. To import the cluster credentials:
-
Retrieve the credentials for the newly created cluster:
gcloud container clusters get-credentials cluster-1 --region us-west1 --project betadocumentation Fetching cluster endpoint and auth data. kubeconfig entry generated for cluster-1. -
Verify the cluster information from the
kubectlenvironment:kubectl cluster-info Kubernetes control plane is running at https://35.233.231.36 GLBCDefaultBackend is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy KubeDNS is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy Metrics-server is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Install the Altinity ClickHouse Operator for ClickHouse
Our cluster is up and ready to go. Time to install the Altinity Kubernetes Operator for ClickHouse through the following steps. Note that we are specifying a specific version of the Altinity Kubernetes Operator for ClickHouse. This insures maximum compatibility with your applications and other Kubernetes environments.
As of the time of this article, the most current version is 0.23.6. The latest version is always available on the project's releases page.
-
Apply the operator manifest by either downloading it and applying it, or referring to the GitHub repository URL. For more information, see the operator's installation guide.
kubectl apply -f https://github.com/Altinity/clickhouse-operator/raw/0.18.1/deploy/operator/clickhouse-operator-install-bundle.yaml -
Verify the installation by running:
kubectl get pods --namespace kube-system NAME READY STATUS RESTARTS AGE clickhouse-operator-77b54889b4-g98kk 2/2 Running 0 53s event-exporter-gke-5479fd58c8-7h6bn 2/2 Running 0 108s fluentbit-gke-b29c2 2/2 Running 0 79s fluentbit-gke-k8f2n 2/2 Running 0 80s fluentbit-gke-vjlqh 2/2 Running 0 80s gke-metrics-agent-4ttdt 1/1 Running 0 79s gke-metrics-agent-qf24p 1/1 Running 0 80s gke-metrics-agent-szktc 1/1 Running 0 80s konnectivity-agent-564f9f6c5f-59nls 1/1 Running 0 40s konnectivity-agent-564f9f6c5f-9nfnl 1/1 Running 0 40s konnectivity-agent-564f9f6c5f-vk7l8 1/1 Running 0 97s konnectivity-agent-autoscaler-5c49cb58bb-zxzlp 1/1 Running 0 97s kube-dns-697dc8fc8b-ddgrx 4/4 Running 0 98s kube-dns-697dc8fc8b-fpnps 4/4 Running 0 71s kube-dns-autoscaler-844c9d9448-pqvqr 1/1 Running 0 98s kube-proxy-gke-cluster-1-default-pool-fd104f22-8rx3 1/1 Running 0 36s kube-proxy-gke-cluster-1-default-pool-fd104f22-gnd0 1/1 Running 0 29s kube-proxy-gke-cluster-1-default-pool-fd104f22-k2sv 1/1 Running 0 12s l7-default-backend-69fb9fd9f9-hk7jq 1/1 Running 0 107s metrics-server-v0.4.4-857776bc9c-bs6sl 2/2 Running 0 44s pdcsi-node-5l9vf 2/2 Running 0 79s pdcsi-node-gfwln 2/2 Running 0 79s pdcsi-node-q6scz 2/2 Running 0 80s
Create a Simple ClickHouse Cluster
The operator allows the easy creation and modification of ClickHouse clusters in whatever format that works best for your organization. Now that the Google Cloud cluster is running and has the operator installed, let’s create a very simple ClickHouse cluster to test on.
The following example will create an operator-controlled cluster with 1 shard and replica, 500 MB of persistent storage, and sets the password of the default user’s password to topsecret. For more information on customizing the operator, see the Altinity Kubernetes Operator for ClickHouse Configuration Guides.
-
Create the following manifest and save it as
gcp-example01.yaml.apiVersion: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "gcp-example" spec: configuration: # What does my cluster look like? clusters: - name: "gcp-example" layout: shardsCount: 1 replicasCount: 1 templates: podTemplate: clickhouse-stable volumeClaimTemplate: pd-ssd # Where is Zookeeper? zookeeper: nodes: - host: zookeeper.zoo1ns port: 2181 # What are my users? users: # Password = topsecret demo/password_sha256_hex: 53336a676c64c1396553b2b7c92f38126768827c93b64d9142069c10eda7a721 demo/profile: default demo/quota: default demo/networks/ip: - 0.0.0.0/0 - ::/0 templates: podTemplates: # What is the definition of my server? - name: clickhouse-stable spec: containers: - name: clickhouse image: altinity/clickhouse-server:21.8.10.1.altinitystable # Keep servers on separate nodes! podDistribution: - scope: ClickHouseInstallation type: ClickHouseAntiAffinity volumeClaimTemplates: # How much storage and which type on each node? - name: pd-ssd # Do not delete PVC if installation is dropped. reclaimPolicy: Retain spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi -
Create a namespace in your GKE environment. For this example, we will be using
test:kubectl create namespace test namespace/test created -
Apply the manifest to the namespace:
kubectl -n test apply -f gcp-example01.yaml clickhouseinstallation.clickhouse.altinity.com/gcp-example created -
Verify the installation is complete when all pods are in a
Runningstate:kubectl -n test get chi -o wide NAME VERSION CLUSTERS SHARDS HOSTS TASKID STATUS UPDATED ADDED DELETED DELETE ENDPOINT gcp-example 0.18.1 1 1 1 f859e396-e2de-47fd-8016-46ad6b0b8508 Completed 1 clickhouse-gcp-example.test.svc.cluster.local
Login to the Cluster
This example does not have any open external ports, but we can still access our ClickHouse database through kubectl exec. In this case, our specific pod we are connecting to is chi-demo-01-demo-01-0-0-0. Replace this with the designation of your pods;
Use the following procedure to verify the Altinity Stable build install in your GKE environment.
-
Login to the
clickhouse-clientin one of your existing pods:kubectl -n test exec -it chi-gcp-example-gcp-example-0-0-0 -- clickhouse-client -
Verify the cluster configuration:
kubectl -n test exec -it chi-gcp-example-gcp-example-0-0-0 -- clickhouse-client -q "SELECT * FROM system.clusters FORMAT PrettyCompactNoEscapes" ┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐ │ all-replicated │ 1 │ 1 │ 1 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ all-sharded │ 1 │ 1 │ 1 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ gcp-example │ 1 │ 1 │ 1 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards_internal_replication │ 1 │ 1 │ 1 │ 127.0.0.1 │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards_internal_replication │ 2 │ 1 │ 1 │ 127.0.0.2 │ 127.0.0.2 │ 9000 │ 0 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_cluster_two_shards_localhost │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │ 0 │ 0 │ 0 │ │ test_unavailable_shard │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 1 │ default │ │ 0 │ 0 │ 0 │ │ test_unavailable_shard │ 2 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 1 │ 0 │ default │ │ 0 │ 0 │ 0 │ └──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘ -
Exit out of your cluster:
chi-gcp-example-gcp-example-0-0-0.chi-gcp-example-gcp-example-0-0.test.svc.cluster.local :) exit Bye.
Further Steps
This simple example demonstrates how to build and manage an operator-run cluster for ClickHouse. Further steps would be to open the cluster to external network connections, set up replication schemes, etc.
For more information, see the Altinity Kubernetes Operator for ClickHouse guides and the Altinity Kubernetes Operator for ClickHouse repository.