Altinity Kubernetes Operator for ClickHouse® on GKE

How to install the Altinity Kubernetes Operator for ClickHouse® using the Google Kubernetes Engine

Organizations can host their Altinity Kubernetes Operator for ClickHouse® on the Google Kubernetes Engine (GKE). This can be done either through Altinity.Cloud or through a separate installation on GKE.

To setup a basic environment for the Altinity Kubernetes Operator for ClickHouse, use the following steps. The steps below use the current free Google Cloud services to set up a minimally viable Kubernetes with ClickHouse environment.

Prerequisites

  1. Register a Google Cloud Account: https://cloud.google.com/.
  2. Create a Google Cloud project: https://cloud.google.com/resource-manager/docs/creating-managing-projects
  3. Install gcloud and run gcloud init or gcloud init --console to set up your environment: https://cloud.google.com/sdk/docs/install
  4. Enable the Google Compute Engine: https://cloud.google.com/endpoints/docs/openapi/enable-api
  5. Enable GKE on your project: https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com.
  6. Select a default Computer Engine zone.
  7. Select a default Compute Engine region.
  8. Install kubectl on your local system. For sample instructions, see the Minikube on Linux installation instructions.

Altinity Kubernetes Operator for ClickHouse Installation instructions for GKE

Installing the Altinity Kubernetes Operator in GKE has the following main steps:

Create the Network

The first step in setting up the Altinity Kubernetes Operator for ClickHouse in GKE is creating the network. The complete details can be found on the Google Cloud documentation site regarding the gcloud compute networks create command. The following command will create a network called kubernetes-1 that will work for our minimal Altinity Kubernetes Operator cluster. Note that this network will not be available to external networks unless additional steps are made. Consult the Google Cloud documentation site for more details.

  1. See a list of current networks available. In this example, there are no networks setup in this project:

    gcloud compute networks list
    NAME     SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
    default  AUTO         REGIONAL
    
  2. Create the network in your Google Cloud project:

    gcloud compute networks create kubernetes-1 --bgp-routing-mode regional --subnet-mode custom
    Created [https://www.googleapis.com/compute/v1/projects/betadocumentation/global/networks/kubernetes-1].
    NAME          SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
    kubernetes-1  CUSTOM       REGIONAL
    
    Instances on this network will not be reachable until firewall rules are created. As an example, you can allow all internal traffic between instances as well as SSH, RDP, and ICMP by running:
    
    $ gcloud compute firewall-rules create <FIREWALL_NAME> --network kubernetes-1 --allow tcp,udp,icmp --source-ranges <IP_RANGE>
    $ gcloud compute firewall-rules create <FIREWALL_NAME> --network kubernetes-1 --allow tcp:22,tcp:3389,icmp
    
  3. Verify its creation:

    gcloud compute networks list
    NAME          SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
    default       AUTO         REGIONAL
    kubernetes-1  CUSTOM       REGIONAL
    

Create the Cluster

Now that the network has been created, we can set up our cluster. The following cluster will be one using the ec2-micro machine type - the is still in the free model, and gives just enough power to run our basic cluster. The cluster will be called cluster-1, but you can replace it whatever name you feel appropriate. It uses the kubernetes-1 network specified earlier and creates a new subnet for the cluster under k-subnet-1.

To create and launch the cluster:

  1. Verify the existing clusters with the gcloud command. For this example there are no pre-existing clusters.

    gcloud container clusters list
    
  2. From the command line, issue the following gcloud command to create the cluster:

    gcloud container clusters create cluster-1 --region us-west1 --node-locations us-west1-a --machine-type e2-micro --network kubernetes-1 --create-subnetwork name=k-subnet-1 --enable-ip-alias &
    
  3. Use the clusters list command to verify when the cluster is available for use:

    gcloud container clusters list
    Created [https://container.googleapis.com/v1/projects/betadocumentation/zones/us-west1/clusters/cluster-1].
    To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1/cluster-1?project=betadocumentation
    kubeconfig entry generated for cluster-1.
    NAME       LOCATION  MASTER_VERSION   MASTER_IP      MACHINE_TYPE  NODE_VERSION     NUM_NODES  STATUS
    cluster-1  us-west1  1.21.6-gke.1500  35.233.231.36  e2-micro      1.21.6-gke.1500  3          RUNNING
    NAME       LOCATION  MASTER_VERSION   MASTER_IP      MACHINE_TYPE  NODE_VERSION     NUM_NODES  STATUS
    cluster-1  us-west1  1.21.6-gke.1500  35.233.231.36  e2-micro      1.21.6-gke.1500  3          RUNNING
    [1]+  Done                    gcloud container clusters create cluster-1 --region us-west1 --node-locations us-west1-a --machine-type e2-micro --network kubernetes-1 --create-subnetwork name=k-subnet-1 --enable-ip-alias
    

Get Cluster Credentials

Importing the cluster credentials into your kubectl environment will allow you to issue commands directly to the cluster on Google Cloud. To import the cluster credentials:

  1. Retrieve the credentials for the newly created cluster:

    gcloud container clusters get-credentials cluster-1 --region us-west1 --project betadocumentation
    Fetching cluster endpoint and auth data.
    kubeconfig entry generated for cluster-1.
    
  2. Verify the cluster information from the kubectl environment:

    kubectl cluster-info
    Kubernetes control plane is running at https://35.233.231.36
    GLBCDefaultBackend is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
    KubeDNS is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
    Metrics-server is running at https://35.233.231.36/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
    
    To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
    

Install the Altinity ClickHouse Operator for ClickHouse

Our cluster is up and ready to go. Time to install the Altinity Kubernetes Operator for ClickHouse through the following steps. Note that we are specifying a specific version of the Altinity Kubernetes Operator for ClickHouse. This insures maximum compatibility with your applications and other Kubernetes environments.

As of the time of this article, the most current version is 0.23.6. The latest version is always available on the project's releases page.

  1. Apply the operator manifest by either downloading it and applying it, or referring to the GitHub repository URL. For more information, see the operator's installation guide.

    kubectl apply -f https://github.com/Altinity/clickhouse-operator/raw/0.18.1/deploy/operator/clickhouse-operator-install-bundle.yaml
    
  2. Verify the installation by running:

    kubectl get pods --namespace kube-system
    NAME                                                  READY   STATUS    RESTARTS   AGE
    clickhouse-operator-77b54889b4-g98kk                  2/2     Running   0          53s
    event-exporter-gke-5479fd58c8-7h6bn                   2/2     Running   0          108s
    fluentbit-gke-b29c2                                   2/2     Running   0          79s
    fluentbit-gke-k8f2n                                   2/2     Running   0          80s
    fluentbit-gke-vjlqh                                   2/2     Running   0          80s
    gke-metrics-agent-4ttdt                               1/1     Running   0          79s
    gke-metrics-agent-qf24p                               1/1     Running   0          80s
    gke-metrics-agent-szktc                               1/1     Running   0          80s
    konnectivity-agent-564f9f6c5f-59nls                   1/1     Running   0          40s
    konnectivity-agent-564f9f6c5f-9nfnl                   1/1     Running   0          40s
    konnectivity-agent-564f9f6c5f-vk7l8                   1/1     Running   0          97s
    konnectivity-agent-autoscaler-5c49cb58bb-zxzlp        1/1     Running   0          97s
    kube-dns-697dc8fc8b-ddgrx                             4/4     Running   0          98s
    kube-dns-697dc8fc8b-fpnps                             4/4     Running   0          71s
    kube-dns-autoscaler-844c9d9448-pqvqr                  1/1     Running   0          98s
    kube-proxy-gke-cluster-1-default-pool-fd104f22-8rx3   1/1     Running   0          36s
    kube-proxy-gke-cluster-1-default-pool-fd104f22-gnd0   1/1     Running   0          29s
    kube-proxy-gke-cluster-1-default-pool-fd104f22-k2sv   1/1     Running   0          12s
    l7-default-backend-69fb9fd9f9-hk7jq                   1/1     Running   0          107s
    metrics-server-v0.4.4-857776bc9c-bs6sl                2/2     Running   0          44s
    pdcsi-node-5l9vf                                      2/2     Running   0          79s
    pdcsi-node-gfwln                                      2/2     Running   0          79s
    pdcsi-node-q6scz                                      2/2     Running   0          80s
    

Create a Simple ClickHouse Cluster

The operator allows the easy creation and modification of ClickHouse clusters in whatever format that works best for your organization. Now that the Google Cloud cluster is running and has the operator installed, let’s create a very simple ClickHouse cluster to test on.

The following example will create an operator-controlled cluster with 1 shard and replica, 500 MB of persistent storage, and sets the password of the default user’s password to topsecret. For more information on customizing the operator, see the Altinity Kubernetes Operator for ClickHouse Configuration Guides.

  1. Create the following manifest and save it as gcp-example01.yaml.

    
    apiVersion: "clickhouse.altinity.com/v1"
    kind: "ClickHouseInstallation"
    metadata:
    name: "gcp-example"
    spec:
    configuration:
        # What does my cluster look like?
        clusters:
        - name: "gcp-example"
        layout:
            shardsCount: 1
            replicasCount: 1
        templates:
            podTemplate: clickhouse-stable
            volumeClaimTemplate: pd-ssd
        # Where is Zookeeper?
        zookeeper:
        nodes:
        - host: zookeeper.zoo1ns
            port: 2181
        # What are my users?
        users:
        # Password = topsecret
        demo/password_sha256_hex: 53336a676c64c1396553b2b7c92f38126768827c93b64d9142069c10eda7a721
        demo/profile: default
        demo/quota: default
        demo/networks/ip:
        - 0.0.0.0/0
        - ::/0
    templates:
        podTemplates:
        # What is the definition of my server?
        - name: clickhouse-stable
        spec:
            containers:
            - name: clickhouse
            image: altinity/clickhouse-server:21.8.10.1.altinitystable
        # Keep servers on separate nodes!
            podDistribution:
            - scope: ClickHouseInstallation
            type: ClickHouseAntiAffinity
        volumeClaimTemplates:
        # How much storage and which type on each node?
        - name: pd-ssd
        # Do not delete PVC if installation is dropped.
        reclaimPolicy: Retain
        spec:
            accessModes:
            - ReadWriteOnce
            resources:
            requests:
                storage: 500Mi
    
  2. Create a namespace in your GKE environment. For this example, we will be using test:

    kubectl create namespace test
    namespace/test created
    
  3. Apply the manifest to the namespace:

    kubectl -n test apply -f gcp-example01.yaml
    clickhouseinstallation.clickhouse.altinity.com/gcp-example created
    
  4. Verify the installation is complete when all pods are in a Running state:

    kubectl -n test get chi -o wide
    NAME          VERSION   CLUSTERS   SHARDS   HOSTS   TASKID                                 STATUS      UPDATED   ADDED   DELETED   DELETE   ENDPOINT
    gcp-example   0.18.1    1          1        1       f859e396-e2de-47fd-8016-46ad6b0b8508   Completed             1                          clickhouse-gcp-example.test.svc.cluster.local
    

Login to the Cluster

This example does not have any open external ports, but we can still access our ClickHouse database through kubectl exec. In this case, our specific pod we are connecting to is chi-demo-01-demo-01-0-0-0. Replace this with the designation of your pods;

Use the following procedure to verify the Altinity Stable build install in your GKE environment.

  1. Login to the clickhouse-client in one of your existing pods:

    kubectl -n test exec -it chi-gcp-example-gcp-example-0-0-0 -- clickhouse-client
    
  2. Verify the cluster configuration:

    kubectl -n test exec -it chi-gcp-example-gcp-example-0-0-0  -- clickhouse-client -q "SELECT * FROM system.clusters  FORMAT PrettyCompactNoEscapes"
    ┌─cluster──────────────────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────────────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─slowdowns_count─┬─estimated_recovery_time─┐
    │ all-replicated                               │         111 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1    │ 90001 │ default │                  │            000│ all-sharded                                  │         111 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1    │ 90001 │ default │                  │            000│ gcp-example                                  │         111 │ chi-gcp-example-gcp-example-0-0 │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_cluster_two_shards                      │         111 │ 127.0.0.1                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_cluster_two_shards                      │         211 │ 127.0.0.2                       │ 127.0.0.2    │ 90000 │ default │                  │            000│ test_cluster_two_shards_internal_replication │         111 │ 127.0.0.1                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_cluster_two_shards_internal_replication │         211 │ 127.0.0.2                       │ 127.0.0.2    │ 90000 │ default │                  │            000│ test_cluster_two_shards_localhost            │         111 │ localhost                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_cluster_two_shards_localhost            │         211 │ localhost                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_shard_localhost                         │         111 │ localhost                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_shard_localhost_secure                  │         111 │ localhost                       │ 127.0.0.1    │ 94400 │ default │                  │            000│ test_unavailable_shard                       │         111 │ localhost                       │ 127.0.0.1    │ 90001 │ default │                  │            000│ test_unavailable_shard                       │         211 │ localhost                       │ 127.0.0.1    │    10 │ default │                  │            000└──────────────────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────┴─────────────────────────┘
    
  3. Exit out of your cluster:

    chi-gcp-example-gcp-example-0-0-0.chi-gcp-example-gcp-example-0-0.test.svc.cluster.local :) exit
    Bye.
    

Further Steps

This simple example demonstrates how to build and manage an operator-run cluster for ClickHouse. Further steps would be to open the cluster to external network connections, set up replication schemes, etc.

For more information, see the Altinity Kubernetes Operator for ClickHouse guides and the Altinity Kubernetes Operator for ClickHouse repository.