Kubernetes requirements

Configuring your Kubernetes environment

Altinity.Cloud Anywhere operates ClickHouse® clusters inside your Kubernetes environment. In this section we’ll cover the general requirements for your Kubernetes environment.

That being said, here are the requirements for your Kubernetes environment:

  • Kubernetes version 1.26 or higher in EKS (AWS), GKE (GCP), or AKS (Azure)
  • Every Node should have the following labels:
    • node.kubernetes.io/instance-type
    • kubernetes.io/arch
    • topology.kubernetes.io/zone
    • altinity.cloud/use=anywhere
  • It is recommended to taint node
    • with altinity.cloud/use=anywhere:NoSchedule only for nodes that are dedicated exclusively to Altinity pods
    • and dedicated=clickhouse:NoSchedule additionally for nodes dedicated to ClickHouse
  • A StorageClass with dynamic provisioning is required
  • LoadBalancer services must be supported

To get the most from Altinity.Cloud Anywhere features:

  • Each StorageClass should preferably allow volume expansion
  • Multiple zones are preferable for high availability
  • Autoscaling is preferable for easier vertical scaling

For platform-specific requirements, see the following sections:

AWS requirements

We recommend setting up karpenter or cluster-autoscaler to launch instances in at least 3 Availability Zones. If you’re using the AWS command line interface, version 2.0 or higher is required.

If you plan on sharing Kubernetes cluster with other workloads, it’s recommended you label Kubernetes Nodes dedicated exclusively to Altinity.Cloud Anywhere with altinity.cloud/use=anywhere and taint them with altinity.cloud/use=anywhere:NoSchedule. Please make sure there are still nodes available to run kube-system pods after applying the Altinity.Cloud Anywhere taints.

Instance types

For Zookeeper infrastructure nodes

  • t3.large or t4g.large*

t4g instances are AWS Graviton2-based (ARM).

For ClickHouse nodes

ClickHouse works the best in AWS when using nodes from ’m’ instance type families:

  • m5 (deprecated)
  • m6i
  • m6a
  • m6g*
  • m7i (new in 2023, may be not available in all regions)
  • m7a (new in 2023, may be not available in all regions)
  • m7g* (new in 2023, may be not available in all regions)

m6g and m7g instances are AWS Graviton-based (ARM).

For RAM-intensive scenarios, r6 and r7 instance type families may be used as well.

Instance sizes from large (2 vCPUs) to 8xlarge (32 vCPUs) are typical.

Storage classes

  • gp2*
  • gp3-encrypted*

We recommend using gp3 storage classes that provide more flexibility and performance over gp2. The gp3 storage classes require the Amazon EBS CSI driver; that driver is not automatically installed. See the AWS CSI driver documentation for details on how to install the driver.

Storage class can be installed with the following manifest:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The default throughput for gp3 is 125MB/s for any volume size. It can be increased in AWS console or using storage class parameters. Here is an example:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted-500
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  throughput: '500'
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Alternatively, you recommend installing the Altinity EBS parameters controller. That allows you to manage EBS volume throughput dynamically through annotations. This is also integrated to Altinity.Cloud UI (ACM).

GCP requirements

Instance types

For Zookeeper and infrastructure nodes

  • e2-standard-2

For ClickHouse nodes

  • n2d-standard-2
  • n2d-standard-4
  • n2d-standard-8
  • n2d-standard-16
  • n2d-standard-32

If GCP is out of n2d-standard-* instances in the region of your choice, we recommend substituting them with n2-standard-*.

It’s recommended to taint node pools with dedicated=clickhouse:NoSchedule (in addition to altinity.cloud/use=anywhere:NoSchedule).

Storage classes

  • standard-rwo
  • premium-rwo

GKE comes pre-configured with both.

Azure requirements

Instance types

For Zookeeper and infrastructure nodes

  • Standard_B2s_v2

For ClickHouse nodes

Use nodes from the Standard Dsv5 series

It’s recommended to set node pool taints with dedicated=clickhouse:NoSchedule.

Storage classes

Be aware that you can define custom throughput for premium2 disks:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium2-disk-sc-500
parameters:
  DiskIOPSReadWrite: "12000"
  DiskMBpsReadWrite: "500"
  cachingMode: None
  skuName: PremiumV2_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true