Altinity.Cloud Anywhere

• 1: Altinity.Cloud Anywhere Quickstart
• 2: Altinity.Cloud connect
• 3: Recommendations for EKS (AWS)
• 4: Recommendations for GKE (GCP)
• 5: Setting up logging

1 - Altinity.Cloud Anywhere Quickstart

This tutorial explains how to connect your Kubernetes cluster to Altinity.Cloud and begin managing ClickHouse clusters.

If you encounter difficulties with any part of the tutorial, check the Troubleshooting section at the end. Contact Altinity support for additional help if the troubleshooting advice does not resolve the problem.

Prepare Kubernetes

Make your Kubernetes cluster ready according to the recommendations below.

Ensure you have a host available access to the cluster and can run kubectl commands as in the following example.

$ kubectl get namespaces
NAME                                STATUS   AGE
default                             Active   184d
kube-node-lease                     Active   184d
kube-public                         Active   184d
kube-system                         Active   184d

Get an Altinity.Cloud Anywhere account

Obtain an account by pointing your browser to the Altinity.Cloud Anywhere signup page. Fill in the page like the example below and press SUBMIT.

Follow the instructions in the signup process to validate your email and provision your account. You will receive an email to login to Altinity.Cloud once your account is ready.

Click on the URL provided in the email and login to Altinity.Cloud. You will need to set a password to continue. Now you are ready to connect your Kubernetes cluster.

Connect Kubernetes to Altinity.Cloud

On initial login to a new Altinity.Cloud Anywhere account you will be directed to the environment setup page to complete setup. If you have an existing account or restart installation, just select the Environments tab on the left side of your screen to reach the setup page. In both cases you will see a tab like the following.

Follow the instructions on the panel using any host that can run kubectl commands on your Kubernetes cluster. Example commands are shown below.

# Download altinitycloud-connect.
curl -sSL https://github.com/altinity/altinitycloud-connect/releases/download/v0.9.3/altinitycloud-connect-0.9.3-linux-amd64 -o altinitycloud-connect \
&& chmod a+x altinitycloud-connect \
&& sudo mv altinitycloud-connect /usr/local/bin/ 

# Login to Altinity.Cloud.
altinitycloud-connect login --token=<registration token>

# Pipe setup commands to deploy connector to your Kubernetes cluster.
altinitycloud-connect kubernetes | kubectl apply -f -

You can run altinitycloud-connect kubernetes before applying to inspect Kubernetes roles and resources that it will create. After applying you should see output like the following.

namespace/altinity-cloud-system created
namespace/altinity-cloud-managed-clickhouse created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
serviceaccount/cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
secret/cloud-connect created
deployment.apps/cloud-connect created

Once these commands have completed, press PROCEED in Altinity.Cloud. If the connection is successful you will advance to Resources Configuration, which allows you to confirm resources used for ClickHouse clusters. Fill the screen out as follows.

1. Select your Kubernetes provider using the Cloud Provider radio button.
2. Inspect the storage classes to ensure classes you need are listed. If necessary, press ADD STORAGE CLASS and add additional storage classes as needed to allocate block storage for nodes in your environment.
3. Inspect the node pool list to ensure availability zones and pools you wish to use are listed. Altinity.Cloud lists availability zones that are currently in use. If you see zones that are missing, add them using the ADD NODE POOL

Here is a sample Resources Configuration tab for a GKE environment.

Press PROCEED to continue. You will see a Confirmation tab showing a JSON representation of the values selected in the previous tab.

Once the connection is fully set up, Altinity.Cloud will display your new environment, similar to the following.

Get to work!

To create your first cluster, press MANAGE CLUSTERS. This will bring you to the Clusters tab. You can now begin creating and managing ClickHouse clusters.

Troubleshooting

Altinity.Cloud Anywhere endpoint not reachable

The altinitycloud-connect command has a –url option that defaults to host anywhere.altinity.cloud on port 443. If this host is not reachable you’ll see failures like the following.

altinitycloud-connect login --token=<token>
Error: Post "https://anywhere.altinity.cloud/sign": dial tcp: lookup anywhere.altinity.cloud on 127.0.0.53:53: no such host

Make sure the name is available in DNS and that the resolved IP address is reachable on port 443 (UDP and TCP), then try again.

Note: if you are using a non-production Altinity.Cloud environment you must specify the correct URL explicitly. Contact Altinity support for help.

Insufficient Kubernetes privileges

Ensure your Kubernetes account has the following permissions:

• cluster-admin for initial provisioning only, it can be revoked afterwards
• full access to ‘altinity-cloud-system’ and ‘altinity-cloud-managed-clickhouse’ namespaces
• a few optional read-only cluster-level permissions (for observability only)

Help! I messed up the resource configuration

Find your environment in the Environment tab. Select ACTIONS->Reset Anywhere. Rerun the environment setup and enter the right values.

2 - Altinity.Cloud connect

What is Altinity.Cloud connect?

Altinity.Cloud connect (altinitycloud-connect) is a tunneling daemon for Altinity.Cloud. It enables management of ClickHouse clusters through Altinity.Cloud Anywhere.

Required permissions

altinitycloud-connect requires following permissions:

Open outbound ports:

• 443 tcp/udp (egress; stateful)

Kubernetes permissions:

• cluster-admin for initial provisioning only, it can be revoked afterwards
• full access to ‘altinity-cloud-system’ and ‘altinity-cloud-managed-clickhouse’ namespaces and a few optional read-only cluster-level permissions (for observability)

Connecting to Altinity.Cloud

See the steps in the Quickstart Connect to Altinity.Cloud procedure.

Batch operation of altinitycloud-connect

altinitycloud-connect login produces cloud-connect.pem used to connect to Altinity.Cloud Anywhere control plane (--token is short-lived while cloud-connect.pem does not expire until revoked). If you need to reconnect the environment in unattended/batch mode (i.e. without requesting the token), you can do so via

altinitycloud-connect kubernetes -i /path/to/cloud-connect.pem | kubectl apply -f -

Disconnecting your environment from Altinity.Cloud

1. Locate your environment in the Environment tab in your Altinity.Cloud account.

2. Select ACTIONS->Delete.

3. Toggle the Delete Clusters switch only if you want to delete managed clusters.

4. Press OK to complete.

After this is complete Altinity.Cloud will no longer be able to see or connect to your Kubernetes environment via the connector.

Cleaning up managed environments in Kubernetes

To clean up managed ClickHouse installations and namespaces in a disconnected Kubernetes cluster, issue the following commands in the exact order shown below.

kubectl -n altinity-cloud-managed-clickhouse delete chi --all
kubectl delete ns altinity-cloud-managed-clickhouse
kubectl delete ns altinity-cloud-system

If you delete the namespaces before deleting the ClickHouse installations (chi) the operation will hang due to missing finalizers on chi resources. Should this occur, issue kubectl edit commands on each ClickHouse installation and remove the finalizer manually from the resource specification. Here is an example.

 kubectl -n altinity-cloud-managed-clickhouse edit clickhouseinstallations.clickhouse.altinity.com/test2

3 - Recommendations for EKS (AWS)

We recommend setting up karpenter or cluster-autoscaler to launch instances in at least 3 Availability Zones.

If you plan on sharing Kubernetes cluster with other workloads, it’s recommended you label Kubernetes Nodes intended for Altinity.Cloud Anywhere with altinity.cloud/use=anywhere & taint with dedicated=anywhere:NoSchedule.

Instance Types

for Zookeeper infrastructure nodes

• t3.large or t4g.large*

* t4g instances are AWS Graviton2-based (ARM).

for ClickHouse nodes

ClickHouse works the best in AWS when using nodes from those instance families:

• m5
• m6i
• m6g*

* m6g instances are AWS Graviton2-based (ARM).

Instance sizes from large to 8xlarge are typical.

Storage Classes

• gp2
• gp2-encrypted
• gp3*
• gp3-encrypted*

* gp3 storage classes require Amazon EBS CSI driver that does not come pre-installed.

Example manifests:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  fsType: ext4
  type: gp2
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2-encrypted
provisioner: kubernetes.io/aws-ebs
parameters:
  encrypted: 'true'
  fsType: ext4
  type: gp2
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  fsType: ext4
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Notes:

• We do not recommend using gp2 storage classes. gp3 is better and less expensive
• gp3 default throughput is 125MB/s for any volume size. It can be increased in AWS console or using storage class parameters. Here is an example:

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted-500
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  throughput: '500'
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

4 - Recommendations for GKE (GCP)

Machine Types

NOTE: Depending on machine types & number of instances you plan to use, you may need to request GCE quota increase.

We recommend setting up each node pool except the default one in at least 3 zones.

If you plan on sharing Kubernetes cluster with other workloads, it’s recommended you label Kubernetes Nodes intended for Altinity.Cloud Anywhere with altinity.cloud/use=anywhere & taint with dedicated=anywhere:NoSchedule.

for Zookeeper and infrastructure nodes

• e2-standard-2

for ClickHouse nodes

It’s recommended to taint node pools below with dedicated=clickhouse:NoSchedule (in addition to altinity.cloud/use=anywhere).

• n2d-standard-2
• n2d-standard-4
• n2d-standard-8
• n2d-standard-16
• n2d-standard-32

If GCP is out of n2d-standard-* instances in the region of your choice, we recommend substituting them with n2-standard-*.

Storage Classes

• standard-rwo
• premium-rwo

GKE comes pre-configured with both.

5 - Setting up logging

Configuring logging

In order for Altinity.Cloud Anywhere to gather/store/query logs, you need to configure access to S3 or GCS bucket. Cloud-specific instructions provided below.

EKS (AWS)

The recommended way is to use IRSA.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: log-storage
  namespace: altinity-cloud-system
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<aws_account_id>:role/<role_arn>"

Alternatively, you can use custom Instance Profile or explicit credentials (shown below).

# create bucket
aws s3api create-bucket --bucket REPLACE_WITH_BUCKET_NAME --region REPLACE_WITH_AWS_REGION

# create user with access to the bucket
aws iam create-user --user-name REPLACE_WITH_USER_NAME
aws iam put-user-policy \
    --user-name REPLACE_WITH_USER_NAME \
    --policy-name REPLACE_WITH_POLICY_NAME \
    --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME/*"
            ],
            "Effect": "Allow"
        }
    ]
}'

# generate access key
aws iam create-access-key --user-name REPLACE_WITH_USER_NAME |
  jq -r '"AWS_ACCESS_KEY_ID="+(.AccessKey.AccessKeyId)+"\nAWS_SECRET_ACCESS_KEY="+(.AccessKey.SecretAccessKey)+"\n"' > credentials.env

# create altinity-cloud-system/log-storage-aws secret containing AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
kubectl create secret -n altinity-cloud-system generic log-storage-aws \
  --from-env-file=credentials.env

rm -i credentials.env

Please send bucket name back to Altinity in order to finish configuration.

GKE (GCP)

The recommended way is to use Workload Identity.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: log-storage
  namespace: altinity-cloud-system
  annotations:
    iam.gke.io/gcp-service-account: "<gcp_sa_name>@<project_id>.iam.gserviceaccount.com"

Alternatively, you can use GCP service account for instance or explicit credentials (shown below).

# create bucket
gsutil mb gs://REPLACE_WITH_BUCKET_NAME

# create GCP SA with access to the bucket
gcloud iam service-accounts create REPLACE_WITH_GCP_SA_NAME \
  --project=REPLACE_WITH_PROJECT_ID \
  --display-name "REPLACE_WITH_DISPLAY_NAME"
gsutil iam ch \
  serviceAccount:REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
  gs://REPLACE_WITH_BUCKET_NAME

# generate GCP SA key
gcloud iam service-accounts keys create credentials.json \
--iam-account=REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com \
--project=REPLACE_WITH_PROJECT_ID

# create altinity-cloud-system/log-storage-gcp secret containing credentials.json
kubectl create secret -n altinity-cloud-system generic log-storage-gcp \
  --from-file=credentials.json

rm -i credentials.json

Please send bucket name back to Altinity in order to finish configuration.