This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Altinity.Cloud Anywhere

Your infrastructure, our managed ClickHouse

Altinity.Cloud Anywhere lets you take advantage of Altinity’s zero-maintenance ClickHouse SaaS platform in your own environment. You provide the infrastructure, Altinity deploys the ClickHouse clusters you need on top of it.

1 - Altinity.Cloud Anywhere 101

Getting started

Introduction and benefits

Altinity.Cloud Anywhere provides the convenient cloud management of Altinity.Cloud but allows users to keep data within their own cloud VPCs and private data centers, all while running managed ClickHouse in their own Kubernetes clusters. We sometimes call this Open Cloud for ClickHouse. The Kubernetes clusters that contain ClickHouse clusters are Altinity.Cloud Anywhere environments.

Altinity.Cloud Anywhere offers several important benefits for users.

  • Compliance - Retain full control of data (including backups) as well as the operating environment and impose your policies for security, privacy, and data sovereignty.
  • Cost - Optimize infrastructure costs by running in your accounts.
  • Location - Place ClickHouse clusters close to data sources and applications.
  • No vendor lock-in - Disconnect the control plane at any time and continue operating ClickHouse using open-source components.

The rest of this document explains Altinity.Cloud Anywhere and its benefits.

The Altinity Cloud Manager UI manages Altinity.Cloud Anywhere environments just like fully hosted Altinity.Cloud environments. Users can control multiple environments from the same Altinity.Cloud account and can mix and match environment types. No matter what environment you’re in, ClickHouse management works the same way.

Open-source analytic stack

Altinity.Cloud Anywhere uses open-source software for the analytic stack and selected management services–the Altinity Kubernetes Operator for ClickHouse, Loki, Prometheus, and Grafana. The following diagram shows how the principal components map to resources in AWS. (GCP is essentially identical.) Open-source components are in dark blue rectangles.

Altinity.Cloud Management Plane
Figure 1 - The open-source analytic stack


Users can terminate the service, disconnect the Altinity.Cloud Anywhere environment from Altinity.Cloud, and run ClickHouse services themselves. There is no migration, since all data, software, and support services are already in the user’s Kubernetes cluster. The only component not in the Kubernetes clustser is the management plane.

Service architecture

The Altinity.Cloud service architecture consists of a shared management plane that serves as a single point of management for all tenants and a data plane that consists of isolated environments for each tenant.

The key component is the Altinity Connector. Altinity Connector is a secure management channel between Altinity.Cloud and the managed environment. It is deployed inside Kubernetes, dials back to the management plane and enables Altinity Cloud to manage ClickHouse and infrastructure components. An important property of Altinity Connector is that it only establishes an outbound connection, which pleases corporate Infosec teams.

We’ll discuss two variations of this architecture:

  • Bring Your Own Kubernetes (BYOK) - You manage your own Kubernetes environment, Altinity runs ClickHouse clusters inside it
  • Bring Your Own Cloud (BYOC) - You provide your cloud account, Altinity builds a Kubernetes cluster inside it, then Altinity runs ClickHouse clusters inside the Kubernetes cluster.

Bring Your Own Kubernetes (BYOK)

The following diagram shows the management plane and data plane relationships.

Altinity.Cloud Management Plane
Figure 2 - The Altinity.Cloud Anywhere service architecture

Each environment is a dedicated Kubernetes cluster. In the case of Altinity.Cloud environments, Kubernetes clusters run on Altinity’s cloud accounts and are completely hidden from users. For Altinity.Cloud Anywhere, Kubernetes clusters run in the user’s cloud account or data center. For example, the user may run an EKS cluster within a VPC belonging to the user’s AWS cloud account. Altinity.Cloud Anywhere environments can also use on-prem Kubernetes clusters. They can even use development versions of Kubernetes running on a user’s PC or laptop.

Connectivity model

Altinity.Cloud Anywhere environments use the Altinity Connector to establish a management connection from the user’s Kubernetes cluster to Altinity.Cloud. The Altinity Connector establishes an outbound HTTPS connection to a management endpoint secured by certificates. This allows management commands and monitoring data to move securely between locations.

Users connect an Altinity.Cloud Anywhere environment to Altinity.Cloud in three simple steps:

  1. Download the Altinity Connector executable program (altinitycloud-connect).
  2. Run and register Altinity Connector with Altinity.Cloud Manager.
    • If Altinity Connector is installed on a separate VM, it may run provisioning of the Kubernetes cluster (EKS or GKE). This process deploys a new instance of Altinity Connector into the provisioned Kubernetes cluster as well.
    • When Altinity Connector is installed directly in Kubernetes, it runs the provisioning of Kubernetes resources.
  3. Complete registration in the Altinity.Cloud Manager.

Altinity.Cloud Anywhere environments run all services in two namespaces:

  • The altinity-cloud-system namespace contains system services including the Altinity Connector.
  • The altinity-cloud-managed-clickhouse namespace contains ClickHouse and ZooKeeper. Users can run services in other namespaces as long as they do not make changes to the Altinity-managed namespaces.

Users should not create any resources in either of the Altinity namespaces.

Preparing your Kubernetes cluster

If you manage the Kubernetes cluster, you are responsible for setting up the Kubernetes cluster properly and deploying the Altinity Connector. We provide guidelines to do that properly, but still it requires the expertise and resources of the user’s IT team, not to mention knowledge of Kubernetes administration.

We cover the requirements for a user-managed Kubernetes cluster in a later section, but at a high level, your Kubernetes cluster must:

  • Configure storage classes that can allocate block storage on-demand
  • Enable auto-scaling
  • Enable Kubernetes pods to connect to S3-compatible object storage.

Bring Your Own Cloud (BYOC)

To make it easier for users, Altinity.Cloud Anywhere can also provision and manage the Kubernetes infrastructure, building the full stack in the user’s cloud account. This is the Bring Your Own Cloud (BYOC) model. With BYOC, the user grants Altinity.Cloud permissions to manage cloud resources directly. This access can be provided in two different ways: via a separate VM running Altinity Connector, or directly, depending on the cloud provider’s security model.

The basic model looks like this:

The basic Altinity.Cloud Anywhere service architecture
Figure 3 - The basic Altinity.Cloud Anywhere service architecture

Altinity.Cloud uses a VM-based approach on AWS. It provides the best isolation between Altinity and user account, with the VM serving as a secure bridge between two. It can be deployed using a CloudFormation template that sets an EC2 instance up and sets up the Altinity Connector. Altinity.Cloud does not have direct access to the user’s account in this case; all management operations are routed via this EC2 instance.

Here’s the architecture diagram for AWS:

The Altinity.Cloud Anywhere service architecture for AWS
Figure 4 - The Altinity.Cloud Anywhere service architecture for AWS

The other approach requires users to grant an Altinity Cloud service account direct access to the user’s cloud infrastructure. It works well in GCP, where it is easy to create a dedicated new project and manage security on the project level.

Here’s the architecture diagram for GCP:

The Altinity.Cloud Anywhere service architecture for GCP
Figure 5 - The Altinity.Cloud Anywhere service architecture for GCP

Administrative responsibilities between Altinity and you

With Altinity.Cloud Anywhere, you assume some of the responsibilities that Altinity.Cloud would handle for you. Exactly what you need to do depends on whether you’re provisioning Kubernetes yourself (Bring Your Own Kubernetes) or providing a cloud environment and giving Altinity permission to provision Kubernetes inside it (Bring Your Own Cloud). This figure shows the division of responsibilities for all three scenarios:

Figure 3 - Administrative Responsibilities
Figure 6 - Altinity.Cloud and Altinity.Cloud Anywhere environments - administrative responsibilities

Note that even though the base Kubernetes environment used by Altinity.Cloud Anywhere is technically managed by the user, in many cases Altinity.Cloud Anywhere can provision Kubernetes resources in your cluster for you.

2 - Bring your own Kubernetes (BYOK)

Using your Kubernetes infrastructure

Altinity.Cloud Anywhere uses your Kubernetes infrastructure to host your ClickHouse clusters. Your Kubernetes cluster needs to be set up a certain way; in this section we’ll go over those requirements.

2.1 - Kubernetes requirements

Configuring your Kubernetes environment

Altinity.Cloud Anywhere operates inside your Kubernetes environment. The general requirements for your Kubernetes environment are:

  • Kubernetes version 1.26 or higher in EKS (AWS), GKE (GCP), or AKS (Azure)
  • Every Node should have the following labels:
    • node.kubernetes.io/instance-type
    • kubernetes.io/arch
    • topology.kubernetes.io/zone
    • altinity.cloud/node-group: infra
  • It is recommended to taint node
    • with altinity.cloud/use=anywhere:NoSchedule for all nodes that are used for Altinity pods
    • and dedicated=clickhouse:NoSchedule additionally for nodes dedicated to ClickHouse
  • A StorageClass with dynamic provisioning is required
  • LoadBalancer services must be supported

To get the most from Altinity.Cloud Anywhere features:

  • Each StorageClass should preferably allow volume expansion
  • Multiple zones are preferable for high availability
  • Autoscaling is preferable for easier vertical scaling

For platform-specific requirements, see the following sections:

AWS requirements

We recommend setting up karpenter or cluster-autoscaler to launch instances in at least 3 Availability Zones.

If you plan on sharing Kubernetes cluster with other workloads, it’s recommended you label Kubernetes Nodes intended for Altinity.Cloud Anywhere with altinity.cloud/use=anywhere and taint them with dedicated=anywhere:NoSchedule.

Instance types

For Zookeeper infrastructure nodes

  • t3.large or t4g.large*

t4g instances are AWS Graviton2-based (ARM).

For ClickHouse nodes

ClickHouse works the best in AWS when using nodes from ’m’ instance type families:

  • m5 (deprecated)
  • m6i
  • m6a
  • m6g*
  • m7i (new in 2023, may be not available in all regions)
  • m7a (new in 2023, may be not available in all regions)
  • m7g* (new in 2023, may be not available in all regions)

m6g and m7g instances are AWS Graviton-based (ARM).

For RAM intesive scenarios, r6 and r7 instance type families may be used as well.

Instance sizes from large (2 vCPUs) to 8xlarge (32 vCPUs) are typical.

Storage classes

  • gp2*
  • gp3-encrypted*

We recommend using gp3 storage classes that provide more flexibility and performance over gp2. The gp3 storage classes require the Amazon EBS CSI driver; that driver is not automatically installed. See the AWS CSI driver documentation for details on how to install the driver.

Storage class can be installed with the following manifest:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The default throughput for gp3 is 125MB/s for any volume size. It can be increased in AWS console or using storage class parameters. Here is an example:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3-encrypted-500
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  throughput: '500'
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Alternatively, you recommend installing the Altinity EBS parameters controller. That allows you to manage EBS volume throughput dynamically through annotations. This is also integrated to Altinity.Cloud UI (ACM).

GCP requirements

Instance types

For Zookeeper and infrastructure nodes

  • e2-standard-2

For ClickHouse nodes

  • n2d-standard-2
  • n2d-standard-4
  • n2d-standard-8
  • n2d-standard-16
  • n2d-standard-32

If GCP is out of n2d-standard-* instances in the region of your choice, we recommend substituting them with n2-standard-*.

It’s recommended to taint node pools with dedicated=clickhouse:NoSchedule (in addition to altinity.cloud/use=anywhere:NoSchedule).

Storage classes

  • standard-rwo
  • premium-rwo

GKE comes pre-configured with both.

Azure requirements

Instance types

For Zookeeper and infrastructure nodes

  • Standard_B2s_v2

For ClickHouse nodes

Use nodes from the Standard Dsv5 series

It’s recommended to set node pool taints with dedicated=clickhouse:NoSchedule.

Storage classes

Be aware that you can define custom throughput for premium2 disks:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium2-disk-sc-500
parameters:
  DiskIOPSReadWrite: "12000"
  DiskMBpsReadWrite: "500"
  cachingMode: None
  skuName: PremiumV2_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

2.2 - Connecting to Altinity.Cloud Anywhere

Tying everything together

This tutorial explains how to use Altinity.Cloud Anywhere to deploy ClickHouse clusters using your choice of a third-party Kubernetes cloud provider, or using your own hardware or private company cloud. The Altinity.Cloud Manager (ACM) is used to manage your ClickHouse clusters.

If you’re just getting started, you can get a trial account in three steps:

  1. Use your business email address to sign up for a free trial on the Altinity.Cloud Anywhere trial page.
    NOTE: This must be a business email address. Addresses like *@gmail.com or *@yahoo.com are not accepted.
  2. You’ll get an email from Altinity. Follow the instructions to validate your email address.
  3. The final email you’ll get contains a login link to create a password to log in to the Altinity Cloud Manager.

Connecting Kubernetes

The first time you log in, you’ll see the Environment Setup dialog. Click the down arrow to see the list of environment types:

Environment Setup - Environment Type
Figure 1 - Choosing an Environment Type

This is the Bring Your Own Kubernetes section, after all, so we’ll select Altinity.Cloud Anywhere Bring Your Own Kubernetes (BYOK). With the environment type set, enter a name for the environment:

Environment Setup dialog
Figure 2 - Naming the Environment

Click OK to continue.

Connection setup

Next you’ll be taken to the Environment Connection screen:

Environment - Connection Setup Tab
Figure 3 - The Connection Setup tab

For Kubernetes Setup, be sure to check Provisioned by User.

Highlighted in red in Figure 3 are the three steps to complete before you select the PROCEED button. NOTE: Be sure kubectl is configured for your Kubernetes cluster.

  1. Install the latest version of Altinity.Cloud connect for your system.

  2. Copy and paste the connection string at the command line.

   altinitycloud-connect login --token=<registration token>

This initiates a TLS handshake that creates a certificate file named cloud-connect.pem on your machine. There is no output at the command line.

  1. Run this command to deploy the connector to your Kubernetes cluster.
   altinitycloud-connect kubernetes | kubectl apply -f -

The altinitycloud-connect kubernetes command generates YAML that includes the .pem file generated in the previous step. This step may take several minutes to complete.


The response will be something like this:
    namespace/altinity-cloud-system created
    namespace/altinity-cloud-managed-clickhouse created 
    clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
    clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
    clusterrole.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
    clusterrole.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
    clusterrole.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
    serviceaccount/cloud-connect created
    clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
    clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
    clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
    clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
    clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
    rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
    rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
    secret/cloud-connect created
    deployment.apps/cloud-connect created

Notice the altinity-cloud-system and altinity-cloud-managed-clickhouse namespaces above. All the resources Altinity.Cloud Anywhere creates are in those namespaces; you should not create anything in those namespaces yourself.

When the command is finished, go back to the ACM and click PROCEED to continue.

Configuring resources

When you click PROCEED on the Connection Setup tab, the ACM connects to the Kubernetes environment you just set up at the command line. You’ll see the Resources Configuration tab:

Resources Configuration Tab
Figure 4 - The Resources Configuration setup page for connecting an environment to Altinity.Cloud.

Field details

  • Cloud Provider - The correct cloud provider should be selected for you. Be sure it’s correct.
  • Region - The region your Kubernetes cluster is running in should be selected for you.
  • Number of AZs - The number of availability zones for your ClickHouse cluster.
  • Storage Classes - One or more storage classes will be defined for you. You can click the trash can icon to delete a storage class, or click the + ADD STORAGE CLASS button to create a new one.
  • Node Pools - One or more node pools from your underlying Kubernetes cluster will be defined for you. You can click the trash can icon to delete a node pool, or click the +ADD NODE POOL button to create a new one.
    • In the Used For section, ClickHouse, Zookeeper, and System must be selected at least once in at least one of the node pools. Selecting multiple node pools for ClickHouse nodes is highly recommended.
    • Be aware that you can add node pools later if needed.
  • View as JSON - If checked, this displays a panel with a JSON description of your new ClickHouse cluster’s configuration when you click PROCEED. If not checked, clicking PROCEED starts the provisioning of the cluster.

Figure 4 above shows the settings for a Kubernetes cluster hosted in the Azure Kubernetes Service. The available controls and options are different for each cloud platform. Click PROCEED to provision the resources you’ve configured.

It will take a few minutes for all the resources to be provisioned.

Connection completed

Once the connection is fully set up, the ACM Environments dashboard will display your new environment:

Provisioned Environment Tab
Figure 5 - Environment dashboard tab showing your running Anywhere cluster.

If you have any problems, see the Troubleshooting section below.

Administering Altinity.Cloud Anywhere

Once your environment is configured, you use the Altinity Cloud Manager (ACM) to perform common user and administrative tasks. The first thing you’ll want to do in the ACM is create a ClickHouse cluster in your environment; the Launch Cluster Wizard makes it easy.

For other tasks, you’ll find all the information you need in the ACM documentation:

Troubleshooting

Q-1. Altinity.Cloud Anywhere endpoint not reachable

Problem

  • By default, the altinitycloud-connect command connects to host anywhere.altinity.cloud on port 443. If this host is not reachable, the following error message appears.

    altinitycloud-connect login --token=<token>
    Error: Post "https://anywhere.altinity.cloud/sign":
       dial tcp: lookup anywhere.altinity.cloud on 127.0.0.53:53: no such host
    

Solution

  • Make sure the name is available in DNS and that the resolved IP address is reachable on port 443 (UDP and TCP), then try again. The altinitycloud-connect command has a --url option if you need to specify a different URL.

Q-2. Insufficient Kubernetes privileges

Problem

  • Your Kubernetes account has insufficient permissions.

Solution

  • Look at the output from the altinitycloud-connect kubernetes | kubectl apply -f - command to see what actions failed, then adjust the permissions for your Kubernetes account accordingly. At a minimum, set the following permissions:

    • cluster-admin for initial provisioning only (it can be revoked afterward)
    • Give full access to the altinity-cloud-system and altinity-cloud-managed-clickhouse namespaces
    • A few optional read-only cluster-level permissions (for observability only)

Q-3. Help! I messed up the resource configuration

Problem

  • The resource configuration settings are not correct.

Solution

  1. From the Environment tab, in the Environment Name column, select the link to your environment.
  2. Select the menu function ACTIONS 》Reconfigure Anywhere.
  3. Rerun the Environment 》Connection Setup and enter the correct values.

Q-4 One of my pods won’t spin up

When you reboot your machine, the Anywhere cluster in your ACM has not started.

Problem

One of the pods won’t start. In the listing below, pod edge-proxy-66d44f7465-lxjjn in the altinity-cloud-system namespace has not started:

    ┌──────────────── Pods(altinity-cloud-system)[8] ──────────────────────────┐
    │ NAME↑                                PF READY RESTARTS STATUS            │
 1  │ cloud-connect-d6ff8499f-bkc5k        ●  1/1       3    Running           │
 2  │ crtd-665fd5cb85-wqkkk                ●  1/1       3    Running           │
 3  │ edge-proxy-66d44f7465-lxjjn          ●  1/2       7    CrashLoopBackOff  │
 4  │ grafana-5b466574d-4scjc              ●  1/1       1    Running           │
 5  │ kube-state-metrics-58d86c747c-7hj79  ●  1/1       6    Running           │
 6  │ node-exporter-762b5                  ●  1/1       3    Running           │
 7  │ prometheus-0                         ●  1/1       3    Running           │
 8  │ statuscheck-f7c9b4d98-2jlt6          ●  1/1       3    Running           │
    └──────────────────────────────────────────────────────────────────────────┘

Solution

Delete the pod using the kubectl delete pod command and it will regenerate.

kubectl -n altinity-cloud-system delete pod edge-proxy-66d44f7465-lxjjn

2.3 - Setting up logging

Configuring storage for logging

In order for Altinity.Cloud Anywhere to gather/store/query logs, you need to configure access to an S3 or GCS bucket. Cloud-specific instructions provided below.

Logs will be scraped only from Node that have the label:

  • altinity.cloud/node-group: infra

EKS (AWS)

The recommended way is to use IRSA.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: log-storage
  namespace: altinity-cloud-system
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<aws_account_id>:role/<role_arn>"

Alternatively, you can use a custom Instance Profile or explicit credentials (shown below).

# create bucket
aws s3api create-bucket --bucket REPLACE_WITH_BUCKET_NAME --region REPLACE_WITH_AWS_REGION

# create user with access to the bucket
aws iam create-user --user-name REPLACE_WITH_USER_NAME
aws iam put-user-policy \
    --user-name REPLACE_WITH_USER_NAME \
    --policy-name REPLACE_WITH_POLICY_NAME \
    --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME/*"
            ],
            "Effect": "Allow"
        }
    ]
}'

# generate access key
aws iam create-access-key --user-name REPLACE_WITH_USER_NAME |
  jq -r '"AWS_ACCESS_KEY_ID="+(.AccessKey.AccessKeyId)+"\nAWS_SECRET_ACCESS_KEY="+(.AccessKey.SecretAccessKey)+"\n"' > credentials.env

# create altinity-cloud-system/log-storage-aws secret containing AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
kubectl create secret -n altinity-cloud-system generic log-storage-aws \
  --from-env-file=credentials.env

rm -i credentials.env

Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.

GKE (GCP)

The recommended way is to use Workload Identity.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: log-storage
  namespace: altinity-cloud-system
  annotations:
    iam.gke.io/gcp-service-account: "<gcp_sa_name>@<project_id>.iam.gserviceaccount.com"

Alternatively, you can use GCP service account for instance or explicit credentials (shown below).

# create bucket
gsutil mb gs://REPLACE_WITH_BUCKET_NAME

# create GCP SA with access to the bucket
gcloud iam service-accounts create REPLACE_WITH_GCP_SA_NAME \
  --project=REPLACE_WITH_PROJECT_ID \
  --display-name "REPLACE_WITH_DISPLAY_NAME"
gsutil iam ch \
  serviceAccount:REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
  gs://REPLACE_WITH_BUCKET_NAME

# generate GCP SA key
gcloud iam service-accounts keys create credentials.json \
--iam-account=REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com \
--project=REPLACE_WITH_PROJECT_ID

# create altinity-cloud-system/log-storage-gcp secret containing credentials.json
kubectl create secret -n altinity-cloud-system generic log-storage-gcp \
  --from-file=credentials.json

rm -i credentials.json

Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.

2.4 - Setting up backups

Configuring storage for backups

In order for Altinity.Cloud Anywhere to work with backups, you need to configure access to an S3 or GCS bucket. Cloud-specific instructions are provided below.

EKS (AWS)

Use a custom Instance Profile or explicit credentials (shown below).

# create bucket
aws s3api create-bucket --bucket REPLACE_WITH_BUCKET_NAME --region REPLACE_WITH_AWS_REGION

# create user with access to the bucket
aws iam create-user --user-name REPLACE_WITH_USER_NAME
aws iam put-user-policy \
    --user-name REPLACE_WITH_USER_NAME \
    --policy-name REPLACE_WITH_POLICY_NAME \
    --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:PutObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_WITH_BUCKET_NAME/*"            
            ],
            "Effect": "Allow"
        }
    ]
}'

# generate access key
aws iam create-access-key --user-name REPLACE_WITH_USER_NAME |
  jq -r '"AWS_ACCESS_KEY_ID="+(.AccessKey.AccessKeyId)+"\nAWS_SECRET_ACCESS_KEY="+(.AccessKey.SecretAccessKey)+"\n"' > credentials.env
  
# create altinity-cloud-system/clickhouse-backup secret containing AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
kubectl create secret -n altinity-cloud-system generic clickhouse-backup \
  --from-env-file=credentials.env

rm -i credentials.env

Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.

GKE (GCP)

Use a GCP service account for the instance or explicit credentials (shown below).

# create bucket
gsutil mb gs://REPLACE_WITH_BUCKET_NAME

# create GCP SA with access to the bucket
gcloud iam service-accounts create REPLACE_WITH_GCP_SA_NAME \
  --project=REPLACE_WITH_PROJECT_ID \
  --display-name "REPLACE_WITH_DISPLAY_NAME"
gsutil iam ch \
  serviceAccount:REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
  gs://REPLACE_WITH_BUCKET_NAME

# generate GCP SA key
gcloud iam service-accounts keys create credentials.json \
--iam-account=REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com \
--project=REPLACE_WITH_PROJECT_ID

# create altinity-cloud-system/clickhouse-backup secret containing GOOGLE_APPLICATION_CREDENTIALS 
kubectl create secret -n altinity-cloud-system generic clickhouse-backup \
  --from-file=GOOGLE_APPLICATION_CREDENTIALS=credentials.json
  
rm -i credentials.json

Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.

2.5 - Disconnecting from Altinity.Cloud Anywhere

Using ClickHouse without Altinity.Cloud Anywhere

Even if you disconnect from Altinity.Cloud Anywhere altogether, your ClickHouse cluster can continue running in your Kubernetes environment. In this section we’ll show you how to do that.

Disconnecting your environment from Altinity.Cloud Anywhere

You can disconnect Altinity Cloud Manager from your Kubernetes environment and the ClickHouse clusters running inside it. This does not delete your running ClickHouse clusters, it merely disconnects them from the Altinity Cloud Manager. Your ClickHouse clusters continue running as usual.

This command disconnects your ClickHouse cluster:

altinitycloud-connect kubernetes-disconnect | kubectl delete -f -

After this command completes, Altinity.Cloud Anywhere will no longer be able to see or connect to your Kubernetes environment.

Deleting managed ClickHouse environments in Kubernetes

If you want to delete the ClickHouse clusters in your environment, enter these two commands in the order shown below.

kubectl -n altinity-cloud-managed-clickhouse delete chi --all
altinitycloud-connect kubernetes | kubectl delete -f -

The first command deletes every ClickHouse installation (chi) that Altinity.Cloud Anywhere created. Those are in the altinity-cloud-managed-clickhouse namespace. With the ClickHouse clusters deleted, the second command deletes the two Altinity namespaces and any remaining resources they contain.

WARNING: If you delete the namespaces before deleting the ClickHouse installations (chi), the operation will hang due to missing finalizers on chi resources. Should this occur, use the kubectl edit command on each ClickHouse installation and remove the finalizer manually from the resource specification. Here is an example:

 kubectl -n altinity-cloud-managed-clickhouse edit clickhouseinstallations.clickhouse.altinity.com/maddie-ch

You can now delete the finalizer from the resource:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  creationTimestamp: "2023-08-29T17:03:58Z"
  finalizers:
  - finalizer.clickhouseinstallation.altinity.com
  generation: 3
  name: maddie-ch
  
. . . 

2.6 - Appendix: Using Altinity.Cloud Anywhere with minikube

For testing and development use only

This guide covers setting up minikube so that you can use Altinity.Cloud Anywhere to provision ClickHouse clusters inside minikube. Any computer or cloud instance that can run minikube and support the resource requirements of the Kubernetes cluster we describe here should work.

Note that while minikube is okay to use for development purposes, it should not be used for production. Seriously. We can’t stress that enough. It’s great for development, but don’t use it for production.

Server requirements

In the deployment you’ll do here, you’ll build a minikube cluster with seven nodes. Using the Docker runtime on a MacBook Pro M2 Max, the system provisioned 6 vCPUs and 7.7 GB of RAM per node, along with roughly 60 GB of disk space per node. It’s unlikely all of your nodes will run at capacity, but there’s no guarantee your machine will have enough resources to do whatever you want to do in your minikube cluster. (Did we mention it’s not for production use?) And, of course, the default provisioning may be different on other operating systems, hardware architectures, or virtualization engines.

Before you get started, you’ll need to sign up for an Altinity.Cloud Anywhere trial account. At the end of that process, you’ll have an email with a link to the Altinity Cloud Manager (ACM). You’ll use that link to set up the connection between minikube and Altinity.

Finally, of course, you’ll need to install minikube itself. See the minikube start page for complete install instructions. Just install the software at this point; we’ll talk about how to start minikube in the next section.

Starting minikube

If you’ve used minikube on your machine before, we recommend that you delete its existing configuration:

minikube delete

Now start a minikube cluster with seven nodes:

minikube start --nodes 7 --kubernetes-version=v1.22.8

You’ll see results like this:

😄  minikube v1.30.1 on Darwin 13.5.2 (arm64)

✨  Automatically selected the docker driver. Other choices: qemu2, parallels, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3500MB) ...
🐳  Preparing Kubernetes v1.22.8 on Docker 23.0.2 ...
❌  Unable to load cached images: loading cached images: stat /Users/dougtidwell/.minikube/cache/images/arm64/registry.k8s.io/kube-apiserver_v1.22.8: no such file or directory
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🔎  Verifying Kubernetes components...

👍  Starting worker node minikube-m02 in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3500MB) ...
🌐  Found network options:
NO_PROXY=192.168.49.2
🐳  Preparing Kubernetes v1.22.8 on Docker 23.0.2 ...
    ▪ env NO_PROXY=192.168.49.2
🔎  Verifying Kubernetes components...

. . .

🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

NOTE: Defining the environment variable MINIKUBE_IN_STYLE=0 disables the emojis that appear in front of every minikube message. You’re welcome.

At this point minikube is up and running. The kubectl get nodes command shows our seven nodes:

NAME           STATUS   ROLES                  AGE     VERSION
minikube       Ready    control-plane,master   4m22s   v1.22.8
minikube-m02   Ready    <none>                 4m2s    v1.22.8
minikube-m03   Ready    <none>                 3m48s   v1.22.8
minikube-m04   Ready    <none>                 3m33s   v1.22.8
minikube-m05   Ready    <none>                 3m17s   v1.22.8
minikube-m06   Ready    <none>                 3m2s    v1.22.8
minikube-m07   Ready    <none>                 2m46s   v1.22.8

When using Altinity.Cloud Anywhere with a traditional cloud vendor, there are node types, availability zones, and storage classes. We need to label our minikube nodes to simulate those things. First, run these commands to define the node types and availability zones:

kubectl --context=minikube label nodes minikube \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-a 
kubectl --context=minikube label nodes minikube-m02 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-a 
kubectl --context=minikube label nodes minikube-m03 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-a
kubectl --context=minikube label nodes minikube-m04 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-b
kubectl --context=minikube label nodes minikube-m05 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-b
kubectl --context=minikube label nodes minikube-m06 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-c
kubectl --context=minikube label nodes minikube-m07 \
  node.kubernetes.io/instance-type=minikube-node \
  topology.kubernetes.io/zone=minikube-zone-c

Now all of our minikube nodes are defined to be of type minikube-node; we’ll see that node type again later. We’ve also defined availability zones named minikube-zone-a, minikube-zone-b, and minikube-zone-c.

On to our storage classes. We want to use the local-path storage class instead of minikube’s default standard storage class. This command defines the new storage class:

curl -sSL https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.22/deploy/local-path-storage.yaml | \
  sed 's/\/opt\/local-path-provisioner/\/var\/opt\/local-path-provisioner/ ' | \
  kubectl --context=minikube apply -f -

Now that we’ve defined the new storage class, we need to tell minikube that the local-path class is the default:

kubectl --context=minikube patch storageclass standard \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl --context=minikube patch storageclass local-path \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Running kubectl get storageclasses shows the new default class:

NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path      Delete          WaitForFirstConsumer   false                  36s
standard               k8s.io/minikube-hostpath   Delete          Immediate              false                  15m

Connecting Altinity.Cloud Anywhere to minikube

Now that we have the minikube cluster running and configured, it’s time to connect it to Altinity.Cloud Anywhere. That’s the final step for enabling Altinity to provision ClickHouse clusters in minikube.

Step 1. Setting up the tunnel

First we need to set up the TLS tunnel between minikube and Altinity. Click the emailed link you got when you signed up for an Altinity.Cloud Anywhere account. You’ll see this screen:

The Connection Setup screen
Figure 1 - The Connection Setup screen

Make sure the “Provisioned by User” box is selected at the top of the page, and make sure you’ve installed altinitycloud-connect from the link beneath it.

Copy and paste the text in the center box at the command line and run it. This doesn’t generate any output at the command line, but it does create a cloud-connect.pem file in the current directory.

Now that you have the cloud-connect.pem file, run the following command to set up the TLS tunnel:

altinitycloud-connect kubernetes --url=https://anywhere.altinity.cloud --release=latest-master | kubectl --context=minikube apply -f -

Note: The command you run is different from the one in the text box at the bottom of Figure 1. Make sure that the --url parameter matches the URL in that text box, as it is dependent on the Altinity.Cloud Anywhere endpoint you’re using.

The altinitycloud-connect kubernetes command generates YAML that has configuration information along with the keys from the .pem file. That YAML data is passed to kubectl.

You’ll see results similar to this:

namespace/altinity-cloud-system created
namespace/altinity-cloud-managed-clickhouse created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:storage-class-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
serviceaccount/cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:storage-class-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
secret/cloud-connect created
deployment.apps/cloud-connect created

Another note: Altinity creates all ClickHouse-related assets in the altinity-cloud-system and altinity-cloud-managed-clickhouse namespaces. You should not create anything in those namespaces yourself.

Click Proceed to go to the next step.

Step 2. Configuring your minikube resources

Next we’ll define aspects of the minikube environment to Altinity. Use the values highlighted in red in Figure 2.

The Resources Configuration screen
Figure 2 - The Resources Configuration screen

The specific values to use are:

  • Cloud Provider: Not Specified (minikube is a special case)
  • Region: minikube-zone (we defined that with a label earlier)
  • Number of AZs: 3
  • Storage Classes: local-path (defined as the default storageclass earlier)
  • Node Pools: A single node pool named minikube-node with a capacity of 2. The boxes for ClickHouse and Zookeeper must be checked.

Click Proceed to go to the Confirmation screen.

Step 3. Confirming your choices

A JSON description of all of your choices appears in the text box at the top of Figure 3:

The Confirmation screen
Figure 3 - The Confirmation screen

You can edit the JSON as needed; currently you need to change the names of the availability zones. Using the values specified in the previous step, the generated availability zones will be minikube-zonea, minikube-zoneb, and minikube-zonec. They should be minikube-zone-a, minikube-zone-b, and minikube-zone-c.

Once everything is correct, click Finish. This begins the process of creating a ClickHouse cluster inside minikube. You’ll see a status bar similar to Figure 4:

The status bar
Figure 4 - The status bar

It’s quite likely the status bar will reach the end before everything is configured. Just keep clicking Finish until things are, well, finished:

Click Finish again to keep waiting
Figure 5 - Click Finish again to keep waiting

When things are finished, you’ll see this screen:

Altinity.Cloud Anywhere is up and running
Figure 6 - Altinity.Cloud Anywhere is up and running

With everything up and running, kubectl get pods -n altinity-cloud-managed-clickhouse shows the pods Altinity.Cloud Anywhere created:

NAME                                   READY   STATUS    RESTARTS        AGE
chi-minikube-ch1-minikube-ch1-0-0-0    2/2     Running   3 (3m12s ago)   4m54s
chi-minikube-ch1-minikube-ch1-0-1-0    2/2     Running   3 (3m23s ago)   4m42s
clickhouse-operator-85c8855c56-qn98x   2/2     Running   0               4m31s
zookeeper-1638-0                       1/1     Running   0               4m43s
zookeeper-1638-1                       1/1     Running   0               2m54s
zookeeper-1638-2                       1/1     Running   0               4m56s

There are two pods for ClickHouse itself, a pod for the Altinity ClickHouse Operator, and three pods for Zookeeper. These pods are managed for you by Altinity.

Working with Altinity.Cloud Anywhere

Now that your environment is configured, you use the Altinity Cloud Manager (ACM) to perform common user and administrative tasks. The steps and tools to manage your ClickHouse clusters are the same for Altinity.Cloud Anywhere and Altinity.Cloud.

Here are some common tasks from the ACM documentation:

The ACM documentation includes:

At the command line you can also connect to a running pod and work with ClickHouse directly.

3 - Bring your own cloud (BYOC)

Configuring your cloud account

Altinity.Cloud Anywhere can use your cloud account to provision cloud resources to host your ClickHouse clusters. Your cloud account needs to have certain permissions and other properties; in this section we’ll go over those requirements.

Introduction

Altinity.Cloud Anywhere can operate inside a user’s cloud account.

Altinity technical support can remotely provision Kubernetes clusters for an Altinity.Cloud Anywhere environment inside your AWS or GCP account. Once the Kubernetes environment is set up, Altinity can provision ClickHouse clustes inside Kubernetes. This section describes how to configure your AWS or GCP accounts to authorize Altinity.Cloud to provision Kubernetes and ClickHouse in your account.

Figure 1 is a high-level view of the Altinity.Cloud Kubernetes infrastructure.

The basic Altinity.Cloud Anywhere service architecture
Figure 1 - The basic Altinity.Cloud Anywhere service architecture

Platform-specific requirements are discussed in the following sections.

3.1 - AWS remote provisioning

Configuring your AWS account

Provisioning process overview

This section summarizes the bootstrap process that lets you use Altinity.Cloud Anywhere to create an EKS environment and deploy ClickHouse clusters to it. You’ll go through these steps:

  1. Get an Altinity.Cloud Anywhere account.
  2. Get an Altinity.Cloud Anywhere environment record.
  3. Get a connection token from the Altinity Cloud Manager.
  4. Provision an AWS EKS cluster using EC2 instance running with a user account.
    The EC2 instance is required in order to deploy altinitycloud-connect, which will establish an outbound connection to Altinity.Cloud and start the EKS provisioning process.
    The EC2 instance can be set up in two ways:
    • Automatically by using the AWS Cloud Formation Template to automate the process.
    • Manually set up by a user following Altinity documentation.
  5. In the Altinity Cloud Manager, complete the configuration of EKS resources.

Automatically provisioning EKS using an EC2 instance created from the AWS Cloud Formation Template

An Amazon AWS EC2 instance is required to deploy altinitycloud-connect, which establishes an outbound connection to Altinity.Cloud Anywhere and starts the EKS provisioning process.

In Altinity.Cloud Anywhere

  1. Get an Altinity.Cloud Anywhere account.

  2. Get an Altinity.Cloud Anywhere environment record.

  3. Get a connection token from the Altinity Cloud Manager. The connection token is everything after token= in the “Connect to Altinity.Cloud” text box. In Figure 1, the token is all of the text in red.

    The Altinity.Cloud Anywhere connection token
    Figure 1 - The Altinity.Cloud Anywhere connection token

  4. Login to your AWS account and navigate to Cloud Formation and click the Create Stack button.

  5. Go to the altinitycloud-connect releases page and download the latest Cloud Formation YAML file, altinitycloud-connect-x.xx.x.aws-cloudformation.yaml. (That’s currently version 0.20.0.) The link to the YAML file is at the bottom of the page.

  6. In the Create Stack panel, choose Upload a template file and select the Altinity Cloud Formation Template YAML file as shown in Figure 2. Click Next to continue:

    The Altinity.Cloud Anywhere connection token
    Figure 2 - Selecting a template file

  7. Fill in any missing fields on the Specify Stack Details page (Figure 3):

    • Set Stack Name to altinitycloud-connect-$ENV_NAME (if your environment name is maddie-byok, the stack name should be altinity-connect-maddie-byok.)

    • Select the Subnets where the altinitycloud-connect EC2 instance(s) should be launched. At least one subnet is required.
      NOTE: The subnet(s) must have the “Auto-assign public IPv4 address” property set to yes.

    • Set the Token presented by https://acm.altinity.cloud/ to the token value from Step 3.

    • Click Next to continue.

    AWS CloudFormation Stack details panel
    Figure 3 - AWS CloudFormation Stack details panel

  8. Take the defaults on the Configure stack options panel (Figure 4). Click Next to continue.

    AWS CloudFormation Configure stack options panel
    Figure 4 - AWS CloudFormation Configure stock options panel

  9. On the Review panel, check the box to allow CloudFormation to create resources as shown in Figure 5. The process will not work unless you click this button. Click Submit to start provisioning the new stack.

    Granting permissions on the AWS CloudFormation Review panel
    Figure 5 - Granting permissions on the AWS CloudFormation Review panel

  10. You’ll see the main CloudFormation panel (Figure 6), which will include your new stack. You can follow its progress in the Events list.

    AWS CloudFormation stack list
    Figure 6 - AWS CloudFormation stack list

EC2 background processing explained

The EC2 instance is processed in the background as follows:

  • EC2 instance gets started from the cloud formation template
  • EC2 gets connected to Altinity.Cloud using altinitycloud-connect
  • EKS cluster gets provisioned
  • EKS cluster gets connected to Altinity.Cloud using altinitycloud-connect

In Altinity Cloud Manager

  1. Once your new stack is created, click the ‘Proceed’ button in the connection wizard. Altinity Cloud Manager will connect to the stack named altinitycloud-connect-$ENV_NAME where $ENV_NAME is the name of your environment. NOTE: It is ok to select Proceed more than once, since provisioning takes some time. Once the EKS cluster is provisioned, the wizard switches to the Resources Configuration page.

  2. Finish configuring your cluster as described in the Configuring resources section.

Manual provisioning of the EC2 instance

The AWS EC2 instance should meet the following requirements:

EC2 Instance Requirements

  • CPU: t2.micro minimum
  • OS: Ubuntu Server v20.04

Creating a Role with AIM policies

Set up a role with IAM policies to access IAM, EC2, VPC, EKS, S3 & Lambda as follows:

  arn:aws:iam::aws:policy/IAMFullAccess
  arn:aws:iam::aws:policy/AmazonEC2FullAccess
  arn:aws:iam::aws:policy/AmazonVPCFullAccess
  arn:aws:iam::aws:policy/AmazonS3FullAccess
  arn:aws:iam::aws:policy/AWSLambda_FullAccess
  arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

NOTE:

Creating a policy for EKS full access

  1. Create a standard policy for EKS full access as follows:
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "eks  ":"*"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":"iam:PassRole",
         "Resource":"*",
         "Condition":{
            "StringEquals":{
               "iam:PassedToService":"eks.amazonaws.com"
            }
         }
      }
   ]
}
  1. To set this instance to have access to the EC2 metadata and Internet, set the Security group to:

    • deny all inbound traffic
    • allow all outbound traffic

Installing Altinity.Cloud Connect

  1. Download altinitycloud-connect. NOTE: The following example is for an Intel Linux installation. Change the filename to match your machine’s architecture.

    curl -sSL https://github.com/altinity/altinitycloud-connect/releases/download/v0.20.0/altinitycloud-connect-0.20.0-linux-amd64 -o altinitycloud-connect \
    
  2. With the file downloaded, run these commands to make the file executable and put it in your path.

    chmod a+x altinitycloud-connect \
    && sudo mv altinitycloud-connect /usr/local/bin/
    
  3. Login to Altinity.Cloud using a connection token as shown in Figure 1 above.

    altinitycloud-connect login --token=<registration token>
    

NOTE: altinitycloud-connect creates a cloud-connect.pem file in the current working directory.

  1. Connect to Altinity.Cloud:

    altinitycloud-connect --capability aws
    

Start EKS provisioning

Provide additional configuration data to Altinity Support

  1. The following data is required in order to create the VPC and EKS cluster properly:
  • The CIDR for the Kubernetes VPC (at least /21 recommended, such as 10.1.0.0/21) that does not overlap with existing VPCs
  • The Number of Availability Zones (3 are recommended)

Please send this information to your Altinity support representative to start the EKS provisioning process. When completed, the Altinity Cloud Manager (ACM) will be updated. At that point, you can create your ClickHouse clusters.

The remainder of the provisioning process is handled by Altinity.Cloud. Users may switch back to ACM and wait for connection to be established in order to finish configuration.

In Altinity.Cloud

  1. Select the Proceed button in the connection wizard. You may repeat this step more than once to see if the connection has completed, since provisioning takes some time. Once the EKS cluster is provisioned, the connection wizard will switch to the Resources Configuration page.

  2. Finish configuration of node pools as described in the Configuring resources section.

Break Glass procedure

The “Break Glass” procedure allows Altinity access to EC2 instance with SSH, using AWS SSM in order to troubleshoot altinitycloud-connect that is running on this instance.

  1. Create an AnywhereAdmin IAM role with trust policy set:
    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Principal":{
                "AWS":"arn:aws:iam::313342380333:role/AnywhereAdmin"
             },
             "Action":"sts:AssumeRole"
          }
       ]
    }
  1. Add a permission policy set:
    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":"ssm:StartSession",
             "Resource":[
                "arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID",
                "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
             ]
          }
       ]
    }
  1. Send the following ARN string to Altinity: NOTE: This is used to revoke the Break Glass Procedure access change, or remove the permission policy.
    arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID

3.2 - Azure remote provisioning

Configuring your Azure account

With an Altinity.Cloud Anywhere account, Altinity technical support can remotely provision Azure AKS clusters in your Azure account. Instructions on this page describe how to configure your Azure account to give Altinity the necessary permissions.

Setting up a Bring Your Own Cloud environment for Azure involves four steps:

  1. Get an Altinity.Cloud Anywhere account.
  2. Create a Resource Group inside your Azure account
  3. Create an Environment in the Altinity Cloud Manager (ACM)
  4. Configure your Environment to connect it to the Azure Resource Group you created.

With these steps completed, Altinity.Cloud Anywhere will have the necessary access to create Kubernetes clusters in your Azure account, then create ClickHouse clusters in those Kubernetes clusters.

Creating a Resource Group

From the Azure web console, click Resource groups:

Intro menu in the Azure console

Click the Create button in the Resource groups tile that appears when you click the icon. Select your Subscription (there’s likely only one), give your Resource group a name, and select the region where your Resource group will be created:

Defining a resource group

You can click the Review + create button at the bottom of the panel, or click Next + Tags > to enter more options. When you’re done, you’ll see this confirmation screen:

Creating a resource group

If everything looks good, click Create to create the Resource group.

Creating an Environment in the ACM

Once the Resource Group is created, go to the Altinity Cloud Manager, and open the Environments tab. Click the button at the top of the screen.

In the Environment Setup dialog, click the down arrow to see the list of environment types:

Environment Setup - Environment Type

Select Altinity.Cloud Anywhere Bring Your Own Cloud (BYOC). With the environment type set, enter a name for the environment:

Naming your environment

Click OK to continue.

Configuring your Environment

With your Altinity.Cloud Anywhere environment created, you need to connect the environment to the Resource Group in your Azure account.

Setting up your connection

The first tab of the Environment Setup wizard is the Connection Setup tab:

AKS provisioned by Altinity

Select AKS Provisioned by Altinity and click PROCEED to continue.

Defining your environment’s properties

Next you’ll see the Resources Configuration tab:

Defining the details of your cloud account

Field Details

  • Cloud Provider - Azure should be selected.
  • Region - Click the down arrow icon and select the appropriate region from the list of Azure regions.
  • Number of AZs - The number of Availability Zones for your Kubernetes clusters. We recommend that you select at least two AZs.
  • Tenant ID - You can find this in the Azure web console.
  • Subscription ID - You can find this in the Azure web console as well.
  • CIDR Block - Enter the CIDR block for your environment. We recommend at least /21 to ensure you have enough addresses for your cluster.
  • Storage Classes - Select as many storage classes as you need. You can create new storage classes with the + ADD STORAGE CLASS button or delete storage classes by clicking the trash can icon.
  • Node Pools - Define as many node pools as you need. At least one pool must be selected for ClickHouse clusters, at least one for Zookeeper nodes, and at least one for System nodes. You can click the + ADD NODE POOL button to add new node pools, or click RESET TO DEFAULTS to restore the original settings. NOTE: You can add more node pools later if you need them.

Click PROCEED to start the configuration process. You’ll see the status of the configuration:

Connecting to your Azure Resource Group

The Altinity Cloud Manager will connect to your Azure account and configure the resources it needs to create and manage Kubernetes clusters and ClickHouse clusters in your Azure account.

Click the CONTINUE WAITING button as many times as necessary.

3.3 - GCP remote provisioning

Configuring your GCP account

Introduction

Altinity.Cloud Anywhere operates inside a user’s cloud account.

Altinity technical support can remotely provision GCP GKE clusters with an Altinity.Cloud Anywhere environment in your GCP account. Instructions on this page describe how to configure your GCP account to provide permission to Altinity to provision ClickHouse to your GCP Kubernetes environment.

Overview - Google GKE Installation

This section covers how to authorize Altinity.Cloud Anywhere to create Kubernetes and ClickHouse resources inside your cloud account.

Creating a project

Creating a separate project makes it easy to isolate resources and do cost management, not to mention security. You can create a project from the command line or in the GCP web UI.

You can use the gcloud projects create command to create a new project:

# Create project 'maddie'
gcloud projects create maddie

You also need to assign a billing account to the project. Currently the gcloud command looks like this:

# Assign a billing account to 'maddie'
gcloud beta billing projects link maddie \
  --billing-account 0X0X0X-0X0X0X-0X0X0X

You can also create a project from the GCP web UI:

The GCP New Project dialog
Figure 1 - The GCP New Project dialog

Granting permissions

For Altinity to be able to create Kubernetes and ClickHouse clusters in your cloud account, you need to grant the following permissions to anywhere-admin@altinity.com inside the project you just created:

  • roles/compute.admin
  • roles/container.admin
  • roles/dns.admin
  • roles/storage.admin
  • roles/storage.hmacKeyAdmin
  • roles/iam.serviceAccountAdmin
  • roles/iam.serviceAccountKeyAdmin
  • roles/iam.serviceAccountTokenCreator
  • roles/iam.serviceAccountUser
  • roles/iam.workloadIdentityPoolAdmin
  • roles/serviceusage.serviceUsageAdmin
  • roles/resourcemanager.projectIamAdmin
  • roles/iap.tunnelResourceAccessor

You can use the gcloud command for each role:

# Add a role for a member of a group associated with project 'maddie'
gcloud projects add-iam-policy-binding maddie \
  --member='group:anywhere-admin@altinity.com' \
  --role='roles/compute.admin'

Alternately, you can use the GCP web UI:

Permissions in the GCP web UI
Figure 2 - Permissions in the GCP web UI

Creating the Kubernetes environment

With the project created and the appropriate permissions granted to the Altinity.Cloud Anywhere admin account, Altinity can create Kubernetes clusters and ClickHouse clusters inside them. The following sections demonstrate how to create the Kubernetes environment.

Set up the environment

In Altinity Cloud Manager, go to the Environments tab. Click the button at the top of the screen.

In the Environment Setup dialog, click the down arrow to see the list of environment types:

Environment Setup - Environment Type
Figure 1 - Choosing an Environment Type

Select Altinity.Cloud Anywhere Bring Your Own Cloud (BYOC). With the environment type set, enter a name for the environment:

Naming your environment
Figure 4 - Naming your environment

Click OK to continue.

Choose the correct Kubernetes setup

To set up your connection, Altinity.Cloud needs to know which Kubernetes setup to use. “GKE Provisioned by Altinity” is the right choice here:

The Connection Setup tab
Figure 5 - The Connection Setup tab

Click PROCEED to continue.

Define your Kubernetes cluster’s resources

The Resources Configuration tab looks like this:

The Resources Configuration Tab
Figure 6 - The Resources Configuration tab for connecting altinity-maddie to Altinity.Cloud.

Field details

  • Cloud Provider - GCP should be selected automatically; select it if it isn’t.
  • Region - Click the arrow next to the field to see a list of available regions.
  • Number of AZs - The number of availability zones for your cluster. NOTE: It is highly recommended that you use at least two availability zones.
  • Project ID - Give your project a name.
  • CIDR Block - The address range allocated to your cluster. NOTE: Be sure you define enough addresses. We recommend x.x.x.x/21 at a minimum. If you run out of addresses, this setting is difficult to change.
  • Storage Classes - Enter the storage classes your cluster will use. You can delete the entries that appear; you can also click the ADD STORAGE CLASS button to add other storage classes.
  • Node Pools - Define the node pools that your cluster will use. At least one node pool must be defined for ClickHouse and Zookeeper and System. In this example, one node pool will host Zookeeper and the System utilities Altinity.Cloud uses; four other node pools will host ClickHouse itself. You can click the ADD NODE POOL button to add more node pools as needed. You can also add more node pools later.

Click PROCEED to continue.

Confirm your settings

The Confirmation screen in Figure 7 displays a JSON representation of the settings you just made. Review these settings; you can edit the JSON directly if needed. When the JSON is correct, select FINISH.

The Confirmation Tab
Figure 7 - The Confirmation tab showing the JSON version of the settings.

It will take a few minutes for all the resources to be provisioned.

Connection completed

Once the connection is fully set up, the ACM Environments dashboard will display your new environment:

Provisioned Environment Tab
Figure 8 - The details of your new Environment