Altinity.Cloud Anywhere lets you take advantage of Altinity’s
zero-maintenance ClickHouse SaaS platform in your own
environment. You provide the infrastructure,
Altinity deploys the ClickHouse clusters you need on top of it.
1 - Altinity.Cloud Anywhere 101
Getting started
Introduction and benefits
Altinity.Cloud Anywhere provides the convenient cloud management of
Altinity.Cloud but allows users to keep data within their own cloud VPCs
and private data centers, all while running managed ClickHouse in their
own Kubernetes clusters. We sometimes call this
Open Cloud for ClickHouse.
The Kubernetes clusters that contain ClickHouse clusters
are Altinity.Cloud Anywhere environments.
Altinity.Cloud Anywhere offers several important benefits for users.
Compliance - Retain full control of data (including backups) as
well as the operating environment and impose your policies for
security, privacy, and data sovereignty.
Cost - Optimize infrastructure costs by running in your accounts.
Location - Place ClickHouse clusters close to data sources and
applications.
No vendor lock-in - Disconnect the control plane at any time and continue operating
ClickHouse using open-source components.
The rest of this document explains
Altinity.Cloud Anywhere and its benefits.
The Altinity Cloud Manager UI manages Altinity.Cloud Anywhere environments
just like fully hosted Altinity.Cloud environments. Users can control
multiple environments from the same Altinity.Cloud account and can mix and
match environment types. No matter what environment you’re in,
ClickHouse management works the same way.
Open-source analytic stack
Altinity.Cloud Anywhere uses open-source software for the analytic stack
and selected management services–the Altinity Kubernetes Operator for ClickHouse,
Loki, Prometheus, and Grafana. The following diagram shows how the principal
components map to resources in AWS. (GCP is essentially identical.)
Open-source components are in dark blue rectangles.
Figure 1 - The open-source analytic stack
Users can terminate the service, disconnect the Altinity.Cloud Anywhere
environment from Altinity.Cloud, and run ClickHouse services themselves.
There is no migration, since all data, software, and support services are
already in the user’s Kubernetes cluster. The only component not in the
Kubernetes clustser is the management plane.
Service architecture
The Altinity.Cloud service architecture consists of a shared management
plane that serves as a single point of management for all tenants and a
data plane that consists of isolated environments for each tenant.
The key component is the Altinity Connector. Altinity Connector is a secure
management channel between Altinity.Cloud and the managed environment. It is
deployed inside Kubernetes, dials back to the management plane and
enables Altinity Cloud to manage ClickHouse and infrastructure components. An
important property of Altinity Connector is that it only establishes an outbound
connection, which pleases corporate Infosec teams.
We’ll discuss two variations of this architecture:
Bring Your Own Kubernetes (BYOK) - You manage your own Kubernetes environment,
Altinity runs ClickHouse clusters inside it
Bring Your Own Cloud (BYOC) - You provide your cloud account, Altinity
builds a Kubernetes cluster inside it, then Altinity runs ClickHouse clusters
inside the Kubernetes cluster.
Bring Your Own Kubernetes (BYOK)
The following diagram shows the management plane and data plane
relationships.
Figure 2 - The Altinity.Cloud Anywhere service architecture
Each environment is a dedicated Kubernetes cluster. In the case of
Altinity.Cloud environments, Kubernetes clusters run on Altinity’s
cloud accounts and are completely hidden from users. For Altinity.Cloud
Anywhere, Kubernetes clusters run in the user’s cloud account or
data center. For example, the user may run an EKS cluster within a VPC
belonging to the user’s AWS cloud account. Altinity.Cloud Anywhere environments can also use on-prem Kubernetes
clusters. They can even use development versions of Kubernetes running
on a user’s PC or laptop.
Connectivity model
Altinity.Cloud Anywhere environments use the Altinity Connector to
establish a management connection from the user’s Kubernetes cluster to
Altinity.Cloud. The Altinity Connector establishes an outbound HTTPS
connection to a management endpoint secured by certificates. This allows
management commands and monitoring data to move securely between locations.
Users connect an Altinity.Cloud Anywhere environment to Altinity.Cloud
in three simple steps:
Download the Altinity Connector executable program
(altinitycloud-connect).
Run and register Altinity Connector with Altinity.Cloud Manager.
If Altinity Connector is installed on a separate VM, it may run
provisioning of the Kubernetes cluster (EKS or GKE). This process
deploys a new instance of Altinity Connector into the provisioned
Kubernetes cluster as well.
When Altinity Connector is installed directly in Kubernetes, it
runs the provisioning of Kubernetes resources.
Complete registration in the Altinity.Cloud Manager.
Altinity.Cloud Anywhere environments run all services in two namespaces:
The altinity-cloud-system namespace contains system services
including the Altinity Connector.
The altinity-cloud-managed-clickhouse namespace contains ClickHouse
and ZooKeeper. Users can run services in other namespaces as long as they
do not make changes to the Altinity-managed namespaces.
Users should not create any resources in either of the Altinity namespaces.
Preparing your Kubernetes cluster
If you manage the Kubernetes cluster, you are responsible for setting
up the Kubernetes cluster properly and deploying the Altinity Connector. We
provide guidelines to do that properly, but still it requires the
expertise and resources of the user’s IT team, not to mention knowledge of
Kubernetes administration.
We cover the requirements
for a user-managed Kubernetes cluster in a later section,
but at a high level, your Kubernetes cluster must:
Configure storage classes that can allocate block storage on-demand
Enable auto-scaling
Enable Kubernetes pods to connect to S3-compatible object storage.
Bring Your Own Cloud (BYOC)
To make it easier for users, Altinity.Cloud Anywhere can also provision and
manage the Kubernetes infrastructure, building the full stack in the
user’s cloud account. This is the Bring Your Own Cloud (BYOC) model. With
BYOC, the user grants Altinity.Cloud permissions to manage cloud resources
directly. This access can be provided in two different ways: via a separate
VM running Altinity Connector, or directly, depending on the cloud provider’s
security model.
The basic model looks like this:
Figure 3 - The basic Altinity.Cloud Anywhere service architecture
Altinity.Cloud uses a VM-based approach on AWS. It provides the best isolation
between Altinity and user account, with the VM serving as a secure bridge between
two. It can be deployed using a CloudFormation template that sets an EC2 instance
up and sets up the Altinity Connector. Altinity.Cloud does not have direct access
to the user’s account in this case; all management operations are routed via this
EC2 instance.
Here’s the architecture diagram for AWS:
Figure 4 - The Altinity.Cloud Anywhere service architecture for AWS
The other approach requires users to grant an Altinity Cloud service account
direct access to the user’s cloud infrastructure. It works well in GCP, where
it is easy to create a dedicated new project and manage security on the project
level.
Here’s the architecture diagram for GCP:
Figure 5 - The Altinity.Cloud Anywhere service architecture for GCP
Administrative responsibilities between Altinity and you
With Altinity.Cloud Anywhere, you assume some of the responsibilities
that Altinity.Cloud would handle for you. Exactly what you need
to do depends on whether you’re provisioning Kubernetes yourself
(Bring Your Own Kubernetes) or providing a cloud
environment and giving Altinity permission to provision Kubernetes
inside it (Bring Your Own Cloud). This
figure shows the division of responsibilities for all three scenarios:
Note that even though the base Kubernetes environment used by Altinity.Cloud
Anywhere is technically managed by the user, in many cases Altinity.Cloud
Anywhere can provision Kubernetes resources in your cluster for you.
2 - Bring your own Kubernetes (BYOK)
Using your Kubernetes infrastructure
Altinity.Cloud Anywhere uses your Kubernetes infrastructure to host
your ClickHouse clusters. Your Kubernetes cluster needs to be set
up a certain way; in this section we’ll go over those requirements.
2.1 - Kubernetes requirements
Configuring your Kubernetes environment
Altinity.Cloud Anywhere operates inside your Kubernetes environment.
The general requirements for your Kubernetes environment are:
Kubernetes version 1.23 or higher in EKS (AWS) or GKE (GCP)
Every Node should have the following labels:
node.kubernetes.io/instance-type
kubernetes.io/arch
topology.kubernetes.io/zone
A StorageClass with dynamic provisioning is required
LoadBalancer services must be supported
To get the most from Altinity.Cloud Anywhere features:
Each StorageClass should preferably allow volume expansion
Multiple zones are preferable for high availability
Autoscaling is preferable for easier vertical scaling
For platform-specific requirements, see the following sections:
We recommend setting up karpenter or
cluster-autoscaler
to launch instances in at least 3 Availability Zones.
If you plan on sharing Kubernetes cluster with other workloads, it’s
recommended you label Kubernetes Nodes intended for Altinity.Cloud Anywhere
with altinity.cloud/use=anywhere and taint them with dedicated=anywhere:NoSchedule.
Instance types
For Zookeeper infrastructure nodes
t3.large or t4g.large*
t4g instances are AWS Graviton2-based (ARM).
For ClickHouse nodes
ClickHouse works best in AWS when using nodes from these instance families:
m5
m6i
m6g*
m6g instances are AWS Graviton2-based (ARM).
Instance sizes from large to 8xlarge are typical.
Storage classes
gp2*
gp3-encrypted*
We recommend using gp3 storage classes that provide more flexibility and performance over gp2. The gp3 storage classes require the Amazon EBS CSI driver;
that driver is not automatically installed. See
the AWS CSI driver documentation
for details on how to install the driver.
Storage class can be installed with the following manifest:
The default throughput for gp3 is 125MB/s for any volume size. It can be increased in AWS console or using storage class parameters. Here is an example:
Alternatively, you recommend installing the Altinity
EBS parameters controller.
That allows you to manage EBS volume throughput dynamically through annotations. This is also integrated to Altinity.Cloud UI (ACM).
GCP requirements
Machine types
For Zookeeper and infrastructure nodes
e2-standard-2
For ClickHouse nodes
It’s recommended to taint node pools with dedicated=clickhouse:NoSchedule (in addition to altinity.cloud/use=anywhere:NoSchedule).
n2d-standard-2
n2d-standard-4
n2d-standard-8
n2d-standard-16
n2d-standard-32
If GCP is out of n2d-standard-* instances in the
region of your choice, we recommend
substituting them with n2-standard-*.
Storage classes
standard-rwo
premium-rwo
GKE comes pre-configured with both.
2.2 - Connecting to Altinity.Cloud Anywhere
Tying everything together
This tutorial explains how to use Altinity.Cloud Anywhere to deploy
ClickHouse clusters using your choice of a third-party Kubernetes cloud
provider, or using your own hardware or private company cloud. The
Altinity.Cloud Manager (ACM) is used to manage your ClickHouse clusters.
If you’re just getting started, you can get a trial account in three steps:
Use your business email address to sign up for a free trial on the Altinity.Cloud Anywhere trial page.
NOTE: This must be a business email address. Addresses like *@gmail.com or *@yahoo.com
are not accepted.
You’ll get an email from Altinity. Follow the instructions to validate your email address.
The final email you’ll get contains a login link to create a password to log in to the Altinity Cloud Manager.
Connecting Kubernetes
The first time you log in, you will be directed to the environment setup tab shown in Figure 1. If you have an existing account or restart the installation, just select the Environments tab on the left side of your screen to reach the setup page.
Be sure to select “Provisioned by User” as shown in Figure 1.
Figure 1 - The Environments > Connection Setup tab in the Altinity.Cloud Manager.
Connection setup
Highlighted in red in Figure 1 are the steps to complete before you select the PROCEED button.
The altinitycloud-connect kubernetes command generates YAML that includes
the .pem file generated in the previous step. This step may take several
minutes to complete.
The response will be something like this:
namespace/altinity-cloud-system created
namespace/altinity-cloud-managed-clickhouse created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
clusterrole.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
serviceaccount/cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:storage-class-view unchanged
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view unchanged
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
secret/cloud-connect created
deployment.apps/cloud-connect created
Notice the altinity-cloud-system and altinity-cloud-managed-clickhouse
namespaces above. All the resources Altinity.Cloud Anywhere creates are in
those namespaces; you should not create anything in those namespaces
yourself.
Configuring resources
Once these commands have completed select the PROCEED button. After the connection is made, you will advance to the Resources Configuration screen as shown in Figure 2.
At the Resources Configuration screen, set the resources used for ClickHouse clusters as follows.
Make sure the correct cloud provider is selected. Altinity.Cloud Anywhere should detect this correctly based on the connection you established earlier.
Add Storage Classes names as needed. These are the block storage classes for your nodes. Use the ADD STORAGE CLASS button to add additional storage classes as needed to allocate block storage for nodes in your environment.
In the Node Pools section, Inspect the node pool list to ensure the availability zones and pools you wish to use are listed.
In the Used For section, ClickHouse, Zookeeper, and System must be selected at least once in at least one of the node pools.
Selecting multiple node pools for ClickHouse nodes is highly recommended.
Listed are the Instance Types that are currently in use. Click ADD NODE POOL to add anything that’s missing.
The following Resources Configuration example shows the settings for a Google Cloud Platform environment.
Figure 2 - The Resources Configuration setup page for connecting cloudv2-gcp to Altinity.Cloud.
Be aware that you can add more node pools later if needed.
Confirming your settings
The Confirmation screen in Figure 3 displays a JSON representation of
the settings you just made. Review these settings; you can
edit the JSON directly if needed. When the JSON is correct,
select FINISH.
Figure 3 - Confirmation page showing the JSON version of the settings.
It will take a few minutes for all the resources to be provisioned.
Connection completed
Once the connection is fully set up, the ACM Environments dashboard will display your new environment as shown in Figure 4.
If you have any problems, see the Troubleshooting section
below.
Administering Altinity.Cloud Anywhere
Once your environment is configured, you use the
Altinity Cloud Manager (ACM) to perform common user and administrative
tasks. The steps and tools to
manage your ClickHouse clusters are the same for
Altinity.Cloud Anywhere
and Altinity.Cloud.
Here are some common tasks from the ACM documentation:
The connection string comes from your cluster (Example: test-gcp-anyw) Connection Details link.
The Copy/Paste for client connections string highlighted in red in Figure 5 is used in your terminal (you supply the password; Example: adminpassword)
Figure 5 - Using the Cluster > EXPLORE > Query tab to create and query ClickHouse databases and tables.
Find your pod name:
kubectl -n altinity-cloud-managed-clickhouse get all
# ResponseNAME READY STATUS RESTARTS AGE
pod/chi-test-anywhere-6-johndoe-anywhere-6-0-0-0 2/2 Running 8(3h25m ago) 2d17h
On your command line terminal, login to that pod using the name you got from step 1:
Login to your ClickHouse database using the clickhouse-client command to get the :) happy face prompt:
clickhouse@chi-test-anywhere-6-johndoe-anywhere-6-0-0-0:/$
clickhouse@chi-test-anywhere-6-johndoe-anywhere-6-0-0-0:/$ clickhouse-client
# Response<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)<jemalloc>: (This is the expected behaviour if you are running under QEMU)ClickHouse client version 22.8.13.21.altinitystable (altinity build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 22.8.13 revision 54460.
test-anywhere-6 :)
Run a show tables sql command:
test-anywhere-6 :) show tables
# ResponseSHOW TABLES
Query id: da01133d-0130-4b98-9090-4ebc6fa4b568
┌─name─────────┐
│ events │
│ events_local │
└──────────────┘
2 rows in set. Elapsed: 0.013 sec.
Run SQL query to show data in the events table:
test-anywhere-6 :) SELECT * FROM events;# ResponseSELECT *
FROM events
Query id: 00fef876-e9b0-44b1-b768-9e662eda0483
┌─event_date─┬─event_type─┬─article_id─┬─title───┐
│ 2023-03-24 │ 1 │ 13 │ Example │
└────────────┴────────────┴────────────┴─────────┘
1 row in set. Elapsed: 0.023 sec.
test-anywhere-6 :)
Troubleshooting
Q-1. Altinity.Cloud Anywhere endpoint not reachable
Problem
By default, the altinitycloud-connect command connects to host anywhere.altinity.cloud on port 443. If this host is not reachable, the following error message appears.
altinitycloud-connect login --token=<token>
Error: Post "https://anywhere.altinity.cloud/sign":
dial tcp: lookup anywhere.altinity.cloud on 127.0.0.53:53: no such host
Solution
Make sure the name is available in DNS and that the resolved IP address is reachable on port 443 (UDP and TCP), then try again.
The altinitycloud-connect command has a --url option if you need to specify a different URL.
Q-2. Insufficient Kubernetes privileges
Problem
Your Kubernetes account has insufficient permissions.
Solution
Look at the output from the altinitycloud-connect kubernetes | kubectl apply -f - command
to see what actions failed, then adjust the permissions for your Kubernetes
account accordingly. At a minimum, set the following permissions:
cluster-admin for initial provisioning only (it can be revoked afterward)
Give full access to the altinity-cloud-system and altinity-cloud-managed-clickhouse namespaces
A few optional read-only cluster-level permissions (for observability only)
Q-3. Help! I messed up the resource configuration
Problem
The resource configuration settings are not correct.
Solution
From the Environment tab, in the Environment Name column, select the link to your environment.
Select the menu function ACTIONS 》Reconfigure Anywhere.
Rerun the Environment 》Connection Setup and enter the correct values.
Q-4 One of my pods won’t spin up
When you reboot your machine, the Anywhere cluster in your ACM has not started.
Problem
One of the pods won’t start. In the listing below,
pod edge-proxy-66d44f7465-lxjjn in the altinity-cloud-system
namespace has not started:
Delete the pod using the kubectl delete pod command and it will regenerate.
kubectl -n altinity-cloud-system delete pod edge-proxy-66d44f7465-lxjjn
2.3 - Setting up logging
Configuring storage for logging
In order for Altinity.Cloud Anywhere to gather/store/query logs, you need to configure access to an S3 or GCS bucket.
Cloud-specific instructions provided below.
# create bucketgsutil mb gs://REPLACE_WITH_BUCKET_NAME
# create GCP SA with access to the bucketgcloud iam service-accounts create REPLACE_WITH_GCP_SA_NAME \
--project=REPLACE_WITH_PROJECT_ID \
--display-name "REPLACE_WITH_DISPLAY_NAME"gsutil iam ch \
serviceAccount:REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
gs://REPLACE_WITH_BUCKET_NAME
# generate GCP SA keygcloud iam service-accounts keys create credentials.json \
--iam-account=REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com \
--project=REPLACE_WITH_PROJECT_ID
# create altinity-cloud-system/log-storage-gcp secret containing credentials.jsonkubectl create secret -n altinity-cloud-system generic log-storage-gcp \
--from-file=credentials.json
rm -i credentials.json
Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.
2.4 - Setting up backups
Configuring storage for backups
In order for Altinity.Cloud Anywhere to work with backups, you need to
configure access to an S3 or GCS bucket. Cloud-specific instructions are
provided below.
EKS (AWS)
Use a custom Instance Profile or explicit credentials (shown below).
# create bucketgsutil mb gs://REPLACE_WITH_BUCKET_NAME
# create GCP SA with access to the bucketgcloud iam service-accounts create REPLACE_WITH_GCP_SA_NAME \
--project=REPLACE_WITH_PROJECT_ID \
--display-name "REPLACE_WITH_DISPLAY_NAME"gsutil iam ch \
serviceAccount:REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com:roles/storage.admin \
gs://REPLACE_WITH_BUCKET_NAME
# generate GCP SA keygcloud iam service-accounts keys create credentials.json \
--iam-account=REPLACE_WITH_GCP_SA_NAME@REPLACE_WITH_PROJECT_ID.iam.gserviceaccount.com \
--project=REPLACE_WITH_PROJECT_ID
# create altinity-cloud-system/clickhouse-backup secret containing GOOGLE_APPLICATION_CREDENTIALS kubectl create secret -n altinity-cloud-system generic clickhouse-backup \
--from-file=GOOGLE_APPLICATION_CREDENTIALS=credentials.json
rm -i credentials.json
Use your private customer Slack channel to send the bucket name to Altinity in order to finish configuration.
2.5 - Disconnecting from Altinity.Cloud Anywhere
Using ClickHouse without Altinity.Cloud Anywhere
Even if you disconnect from Altinity.Cloud Anywhere
altogether, your ClickHouse cluster
can continue running in your Kubernetes environment.
In this section we’ll show you how to do that.
Disconnecting your environment from Altinity.Cloud Anywhere
You can disconnect Altinity Cloud Manager from your
Kubernetes environment and the ClickHouse
clusters running inside it. This does not delete your running
ClickHouse clusters, it merely disconnects them from
the Altinity Cloud Manager. Your ClickHouse clusters continue
running as usual.
The first command deletes every ClickHouse installation (chi) that
Altinity.Cloud Anywhere created. Those are in the
altinity-cloud-managed-clickhouse namespace. With the ClickHouse
clusters deleted, the second command deletes the two Altinity
namespaces and any remaining resources they contain.
WARNING: If you delete the namespaces before deleting the ClickHouse
installations (chi), the operation will hang due to missing
finalizers on chi resources.
Should this occur, use the kubectl edit command on each ClickHouse
installation and remove the finalizer manually from the resource
specification. Here is an example:
You can now delete the finalizer from the resource:
# Please edit the object below. Lines beginning with a '#' will be ignored,# and an empty file will abort the edit. If an error occurs while saving this file will be# reopened with the relevant failures.#apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
creationTimestamp: "2023-08-29T17:03:58Z" finalizers:
- finalizer.clickhouseinstallation.altinity.com
generation: 3 name: maddie-ch
. . .
2.6 - Appendix: Using Altinity.Cloud Anywhere with minikube
For testing and development use only
This guide covers setting up minikube so that you can use Altinity.Cloud Anywhere
to provision ClickHouse clusters inside minikube.
Any computer or cloud instance that can run minikube and support
the resource requirements of the Kubernetes cluster we
describe here should work.
Note that while minikube is okay to use for development purposes,
it should not be used for production. Seriously. We can’t stress that enough.
It’s great for development, but don’t use it for production.
Server requirements
In the deployment you’ll do
here, you’ll build a minikube cluster with seven nodes.
Using the Docker runtime on a MacBook Pro M2 Max,
the system provisioned 6 vCPUs and 7.7 GB of RAM per node, along
with roughly 60 GB of disk space per node.
It’s unlikely all of your nodes will run at capacity, but there’s
no guarantee your machine will have enough resources to do whatever
you want to do in your minikube cluster. (Did we mention it’s
not for production use?)
And, of course, the default provisioning may be different
on other operating systems, hardware architectures, or
virtualization engines.
Before you get started, you’ll need to
sign up for an Altinity.Cloud Anywhere trial account.
At the end of that process, you’ll have an email with a link to
the Altinity Cloud Manager (ACM). You’ll use that link
to set up the connection between minikube and Altinity.
Finally, of course, you’ll need to install minikube itself.
See the minikube start page
for complete install instructions. Just install the software at this
point; we’ll talk about how to start minikube in the next section.
Starting minikube
If you’ve used minikube on your machine before, we recommend that
you delete its existing configuration:
😄 minikube v1.30.1 on Darwin 13.5.2 (arm64)✨ Automatically selected the docker driver. Other choices: qemu2, parallels, ssh
📌 Using Docker Desktop driver with root privileges
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=3500MB) ...
🐳 Preparing Kubernetes v1.22.8 on Docker 23.0.2 ...
❌ Unable to load cached images: loading cached images: stat /Users/dougtidwell/.minikube/cache/images/arm64/registry.k8s.io/kube-apiserver_v1.22.8: no such file or directory
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🔎 Verifying Kubernetes components...
👍 Starting worker node minikube-m02 in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=3500MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.22.8 on Docker 23.0.2 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...
. . .
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
NOTE: Defining the environment variable MINIKUBE_IN_STYLE=0 disables
the emojis that appear in front of every minikube message. You’re welcome.
At this point minikube is up and running. The kubectl get nodes command
shows our seven nodes:
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 4m22s v1.22.8
minikube-m02 Ready <none> 4m2s v1.22.8
minikube-m03 Ready <none> 3m48s v1.22.8
minikube-m04 Ready <none> 3m33s v1.22.8
minikube-m05 Ready <none> 3m17s v1.22.8
minikube-m06 Ready <none> 3m2s v1.22.8
minikube-m07 Ready <none> 2m46s v1.22.8
When using Altinity.Cloud Anywhere with a traditional cloud vendor, there are node types,
availability zones, and storage classes. We need to label our
minikube nodes to simulate those things. First, run these commands
to define the node types and availability zones:
Now all of our minikube nodes are defined to be of type minikube-node;
we’ll see that node type again later. We’ve also defined availability
zones named minikube-zone-a, minikube-zone-b, and minikube-zone-c.
On to our
storage classes. We want to use the local-path storage class instead
of minikube’s default standard storage class. This command defines
the new storage class:
Running kubectl get storageclasses shows the new default class:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 36s
standard k8s.io/minikube-hostpath Delete Immediate false 15m
Connecting Altinity.Cloud Anywhere to minikube
Now that we have the minikube cluster running and configured, it’s time
to connect it to Altinity.Cloud Anywhere. That’s the final step for enabling
Altinity to provision ClickHouse clusters in minikube.
Step 1. Setting up the tunnel
First we need to set up the TLS tunnel between minikube and Altinity.
Click the emailed link you got when you signed up for an Altinity.Cloud Anywhere
account. You’ll see this screen:
Figure 1 - The Connection Setup screen
Make sure the “Provisioned by User” box is selected at the top of the page, and
make sure you’ve installed altinitycloud-connect from the link beneath it.
Copy and paste the text in the center box at the command line and run it. This
doesn’t generate any output at the command line, but it does create a
cloud-connect.pem file in the current directory.
Now that you have the cloud-connect.pem file, run the following command
to set up the TLS tunnel:
Note: The command you run is different from the one in the text box at the
bottom of Figure 1.
Make sure that the --url parameter matches the URL in that text box,
as it is dependent on the Altinity.Cloud Anywhere
endpoint you’re using.
The altinitycloud-connect kubernetes command generates YAML that has configuration
information along with the keys from the .pem file. That YAML data is
passed to kubectl.
You’ll see results similar to this:
namespace/altinity-cloud-system created
namespace/altinity-cloud-managed-clickhouse created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:storage-class-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view created
clusterrole.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
serviceaccount/cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:node-metrics-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:storage-class-view created
clusterrolebinding.rbac.authorization.k8s.io/altinity-cloud:persistent-volume-view created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
rolebinding.rbac.authorization.k8s.io/altinity-cloud:cloud-connect created
secret/cloud-connect created
deployment.apps/cloud-connect created
Another note: Altinity creates all ClickHouse-related
assets in the altinity-cloud-system and altinity-cloud-managed-clickhouse
namespaces. You should not create anything in those namespaces yourself.
Click Proceed to go to the next step.
Step 2. Configuring your minikube resources
Next we’ll define aspects of the minikube environment to Altinity. Use
the values highlighted in red in
Figure 2.
Figure 2 - The Resources Configuration screen
The specific values to use are:
Cloud Provider: Not Specified (minikube is a special case)
Region: minikube-zone (we defined that with a label earlier)
Number of AZs: 3
Storage Classes: local-path (defined as the default storageclass earlier)
Node Pools: A single node pool named minikube-node with a capacity of 2.
The boxes for ClickHouse and Zookeeper must be checked.
Click Proceed to go to the Confirmation screen.
Step 3. Confirming your choices
A JSON description of all of your choices appears in the text box
at the top of Figure 3:
Figure 3 - The Confirmation screen
You can edit the JSON as needed;
currently you need to change the names of the availability zones. Using
the values specified in the previous step, the generated availability zones will be
minikube-zonea, minikube-zoneb, and minikube-zonec. They should be
minikube-zone-a, minikube-zone-b, and minikube-zone-c.
Once everything is correct, click Finish. This begins the process
of creating a ClickHouse cluster inside minikube. You’ll see a status bar
similar to Figure 4:
Figure 4 - The status bar
It’s quite likely the status bar will reach the end before everything
is configured. Just keep clicking Finish until things are, well, finished:
Figure 5 - Click Finish again to keep waiting
When things are finished, you’ll see this screen:
Figure 6 - Altinity.Cloud Anywhere is up and running
With everything up and running, kubectl get pods -n altinity-cloud-managed-clickhouse
shows the pods Altinity.Cloud Anywhere created:
There are two pods for ClickHouse itself, a pod for the Altinity ClickHouse Operator,
and three pods for Zookeeper. These pods are managed for you by Altinity.
Working with Altinity.Cloud Anywhere
Now that your environment is configured, you use the
Altinity Cloud Manager (ACM) to perform common user and administrative
tasks. The steps and tools to
manage your ClickHouse clusters are the same for
Altinity.Cloud Anywhere
and Altinity.Cloud.
Here are some common tasks from the ACM documentation:
Altinity.Cloud Anywhere can use your cloud account
to provision cloud resources to
host your ClickHouse clusters. Your cloud account needs to have
certain permissions and other properties; in this section
we’ll go over those requirements.
Introduction
Altinity.Cloud Anywhere can operate inside a user’s cloud account.
Altinity technical support can remotely provision Kubernetes clusters
for an Altinity.Cloud Anywhere environment inside your AWS or GCP
account. Once the Kubernetes environment is set up, Altinity can
provision ClickHouse clustes inside Kubernetes.
This section describes how to configure your
AWS or GCP accounts to authorize
Altinity.Cloud to provision Kubernetes and ClickHouse in your account.
Figure 1 is a high-level view of the
Altinity.Cloud Kubernetes infrastructure.
Figure 1 - The basic Altinity.Cloud Anywhere service architecture
Platform-specific requirements are discussed in the following
sections.
3.1 - AWS remote provisioning
Configuring your AWS account
Provisioning process overview
This section summarizes the bootstrap process that lets you use
Altinity.Cloud Anywhere to create an EKS environment and deploy
ClickHouse clusters to it. You’ll go through these steps:
Get an Altinity.Cloud Anywhere account.
Get an Altinity.Cloud Anywhere environment record.
Get a connection token from the Altinity Cloud Manager.
Provision an AWS EKS cluster using EC2 instance running with a user account.
The EC2 instance is required in order to deploy altinitycloud-connect, which will establish an outbound connection to Altinity.Cloud and start the EKS provisioning process.
The EC2 instance can be set up in two ways:
Automatically by using the AWS Cloud Formation Template to automate the process.
Manually set up by a user following Altinity documentation.
In the Altinity Cloud Manager, complete the configuration of EKS resources.
Automatically provisioning EKS using an EC2 instance created from the AWS Cloud Formation Template
An Amazon AWS EC2 instance is required to deploy
altinitycloud-connect, which establishes an outbound
connection to Altinity.Cloud Anywhere and starts the EKS provisioning process.
Get a connection token from the Altinity Cloud Manager. The
connection token is everything after token= in the “Connect to
Altinity.Cloud” text box. In Figure 1, the token is all of the
text in red.
Figure 1 - The Altinity.Cloud Anywhere connection token
Login to your AWS account and navigate to Cloud Formation and click the Create Stack button.
Go to the altinitycloud-connect releases page and
download the latest Cloud Formation YAML file,
altinitycloud-connect-x.xx.x.aws-cloudformation.yaml.
(That’s currently version 0.20.0.)
The link to the YAML file is at the bottom of the page.
In the Create Stack panel, choose Upload a template file and select the Altinity Cloud Formation Template YAML file as shown in Figure 2. Click Next to continue.
Fill in any missing fields on the Specify Stack Details page (Figure 3):
Set Stack Name to
altinitycloud-connect-$ENV_NAME
(if your environment name is maddie-byok, the stack name should be altinity-connect-maddie-byok.)
Select the Subnets where the altinitycloud-connect EC2 instance(s) should be launched. At least one subnet is required. NOTE: The subnet(s) must have the “Auto-assign public IPv4 address” property set to yes.
Set the Token presented by https://acm.altinity.cloud/ to the token value from Step 3.
Click Next to continue.
Figure 3 - AWS CloudFormation Stack details panel
Take the defaults on the Configure stack options panel (Figure 4). Click Next to continue.
On the Review panel, check the box to allow CloudFormation to create
resources as shown in Figure 5. The process will not work unless you click this button.
Click Submit to start provisioning the new stack.
Figure 5 - Granting permissions on the AWS CloudFormation Review panel
You’ll see the main CloudFormation panel (Figure 6), which will include your
new stack. You can follow its progress in the Events list.
Figure 6 - AWS CloudFormation stack list
EC2 background processing explained
The EC2 instance is processed in the background as follows:
EC2 instance gets started from the cloud formation template
EC2 gets connected to Altinity.Cloud using altinitycloud-connect
EKS cluster gets provisioned
EKS cluster gets connected to Altinity.Cloud using altinitycloud-connect
In Altinity Cloud Manager
Once your new stack is created, click the ‘Proceed’ button in the connection wizard.
Altinity Cloud Manager will connect to the stack named altinitycloud-connect-$ENV_NAME
where $ENV_NAME is the name of your environment.
NOTE: It is ok to select Proceed more than once, since provisioning takes some time.
Once the EKS cluster is provisioned, the wizard switches to the Resources Configuration page.
To set this instance to have access to the EC2 metadata and Internet, set the Security group to:
deny all inbound traffic
allow all outbound traffic
Installing Altinity.Cloud Connect
Download altinitycloud-connect.
NOTE: The following example is for an Intel Linux installation.
Change the filename to match your machine’s architecture.
NOTE:altinitycloud-connect creates a cloud-connect.pem file in the current working directory.
Connect to Altinity.Cloud:
altinitycloud-connect --capability aws
Start EKS provisioning
Provide additional configuration data to Altinity Support
The following data is required in order to create the VPC and EKS cluster properly:
The CIDR for the Kubernetes VPC (at least /21 recommended, such as 10.1.0.0/21) that does not overlap with existing VPCs
The Number of Availability Zones (3 are recommended)
Please send this information to your Altinity support representative to start the EKS provisioning process.
When completed, the Altinity Cloud Manager (ACM) will be updated. At that point, you can create your ClickHouse clusters.
The remainder of the provisioning process is handled by Altinity.Cloud.
Users may switch back to ACM and wait for connection to be established in order to finish configuration.
In Altinity.Cloud
Select the Proceed button in the connection wizard.
You may repeat this step more than once to see if the connection has completed, since provisioning takes some time.
Once the EKS cluster is provisioned, the connection wizard will switch to the Resources Configuration page.
Finish configuration of node pools as described in the Configuring resources section.
Break Glass procedure
The “Break Glass” procedure allows Altinity access to EC2 instance with SSH, using AWS SSM in order to troubleshoot altinitycloud-connect that is running on this instance.
Create an AnywhereAdmin IAM role with trust policy set:
Altinity.Cloud Anywhere operates inside a user’s cloud account.
Altinity technical support can remotely provision GCP GKE clusters
with an Altinity.Cloud Anywhere environment in your GCP account.
Instructions on this page describe how to configure your GCP account
to provide permission to Altinity to provision ClickHouse
to your GCP Kubernetes environment.
Overview - Google GKE Installation
This section covers how to authorize Altinity.Cloud Anywhere to
create Kubernetes and ClickHouse resources inside your cloud account.
Creating a project
Creating a separate project makes it easy to isolate resources and do cost
management, not to mention security. You can create a project from the command
line or in the GCP web UI.
You can use the gcloud projects create command to create a new project:
You also need to assign a billing account to the project.
Currently the gcloud command looks like this:
# Assign a billing account to 'maddie'gcloud beta billing projects link maddie \
--billing-account 0X0X0X-0X0X0X-0X0X0X
You can also create a project from the GCP web UI:
Figure 1 - The GCP New Project dialog
Granting permissions
For Altinity to be able to create Kubernetes and ClickHouse clusters
in your cloud account, you need to grant the following permissions
to anywhere-admin@altinity.com inside the project you just created:
roles/compute.admin
roles/container.admin
roles/dns.admin
roles/storage.admin
roles/storage.hmacKeyAdmin
roles/iam.serviceAccountAdmin
roles/iam.serviceAccountKeyAdmin
roles/iam.serviceAccountTokenCreator
roles/iam.serviceAccountUser
roles/iam.workloadIdentityPoolAdmin
roles/serviceusage.serviceUsageAdmin
roles/resourcemanager.projectIamAdmin
roles/iap.tunnelResourceAccessor
You can use the gcloud command for each role:
# Add a role for a member of a group associated with project 'maddie'gcloud projects add-iam-policy-binding maddie \
--member='group:anywhere-admin@altinity.com'\
--role='roles/compute.admin'
Alternately, you can use the GCP web UI:
Figure 2 - Permissions in the GCP web UI
Creating the Kubernetes environment
With the project created and the appropriate permissions
granted to the Altinity.Cloud Anywhere admin account, Altinity
can create Kubernetes clusters and ClickHouse clusters inside them.
The following sections demonstrate how to create the Kubernetes environment.
With the Kubernetes environment created, Altinity.Cloud Anywhere
can create ClickHouse clusters inside it.
Define the environment type
In Altinity Cloud Manager, go to the Environments tab. Click the
“+ SETUP ENVIRONMENT” button at the top of the screen.
Select “Altinity.Cloud Anywhere Bring Your Own Cloud (BYOC)” as the
environment type:
Figure 3 - Selecting your environment type
Name your environment
With the environment type selected, give your new environment a name:
Figure 4 - Naming your environment
Click OK to continue.
Choose the correct Kubernetes setup
To set up your connection, Altinity.Cloud needs to know which Kubernetes setup to use. “GKE Provisioned by Altinity” is
the right choice here:
Figure 5 - The Connection Setup tab
Click PROCEED to continue.
Define your Kubernetes cluster’s resources
The Resources Configuration tab looks like this:
Figure 6 - The Resources Configuration tab for connecting altinity-maddie to Altinity.Cloud.
Fields
Cloud Provider - GCP should be selected automatically; select it if it isn’t.
Region - Click the arrow next to the field to see a list of available regions.
Number of AZs - The number of availability zones for your cluster. NOTE: It is highly recommended that
you use at least two availability zones.
Project ID - Give your project a name.
CIDR Block - The address range allocated to your cluster. NOTE: Be sure
you define enough addresses. We recommend x.x.x.x/21 at a minimum.
If you run out of addresses, this setting is difficult to change.
Storage Classes - Enter the storage classes your cluster will use. You can
delete the entries that appear; you can also click the ADD STORAGE CLASS button
to add other storage classes.
Node Pools - Define the node pools that your cluster will use. At least one node pool
must be defined for ClickHouse and Zookeeper and System. In this example,
one node pool will host Zookeeper and the System utilities Altinity.Cloud uses;
four other node pools will host ClickHouse itself. You can click the ADD NODE POOL
button to add more node pools as needed. You can also add more node pools later.
Click PROCEED to continue.
Confirm your settings
The Confirmation screen in Figure 7 displays a JSON representation of
the settings you just made. Review these settings; you can
edit the JSON directly if needed. When the JSON is correct,
select FINISH.
Figure 7 - The Confirmation tab showing the JSON version of the settings.
It will take a few minutes for all the resources to be provisioned.
Connection completed
Once the connection is fully set up, the ACM Environments dashboard will display your new environment as shown in Figure 8.
