Working with Environments
An Environment is a way of grouping ClickHouse® clusters together in your account. There are several things you’ll want to do with your environments; we’ve broken the details into several sections:
- The environments view
- Setting up an environment
- Configuring an environment
There’s a lot of material in this section, so we suggest you go directly to the subsection that’s right for you: - Other configuration tasks
- Deleting an environment
The environments view
In working with Environments, you’ll always start by selecting the Environments tab to load the list of Environments:
Figure 1 - The Environments tab
You’ll see the Environment list view:
Figure 2 - The list of Altinity.Cloud Environments
The columns in the table are:
- Select button - Lets you select an Environment for operations and includes the vertical dots icon, which includes options to Edit, Export, or Delete the environment. (More on those tasks later.)
- Environment number - Each Environment has a unique number. If you’re using the Altinity Cloud Manager API, you will need this number when making certain API calls.
- Environment Name - Um, the name of the environment.
- Status - The status of the cluster. is what we’re looking for here.
- Deployment Type - The kind of deployment, either Kubernetes (Altinity.Cloud) or Anywhere (Altinity.Cloud Anywhere).
- Cloud Provider - Either AWS, GCP, or Azure.
- Created - The time the Environment was created.
- Owner - The ID of the Environment’s administrator.
- Organization - The Organization that owns this Environment.
- The button - Clicking this button takes you to the Clusters (plural) view of the ClickHouse clusters in this environment.
Setting up an Environment
The first step in setting up an environment is clicking the button. In the Environment Setup dialog, click a radio button to select the type of environment you need:
Figure 3 - The list of environment types
The environment types are:
Bring Your Own Kubernetes (BYOK) - Altinity.Cloud runs in your Kubernetes environment. See the Bring Your Own Kubernetes documentation for complete details.
Bring Your Own Cloud (BYOC) - Altinity.Cloud provisions Kubernetes clusters in your cloud account, then provision ClickHouse clusters inside those Kubernetes clusters. When you select this option, you’ll see radio buttons that let you choose your cloud provider:
Figure 4 - BYOC environment - Choose your cloud provider
See the Bring Your Own Cloud documentation for complete details.
SaaS Provisioned by Altinity - ClickHouse as a service, hosted and managed by Altinity. When you select this option, you’ll see radio buttons and a dropdown list that let you choose a cloud provider and region for your new environment:
Figure 5 - SaaS environment - Choose a cloud provider and region
Choose an environment type and give your new environment a name. Depending on the environment type you choose, you may need to specify a cloud provider or region.
Click OK to create the environment.
Configuring an Environment
To configure an environment, click the vertical dots icon and select Edit from the menu:
Figure 6 - The Edit Environment menu
This takes you to the Environment Configuration dialog. The dialog has seven tabs:
We’ll cover each of these next.
The General tab
The most high-level settings are on the General tab:
Figure 7 - The General tab
Field details
Name
The name of the Environment.
Deployment Type
This field is read-only for most users. Under normal circumstances, the only values that will appear here are Kubernetes or Anywhere.
Availability Zones
Enter the availability zones for your cloud provider.
Use External DNS
The names of the endpoints of ClickHouse resources used by ACM are based on a naming convention that includes the cluster name and altinity.cloud
. Checking this box means the ACM will access those resources through endpoints resolved through an external DNS. Contact Altinity support for help in setting this up.
Domain
You can define a custom domain; contact Altinity support to set this up. The default domain uses a combination of your ACM environment name and altinity.cloud
.
Database Login
Database Password
The username and password for the ClickHouse cluster. The ACM uses these values to retrieve information about schemas, users, and other high-level information from the ClickHouse cluster. These values are set for you by default.
If you change the username to default
and leave the password blank, the ACM will attempt to retrieve the password from the ClickHouse operator’s configuration.
Automatic Publish
If selected, any changes made to the Environment’s configuration are automatically published when you make them. Turning this off lets you decide when a change is published, giving you more control over the configuration. For example, you might want to make a number of configuration changes and publish them at one time.
Monitoring
Monitoring is turned on and the button is disabled. This control is a legacy item that will disappear in future versions of the ACM.
The Kubernetes tab
The Kubernetes tab, as you would expect, lets you configure Kubernetes settings:
Figure 8 - The Kubernetes tab
Field details
Master API URL
Under normal circumstances this value should not be changed. It can be useful in a Bring Your Own Kubernetes environment when defining connections between the ACM and Kubernetes. Contact Altinity support for help setting this up.
Auth Options - TLS Handshake + Token
This authentication scheme is the most common:
Figure 9 - Using the TLS Handshake and Token authentication scheme
The Client Key and Client Cert are defined for you. Under normal circumstances, you won’t need to change these values at all.
Auth Options - No Auth (Proxy)
This approach is used for development purposes only. It should never be used in production.
Figure 10 - Using the No Auth authentication scheme
Auth Options - Access Token
This is a legacy technique that is no longer used. It will be removed in a future version of the ACM.
Figure 11 - Using the Access Token authentication scheme
Namespace
The namespace used for the ClickHouse clusters deployed by Altinity. The default value is altinity-cloud-managed-clickhouse
. Once the ClickHouse cluster is created, the namespace cannot be changed.
Manage Namespace
If this switch is turned on, namespaces will be managed by the ACM.
Certificate ARN
The ARN (Amazon Resource Name) for the certificate used by this environment. This is managed for you in Altinity.Cloud environments, and is typically not needed in an Altinity.Cloud Anywhere environment.
Load Balancer Type
The default value (and likely the only value available) is Altinity Edge Ingress
. If your cloud provider is AWS, AWS NLB
may be an option as well. In some cases, you’ll see checkboxes that let you enable the load balancer for public and/or private traffic. If you need to use a different load balancer, contact Altinity support.
Node Scope Label
Defines a label that will be added to nodes created by the ACM. This is useful for Kubernetes clusters that have user-created nodes. With a node scope label, the ACM will only deploy pods to nodes with that label.
Subnets CIDR
This is used in Bring Your Own Cloud (BYOC) environments. You define subnets in your cloud account, then reference them when you use the ACM to create a Kubernetes cluster in your cloud account. The subnets you provided when you defined the environment appear here.
K8S Dashboard URL
You have the option of installing a Kubernetes monitoring tool inside the Kubernetes cluster that hosts your ClickHouse clusters. If you do, enter the URL of the monitoring tool here. Most customers use tools that run outside the Kubernetes cluster (k9s, for example), so this option is rarely used.
Server Startup Mode
The startup modes affect how Altinity Cloud Manager starts a ClickHouse cluster when you change its configuration. The three modes are:
- Default mode
- Direct mode - If something goes wrong during startup, the ACM will retry the startup several times before giving up.
- Troubleshooting mode - If something goes wrong during startup, the ACM will not try to restart it if it fails. This is useful for debugging any problems with your cluster’s updated configuration.
Server Startup Time
Depending on the number of tables and the amount of data in your ClickHouse cluster, it may take longer than normal to start. That means it’s possible that the Kubernetes cluster hosting your ClickHouse cluster will delete and restart the pods needed to run ClickHouse before ClickHouse can start. For that reason, you can define a startup time, which is the number of seconds the Kubernetes cluster should wait for your ClickHouse cluster to start.
SNI Proxy
In an Altinity.Cloud Anywhere environment, the endpoints the ACM needs to access may not be publicly accessible. This field lets you define an SNI proxy to route requests from the ACM to the correct endpoint inside your Altinity.Cloud Anywhere environment. (Server Name Indication is an extension to TLS that allows multiple hostnames to be served over a single HTTP endpoint.)
Use SNI Proxy to access ClickHouse
Turns the SNI proxy on or off.
Use Operator Managed PVs
If selected, the persistent volumes used by your ClickHouse clusters will be managed by the Altinity Kubernetes Operator. This is the default; only legacy Altinity.Cloud environments should disable this option.
The Node Defaults tab
This tab lets you define the default ClickHouse version, default node types for ClickHouse and Zookeeper nodes, and storage settings. In addition, you can define the URL of a private image registry if you want your Kubernetes cluster to use a custom registry.
Figure 12 - The Node Defaults tab
Field details
ClickHouse Version
Click the down arrow icon to see a list of available ClickHouse versions.
ClickHouse Node Type
Click the down arrow icon to see a list of available node types.
ClickHouse Storage Multiplier
Defines the amount of storage that will be allocated to your ClickHouse cluster. The ACM takes the amount of RAM allocated to the cluster (in GB), multiplies it by the storage multiplier, then rounds up to the nearest 100GB. For the mathematically inclined, the formula’s notation is , where x
is the RAM in GB and y
is the storage multiplier.
Here are a couple of examples:
- With 8GB of RAM and a storage multiplier of 10, the amount of storage is 100GB (
8 * 10
is80
, rounded up to100
). - With 16GB of RAM and a storage multiplier of 10, the amount of storage is 200GB (
16 * 10
is160
, rounded up to200
).
All that being said, it’s unlikely you’ll ever need to change the default value of 10
.
Zookeeper Node Type
Click the down arrow icon to see a list of available node types.
Zookeeper Storage (GB)
Defines the storage that should be allocated for each Zookeeper node.
Private Image Registry
Enter the URL of your private image registry in the format https://username:password@imageurl.com
. If a private registry is defined, the ACM will pull container images for the ClickHouse server, Zookeeper, and the ClickHouse Backup tool from the private registry.
The Resource Limits tab
Figure 13 - The Resource Limits tab
Field details
Shards per Cluster
Enter the maximum number of shards per cluster.
Replicas per Cluster
Enter the maximum number of replicas per cluster.
Total Storage, GiB
Enter, in gigabytes, the total amount of storage used by the ClickHouse cluster.
Total CPU Cores
Enter the maximum number of cores across all CPUs.
Volume Rescale Percent
Enter the minimum percentage of space that must be used on a given disk before it can be rescaled.
Minimum Volume Size, GiB
Enter, in gigabytes, the minimum volume size for a multi-volume configuration.
Minimum st1
Volume Size, GiB
Enter, in gigabytes, the minimum volume size for EBS st1
volumes. If your cloud provider is not AWS, this field is disabled.
Total Nodes
Enter the maximum number of nodes for your ClickHouse cluster. A value of 0
means there is no limit.
The Backups tab
Figure 14 - The Backups tab
Field details
Turn On Backups
Select this to, um, turn on backups. If not selected, no backups of your data will be made.
Backup Schedule
Allows you to define a schedule of backups. See the Configuring backups documentation in the User Guide for complete details.
Backup Tool Image
The name and tag of the container image used to create and manage backups. If left blank, the default is altinity/clickhouse-backup:stable
.
Compression Format
The default is tar
; other options are gzip
and zstd
. Be aware that creating a tar
file has the lowest impact on the CPU, but it creates the largest file because a tar
file isn’t compressed. On the other hand, the other compression formats take more CPU cycles to create but have smaller file sizes. Choose accordingly.
Enable Objects Labeling
If selected, everything in a backup is labeled with the name of the cluster. This can be useful if you’re working directly with the bucket where the backups are stored.
Backup Storage Provider
This lets you select the cloud provider that the ACM should use to store your backups. The details are different depending on the cloud provider you select.
NOTE: You can, in fact, choose to store your backups with a different cloud provider than the provider that hosts your ClickHouse clusters. That will likely incur massive data transfer charges, but you can do it. To be safe, you’ll be asked to confirm that decision:
Figure 15 - Confirming that you want to store backups on a different cloud
Now that you’ve chosen a cloud backup provider, let’s move on to the details…
Backup Storage Provider - AWS
If your backup storage provider is AWS, you’ll see these fields:
Figure 16 - Using AWS as the backup storage provider
Bucket
The name of the S3 bucket.
Region
The region where the bucket is hosted.
Access Key
The access key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.
Secret Key
The secret key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.
Assume ARN
The ARN (Amazon Resource Name) for the bucket. If you have a value in this field, ACM ignores any values in the Access Key and Secret Key fields above. NOTE - Configuring an ARN that works with your service accounts and the Altinity Cloud Manager is complicated. Contact Altinity support for help in setting up your ARN.
Path (optional)
The path inside the bucket where the backups should be stored. The default value is altinity-cloud-managed-clickhouse
.
TEST CONNECTION
If you change the Secret Key, the TEST CONNECTION button becomes active. If the connection works, you’ll see this message:
Figure 17 - A successful connection test
Otherwise you’ll see a message that tells you what went wrong:
Figure 18 - An unsuccessful connection test
Backup Storage Provider - GCP
If your backup storage provider is GCP, you’ll see these fields:
Figure 19 - Using GCP as the backup storage provider
Bucket
The name of the bucket.
Credentials JSON
JSON data that contains credentials associated with a GCP service account. That service account can have access to your entire GCP project, or it may be restricted to a single bucket or even a single folder within a single bucket. See the Google Cloud documentation for details on Creating a service account and Creating a credentials.json file with the gcloud iam service-accounts keys create command
Path (optional)
The path to the directory inside the bucket where your data is stored. The default value is altinity-cloud-managed-clickhouse
.
TEST CONNECTION
When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:
Figure 20 - A successful connection test
Figure 21 - An unsuccessful connection test
You’ll of course need to correct any errors before you can continue.
Note that when you return to this panel, the value of the Credentials JSON field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your Credentials JSON again if you want to re-test the connection.
Backup Storage Provider - Azure
If your backup storage provider is Azure, you’ll see these fields:
Figure 22 - Using Azure as the backup storage provider
Container
The name of the container.
Account Name
The organization name for your Azure account.
Account Key
The key for your Azure account.
Path (optional)
The path inside the container where the backups should be stored. The default value is altinity-cloud-managed-clickhouse
.
TEST CONNECTION
When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:
Figure 23 - Successful and unsuccessful connection test messages
You’ll of course need to correct any errors before you can continue.
Note that when you return to this panel, the value of the Account Key field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your Account Key again if you want to re-test the connection.
The Metrics tab
Figure 24 - The Metrics tab
Field details
Datadog options
Turn on Datadog
Enables Datadog for clusters in this environment. See Datadog's ClickHouse integration documentation for the details on configuring Datadog and the ClickHouse metrics that Datadog can monitor. Complete details on configuring Datadog access in the Altinity Cloud Monitor are in the section Enabling Datadog at the environment level.
API Key
Your Datadog API key.
Region
The region where your Datadog server is located.
Send Logs
Send Metrics
Send Table Metrics
These buttons allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.
External Prometheus configuration options
Remote URL
The URL of your remote Prometheus server. In Figure 24 above, the remote write endpoint is https://logs-prod-006.grafana.net
.
Auth User
Auth Password
The username and password for your remote Prometheus server. In the example above, the User ID is 4739826. The password in this example is a token generated by Grafana Cloud.
Metric Storage configuration option
Retention Period, Days
The number of days backups should be kept. The default value for Altinity.Cloud Anywhere environments is 30
. For all other environments, this value is 180
and cannot be changed.
The Logs tab
Figure 25 - The Logs tab
Field details
Loki Logs Storage Configuration
Logs Bucket
Logs Region
The name of the bucket where your Loki logs should be stored and the region where the bucket is hosted. The bucket is used internally by the ACM; it’s not accessible by users. In addition, this setting is only used for Bring Your Own Kubernetes (BYOK) environments. For Bring Your Own Cloud (BYOC) environments, the bucket is configured automatically and these fields are not editable.
Send Logs to an External Loki service
External Loki URL
The URL of your external Loki server in the format https://username:password@lokiserver.com/api/prom/push
. For complete details on integrating your Altinity.Cloud environment with an external Loki server, see the Administrator Guide section on Integrating Loki.
Other configuration tasks
Defining Cost Allocation tags
For Bring Your Own Cloud (BYOC) environments, you can assign tags (name/value pairs) to storage and compute assets created through the ACM. That gives you more flexibility in tracking costs and in doing chargebacks. To define them, select Cost Allocation Tags from the ACTIONS menu on the Environment Overview page:
Figure 26 - The Cost Allocation Tags menu
This brings up the Cost Allocation Tags dialog:
Figure 27 - The Cost Allocation Tags dialog
The dialog is straightforward; the button creates a new tag, and the trash can icon deletes one.
The fourth tag in Figure 27 is a dynamic tag. Dynamic tags only apply to AWS environments, and they allow you to define templates whose values are evaluated at runtime. In the example here, $(label:clickhouse.altinity.com/chi)
is the name of the current ClickHouse cluster. That sets the value of the example.com/role
tag to be altinity-[clusterName]
.
Defining a maintenance window
You can define maintenance windows that tell Altinity when certain components of your environment can be updated. That can mean configuring node pools or environment connectivity as well as an update of the ACM itself. To define a maintenance window, select Maintenance Window from the ACTIONS menu on the Environment Overview page:
Figure 28 - The Maintenance Window menu
Click the button to add a new schedule. Each schedule must include at least two days:
Figure 29 - The Maintenance Window dialog with a maintenance window defined
In Figure 29 above, there is a single maintenance window that runs on Saturday and Sunday starting at 20:00 UTC for four hours. Note that all starting times are in UTC.
Defining custom node types
When you create a ClickHouse cluster in your environment, the ACM automatically includes a number of node types. The names and capabilities of those node types are based on the cloud provider associated with your environment.
To see the list of node types defined for your environment, go to the summary view of the environment and click the NODE TYPES menu at the top:
Figure 30 - The Node Types menu
You’ll see the list of node types defined in your environment:
Figure 31 - The list of node types
How (or if) you define custom node types depends on your environment:
- If you’re running Altinity.Cloud as a Service, the node types are managed for you by Altinity.Cloud. If you need to add or edit node types, contact Altinity support.
- If you’re running Altinity.Cloud in a Bring Your Own Cloud (BYOC) environment, Altinity defines a number of node types for you. It’s highly unlikely you’ll need to edit or create node types, but if you do, you can.
- If you’re running Altinity.Cloud in a Bring Your Own Kubernetes (BYOK) environment, the ACM has no way of knowing what kinds of nodes you have in your environment. If you configure autoscaling with Karpenter (AWS), the AKS cluster autoscaler (Azure), or the GKE Autopilot (GCP), you can use the ACM to refer to those node types.
To work with node types, click the button to add a new node type.
It’s not common, but you can also click the vertical dots icon next to a node type in the list and select Edit to edit an existing node type:
Figure 32 - The edit node type menu
You’ll see this dialog:
Figure 33 - The Node Type Details dialog
Field details
Name - The name of the node. The node name and the instance type are typically the same, although that’s not a requirement.
Scope - Either ClickHouse, Zookeeper, or System. The scope allows you to define which node types you want to use with each component of the system.
Instance Type - Clicking the down arrow displays a list of dozens of cloud-provider-specific instance types. The example here is from a ClickHouse environment hosted on Azure, so Standard_D2s_v5
is an option. Similarly, n2-standard-4
would be an option for GCP and t4g.large
would be an option for AWS.
Memory, MB - The amount of memory assigned to this node type. A value of at least 4096
is recommended.
CPU - The number of CPUs assigned to this node type.
Kubernetes options
These options are typically used in BYOK environments only. In many cases, the underlying BYOK infrastructure hosts workloads other than ClickHouse. That means you’ll want to make sure the right nodes are used for the ClickHouse, Zookeeper, and System nodes, as well as all of the other nodes in your environment. Defining Kubernetes tolerations or nodeSelectors gives you control over how nodes are allocated and managed in your environment. (In some cases BYOC environments are shared with other workloads; if that’s true, BYOC users may need Tolerations or NodeSelectors as well.)
Tolerations - One or more Kubernetes tolerations associated with this node type. Multiple tolerations should be separated by semicolons (;
).
Node Selector - One or more Kubernetes nodeSelectors associated with this node type. Multiple nodeSelectors should be separated by commas (,
).
Storage Class - Leave this field as is. It will be removed in a future version of the ACM.
Viewing environment audit logs
To view an environment’s audit logs, go to the list of environments, click the vertical dots icon next to your environment name and select Audit Logs from the menu:
Figure 34 - The Audit Logs Environment menu
You’ll see the logs for the ACM by default:
Figure 35 - The Audit Logs dialog
This is a list of all of the things that have happened in the ACM, such as configuration changes. You can click the arrow to expand any item in the log to get more details. The Anywhere Logs tab shows similar information for your Altinity.Cloud Anywhere environment.
Deleting an Environment
When you’re ready to delete the Altinity.Cloud environment, go to the list of environments, click the vertical dots icon next to your environment name and select Delete from the menu:
Figure 36 - The Delete Environment menu
There are three choices in the dialog:
- Do not de-provision Kubernetes resources - By default the Kubernetes resources created in a BYOK Altinity.Cloud environment will be deleted. Check this button if you plan to delete those resources some other way.
- Do not de-provision cloud resources - By default cloud resources created in a BYOC Altinity.Cloud environment will be deleted. Check this button if you plan to delete those resources some other way.
- Delete Clusters - By default the data for the ClickHouse clusters in the environment is not deleted. Check this button to delete everything associated with your ClickHouse clusters along with your environment. If you check this button, you’ll have to type the name of your environment before it can be deleted.
How and when to use each choice depends on the status of your Altinity.Cloud environment and its underlying infrastructure. There are three scenarios when deleting an Altinity.Cloud environment:
-
You want to delete everything in your environment. That includes any Kubernetes or cloud resources Altinity.Cloud might have created.
The simplest case is an environment that doesn’t have any ClickHouse clusters:
Figure 37 - Deleting an Altinity.Cloud environment with no ClickHouse clusters
Clicking OK deletes the Altinity.Cloud environment and its associated Kubernetes or cloud resources. If the environment does have ClickHouse clusters, though, things are slightly more complicated:
Figure 38 - Deleting an Altinity.Cloud environment with ClickHouse clusters
You should delete your ClickHouse clusters by hand before deleting your Altinity.Cloud environment. However, you can click the Delete Clusters slider and have the ACM delete your ClickHouse clusters along with the resources that host them. If that’s your choice, you need to type the name of your environment before you click OK. The choices selected and not selected in Figure 38 delete everything. -
Another scenario is when Altinity.Cloud can no longer connect to your environment. In this case, you need to tell the ACM that you’ll take the responsibility of deleting any Kubernetes or cloud resources that Altinity.Cloud created:
Figure 39 - Deleting an Altinity.Cloud environment, deleting resources by hand
The ACM deletes the Altinity.Cloud environment, but it can’t access the resources underneath it. You’ll have to delete those resources by hand. -
Finally, it’s possible that you’ve decided you want to remove Altinity.Cloud’s access to your ClickHouse clusters, but keep those ClickHouse clusters running. (This is a good time to mention that we don’t have vendor lock-in.) You can delete Altinity.Cloud’s access to your ClickHouse clusters without deleting your ClickHouse clusters and their data. In that case, tell the ACM not to delete the Kubernetes and cloud resources it created or your ClickHouse clusters.
Figure 40 - Deleting the Altinity.Cloud environment, but nothing else
The first two choices feel like a double negative (“Yes, don’t delete my resources), but these three options (yes / yes / no) delete the Altinity.Cloud environment without disturbing your ClickHouse clusters, their data, and the infrastructure where they’re running.