Working with Environments

Creating an Environment for your ClickHouse® clusters

An Environment is a way of grouping ClickHouse® clusters together in your account. There are several things you’ll want to do with your environments; we’ve broken the details into several sections:

The environments view

In working with Environments, you’ll always start by selecting the Environments tab to load the list of Environments:

The Environments tab

Figure 1 - The Environments tab

You’ll see the Environment list view:

The Environment list

Figure 2 - The list of Altinity.Cloud Environments

The columns in the table are:

  • Select button - Lets you select an Environment for operations and includes the vertical dots icon, which includes options to Edit, Export, or Delete the environment. (More on those tasks later.)
  • Environment number - Each Environment has a unique number. If you’re using the Altinity Cloud Manager API, you will need this number when making certain API calls.
  • Environment Name - Um, the name of the environment.
  • Status - The status of the cluster. is what we’re looking for here.
  • Deployment Type - The kind of deployment, either Kubernetes (Altinity.Cloud) or Anywhere (Altinity.Cloud Anywhere).
  • Cloud Provider - Either AWS, GCP, or Azure.
  • Created - The time the Environment was created.
  • Owner - The ID of the Environment’s administrator.
  • Organization - The Organization that owns this Environment.

Setting up an Environment

The first step in setting up an environment is clicking the button. In the Environment Setup dialog, click a radio button to select the type of environment you need:

Environment Setup - Environment Type

Figure 3 - The list of environment types

The environment types are:

Bring Your Own Kubernetes (BYOK) - Altinity.Cloud runs in your Kubernetes environment. See the Bring Your Own Kubernetes documentation for complete details.

Bring Your Own Cloud (BYOC) - Altinity.Cloud provisions Kubernetes clusters in your cloud account, then provision ClickHouse clusters inside those Kubernetes clusters. When you select this option, you’ll see radio buttons that let you choose your cloud provider:

BYOC environment - Choose your cloud provider

Figure 4 - BYOC environment - Choose your cloud provider

See the Bring Your Own Cloud documentation for complete details.

SaaS Provisioned by Altinity - ClickHouse as a service, hosted and managed by Altinity. When you select this option, you’ll see radio buttons and a dropdown list that let you choose a cloud provider and region for your new environment:

SaaS environment - Choose a cloud provider and region

Figure 5 - SaaS environment - Choose a cloud provider and region

Choose an environment type and give your new environment a name. Depending on the environment type you choose, you may need to specify a cloud provider or region.

Click OK to create the environment.

Configuring an Environment

To configure an environment, click the vertical dots icon and select Edit from the menu:

The Edit Environment menu

Figure 6 - The Edit Environment menu

This takes you to the Environment Configuration dialog. The dialog has seven tabs:

We’ll cover each of these next.

The General tab

The most high-level settings are on the General tab:

The General tab

Figure 7 - The General tab

Field details

Name

The name of the Environment.

Deployment Type

This field is read-only for most users. Under normal circumstances, the only values that will appear here are Kubernetes or Anywhere.

Availability Zones

Enter the availability zones for your cloud provider.

Use External DNS

The names of the endpoints of ClickHouse resources used by ACM are based on a naming convention that includes the cluster name and altinity.cloud. Checking this box means the ACM will access those resources through endpoints resolved through an external DNS. Contact Altinity support for help in setting this up.

Domain

You can define a custom domain; contact Altinity support to set this up. The default domain uses a combination of your ACM environment name and altinity.cloud.

Database Login
Database Password

The username and password for the ClickHouse cluster. The ACM uses these values to retrieve information about schemas, users, and other high-level information from the ClickHouse cluster. These values are set for you by default.

If you change the username to default and leave the password blank, the ACM will attempt to retrieve the password from the ClickHouse operator’s configuration.

Automatic Publish

If selected, any changes made to the Environment’s configuration are automatically published when you make them. Turning this off lets you decide when a change is published, giving you more control over the configuration. For example, you might want to make a number of configuration changes and publish them at one time.

Monitoring

Monitoring is turned on and the button is disabled. This control is a legacy item that will disappear in future versions of the ACM.

The Kubernetes tab

The Kubernetes tab, as you would expect, lets you configure Kubernetes settings:

The Kubernetes tab

Figure 8 - The Kubernetes tab

Field details

Master API URL

Under normal circumstances this value should not be changed. It can be useful in a Bring Your Own Kubernetes environment when defining connections between the ACM and Kubernetes. Contact Altinity support for help setting this up.

Auth Options - TLS Handshake + Token

This authentication scheme is the most common:

The Kubernetes tab - TLS Handshake and Token

Figure 9 - Using the TLS Handshake and Token authentication scheme

The Client Key and Client Cert are defined for you. Under normal circumstances, you won’t need to change these values at all.

Auth Options - No Auth (Proxy)

This approach is used for development purposes only. It should never be used in production.

The Kubernetes tab - No Auth

Figure 10 - Using the No Auth authentication scheme

Auth Options - Access Token

This is a legacy technique that is no longer used. It will be removed in a future version of the ACM.

The Kubernetes tab - Access Token

Figure 11 - Using the Access Token authentication scheme

Namespace

The namespace used for the ClickHouse clusters deployed by Altinity. The default value is altinity-cloud-managed-clickhouse. Once the ClickHouse cluster is created, the namespace cannot be changed.

Manage Namespace

If this switch is turned on, namespaces will be managed by the ACM.

Certificate ARN

The ARN (Amazon Resource Name) for the certificate used by this environment. This is managed for you in Altinity.Cloud environments, and is typically not needed in an Altinity.Cloud Anywhere environment.

Load Balancer Type

The default value (and likely the only value available) is Altinity Edge Ingress. If your cloud provider is AWS, AWS NLB may be an option as well. In some cases, you’ll see checkboxes that let you enable the load balancer for public and/or private traffic. If you need to use a different load balancer, contact Altinity support.

Node Scope Label

Defines a label that will be added to nodes created by the ACM. This is useful for Kubernetes clusters that have user-created nodes. With a node scope label, the ACM will only deploy pods to nodes with that label.

Subnets CIDR

This is used in Bring Your Own Cloud (BYOC) environments. You define subnets in your cloud account, then reference them when you use the ACM to create a Kubernetes cluster in your cloud account. The subnets you provided when you defined the environment appear here.

K8S Dashboard URL

You have the option of installing a Kubernetes monitoring tool inside the Kubernetes cluster that hosts your ClickHouse clusters. If you do, enter the URL of the monitoring tool here. Most customers use tools that run outside the Kubernetes cluster (k9s, for example), so this option is rarely used.

Server Startup Mode

The startup modes affect how Altinity Cloud Manager starts a ClickHouse cluster when you change its configuration. The three modes are:

  • Default mode
  • Direct mode - If something goes wrong during startup, the ACM will retry the startup several times before giving up.
  • Troubleshooting mode - If something goes wrong during startup, the ACM will not try to restart it if it fails. This is useful for debugging any problems with your cluster’s updated configuration.

Server Startup Time

Depending on the number of tables and the amount of data in your ClickHouse cluster, it may take longer than normal to start. That means it’s possible that the Kubernetes cluster hosting your ClickHouse cluster will delete and restart the pods needed to run ClickHouse before ClickHouse can start. For that reason, you can define a startup time, which is the number of seconds the Kubernetes cluster should wait for your ClickHouse cluster to start.

SNI Proxy

In an Altinity.Cloud Anywhere environment, the endpoints the ACM needs to access may not be publicly accessible. This field lets you define an SNI proxy to route requests from the ACM to the correct endpoint inside your Altinity.Cloud Anywhere environment. (Server Name Indication is an extension to TLS that allows multiple hostnames to be served over a single HTTP endpoint.)

Use SNI Proxy to access ClickHouse

Turns the SNI proxy on or off.

Use Operator Managed PVs

If selected, the persistent volumes used by your ClickHouse clusters will be managed by the Altinity Kubernetes Operator. This is the default; only legacy Altinity.Cloud environments should disable this option.

The Node Defaults tab

This tab lets you define the default ClickHouse version, default node types for ClickHouse and Zookeeper nodes, and storage settings. In addition, you can define the URL of a private image registry if you want your Kubernetes cluster to use a custom registry.

The Node Defaults tab

Figure 12 - The Node Defaults tab

Field details

ClickHouse Version

Click the down arrow icon to see a list of available ClickHouse versions.

ClickHouse Node Type

Click the down arrow icon to see a list of available node types.

ClickHouse Storage Multiplier

Defines the amount of storage that will be allocated to your ClickHouse cluster. The ACM takes the amount of RAM allocated to the cluster (in GB), multiplies it by the storage multiplier, then rounds up to the nearest 100GB. For the mathematically inclined, the formula’s notation is , where x is the RAM in GB and y is the storage multiplier.

Here are a couple of examples:

  • With 8GB of RAM and a storage multiplier of 10, the amount of storage is 100GB (8 * 10 is 80, rounded up to 100).
  • With 16GB of RAM and a storage multiplier of 10, the amount of storage is 200GB (16 * 10 is 160, rounded up to 200).

All that being said, it’s unlikely you’ll ever need to change the default value of 10.

Zookeeper Node Type

Click the down arrow icon to see a list of available node types.

Zookeeper Storage (GB)

Defines the storage that should be allocated for each Zookeeper node.

Private Image Registry

Enter the URL of your private image registry in the format https://username:password@imageurl.com. If a private registry is defined, the ACM will pull container images for the ClickHouse server, Zookeeper, and the ClickHouse Backup tool from the private registry.

The Resource Limits tab

The Resource Limits tab

Figure 13 - The Resource Limits tab

Field details

Shards per Cluster

Enter the maximum number of shards per cluster.

Replicas per Cluster

Enter the maximum number of replicas per cluster.

Total Storage, GiB

Enter, in gigabytes, the total amount of storage used by the ClickHouse cluster.

Total CPU Cores

Enter the maximum number of cores across all CPUs.

Volume Rescale Percent

Enter the minimum percentage of space that must be used on a given disk before it can be rescaled.

Minimum Volume Size, GiB

Enter, in gigabytes, the minimum volume size for a multi-volume configuration.

Minimum st1 Volume Size, GiB

Enter, in gigabytes, the minimum volume size for EBS st1 volumes. If your cloud provider is not AWS, this field is disabled.

Total Nodes

Enter the maximum number of nodes for your ClickHouse cluster. A value of 0 means there is no limit.

The Backups tab

The Backups tab

Figure 14 - The Backups tab

Field details

Turn On Backups

Select this to, um, turn on backups. If not selected, no backups of your data will be made.

Backup Schedule

Allows you to define a schedule of backups. See the Configuring backups documentation in the User Guide for complete details.

Backup Tool Image

The name and tag of the container image used to create and manage backups. If left blank, the default is altinity/clickhouse-backup:stable.

Compression Format

The default is tar; other options are gzip and zstd. Be aware that creating a tar file has the lowest impact on the CPU, but it creates the largest file because a tar file isn’t compressed. On the other hand, the other compression formats take more CPU cycles to create but have smaller file sizes. Choose accordingly.

Enable Objects Labeling

If selected, everything in a backup is labeled with the name of the cluster. This can be useful if you’re working directly with the bucket where the backups are stored.

Backup Storage Provider

This lets you select the cloud provider that the ACM should use to store your backups. The details are different depending on the cloud provider you select.

NOTE: You can, in fact, choose to store your backups with a different cloud provider than the provider that hosts your ClickHouse clusters. That will likely incur massive data transfer charges, but you can do it. To be safe, you’ll be asked to confirm that decision:

Confirm backing up to another cloud

Figure 15 - Confirming that you want to store backups on a different cloud

Now that you’ve chosen a cloud backup provider, let’s move on to the details…

Backup Storage Provider - AWS

If your backup storage provider is AWS, you’ll see these fields:

Backup storage provider is AWS

Figure 16 - Using AWS as the backup storage provider

Bucket

The name of the S3 bucket.

Region

The region where the bucket is hosted.

Access Key

The access key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.

Secret Key

The secret key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.

Assume ARN

The ARN (Amazon Resource Name) for the bucket. If you have a value in this field, ACM ignores any values in the Access Key and Secret Key fields above. NOTE - Configuring an ARN that works with your service accounts and the Altinity Cloud Manager is complicated. Contact Altinity support for help in setting up your ARN.

Path (optional)

The path inside the bucket where the backups should be stored. The default value is altinity-cloud-managed-clickhouse.

TEST CONNECTION

If you change the Secret Key, the TEST CONNECTION button becomes active. If the connection works, you’ll see this message:

Connection test worked

Figure 17 - A successful connection test

Otherwise you’ll see a message that tells you what went wrong:

AWS connection test failed

Figure 18 - An unsuccessful connection test

Backup Storage Provider - GCP

If your backup storage provider is GCP, you’ll see these fields:

Backup storage provider is GCP

Figure 19 - Using GCP as the backup storage provider

Bucket

The name of the bucket.

Credentials JSON

JSON data that contains credentials associated with a GCP service account. That service account can have access to your entire GCP project, or it may be restricted to a single bucket or even a single folder within a single bucket. See the Google Cloud documentation for details on Creating a service account and Creating a credentials.json file with the gcloud iam service-accounts keys create command

Path (optional)

The path to the directory inside the bucket where your data is stored. The default value is altinity-cloud-managed-clickhouse.

TEST CONNECTION

When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:

Successful connection test

Figure 20 - A successful connection test

Unsuccessful connection test

Figure 21 - An unsuccessful connection test

You’ll of course need to correct any errors before you can continue.

Note that when you return to this panel, the value of the Credentials JSON field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your Credentials JSON again if you want to re-test the connection.

Backup Storage Provider - Azure

If your backup storage provider is Azure, you’ll see these fields:

Backup storage provider is Azure

Figure 22 - Using Azure as the backup storage provider

Container

The name of the container.

Account Name

The organization name for your Azure account.

Account Key

The key for your Azure account.

Path (optional)

The path inside the container where the backups should be stored. The default value is altinity-cloud-managed-clickhouse.

TEST CONNECTION

When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:

Successful connection test
Unsuccessful connection test

Figure 23 - Successful and unsuccessful connection test messages

You’ll of course need to correct any errors before you can continue.

Note that when you return to this panel, the value of the Account Key field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your Account Key again if you want to re-test the connection.

The Metrics tab

The Metrics tab

Figure 24 - The Metrics tab

Field details

Datadog options

Turn on Datadog

Enables Datadog for clusters in this environment. See Datadog's ClickHouse integration documentation for the details on configuring Datadog and the ClickHouse metrics that Datadog can monitor. Complete details on configuring Datadog access in the Altinity Cloud Monitor are in the section Enabling Datadog at the environment level.

API Key

Your Datadog API key.

Region

The region where your Datadog server is located.

Send Logs
Send Metrics
Send Table Metrics

These buttons allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.

External Prometheus configuration options

Remote URL

The URL of your remote Prometheus server. In Figure 24 above, the remote write endpoint is https://logs-prod-006.grafana.net.

Auth User
Auth Password

The username and password for your remote Prometheus server. In the example above, the User ID is 4739826. The password in this example is a token generated by Grafana Cloud.

Metric Storage configuration option

Retention Period, Days

The number of days backups should be kept. The default value for Altinity.Cloud Anywhere environments is 30. For all other environments, this value is 180 and cannot be changed.

The Logs tab

The Logs tab

Figure 25 - The Logs tab

Field details

Loki Logs Storage Configuration

Logs Bucket
Logs Region

The name of the bucket where your Loki logs should be stored and the region where the bucket is hosted. The bucket is used internally by the ACM; it’s not accessible by users. In addition, this setting is only used for Bring Your Own Kubernetes (BYOK) environments. For Bring Your Own Cloud (BYOC) environments, the bucket is configured automatically and these fields are not editable.

Send Logs to an External Loki service

External Loki URL

The URL of your external Loki server in the format https://username:password@lokiserver.com/api/prom/push. For complete details on integrating your Altinity.Cloud environment with an external Loki server, see the Administrator Guide section on Integrating Loki.

Other configuration tasks

Defining Cost Allocation tags

For Bring Your Own Cloud (BYOC) environments, you can assign tags (name/value pairs) to storage and compute assets created through the ACM. That gives you more flexibility in tracking costs and in doing chargebacks. To define them, select Cost Allocation Tags from the ACTIONS menu on the Environment Overview page:

The Cost Allocation Tags menu

Figure 26 - The Cost Allocation Tags menu

This brings up the Cost Allocation Tags dialog:

The Cost Allocation Tags dialog

Figure 27 - The Cost Allocation Tags dialog

The dialog is straightforward; the button creates a new tag, and the trash can icon deletes one.

The fourth tag in Figure 27 is a dynamic tag. Dynamic tags only apply to AWS environments, and they allow you to define templates whose values are evaluated at runtime. In the example here, $(label:clickhouse.altinity.com/chi)is the name of the current ClickHouse cluster. That sets the value of the example.com/role tag to be altinity-[clusterName].

Defining a maintenance window

You can define maintenance windows that tell Altinity when certain components of your environment can be updated. That can mean configuring node pools or environment connectivity as well as an update of the ACM itself. To define a maintenance window, select Maintenance Window from the ACTIONS menu on the Environment Overview page:

The Maintenance Window menu

Figure 28 - The Maintenance Window menu

Click the button to add a new schedule. Each schedule must include at least two days:

The Maintenance Window dialog

Figure 29 - The Maintenance Window dialog with a maintenance window defined

In Figure 29 above, there is a single maintenance window that runs on Saturday and Sunday starting at 20:00 UTC for four hours. Note that all starting times are in UTC.

Defining custom node types

When you create a ClickHouse cluster in your environment, the ACM automatically includes a number of node types. The names and capabilities of those node types are based on the cloud provider associated with your environment.

To see the list of node types defined for your environment, go to the summary view of the environment and click the NODE TYPES menu at the top:

The Node Types menu

Figure 30 - The Node Types menu

You’ll see the list of node types defined in your environment:

The list of Node Types

Figure 31 - The list of node types

How (or if) you define custom node types depends on your environment:

  • If you’re running Altinity.Cloud as a Service, the node types are managed for you by Altinity.Cloud. If you need to add or edit node types, contact Altinity support.
  • If you’re running Altinity.Cloud in a Bring Your Own Cloud (BYOC) environment, Altinity defines a number of node types for you. It’s highly unlikely you’ll need to edit or create node types, but if you do, you can.
  • If you’re running Altinity.Cloud in a Bring Your Own Kubernetes (BYOK) environment, the ACM has no way of knowing what kinds of nodes you have in your environment. If you configure autoscaling with Karpenter (AWS), the AKS cluster autoscaler (Azure), or the GKE Autopilot (GCP), you can use the ACM to refer to those node types.

To work with node types, click the button to add a new node type.

It’s not common, but you can also click the vertical dots icon next to a node type in the list and select Edit to edit an existing node type:

The edit node type menu

Figure 32 - The edit node type menu

You’ll see this dialog:

The Node Type Details dialog

Figure 33 - The Node Type Details dialog

Field details

Name - The name of the node. The node name and the instance type are typically the same, although that’s not a requirement.

Scope - Either ClickHouse, Zookeeper, or System. The scope allows you to define which node types you want to use with each component of the system.

Instance Type - Clicking the down arrow displays a list of dozens of cloud-provider-specific instance types. The example here is from a ClickHouse environment hosted on Azure, so Standard_D2s_v5 is an option. Similarly, n2-standard-4 would be an option for GCP and t4g.large would be an option for AWS.

Memory, MB - The amount of memory assigned to this node type. A value of at least 4096 is recommended.

CPU - The number of CPUs assigned to this node type.

Kubernetes options

These options are typically used in BYOK environments only. In many cases, the underlying BYOK infrastructure hosts workloads other than ClickHouse. That means you’ll want to make sure the right nodes are used for the ClickHouse, Zookeeper, and System nodes, as well as all of the other nodes in your environment. Defining Kubernetes tolerations or nodeSelectors gives you control over how nodes are allocated and managed in your environment. (In some cases BYOC environments are shared with other workloads; if that’s true, BYOC users may need Tolerations or NodeSelectors as well.)

Tolerations - One or more Kubernetes tolerations associated with this node type. Multiple tolerations should be separated by semicolons (;).

Node Selector - One or more Kubernetes nodeSelectors associated with this node type. Multiple nodeSelectors should be separated by commas (,).

Storage Class - Leave this field as is. It will be removed in a future version of the ACM.

Viewing environment audit logs

To view an environment’s audit logs, go to the list of environments, click the vertical dots icon next to your environment name and select Audit Logs from the menu:

The Audit Logs Environment menu

Figure 34 - The Audit Logs Environment menu

You’ll see the logs for the ACM by default:

The Audit Logs dialog

Figure 35 - The Audit Logs dialog

This is a list of all of the things that have happened in the ACM, such as configuration changes. You can click the arrow to expand any item in the log to get more details. The Anywhere Logs tab shows similar information for your Altinity.Cloud Anywhere environment.

Deleting an Environment

When you’re ready to delete the Altinity.Cloud environment, go to the list of environments, click the vertical dots icon next to your environment name and select Delete from the menu:

The Delete Environment menu

Figure 36 - The Delete Environment menu

There are three choices in the dialog:

  • Do not de-provision Kubernetes resources - By default the Kubernetes resources created in a BYOK Altinity.Cloud environment will be deleted. Check this button if you plan to delete those resources some other way.
  • Do not de-provision cloud resources - By default cloud resources created in a BYOC Altinity.Cloud environment will be deleted. Check this button if you plan to delete those resources some other way.
  • Delete Clusters - By default the data for the ClickHouse clusters in the environment is not deleted. Check this button to delete everything associated with your ClickHouse clusters along with your environment. If you check this button, you’ll have to type the name of your environment before it can be deleted.

How and when to use each choice depends on the status of your Altinity.Cloud environment and its underlying infrastructure. There are three scenarios when deleting an Altinity.Cloud environment:

  • You want to delete everything in your environment. That includes any Kubernetes or cloud resources Altinity.Cloud might have created.

    The simplest case is an environment that doesn’t have any ClickHouse clusters:

    Deleting an Altinity.Cloud environment with no ClickHouse clusters

    Figure 37 - Deleting an Altinity.Cloud environment with no ClickHouse clusters

    Clicking OK deletes the Altinity.Cloud environment and its associated Kubernetes or cloud resources. If the environment does have ClickHouse clusters, though, things are slightly more complicated:

    Deleting an Altinity.Cloud environment with ClickHouse clusters

    Figure 38 - Deleting an Altinity.Cloud environment with ClickHouse clusters

    You should delete your ClickHouse clusters by hand before deleting your Altinity.Cloud environment. However, you can click the Delete Clusters slider and have the ACM delete your ClickHouse clusters along with the resources that host them. If that’s your choice, you need to type the name of your environment before you click OK. The choices selected and not selected in Figure 38 delete everything.

  • Another scenario is when Altinity.Cloud can no longer connect to your environment. In this case, you need to tell the ACM that you’ll take the responsibility of deleting any Kubernetes or cloud resources that Altinity.Cloud created:

    Deleting an Altinity.Cloud environment, deleting resources by hand

    Figure 39 - Deleting an Altinity.Cloud environment, deleting resources by hand

    The ACM deletes the Altinity.Cloud environment, but it can’t access the resources underneath it. You’ll have to delete those resources by hand.

  • Finally, it’s possible that you’ve decided you want to remove Altinity.Cloud’s access to your ClickHouse clusters, but keep those ClickHouse clusters running. (This is a good time to mention that we don’t have vendor lock-in.) You can delete Altinity.Cloud’s access to your ClickHouse clusters without deleting your ClickHouse clusters and their data. In that case, tell the ACM not to delete the Kubernetes and cloud resources it created or your ClickHouse clusters.

    Deleting an Altinity.Cloud environment, but nothing else

    Figure 40 - Deleting the Altinity.Cloud environment, but nothing else

    The first two choices feel like a double negative (“Yes, don’t delete my resources), but these three options (yes / yes / no) delete the Altinity.Cloud environment without disturbing your ClickHouse clusters, their data, and the infrastructure where they’re running.