This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

User Guide

Instructions on general use of Altinity.Cloud

Altinity.Cloud is made to be both convenient and powerful for ClickHouse users. Whether you’re a ClickHouse administrator or a developer, these are the concepts and procedures common to both.

1 - Working with your account

Creating and configuring your Altinity.Cloud account

This section covers the basics of working with your Altinity Cloud Manager account. Additional instructions for managing an Altinity.Cloud subscription in the AWS Marketplace and GCP Marketplace are included as well.

The tasks we cover for your Altinity.Cloud account are:

Logging in

To log in, go to the Altinity Cloud Manager login page. Enter your email and password and click SIGN IN:

The Altinity.Cloud login screen

You can also click the Auth0 link to log in through a third-party authentication provider. The address you use with Auth0 must be the same one you used to create your account.

Figure 1 - The login screen

The My Account menu

The rest of the tasks we’ll cover are in the My Account menu. To access it, click your username in the upper right corner:

Access user account

Figure 2 - The My Account menu

NOTE: Depending on your role, some of these menu items may not appear.

Accessing basic account settings

Cluster Lock button

To access basic settings for your account, click the My Account menu item. This displays the My Account panel:

Access user account

Figure 3 - The four tabs of the My Account panel

The panel has four tabs, which we’ll cover next.

The Common Information tab

The Common Information tab displays basic information, lets you update your password, and lets you switch between light theme and dark theme.

Altinity Cloud user common settings

Figure 4 - The Common Information tab

The Email/Login and Role fields are read-only.

To update your account password, type the new password and confirm it. Click SAVE to update the password.

NOTE: The SAVE button is disabled until both passwords are at least 12 characters long and both passwords match.

The API Access and and Anywhere API Access tabs

These tabs let you configure access keys and tokens for your account. Details of API configuration are in the Altinity API Guide.

The Access Rights tab

The Access Rights tab displays the permissions your account has. You need administrator privileges to modify them.

The Access Rights tab

Figure 5 - The Access Rights tab

The permissions are listed in three columns:

  • Section: The area of access within Altinity.Cloud, such as Accounts, Environments, and Console. A value of * means the user has access to all areas.
  • Action: What actions the access right rule allows within the section. Actions marked as * include all actions within the section.
  • Rule: Whether the action is allowed (marked with a check mark), or denied (marked with an X).

The user account in Figure 5 has complete access to everything. A user with a different role and different privileges will see a comprehensive list of what they can do:

The Access Rights tab with fewer privileges

Figure 6 - The Access Rights tab for a user with fewer privileges

In this excerpt (there are dozens of sections and actions), the user cannot access the Console or Billing sections, although they can display and edit settings for their account. The global permission for everything (sections and actions * *) is disallowed as well. Finally, note that the Anywhere API Access tab does not exist for this user.

Viewing notifications

Cluster Lock button

The Notifications menu item lets you view any notifications you have received:

The Notification History panel

Figure 7 - The Notification History dialog

Here the history shows a single message. The text of the message, its severity (Info, News, Warning, or Danger), and the time the message was received and acknowledged are displayed. The meanings of the message severities are:

  • Info: Updates for general information
  • News: Notifications of general news and updates in Altinity.Cloud
  • Warning: Notifications of possible issues that are less than critical
  • Danger: Critical notifications that can effect your clusters or account

Working with billing information

Cluster Lock button

If you have orgadmin access, you’ll see the Billing menu item. See the Billing page in the Administrator Guide for more information.

Checking system status

Access user account

The System Status menu item lets you check the status of the Altinity.Cloud service and your ClickHouse clusters. This gives a quick glance to help devops staff determine where issues might be when communicating with their ClickHouse clusters.

The System Status page displays the status of the Altinity.Cloud services. To send a message to Altinity.Cloud support, select Get in touch at the top of the page:

Get in touch

Figure 8 - The Get in touch button

The rest of the page displays the status of the system:

Altinity.Cloud system statut page

Figure 9 - Status of your clusters and ACM

The three graphs show the status for the last 90 days. The first two lines here refer to the ClickHouse clusters in your environment. In this example, the clusters were defined just over a month ago, so there isn’t as much data to display. The bottom line is the status of the Altinity.Cloud management console,

Hovering over any bar (red or green) displays more details for that day:

Altinity.Cloud system status outage info

Figure 10 - Details of an entry in the status bar

To subscribe to email updates about any system outages, enter your email and click Subscribe at the bottom of the page.

Viewing Altinity.Cloud documentation

Cluster Lock button

Selecting this menu item takes you to the Altinity documentation page.

Checking the version of ACM you’re using

Cluster Lock button

Selecting this menu item shows you the version of ACM you’re using:

The version of ACM you're using

Figure 11 - The Altinity.Cloud Manager version

You’re using ACM Version 23.7.43. This can be useful when diagnosing a problem.

Logging out

To log out, click your username in the upper right corner, then click Log out.

Log out of your account

Your session will be ended, and you will have to authenticate again to log back into Altinity.Cloud.

NOTE: Your session will automatically be logged out after 60 minutes of inactivity.

Managing your Altinity.Cloud subscription at AWS Marketplace

Go to your AWS Marketplace account and click the Manage your software subscriptions link in the Your Software section:

Figure 12 - Managing your account in the AWS Marketplace Console

Select Altinity.Cloud for ClickHouse in your subscription list and click the Manage button:


Figure 13 - Managing your Altinity.Cloud subscription

Select the Actions button to manage your subscription. The Actions menu has a variety of options:

Figure 14 - The Actions menu

Terminating your Altinity.Cloud subscription at AWS Marketplace

  1. Backup and save your data. Your data will be available for retrieval for the period defined in Altinity.Cloud Terms of Service agreement.

  2. Stop and delete your clusters.

  3. Terminate your service as described in the Altinity.Cloud Terms of Service agreement.

  4. Unsubscribe from the service in your AWS Marketplace account

  5. Go to your AWS Marketplace account and click the Manage your software subscriptions link in the Your Software section:

Figure 15 - Managing your account in the AWS Marketplace Console

  1. Select Altinity.Cloud for ClickHouse in your subscription list and click the Manage button:

Figure 16 - The Manage button

  1. Select the Actions button and select Cancel subscription in the menu:

Figure 17 - The Cancel Subscription menu item

You are now unsubscribed from Altinity.Cloud.

Managing your Altinity.Cloud subscription at GCP Marketplace

Go to GCP Marketplace and click the Your orders link on the left panel:

Figure 13 - Managing your orders in the GCP Marketplace

Select Altinity.Cloud for ClickHouse in your orders list for the billing account you subscribed from and click the vertical dots:


Figure 14 - Managing your Altinity.Cloud subscription

Select needed Action:

  • Sign up with Altinity Inc - redirect you to sign up form
  • Cancel order - terminates your order

Terminating your Altinity.Cloud subscription at GCP Marketplace

  1. Backup and save your data. Your data will be available for retrieval for the period defined in Altinity.Cloud Terms of Service agreement.
  2. Stop and delete your clusters.
  3. Terminate your service as described in the Altinity.Cloud Terms of Service agreement.
  4. Unsubscribe from the service in your GCP Marketplace

Follow Managing your Altinity.Cloud subscription at GCP Marketplace and Cancel your order.


Figure 14 - Cancel your Altinity.Cloud subscription

You are now unsubscribed from Altinity.Cloud.

2 - Viewing clusters in the ACM

Details of the ACM user interface

You’ll spend most of your time in the Altinity Cloud Manager working with ClickHouse clusters. In this section we’ll look at the parts of the ACM user interface that make that easy. Along the way we’ll include links to more details about how to use each set of features.

The Clusters (plural) view

The Clusters view displays your ClickHouse clusters:

Altinity.Cloud Management Plane

Figure 1 - The Clusters (plural) view, featuring two clusters

Key items in Figure 1 above include:

  • The button, which makes it easy to create a new ClickHouse cluster. If you’d like to exit the tour and create a new cluster now, see the Launch Cluster Wizard documentation.
  • The Environment menu. Clicking on the current environment name (altinity-maddie-byok) displays a list of environments you can access.
  • The My Account menu. Clicking on your username (Doug Tidwell) displays a list of options for managing your account. See the My Account menu documentation for all the details of managing your account.
  • The clusters in this environment. In this view there are two ClickHouse clusters, cluster2 and maddie-byok. Your clusters are displayed in panel view (shown) or the more compact list view. The panel view shows details about the cluster such as its resources and configuration, the version of ClickHouse it’s running, and whether backups are set up.

Clicking on a cluster name takes you to the Cluster view for that cluster. And speaking of the Cluster view….

The Cluster (singular) view

Here’s the Cluster view for the maddie-byok cluster:

Cluster Details Page

Figure 2 - The Cluster (singular) view for the maddie-byok cluster

The buttons and menus across the top of the panel are:

The top row of buttons on the Cluster view page
  • ACTIONS - a menu of actions you can take on this cluster. See the Cluster Actions documentation for complete details.
  • TOOLS - Lets you configure a Kafka connection for this cluster.
  • CONFIGURE - A menu of configuration options. See Configuring a Cluster for complete details.
  • EXPLORE - Takes you to the Cluster Explorer, a panel that lets you work with data in the cluster, import data, view database schemas, and other useful things. See the Cluster Explorer guide for complete details.
  • ALERTS - Lets you define alerts that should be triggered in response to certain events. See the Cluster alerts documentation for complete details.
  • LOGS - Takes you to the Logs view, which contains a number of different logs to give you insight into your cluster. See the Cluster logs documentation for complete details.
  • ALTINITY ACCESS - Lets you define the access privileges Altinity support personnal should have into your ClickHouse clusters. See the Altinity Access to ClickHouse documentation for complete details.
  • - The cluster lock button, which lets you prohibit any changes to the cluster. See the Locking a cluster documentation for all the details.
  • - The refresh button. Refreshes the display, as you would expect.

The rest of the view shows statistics and configuration information for the cluster. Most of that data is read-only. You can, however, change the name of the cluster owner (Doug Tidwell in Figure 2) or the Cluster Role (Development) by clicking the current value. You’ll get a simple dialog that lets you set a new value.

The Connection Details link highlighted in Figure 2 shows you the hostname, ports, and other details you’ll need to connect to your ClickHouse cluster from an application, a monitoring tool, or the clickhouse-client tool:

Cluster connection details
Figure 3 - The Connection Details for the cluster

The Nodes (plural) view

Clicking the Nodes tab highlighted below the cluster name at the top of Figure 2 above shows you the list of nodes in this cluster:

The Nodes view for a cluster

Figure 4 - The Nodes (plural) view for a cluster

Clicking the NODE VIEW button above takes you to, well, the view of a single node.

The Node (singular) view

The view of a single node looks like this:

Connection details for a single node

Figure 5 - The Node (singular) view

The buttons across the top of the panel are:

  • ACTIONS - The ACTIONS menu has two items:
    The Action menu for a node
    • Restart - Restarts the node.
    • Exclude from Load Balancer - Excludes this node from the load balancer, as you would expect. Selecting this item means that the node will not get any traffic from the load balancer. (You can still access the node directly if you have its connection information as shown in Figure 6 below.) When a node is removed from the load balancer, the text of this menu item becomes Include in Load Balancer; clicking that item returns the node to the load balancer. The value of LB Status in the display is Included or Excluded as appropriate:
      A node's load balancer status
  • EXPLORE - Takes you to the Cluster Explorer for this node. This is the same Cluster Explorer available from the Cluster view, but it is scoped to work with the current node only. See the Cluster Explorer guide for complete details.
  • STORAGE - Lets you view, modify, and manage the storage used by this node. See the Configuring storage documentation for complete details.
  • VIEW LOGS - Takes you to the Logs view, which contains a number of different logs to give you insight into your cluster. See the Cluster logs documentation for complete details.

Clicking the Connection Details link highlighted in Figure 5 above shows the information you need to connect directly to this node:

Node connection details
Figure 6 - The Connection Details for a particular node

This dialog is similar to Figure 3 above, the difference being that the host is the URL for this node, not the entire cluster. Depending on how you have configured replication and sharding, connecting to an individual node may be useful.

3 - Cluster actions

Common tasks for working with ClickHouse clusters

In this section we’ll look at all the actions you can take with ClickHouse clusters through the Altinity Cloud Manager (ACM), expanding on the basics we covered in the ACM introduction.

There are a number of common ClickHouse cluster tasks you’ll likely need to do:

Launching a new Cluster

Launching a new cluster is incredibly easy, and only takes a few minutes. To launch a new ClickHouse cluster, click the LAUNCH CLUSTER button at the top of the Clusters view page: See the Launch Cluster wizard documentation for complete details on the options for creating and configuring a new cluster.

Launch New Cluster

Figure 1 - The LAUNCH CLUSTER button

NOTE: As you would expect, if you have one or more ClickHouse clusters running already, they’ll appear here in place of the “You don’t have any clusters” message.

Locking a cluster

If you’re looking at a particular cluster, there is a lock button at the top of the panel that allows you to prevent any changes to the cluster. Be aware that anyone with appropriate access can lock or unlock the cluster at any time.

To lock or unlock the cluster, simply click the lock button:

Cluster Lock button

Figure 2 - The Lock Cluster button

The button turns red when locked and returns to transparent when unlocked.

The Actions menu

The rest of the items we’ll go through are in the Actions menu, which appears at the bottom of the cluster explorer view or the top of a cluster panel:

The Actions menu

Upgrading a cluster

Launch New Cluster

Clusters can be upgraded to newer versions of ClickHouse. Before you upgrade, we strongly encourage you to open a support ticket with normal priority and label the ticket “Upgrade.”

Select the ClickHouse version to update to. You can select from Altinity Stable Builds or Community Builds. As you choose which type of build you want to use, the dropdown menu below changes to include the latest builds of that type. There is also a Release notes link beneath the menu; clicking that link takes you to the release notes for that release. We also strongly recommend reading the release notes for the version you’re upgrading to.

Cluster Upgrade

Figure 3 - The Upgrade Cluster dialog

Be sure to read any warning messages carefully. If the version you’ve selected will downgrade your cluster by at least one major version, you’re required to acknowledge that before the process will start.

The upgrade process time varies with the size of the cluster, as each server is upgraded individually. This may cause downtime while the cluster is upgraded.

Rescaling a cluster

Launch New Cluster

You may need to change the size and structure of your ClickHouse cluster at some point. Clicking the Rescale menu item displays this dialog:

Cluster Rescale

Figure 4 - The Rescale Cluster dialog

The current properties of your cluster are shown on the left side; you can enter new values or select a new node type on the right. Click OK to rescale the cluster with the new values. As you would expect, the rescaling time varies with the size of the cluster, as each node is rescaled individually.

The rescaling operation takes place one node at a time, so the impact of rescaling should have a minimal impact on the performance of your applications. (Assuming your cluster has more than one replica, of course.)

Stopping, resuming, or restarting a cluster

Launch New Cluster
Launch New Cluster

You can stop, resume, or restart a cluster by clicking the appropriate menu item. As you would expect, the text of the first menu item is Stop if the cluster is running and Resume if the cluster is stopped. Depending on the size of your cluster, it may take a few minutes until it is fully stopped, resumed, or restarted.

If you are resuming a cluster, you can change the node type uaed in the cluster:

Resuming a cluster

If you are restarting a cluster, you have the option of doing a Hard Cluster Reset, which kills all of the ClickHouse cluster’s Kubernetes pods before restarting the cluster:

Hard cluster reset

Be aware that a cluster may be stopped or resumed automatically based on any activity schedule settings you may have configured.

Exporting a cluster’s configuration

Launch New Cluster

The structure of an Altinity Cloud cluster can be exported as JSON. Selecting this menu item creates a JSON file that is downloaded to your machine. As an example, if your cluster is named maddie-byok, the JSON file will be named maddie-byok.json.

Publishing a cluster’s configuration

Launch New Cluster

The Publish Configuration menu item applies any configuration changes you have made to your cluster. This gives you more control over how changes are applied.

This feature is related to the Automatic Publish setting on your environment:

The Automatic Publish setting

Automatic publishing is enabled by default. That means any changes to the configuration of any cluster in the environment are automatically applied as they are made in the ACM. However, you (or your administrator) may want more control over how changes are applied. Publish Configuration lets you decide when changes should be applied. (You must have administrator-level access to change an environment’s Automatic Publish option setting.)

If you have configuration changes that haven’t been published yet, you’ll see the PUBLISH button at the top of the cluster view:

The Publish button

Clicking the button displays the Pending Cluster Updates dialog:

The Pending Cluster Updates dialog

You can click the arrow icon to see the details of any particular change. Be aware that you can’t modify your changes from this dialog. If you need to modify something, click CANCEL, edit your settings and environment variables as needed, then come back to the Publish Configuration function to continue.

Click CONFIRM to publish the changes.

Working with a cluster’s history

Cluster History

The Cluster Update History view lets you rollback changes that have been made to your cluster:

Cluster History

Figure 5 - The Cluster Update History view

In this example, several cluster settings have been modified. You can click the arrow icon to expand an item and see its details. The highlighted change added the logger/level attribute to the cluster’s configuration.

NOTE: Rolling back an action also rolls back every other action made since the one you’re rolling back. The text for the two actions above the highlighted one are crossed out as a visual reminder.

Launching a replica cluster

Launch New Cluster

WARNING: FOR ADVANCED USERS ONLY.

Clusters can be replicated easily. A replica can include the same database schema as the original cluster, or it can be launched without the schema.

NOTE: Be aware this a true replica, not a one-time copy. Any changes made to the replica’s data will be replicated back to the source cluster.

This menu item starts a slightly modified version of the Launch Cluster Wizard. The only difference is on the ClickHouse Setup panel:

Replicate the cluster schema

Figure 6 - The modified ClickHouse Setup panel

You have the option to replicate the schema of the existing cluster. With that exception, this version of the Launch Cluster Wizard is identical to the one covered in the Launch Cluster Wizard documentation.

Once you click Launch on the Review & Launch tab at the end of the wizard, the replica cluster will be available within a few minutes.

Creating a backup

Launch New Cluster

Creating a backup is straightforward: simply click the Create Backup item on the ACTIONS menu. You’ll see an informative message like this:

Creating a backup

As the dialog points out, backups are stored separately from the cluster, so you can restore your cluster from a backup even if you delete the cluster. Click OK to create the backup.

Restoring a backup

Launch New Cluster

WARNING: FOR ADVANCED USERS ONLY.

You must have Administrator access to perform this action. See Restoring a backup in the Administrator Guide for all the details.

Destroying a cluster

Destroy a cluster

When you no longer need a cluster, the entire cluster and all of its data can be destroyed. Enter the name of the cluster and click OK to delete it:

Destroy a cluster

Figure 7 - The Delete Cluster dialog

NOTE: Once destroyed, a cluster cannot be recovered, it must be manually recreated.

4 - Configuring a Cluster

How to configure ClickHouse clusters

In this section we’ll look at all the configuration settings you can change or modify for a ClickHouse cluster through the Altinity Cloud Manager (ACM), expanding on the basics we covered in the ACM introduction. The items we’ll go through are in the Configure menu:

The cluster configuration menu

You can skip ahead to a specific configuration task if you like:

Configuring cluster settings

Every cluster has a number of settings. The Settings view has a Server Settings tab that lets you view, add, edit, or delete the server’s current settings, or restore the settings to their defaults. There is also an Environment Variables tab that lets you work with environment variables. In addition to configuring settings and environment variables themselves, there are also a couple of global settings that control how the cluster starts when you apply your changes.

WARNING: FOR ADVANCED USERS ONLY. Many ClickHouse settings can be configured through the Altinity Cloud Manager UI. We strongly recommend using the UI wherever possible; it protects you from syntax errors and other common mistakes.

An important point before we go on: By default, the environment that contains your cluster is configured so that any changes to settings and environment variables are automatically published to your cluster. However, if automatic publishing is turned off for your cluster, you’ll need to use the Publish Configuration feature to make your changes permanent. See Publishing a cluster's configuration for more information.

Working with server settings

The Settings View looks like this:

The System Settings tab in the Settings view

Figure 1 - The Server Settings tab in the Settings View

Settings with a lock icon next to their names can’t be changed through the Settings View. For any other setting, click the vertical dots icon next to the setting name, then click Edit or Delete.

Adding a setting

Clicking the button at the top of the panel lets you add a setting. There are three types of settings:

  • Attribute - A value in the config.xml file
  • config.d file - A value in a file stored in the config.d directory
  • users.d file - A value in a file stored in the users.d directory

Setting an attribute in the config.xml file

The most straightforward kind of setting is an attribute:

Setting an attribute

Figure 2 - The dialog for setting an attribute

The attribute is added to the config.xml file. The Name is the name of the XML element that contains the value. Note that if the XML element is contained in other elements, you need to specify the element’s entire path below the root <clickhouse> element. In the example above, the name logger/level stores the value in config.xml like this:

<clickhouse>
  <logger>
    <level>
      debug
    </level>
  </logger>
</clickhouse>

Setting a value in a file in the config.d directory

The ACM UI makes it easy to create a file with configuration data in the config.d directory. The values defined in that file become part of the ClickHouse system configuration. To set a value, enter a filename and its contents:

Setting a config.d value

Figure 3 - Setting a value in the config.d directory

This example writes the XML in the text box to the file config.d/query_log.xml. When applied to the ClickHouse cluster, logging settings for the query_log table in the system database are updated. Specifically, the retention period for the table is now 90 days (the default is 30).

The value of the Filename field must include an extension. If the contents of the text box are XML, it must be well-formed, and the root element must be either <yandex> or <clickhouse>.

Setting a value in a file in the users.d directory

Similar to the config.d directory, the ACM UI makes it easy to create a file with configuration data in the users.d directory. The values defined in that file become part of the ClickHouse system configuration. To set a value, enter a filename and its contents:

Setting a users.d value

Figure 4 - Setting a value in the users.d directory

This example contains an XML document that defines new users. If you wanted to define multiple users for your ClickHouse cluster, you could go through the ACM UI and create each one individually. However, it might be simpler and faster to define all the users in XML and use this dialog to create the file users.d/myusers.xml. When the new setting is applied to the ClickHouse cluster, there will be new users based on the data in the XML file.

The value of the Filename field must include an extension. As with config.d values, if the contents are XML, it must be well-formed, and the root element must be either <yandex> or <clickhouse>.

Editing a setting

When you click the vertical dots icon next to a setting name, you can click the Edit menu to, well, edit that setting. The dialogs to edit a setting are exactly the same as the dialogs in Figures 2, 3, and 4 above.

Deleting a setting

Clicking the Delete button in the menu next to a setting name brings up the Delete Setting confirmation dialog:

Deleting a setting

Figure 5 - Delete Setting configuration dialog

Click OK to delete the setting.

Example: Configuring protobuf schema

As an example, we’ll look at how to use protobuf in your ClickHouse cluster. You need to add two settings:

  1. The attribute format_schema_path
  2. The file events.proto in users.d

To get started, click the button, then create the format_schema_path attribute and set its value to /etc/clickhouse-server/users.d/:

Setting the format_schema_path attribute

Figure 12 - Adding the format_schema_path attribute

Click OK to create the attribute. Next, click the button again and create a file in users.d named events.proto with the contents syntax = "proto3":

Creating the events.proto file

Figure 13 - Defining the events.proto file

Click OK to create the setting.

Working with environment variables

To work with environment variables, click the Environment Variables tab in the Settings View:

Environment Variables tab in the Settings view

Figure 6 - The Environment Variables tab in the Settings View

We’ll look at the three environment variables listed in Figure 6 as an example. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY define the credentials used to access resources. S3_OBJECT_DISK_PATH is a simple string.

Adding an environment variable

Clicking the button at the top of the view opens the Environment Variable Details dialog:

Environment Variables Details dialog

Figure 7 - Using Value in the Environment Variable Details dialog

The fields in the dialog are:

  • Name - The name of the environment variable. The name can contain only letters [a-zA-Z], numbers [0-9], underscores [_], dots [.], and dashes [-], it can’t start with a digit, and it can’t be more than 50 characters long.
  • Value - The value of the variable. This field is mutually exclusive with the Value From field.
  • Value From - The name of a key inside a Kubernetes secret whose value should be used as the value for this environment variable. This field is mutually exclusive with the Value field.

In Figure 7 above, we’re using the Value field to define the string object_disks/demo/github as the value of the S3_OBJECT_DISK_PATH variable. Using the Value From field is slightly more complicated:

Environment Variables Details dialog

Figure 8 - Using Value From in the Environment Variable Details dialog

The format for Value From is secret-name/secret-key. In the example above x, the value clickhouse-data/clickhouse_data_s3_access_key refers to the first key in this secret:

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-data
type: Opaque
data:
  clickhouse_data_s3_access_key: YXNkZmtqaGJhc2Rma2poYXNrYmpoYXNkZmtqaA==
  clickhouse_data_s3_secret_key: YXNncXd0c2Zid2FlcmFzbGtkamZobGFrc2poZg==

Continuing our example, when using S3 for data storage in ClickHouse, the configuration data must include the <access_key_id> and <secret_access_key> elements. With the contents of the keys inside the Kubernetes secret defined as environment variables, they can be used in ClickHouse configuration files via the from_env attribute:

<access_key_id from_env="AWS_ACCESS_KEY_ID"></access_key_id>
<secret_access_key from_env="AWS_SECRET_ACCESS_KEY"></secret_access_key>

With this technique, anyone with access to the cluster can use the values from the Kubernetes secret, but no one can get the actual values.

Editing an environment variable

To edit an environment variable, click the vertical dots icon next to the variable name and select Edit from the menu. That takes you to the Environment Variables Details dialog seen in Figures 7 and 8 above.

Deleting an environment variable

Clicking the vertical dots icon next to the variable name and selecting Delete takes you to the Delete Environment Setting dialog:

Delete Environment Settings dialog

Figure 9 - The Delete Environment Setting dialog

Setting the Startup Mode

The Startup Mode menu lets you set how the cluster responds to changes in its settings:

Startup mode

Figure 10 - The startup modes for a ClickHouse cluster

The startup modes affect how Altinity Cloud Manager starts your ClickHouse cluster when you apply your new settings. The two modes are:

  • Direct mode - If something goes wrong during startup, ACM will retry the startup several times before giving up.
  • Troubleshooting mode - If something goes wrong during startup, ACM will not try to restart it if it fails. This is useful for debugging any problems with your cluster’s updated configuration.

Setting the Startup Time

Depending on the number of tables and the amount of data in your ClickHouse cluster, it may take longer than normal to start. That means it’s possible that the Kubernetes cluster hosting your ClickHouse cluster will delete and restart the pods needed to run ClickHouse before ClickHouse can start. For that reason, you can define a startup time, which is the number of seconds the Kubernetes cluster should wait for your ClickHouse cluster to start.

Startup mode

Figure 11 - The startup time parameter

If your ClickHouse cluster fails to start, you can check the ClickHouse Logs tab in your cluster’s Logs view for details. See the Cluster logs documentation for more information.

Click the button to set the startup time parameter. Note: this button only, uh, applies to the startup time parameter. It does not apply any changes you’ve made to settings or environment variables.

Resetting everything

The button restores all the standard settings to their default values. Any additional settings you have configured are deleted.

Configuring user settings profiles

Profiles allow you to give a name to a group of user settings, then easily apply those settings to a user.

Configuring a profile

Figure 14 - The Profiles view

Clicking the button takes you to the Profile Details dialog:

Defining a profile

Figure 15 - The Profile Details dialog

To edit the settings in the profile, click Edit Settings on the right-hand side of the list of profiles as shown in Figure 14 above. You’ll see a list of all the settings in this profile:

Defining a profile

Figure 16 - The Profile Settings list

Clicking the button takes you to the Profile Setting Details dialog:

Details of a profile setting

Figure 17 - The Profile Setting Details dialog

Clicking the down arrow displays a drop-down menu that lists all the settings. Selecting any setting in the list updates the dialog with a hint for that setting:

Details of a profile setting

Figure 18 - An different hint for a different setting

In the Profile list, clicking the vertical dots icon lets you either Edit or Delete a profile. Clicking Edit on the vertical dots menu takes you back to the Profile Details dialog shown in Figure 15 above. As you would expect, the delete button takes you to the Delete Profile dialog:

Delete a profile

Figure 19 - The Delete Profile dialog

Managing users

Depending on your account’s privileges, you may be able to add, edit, or delete users for the cluster. The Users panel lists all the users, their access, where their accounts are defined, and their profiles:

Edit a setting

Figure 20 - The Users View

In the display here, the lock icon next to the altinity and datadog user accounts mean that their access and privileges can’t be changed.

Clicking the button lets you add a new user:

Edit a setting

Figure 21 - The User Details dialog

Field details

  • Login - The username.
  • Password - The passwords have to be 12 characters long, and they have to match. The OK button is disabled until the password fields are correct.
  • Databases - A comma-separated list of databases this user is allowed to access. If left blank, the user has access to all databases.
  • Profile - The user’s profile. See Configuring user settings profiles above for details on working with profiles.
  • Access Management - If selected, this user will be able to create, delete, and modify user accounts via SQL statements. This is useful for users who may not have access to the ACM UI.

As you would expect, clicking the vertical dots icon next to a username lets you edit or delete that user. The edit dialog is identical to the User Details dialog shown in Figure 21 above. The delete dialog is straightforward:

Delete a user

Figure 22 - The Delete User dialog

Configuring storage

Clicking the Storage menu item displays the list of volumes for your cluster:

Edit a setting

Figure 23 - The Volumes view

There are several buttons at the top of the display:

Edit a setting
  • MODIFY VOLUME - Lets you make changes to the selected volume.
  • ADD VOLUME - Lets you add another volume to your cluster.
  • FREE VOLUME - Moves all data from the selected volume to non-cordoned volumes in the cluster.
  • CORDON VOLUME - Changes the selected volume’s status to cordoned. A cordoned volume will not receive any new data; cordoning a volume is the first step towards removing it.
  • REMOVE VOLUME - Removes the selected volume from the cluster.
  • STORAGE POLICY - Lets you change the storage policy for the cluster. Available policies are JBOD (just a bunch of disks) or Tiered.

We’ll cover these options next.

Modifying a volume

Selecting a volume and clicking the button lets you change the properties of the selected volume. At a minimum, this allows you to change the type of disk and its size:

Edit a setting

Figure 24 - The Modify Volume dialog

Clicking the down arrow icon displays a menu of available disk types based on the cloud provider hosting your ClickHouse cluster. You can also change the size of the volume.

The dialog may have other options based on your cloud provider. For example, if your ClickHouse cluster is hosted on AWS, you can change the throughput of the volume:

Changing a volume's throughput

Figure 25 - Setting throughput for an AWS volume

Finally, if your storage policy is JBOD, you’ll get a warning message if the volume type you’ve selected is different from the other volumes in your cluster:

Warning message for JBOD volumes

Figure 26 - The JBOD warning message for mixed volume types

Using different types of volumes with a JBOD policy will give inconsistent performance, as the different volumes may not have the same capabilities.

Adding a volume

Clicking the button lets you add another volume to your cluster:

Adding a volume

Figure 27 - The Add New Volume dialog

Be aware that the size of each volume must be at least 350 GB to use multiple volumes. As with modifying a volume, if your storage policy is JBOD, you’ll get a warning message if the new volume is of a different type from the other volumes in your cluster:

Warning message for JBOD volumes

Figure 28 - The JBOD warning message for mixed volume types

Click SAVE to add the new volume. It will appear in the list of volumes.

Freeing a volume

Selecting a volume and clicking the button moves all data from the selected volume to non-cordoned volumes in the cluster. You must first cordon the volume for the FREE VOLUME button to be enabled. You’ll be asked to confirm that you want to free the volume:

Warning message for JBOD volumes

Figure 29 - The Free Volume dialog

When all data is moved off of this volume, the REMOVE VOLUME button will become active.

Cordoning a volume

Selecting a volume and clicking the button cordons the volume, which means no new data will be written to that volume. Clicking the button changes its text to UNCORDON VOLUME, which reverses the operation. A cordoned volume can be freed, which moves all data from the volume to non-cordoned volumes in the cluster.

Removing a volume

If the selected volume has no data, the button will be active. To remove a volume, you must cordon it, which means no new data will be written to it, then free the volume, which moves any data on the volume to non-cordoned volumes. As you would expect, clicking the button gives you a confirmation message:

Warning message for JBOD volumes

Figure 30 - The Remove Volume dialog

Click OK to remove the volume.

Setting the storage policy

Clicking the button lets you set the storage policy. There are two options: JBOD (just a bunch of disks) and Tiered (hot and cold data are stored separately). If the storage policy is JBOD, the dialog is simple:

Edit a setting

On the other hand, Tiered storage is more complicated. Tiered storage is useful, for example, in a hot/cold architecture. You could keep your newest data in faster, more expensive storage, while moving older data to a slower, cheaper volume.

In ClickHouse you can define a Time-To-Live (TTL) period for data. The most common use of TTL is to delete old data that is no longer needed, but ClickHouse also lets you use TTL to move data from one volume to another when it reaches a certain age. To make the most of Tiered storage, you’ll need to define your own TTL properties.

See Manage Data with TTL in the ClickHouse documentation for all the details on TTL.

The dialog for Tiered storage has an additional field:

Edit a setting

The Move factor lets you define when the cluster should move data to alternate storage. The default value is 0.1, which means data is moved when the amount of available space on a volume is less than 10%. In the example here, the move factor is 0.2, which sets the threshold to 20%.

Changing the storage policy may result in data being moved between volumes. That can impact performance of running applications. For that reason, you’re asked to confirm any changes to the policy:

Edit a setting

Click OK to change the storage policy.

Configuring connections

The Connection Configuration dialog makes it easy to configure your ClickHouse cluster’s connections to the world:

Edit a setting

Field details

Endpoint

This is the endpoint of your cluster. The value here is your cluster name combined with an Altinity domain.

Alternate Endpoints

Defining an Alternate Endpoint

You can define alternate endpoints for your cluster. The name of the alternate endpoint can contain lowercase letters, numbers, and hyphens. It must start with a letter, and it cannot end with a hyphen.

Contact Altinity support to set up an alternate endpoint.

VPC Endpoint Enabled

Contact Altinity support to set up a VPC endpoint.

IP restrictions

If enabled, only ClickHouse applications or clients coming from specific IP addresses are allowed to connect to the cluster. You can add other addresses to the list, including ranges of IP addresses in CIDR format.

Disabling IP restrictions means that any ClickHouse application or client can connect to your ClickHouse cluster from any IP address.

NOTE: This restriction only applies to ClickHouse applications or clients. Anyone with the proper credentials can access the Altinity Cloud Manager UI from any IP address.

Protocols

Port 9440 enables the ClickHouse binary protocl, and port 8443 enables HTTP connections.

Datadog Integration

You can use Datadog to monitor your ClickHouse cluster. The Datadog options are only enabled if your cluster’s environment is enabled for Datadog support. See the section Enabling Datadog at the environment level for the details. Be aware that you must have the appropriate privileges to edit an environment’s settings, so you may need to contact your administrator.

Zone Awareness

When Zone Awareness is enabled, Altinity.Cloud keeps traffic between client connections and your ClickHouse cluster in a single availability zone whenever possible. This allows you to avoid cross-zone hops.

However, if all of your client connections come from a single zone, this feature will route all requests to a single ClickHouse node. In that case, turning Zone Awareness off will ensure that your load balancer will distribute requests across all the nodes in the cluster.

Setting an activity schedule

The Activity Schedule settings let you control when your ClickHouse clusters run. Altinity.Cloud does not bill you for compute resources or support for non-running clusters, so shutting down ClickHouse clusters that don’t need to run constantly can cut your costs. NOTE: This cost saving does not apply to storage and backups.

The available activity schedules are:

  • ALWAYS ON - the cluster is always active
  • STOP WHEN INACTIVE - the cluster shuts off after some number of hours of inactivity
  • ON SCHEDULE - the cluster is active only on certain days of the week as well as certain times during each day.

If a cluster’s status is STOP WHEN INACTIVE or ON SCHEDULE, a clock icon appears along with the cluster name in the panel view:

Cluster with a schedule

Figure 1 - A cluster with a schedule

We’ll look at the three schedule options now.

ALWAYS ON

Used for mission-critical ClickHouse clusters that must run 24/7.

Figure 3 – The ALWAYS ON Activity Schedule setting

Click ALWAYS ON and click CONFIRM to save. Simple as that.

STOP WHEN INACTIVE

Used to turn off non-critical clusters after a set number of hours of inactivity. For non-running clusters, Altinity.Cloud does not bill you for compute resources or support, although charges for storage and backups continue.

Figure 4STOP WHEN INACTIVE Activity Schedule setting

Click STOP WhEN INACTIVE, then select the number of hours by typing in a value or by using the up and down arrows. Click CONFIRM to save. A clock icon will appear next to the cluster name in the Clusters Dashboard.

NOTE: If a cluster is stopped for more than 30 days, you’ll get a warning message suggesting that you delete the cluster to avoid storage and backup charges for the unused cluster:

Figure 5 - Unused cluster warning message

ON SCHEDULE

Sets the days of the week your ClickHouse clusters will run, with the option of defining From and To times (expressed as GMT) that your clusters will run for each day.

The schedule for maddie-byok below defines the following settings:

  • On Tuesday and Thursday, the cluster is active from 8:00 PM to 5:00 PM GMT.
  • On Monday, Wednesday, and Friday, the cluster is active all day.
  • On Saturday and Sunday, the cluster is down.

Figure 6 ON SCHEDULE Activity Schedule setting

Click CONFIRM to save. A clock icon will appear next to the cluster name in the Clusters Dashboard.

Configuring backups

You can define a schedule for creating backups of your cluster. The Backup Settings dialog lists the current schedule:

Edit a setting

There are five options to define the Period when backups should occur:

  • Monthly - Define the day of the month
  • Weekly - Define the day of the week
  • Daily - Define the time of day
  • Every 6 hours - Backups occur every six hours
  • Every hour - Backups occur every hour.

NOTE: All times are defined in GMT.

In addition to defining the period, you can also define the number of backups to keep. The default is seven.

The button lets you define multiple schedules. For example, if you only want backups to occur on Friday and Saturday, create two Weekly schedules, one for Friday and one for Saturday. You can define up to three schedules.

5 - Cluster Explorer Guide

Working directly with a ClickHouse cluster

The Altinity Cloud Manager UI provides the Cluster Explorer, a panel that lets you run queries against your ClickHouse databases, work with database schemas, look at processes running inside the cluster, and use a number of useful DBA tools.

If you’re looking for help on a particular aspect of the Cluster Explorer, you can skip ahead to any of the tasks:

Panel buttons and menus

The top of every tab looks like this:

Menus and buttons in the Cluster Explorer

Figure 1 - Cluster Explorer menus and buttons

The functions of the buttons and menus are:

  • - The back button. Takes you back to the cluster view for this cluster.
  • CLUSTER menu - Clicking this menu displays a dropdown list of all of the clusters in this environment. Whatever you do in the Cluster Explorer will be done on the selected cluster.
    The Cluster menu
  • RUN DDLs ON CLUSTER - If selected, Distributed DDL Queries are run on the cluster.
  • NODE menu - Lets you select the scope of anything you do in the Cluster Explorer. For the maddie-byok cluster, the menu looks like this:
    Menus and buttons in the Cluster Explorer
    Selecting Any means everything you do will be executed on a single node selected by the load balancer. All executes everything on every node. Beyond those two choices, you can select a node from the list of all nodes in this cluster.
  • IMPORT DATASET - lets you import a sample dataset into your ClickHouse cluster. See Importing a dataset below.
  • - The refresh button. Refreshes the display, as you would expect.

The Query tab

When you open the Cluster Explorer, you’ll be on the Query tab:

Query Page

Figure 2 - The Query tab

The text area at the top of the panel lets you run SQL statements by clicking the EXECUTE button. (You can also keep your hands on the keyboard by typing Ctrl or CMD + Enter.)

When you execute a query, the results appear under the text area:

Query results

Beneath the text area are helpful notes with links to ClickHouse documentation and a couple of example queries.

Above the text area, the left and right arrows let you move through the queries you’ve executed. The button brings up the Select User dialog:

The Select User dialog

This lets you select the user (and their permissions) to run the query. The down arrow lets you choose from the users defined for this cluster. If your account has the necessary permissions, you can also click the link in Manage ClickHouse users here to go directly to the user management panel.

The Schema tab

The Schema tab lets you view the databases and tables in your cluster. You can then select a table and see its details.

The schema tab

Figure 3 - The Schema tab

You can click the Show system tables checkbox to see the system tables as well. All of the table names are links; click on a table name to see the Table Details dialog, which has four subtabs:

The Table Description tab

The Table Description tab has a table of the database’s properties. There are dozens of them, some of which are shown here:

Table description column

The Table Schema tab

The Table Schema tab displays the table’s schema as an SQL statement:

Table schema column

Click the Copy to Clipboard link to put the schema on the clipboard.

The Sample Rows tab

The Sample Rows tab shows randomly selected rows from the table:

Sample rows column

The Column Compression tab

Finally, the Column Compression tab shows the details of the columns in the table and how much they have been compressed:

Column compression column

The Workload tab

The Workload tab shows you what’s happening in the system now:

Workloads page

Figure 4 - The Workload tab

It has four subtabs:

The Running Queries tab

The Running Queries tab shows the details of all the currently running queries:

CUrrently running queries

If you select one or more queries, the button becomes active.

The Query Stats tab

The Replication Queue tab

The Mutations tab

The Processes view displays the currently running processes on a cluster or node.

To view the processes view:

  1. Select Explore from either the Clusters View or the Clusters Detail Page.

  2. Select Processes from the top tab.

  3. Select the following node options:

    Select node for query.
    1. Any: Any node selected from the Zookeeper parameters.
    2. All: Run the query against all nodes in the cluster.
    3. Node: Select a specific node to run the query against.

The following information is displayed:

  • Query ID: The ClickHouse ID of the query.
  • Query: The ClickHouse query that the process is running.
  • Time: The elapsed time of the process.
  • User: The ClickHouse user running the process.
  • Client Address: The address of the client submitting the process.
  • Action: Stop or restart a process.

The DBA Tools tab

DBA Tools page

Figure 5 - The DBA Tools tab

Importing a dataset

The Cluster Explorer makes it easy to import a sample dataset. When you click the IMPORT DATASET button, you’ll see this dialog:

Select a dataset to import

Select the dataset you want and click NEXT. Next you’ll be asked about the database schema:

Use the database schema

In this example, three tables will be created when you click IMPORT. You can expand each of the entries to look at the database schema and change engine parameters if you want.

Click IMPORT to import the data. Notice that even though there’s another step in the wizard, clicking the button in the lower right (as you normally would) starts the import immediately.

6 - Monitoring a Cluster

How to monitor and manager your clusters’ performance.

There are several ways to monitor your ClickHouse cluster:

Grafana dashboards

Altinity.Cloud uses Grafana as its default monitoring tool. You can access Grafana from the Monitoring section of a cluster panel:

Cluster Monitoring View

Figure 1 - The Monitoring section of the cluster panel

Clicking the View in Grafana link displays the following menu:

Cluster Monitoring menu

Figure 2 - The Grafana monitoring menu

We’ll go through those menu items next. If you’d like to jump to a particular Grafana view, click any of these links:

The Cluster Metrics view

The Cluster Metrics view

Selecting Cluster Metrics opens this Grafana dashboard in another browser tab:

The Cluster Metrics view

Figure 3 - The Cluster Metrics dashboard

Cluster metrics include things like the number of bytes and rows inserted into databases in the ClickHouse cluster, merges, queries, connections, and memory / CPU usage.

The System Metrics view

The System Metrics view

Selecting System Metrics opens this Grafana dashboard in another browser tab:

The System Metrics view

Figure 4 - The System Metrics dashboard

System metrics include things like CPU load, OS threads and processes, network traffic for each network connection, and activity on storage devices.

The Queries view

The Queries view

Selecting Queries opens this Grafana dashboard in another browser tab:

The Queries view

Figure 5 - The Queries dashboard

The Queries dashboard includes information about your most common queries, slow queries, failed queries, and the queries that used the most memory.

The Logs view

The Logs view

Selecting Logs opens this Grafana dashboard in another browser tab:

The Logs view

Figure 6 - The Logs dashboard

The Logs dashboard shows all of the log messages as well as the frequency of messages over time. You can add a query to the Logs visualization to filter the view for particular messages.

Cluster alerts

You can define cluster alerts to notify users when certain events occur. You can access alerts from the button on a cluster panel:

Cluster Alerts

Figure 7 - The ALERTS item in the cluster panel

Clicking on the button displays the Cluster Alerts dialog:

Cluster Alerts dialog

Figure 8 - The Cluster Alerts dialog

Enter the email address of the user who should be alerted when particular events occur. For each event, you can send them a popup message in the Altinity Cloud Manager UI and/or send an email.

The different types of alerts are:

  • System Alerts: Triggered by a significant system event such as a network outage.
  • ClickHouse Version Upgrade: Triggered by an update to the version of ClickHouse installed in the cluster.
  • Cluster Rescale: Triggered when the cluster is rescaled.
  • Cluster Stop: Triggered when some event has caused the cluster to stop running. This could be some event that caused a problem, a user stopping the cluster, or a stop caused by your cluster uptime settings.
  • Cluster Resume: Triggered when a previously stopped cluster is restarted.

A popup alert appears at the top of the ACM UI:

Cluster resumed alert

Figure 9 - An alert for a resumed cluster

Health checks

You can check the health of a cluster or node from the ACM. For clusters, there are two basic checks: the health of the nodes in the cluster and the health of the cluster itself. The health checks for a node are whether the node is online and, as you would expect, the health of the node itself.

Cluster health checks

Cluster health checks appear near the top of a Cluster view. For example, here is the panel view of a cluster with the two health checks:

Cluster Alerts

Figure 10 - A cluster panel with its two health checks

The health check at the top of the panel indicates that 2 of the 2 nodes in the cluster are online:

All cluster nodes online

Clicking on this green bar takes you to the detailed view of the cluster. From there you can see the individual nodes and their status.

The second health check indicates that 6 of the 6 cluster health checks passed:

All cluster checks passed

Clicking on this green bar shows you the health check dialog:

Details of the cluster health checks

Figure 11 - The Health Checks dialog

The cluster health checks are based on six SELECT statements executed against the cluster and its infrastructure. The six statements look at the following cluster properties:

  • Access point availability
  • Distributed query availability
  • Zookeeper availability
  • Zookeeper contents
  • Readonly replicas
  • Delayed inserts

Clicking any of the checks shows the SQL statement used in the check along with its results:

Details of the access point check

Figure 12 - Details of a particular cluster health check

Depending on the cluster’s status, you may see other indicators:

Health check Meaning
A cluster or node that is restarting
The cluster or node is rescaling
A cluster or node that is being terminated
The cluster or node is being terminated
A cluster or node that is stopped
The cluster or node is stopped

Node health checks

The basic “Node is online” check appears next to the node name in the Nodes view of the cluster:

The Nodes view

Figure 13 - The Nodes view of a cluster

Opening the Node view shows more details:

Node health

Figure 14 - The health checks for a single node in the cluster

The first health check indicates that the node is online:

Node is online

The second health check indicates that 5 of the 5 node health checks passed:

All node checks passed

Clicking on this green bar takes you to a more detailed view of the health checks and their results, similar to Figure 11 above.

Cluster logs

You can look at a variety of logs by clicking the button on a cluster panel:

Cluster Logs

Figure 15 - The LOGS item in the cluster panel

You’ll see this panel:

Cluster Logs view

Figure 16 - The Logs panel

Notice at the top of the panel that there are five different logs available:

  • ACM Logs: messages issued by the Altinity Cloud Manager
  • ClickHouse Logs: messages issued by ClickHouse itself
  • Backup Logs: messages related to system backups
  • Operator Logs: messages issued by Altinity's ClickHouse Kubernetes operator
  • Audit Logs: messages related to significant system events initiated by a user

The upper right corner of the Logs panel includes the Download Logs button and the Refresh button.

Notifications

You can see your notifications by clicking on your username in the upper right corner of Altinity Cloud Manager:

Cluster Lock button

The Notifications menu item lets you view any notifications you have received:

The Notification History panel

Figure 17 - The Notification History dialog

Here the history shows a single message. The text of the message, its severity (Info, News, Warning, or Danger), and the time the message was received and acknowledged are displayed. The meanings of the message severities are:

  • - Updates for general information
  • - Notifications of general news and updates in Altinity.Cloud
  • - Notifications of possible issues that are less than critical
  • - Critical notifications that can effect your clusters or account