This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Administrator Guide

How to manage Altinity.Cloud.

Altinity.Cloud allows administrators to manage clusters, users, and keep control of their ClickHouse environments with a few clicks. Monitoring tools are provided so you can keep track of everything in your environment to keep on top of your business.

1 - Access Control

How to control access to your organizations, environments, and clusters.

Altinity.Cloud provides role based access control. Depending the role granted to an Altinity.Cloud Account, they can assign other Altinity.Cloud accounts roles and grant permissions to access organizations, environments, or clusters.

1.1 - Role-Based Access and Security Tiers

Altinity.Cloud hierarchy and role-based access.

Introduction

Access to ClickHouse data hosted in Altinity.Cloud is controlled through a combination of security tiers and account roles. This allows companies to tailor access to data in a way that maximizes security while still allowing ease of access.

Security Tiers

Altinity.Cloud groups a set of clusters together in ways that allow companies to provide Accounts access only to the clusters or groups of clusters that they need.

Altinity.Cloud groups clusters into the following security-related tiers:

Security Tiers
Figure 1 - Security tier showing the relationship between an organization, environment, cluster, and ClickHouse database nodes.

Hierarchy

  • ClickHouse databases and tables live inside clusters.
  • Clusters contain ClickHouse databases and manage access.
  • Environments contain one or more clusters.
  • Organizations contain one or more environments.

Security Tier

Account access is controlled by assigning an account a single role and a security tier depending on their role.
A single account can be assigned to:

  • A single Organization
  • One or multiple Environments
  • One or multiple Clusters within an environment

Account Roles

The actions that can be taken by Altinity.Cloud accounts are based on the role they are assigned.
The following roles and their actions based on the security tier are detailed in Table 1.

Role Descriptions

  • orgadmin manages the Organization, including all user accounts and Environment settings, and has full access to any cluster.
  • envadmin is a member of an Organization and can edit assigned Environments and has full access to any cluster.
  • envuser is a member of an Organization and has full access to specified clusters.
  • envsupport is a member of an Organization and has read access to specified environments and clusters.
  • grafanauser is a member of an Organization and has read access to specified environments.
  • billing can access the billing page only. From there they can view invoices and update payment details.

Roles and Permissions
Table 1 - Role names and their levels of access to Environments and Clusters.

Organization Example

The following example in Table 2 shows an Organization called HappyDragon that shows how Accounts and Role assignments are configured. Role names are also shown in Figure 1. The account roles are tied into the security tiers and allow an account to access multiple environments and clusters depending on what type of tier they are assigned to.

Account Title Role Organization Accounts Billing Environments Clusters
mary Administrator orgadmin HappyDragon all all Access to all Env. all
jessica Operations envadmin HappyDragon n/a n/a HappyDragon_Prod
HappyDragon_Dev
all
peter Developer envadmin HappyDragon n/a n/a HappyDragon_Dev all
paul Marketing envuser HappyDragon marketing n/a HappyDragon_Prod marketing

Table 2 - Accounts and their roles and security tiers.

Account and Roles

Mary (Administrator, Role: orgadmin)

Mary is the orgadmin role, which has the highest level of access in this example.

  • Has full access to the organization account
  • Can create and manage access for other users
  • Can create and manage new environments
  • Can create and manage new clusters

Jessica (Operations, Role: envadmin)

  • Has read access (but not write or delete access) to both Dev and Prod environments
  • Has full access to read, write and delete clusters in both environments

Peter (Developer, Role: envadmin)

  • Has read access (but not write or delete access) to the Dev environment
  • Has full access to read, write and delete any cluster in the Dev environment

Paul (Marketing user, Role: envuser)

  • Has read, write and delete access to the cluster marketing in the environment HappyDragon_Prod

1.2 - Account Management

How to manage Altinity.Cloud accounts.

Altinity.Cloud accounts with the role orgadmin are able to create new Altinity.Cloud accounts and associate them with organizations, environments, and one or more clusters depending on their role. For more information on roles, see Role Based Access and Security Tiers.

Account Page

The Account Page displays all accounts assigned to the same Organization and Environments as the logged in account.

For example: the accounts mario, luigi, and peach and todd are members of the organizations MushroomFactory and BeanFactory as follows:

Account Role Organization: MushroomFactory Organization: BeanFactory
peach orgadmin *  
mario orgadmin   *
luigi envuser   *
todd envuser *  
  • peach will be able to see their account and todd in the Account Page, while accounts mario and luigi will be hidden from them.
  • mario will be able to see their account and luigi.

Access Accounts

To access the accounts that are assigned to the same Organizations and Environments as the logged in user with the account role orgadmin:

  1. Login to Altinity.Cloud with an account granted the orgadmin role.
  2. From the left navigation panel, select Accounts.
  3. All accounts that are in the same Organizations and Environments as the logged in account will be displayed.

Account Details

Accounts have the following details that can be set by an account with the orgadmin role:

  1. Common Information:
    1. Name: The name of the account.
    2. Email (Required): The email address of the account. This will be used to login, reset passwords, notifications, and other uses. This must be a working email for these functions to work.
    3. Password: The password for the account. Once a user has authenticated to the account, they can change their password.
    4. Confirm Password: Confirm the password for the account.
    5. Role (Required): The role assigned to the account. For more information on roles, see Role Based Access and Security Tiers.
    6. Organization: The organization assigned to the account. Note that the orgadmin can only assign accounts the same organizations that the orgadmin account also belongs to.
    7. Suspended: When enabled, this prevents the account from logging into Altinity.Cloud.
  2. Environment Access:
    1. Select the environments that the account will require access to. Note that the orgadmin can only assign accounts the same environments that the orgadmin account also belongs to.
  3. Cluster Access:
    1. This is only visible if the Role is set to envuser. This allows one or more clusters in the environments the new account was assigned to in Environmental Access to be accessed by them.
  4. API Access:
    1. Allows the new account to make API calls from the listed domain names.

Account Actions

Create Account

orgadmin accounts can create new accounts and assign them to the same organization and environments they are assigned to. For example, continuing the scenario from above, if account peach is assigned to the organization MushroomFactory and the environments MushroomProd and MushroomDev, they can assign new accounts to the organization MushroomFactory, and to the environments MushroomProd or MushroomDev or both.

To create a new account:

  1. Login to Altinity.Cloud with an account granted the orgadmin role.

  2. From the left navigation panel, select Accounts.

  3. Select Add Account.

  4. Set the fields as listed in the Account Details section.

    New User Settings
  5. Once all settings are completed, select Save. The account will be able to login with the username and password, or if their email address is registered through Google, Auth0.

Edit Account

  1. Login to Altinity.Cloud with an account granted the orgadmin role.
  2. From the left navigation panel, select Accounts.
  3. From the left hand side of the Accounts table, select the menu icon for the account to update and select Edit.
  4. Update the fields as listed in the Account Details section.
  5. When finished, select Save.

Suspend Account

Instead of deleting an account, setting an account to Suspended may be more efficient to preserve the accounts name and other settings. A suspended account is unable to login to Altinity.Cloud. This includes directly logging through a browser and API calls made under the account.

To suspend or activate an account:

  1. Login to Altinity.Cloud with an account granted the orgadmin role.
  2. From the left navigation panel, select Accounts.
  3. From the left hand side of the Accounts table, select the menu icon for the account to update and select Edit.
    1. To suspend an account, toggle Suspended to on.
    2. To activate a suspended account, toggle Suspended to off.
  4. When finished, select Save.

Delete Account

Accounts can be deleted which removes all information on the account. Clusters and environments created by account will remain.

To delete an existing account:

  1. Login to Altinity.Cloud with an account granted the orgadmin role.
  2. From the left navigation panel, select Accounts.
  3. From the left hand side of the Accounts table, select the menu icon for the account to update and select Delete.
  4. Verify the account is to be deleted by selecting OK.

1.3 - Integrating Okta into the Altinity.Cloud login page

How to set up Okta integration with Auth0 in Altinity.Cloud

Overview - Okta Integration

Altinity uses Auth0 so that customers who are already logged into other identity providers such as Google or Okta are automatically granted access to Altinity.Cloud.

The following diagram shows the Altinity login process using Auth0, plus adding Okta as discussed on this page.

  1. Logging in to Altinity.Cloud using a Login Email and Password.
  2. The Auth0 login link to use a 3rd party authenticator such as Google or Okta. (See Okta/Auth0 Altinity Integration)
  3. Using Okta allows previously authorized logged-in employees to gain immediate access to Altinity.Cloud.
Launch Cluster wizard screens

Figure 1 – Altinity integration of an Okta customer to Auth0.

Set-up Okta Integration

  1. Go to Okta Dashboard -> Applications -> Create App Integration
  2. Choose OIDC as Sign-in method and Web Application as Application type, then click Next
  3. Provide a name, and check all Grant type checkboxes
  4. Provide https://altinity.auth0.com/login/callback as Sign-in redirect URI
  5. Choose preferred option for Assignments
  6. Click Save and leave page open.

Provide Okta Client Details

Contact Altinity to add the Okta domain and Client ID to the Altinity.Cloud access list. Please provide the following:

  • The domain you want to sign in on the Okta side, e.g. somemail.com. Note: it should match your organization domain in Altinity.Cloud.
  • Okta Domain, e.g. somemail.okta.com
  • Okta Client ID

Okta/Auth0 Altinity Integration

These steps are for Altinity technical support to add an Okta connection to Auth0.


Setting up the Auth0 connection

  1. Go to Auth0 Dashboard -> Authentication -> Enterprise.
  2. Click Create (plus icon) next to OpenID Connect.
  3. Provide a name.
  4. Copy the Okta domain provided by a customer to Issuer URL.
  5. Copy the Client ID provided by a customer to the Client ID.
  6. Click Create.

Enabling the connection

  1. Go to Auth0 Dashboard -> Applications.
  2. Click the application you wish to use with the connection.
  3. Go to the Connections tab, find your newly created connection, and switch it on.

Testing the connection

  1. Go to Auth0 Dashboard -> Authentication -> Enterprise.
  2. Click OpenID Connect (not the plus sign, the text).
  3. Find the newly created connection.
  4. Click the three dots on the right -> Try.
    • You should be greeted with an Okta password prompt, or if there is a problem, an error is shown.

Enabling the button

  1. Go to Auth0 Dashboard -> Authentication -> Enterprise.
  2. Click OpenID Connect (not the plus sign, the text).
  3. Find the newly created connection and click its name.
  4. Go to the Login Experience tab.
  5. Check the Connection button -> Display connection as a button.
  6. Configure the Button display name and logo URL.
  7. Click Save.

Testing

  1. Go to the https://acm.altinity.cloud login page.
  2. Click Sign in with Auth0.
  3. A button for the new connection should be shown.
  4. Upon clicking the button, it should either ask for Okta credentials or log straight into the app.

Altinity blog post

The following Altinity blog post provides an in-depth discussion of adding Okta as an identity provider.

1.4 - Altinity Access to ClickHouse

Controlling Altinity personnel’s access to data and cluster administration

Introduction

Altinity Access settings allow Altinity.Cloud users to limit the level of access Altinity personnel have to customer ClickHouse data or administrative operations. Altinity.Cloud provides two types of limits:

  • Data Access - Control the ability of Altinity personnel to view or change data in ClickHouse tables.
  • Management Access - Control ability to change cluster configuration or perform administrative actions.

If you restrict access to data or management functions you may choose to lift them from time to time to allow Altinity support to diagnose problems or perform operations on your behalf. You can apply the restrictions again afterward.

Viewing and Changing Altinity Access Settings

Access the Cluster Dashboard view of any cluster. The ALTINITY ACCESS button appears in the upper right-hand side of the dashboard view.

Altinity Access button
Figure 1 - The Altinity Access Button in Cluster Dashboard

Button Colors

The button color indicates Altinity personnel’s access level to your ClickHouse clusters and data.

No access

No Access - Altinity personnel cannot use the ACM Query Browser or Schema Browser. They cannot look at data or schema. This is shown in Figure 1 above.

No access

System Access - Altinity personnel can use the ACM Query browser to query system tables and look at table definitions in the Schema Browser, but they cannot look at data. This setting provides a good balance between protecting data and providing the access required for quick support from Altinity. This is the default value.

No access

Read-Only Access - In addition to the above, Altinity personnel can use the ACM Query Browser to run SELECT statements that read data from any table.

No access

Full Access - In addition to the above, Altinity personnel can use the ACM Query Browser to run SQL statements that alter data.

Modifying Altinity access settings

Press the ALTINITY ACCESS button, whatever color it may be, to manage access settings for Altinity personnel.

Data Access Level settings

The four data access levels are at the top of the panel:

Altinity Access settings
Figure 2 - Altinity Access Dialog

Select the new access level and press CONFIRM to save it. The CONFIRM button is disabled until you enter a reason for the change. Changing settings requires an account with EnvUser role or higher.

Management Access settings

You may similarly enable or disable management access using the check boxes in the middle of the panel.

Enable Cluster Configuration Management

Checking this box allows Altinity personnel to perform any of the following cluster configuration operations:

  • Changing cluster configuration settings
  • Changing users or profiles
  • Setting connection configuration
  • Altering backup settings
  • Setting uptime schedules
  • Setting alerts

The above actions can cause your server to restart, alter user passwords, or change the information that you receive from Altinity.Cloud about your clusters. If the box is unchecked, only you can make these changes.

Enable Cluster Actions

Checking this box allows Altinity staff to perform any of the following cluster administration operations:

  • Creating new clusters
  • Rescaling clusters
  • Upgrading clusters
  • Restarting clusters

The above actions may cause your server to restart, behave differently for applications, or affect operating costs. If the box is unchecked, only you can make these changes.

2 - Integrating other tools with Altinity.Cloud

Connecting Altinity.Cloud to other tools

2.1 - Integrating Datadog

Connecting Datadog to your Altinity.Cloud environment

Datadog is a popular observability platform for monitoring applications and infrastructure. Datadog’s access to Altinity.Cloud is controlled at two levels:

  • The Environment level - To use Datadog, you must enable it for your environment.
  • The Cluster level - Once Datadog is enabled at the environment level, you can enable Datadog monitoring for individual clusters.

Enabling Datadog at the environment level

To enable Datadog at the environment level, click the Environments tab on the left. You’ll see the list of environments in your account.

Environment list

Figure 1 - Environment list

Click the radio button next to the environment you want to enable, then click the vertical dots icon next to it. Select Edit from the menu.

The Edit Environment menu

Figure 2 - The Edit Environment menu

In the Environment Details panel, click the Logs tab. Click the Turn On Datadog slider to turn Datadog on or off. When Datadog is turned on, you need to enter the API key from your Datadog account. (The Datadog API and Application Keys page has complete details on how to create and manage API keys.) Click the down arrow to select your nearest Datadog region.

Turning on Datadog

Figure 3 - Turning on Datadog

Once Datadog is turned on, the three options below the region allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.

See Datadog's ClickHouse integration documentation for the details on configuring Datadog and the ClickHouse metrics that Datadog can monitor.

Finally, click OK to save your changes.

Enabling Datadog at the cluster level

If you’re creating a new ClickHouse cluster and your environment is enabled for Datadog, the Datadog integration options are enabled on the Connection Configuration tab of the Launch Cluster Wizard. Simply click the checkboxes to send logs and/or metrics to Datadog. (If the environment isn’t enabled for Datadog, the Datadog section of the Connection Configuration tab is disabled.)

If your ClickHouse cluster has already been created, you can use the Cluster Settings dialog to enable Datadog. From the Clusters tab of the ACM, click the Configure button, then choose Connections from the menu:

Editing a cluster's connections

Figure 4 - Editing a cluster’s connections

In the Connection Configuration dialog, select the checkboxes in the Datadog integration section:

Enabling Datadog at the cluster level

Figure 5 - Enabling Datadog at the cluster level

The three options here allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.

Click CONFIRM to save your changes.

2.2 - Integrating Grafana

Connecting Grafana to your Altinity.Cloud environment

Grafana is a popular open-source observability platform for monitoring applications and infrastructure. We’ll look at connecting Altinity.Cloud to Grafana Cloud, although connecting to any Grafana instance works basically the same.

To connect your ClickHouse cluster to Grafana, you’ll need the connection details for your cluster. In the Clusters view, click the Connection Details link:

Getting connection details for the cluster

Figure 1 - A cluster panel

You’ll need the host name and the HTTP port:

Getting connection details for the cluster

Figure 2 - Connection details for the ClickHouse cluster

With those details, go ahead and log in to your Grafana Cloud account. Click the Grafana menu in the upper left corner, then select the Data sources item in the Connections section:

Connect a data source

Figure 3 - The create new data source menu

On the Data sources page, click the + Add new data source button in the upper right:

Connect a data source

Figur 4 - The Add new data source button

On the Add data source page, type click in the search bar, then click on the Altinity plugin for ClickHouse item:

Connect a data source

Figure 5 - Selecting the Altinity plugin for ClickHouse

To start, give your new data source a name at the top of the panel:

Connect a data source

Figure 6 - Naming your new data source

In the HTTP section of the page, enter https:// followed by the host and HTTP port from the ACM Connection Details page:

HTTP details for the new data source

Figure 7 - Entering the connection data for your ClickHouse cluster

Scroll down to the Auth section, select Basic auth, then enter your user ID and password:

Credentials for the new data source

Figure 8 - Entering the credentials for your ClickHouse cluster

NOTE: The user ID is the ID of your ClickHouse cluster, which is probably not the same as the ID of your Altinity.Cloud account. (admin is the default.)

To complete the connection, scroll to the bottom of the page and click the Save & test button:

Save and test the new data source

Figure 9 - The Save & Test button

If all goes well, you’ll get a Data source is working message from Grafana:

New data source is working

Figure 10 - A successfully created data source

If anything goes wrong, the dialog will contain an error message:

Connection to new data source failed

Figure 11 - An unsuccessfully created data source

From the Data source is working message, you can take either the Building a dashboard or the Explore view link. We’ll look at the Explore view; if you want to build a dashboard, you’ll use the Explore view to define a query against your ClickHouse cluster.

The Explore view starts with an empty query:

Connection to new data source failed

Figure 12 - The initial data source explorer view

The data source you just created is displayed at the top of the panel. Click the –database– and –table– fields to see a list of all of the tables and database available:

Connection to new data source failed

Figure 13 - Selecting a database

Here we’re selecting the system database. Clicking the –table– field displays all of the tables in the system database:

Connection to new data source failed

Figure 14 - Selecting a table within a database

We’ll select the query_log table. With the database and table selected, click the Go to Query button:

Connection to new data source failed

Figure 15 - The Go to Query button

You’ll see the query source and a graph of the results:

Connection to new data source failed

Figure 16 - The syntax of the query and its results

2.3 - Integrating Loki

Connecting Loki to your Altinity.Cloud environment

Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure.

2.4 - Integrating Prometheus

Connecting Prometheus to your Altinity.Cloud environment

Prometheus is popular open-source library used for event monitoring and alerting.

2.5 - Using an external S3 bucket for ClickHouse data

Control where your ClickHouse data is stored

For data sovereignty or other reasons, you might want to store your ClickHouse data in an external S3 bucket.

2.6 - Replicating Altinity.Cloud backups to S3

Configuring external S3 buckets for Altinity.Cloud backups

You can use external S3 buckets to store replicas of Altinity.Cloud backups. To configure them, you’ll need to contact us to get an Amazon ARN (Amazon Resource Name) that identifies the S3 bucket you’ll use.

Here are the steps to set everything up:

  1. Create the S3 bucket you want to use.
  2. Contact Altinity support and give us the bucket name. Support will use that bucket name to create the ARN.
  3. Set the ARN and bucket name in the following JSON, then use kubectl to apply it:
{
  "Id": "altinity-clickhouse-backup-for-s3-tenjin-prod",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "altinity-clickhouse-backup-for-s3-tenjin-prod",
      "Action": [
        "s3:GetBucketVersioning",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::tenjin-clickhouse-production",
      "Principal": {
        "AWS": [
          "arn:aws:iam::313342380333:user/clickhouse-backup-tenjin-prod"
        ]
      }
    },
    {
      "Sid": "altinity-clickhouse-backup-for-s3-tenjin-prod",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::tenjin-clickhouse-production/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::313342380333:user/clickhouse-backup-tenjin-prod"
        ]
      }
    }
  ]
}

3 - Altinity API Guide

How to use Altinity APIs

To work with Altinity API tokens and control their access, click your username in the upper right corner and select the My Account menu item:

The My Account panel has two API-related tabs, API Access and Anywhere API Access.

The API Access tab

This page lets you generate Altinity Cloud Manager API keys for your account. The API keys have the same access and privileges as your account. By default, each key is set to expire in 24 hours, although you can adjust the expiration time. (Be aware that the time is always in GMT.)

In Figure 1 below, there are two keys for this account:

Create Altinity.Cloud API Key

Figure 1 - Altinity Cloud Manager API keys for this account

In the API Keys section, working with keys is straightforward:

  • Use the button to add an Altinity.Cloud API key.
  • Use the copy icon to copy a key to the clipboard. Be aware that the API key may be too wide to display completely, so clicking the copy icon is more reliable than highlighting and copying the text directly.
  • Use the calendar icon to change the expiration time of the key.
  • Finally, use the trash can icon to delete a key.

The Allow Domains section let you restrict API calls to specific domains. This provides enhanced security by allowing API connections only from certain IP addresses. To specify multiple domains, separate them by commas or put each one on a separate line.

You can find documentation on the Altinity Cloud Manager API below.

The Anywhere API Access tab

Create Anywhere API Key

You can also create an API token to access the Altinity.Cloud Anywhere API. Be aware that an account can have only one Altinity.Cloud Anywhere API key.

Click the button to create a new key. Generating a new key disables your existing key. Click OK in the confirmation dialog to confirm that you want to create a new key.

After creating the token, you’ll see the new token in an entry field on the screen. Copy the token; you won’t be able to see it again. (You can copy the key with the icon.)

If you’re using the Altinity.Cloud Terraform provider, use this key as the api_token field in the provider section:

terraform {
  required_providers {
    altinitycloud = {
      source = "altinity/altinitycloud"
      version = "v0.1.2"
    }
  }
}

provider "altinitycloud" {
  # `api_token` can be omitted if ALTINITYCLOUD_API_TOKEN env var is set.
  api_token = "<TOKEN>"
}

See the Altinity.Cloud Terraform provider documentation for all the details. If you want to interact with the API directly, there’s a GraphQL console at anywhere.altinity.cloud/api/v1/graphql/console.

The Altinity Cloud Manager API

The Altinity Cloud Manager API lets organizations submit commands to manage their Altinity.Cloud environments and clusters.

Endpoints and authentication

The endpoint for the Altinity Cloud Manager API is https://acm.altinity.cloud/api. You can try out API calls through the Swagger UI at https://acm.altinity.cloud/docs/. You can also see the JSON description of the API at https://acm.altinity.cloud/api/refrence.json.

To authenticate any request, you can use any unexpired ACM API key (the keys in Figure 1 above, for example) with an HTTP X-Auth-Token header.

NOTE: By default, the Swagger UI only displays methods that don’t require authentication. To see the complete UI, use the /login method to log in:

The Swagger UI for the login method

Figure 3 - The Swagger UI for the /login method

Use the username and password from your Altinity.Cloud account, not your ClickHouse cluster. Click Execute and the results should include a new authentication token:

A typical response from the login method

Figure 4 - A typical response from the /login method

Once you have logged in through the Swagger UI, refreshing the page displays the complete ACM API, including all the methods that require an authentication token.

Similar to the Swagger UI, the JSON definition of the API only contains methods that don’t require authentication. One way to get the complete API in JSON format is to use curl, replacing <TOKEN> with your authentication token:

curl -X GET "https://acm.altinity.cloud/api/reference.json" \
-H "accept: application/json" \
-H "X-Auth-Token: <TOKEN>"

Example: Launch a cluster

The following example demonstrates how to launch a new cluster in Altinity.Cloud. The endpoint is /{environment}/cluster-launch, where {environment} is the ID of your environment. To get that ID, though, you need to call the /environments endpoint to get a list of all environments and their details, replacing <TOKEN> with your API key:

curl -X GET "https://acm.altinity.cloud/api/environments" \
-H "accept: application/json" \
-H "X-Auth-Token: <TOKEN>"

You’ll get a long, unformatted JSON document with extensive details on every environment. The formatted version looks something like this:

{
  "data": [
    {
      "id": "176",
      "name": "altinity-gke-test",
      "created": "2023-12-28 21:59:00",
      "type": "kubernetes",
      "domain": "altinity-gke-test.altinity.cloud",
      "externalDNS": "1",
. . .

All we care about here is the ID (176) of the environment (altinity-gke-test) where we want to launch the new cluster.

PRO TIP: If you have the awesome jq tool, you can pipe the curl command to jq '.data[] | {id, name}' to see just the ID and name of each environment:

{
  "id": "176",
  "name": "altinity-gke-test"
}
{
  "id": "175",
  "name": "altinity-maddie-gke-proj"
}
. . . 

With the environment ID, you’re ready to call the cluster-launch endpoint. As you would expect, there are many parameters to the method, corresponding to the fields in the Launch Cluster Wizard.

Our sample cluster will have the following settings:

  • name (name of the cluster): api-test
  • adminPass: dougspassword
  • adminUser: admin
  • https_port: 8443
  • instanceType: n2d-standard-2
  • nodes: 2
  • replicas: 2
  • secure: true
  • shards: 1
  • size (disk storage per node in GB): 100
  • tcp_port_secure: 9440
  • version (the tag for the altinity/clickhouse-server container image): 23.8.8.21.altinitystable
  • zookeeper: launch
  • zookeeperOptions: {"tag": "", size": "single"}

The following command will launch the cluster with the specifications above, replacing <ENV_ID> with the ID of your environment and <TOKEN> with your API token:

curl -X POST https://acm.altinity.cloud/api/<ENV_ID>/cluster-launch \
-d '{"name": "api-test", "adminPass": "dougspassword", "adminUser": "admin", "https_port": 8443, "instanceType": "n2d-standard-2", "nodes": 2, "replicas": 2, "secure": true, "shards": 1, "size": 100, "tcp_port_secure": 9440, "version": "23.8.8.21.altinitystable", "zookeeper": "launch", "zookeeperOptions": {"tag": "", "size": "single"}}' \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <TOKEN>"

4 - Backing up and Restoring Data

Working with backups

Working with backups is a crucial part of any analytics infrastructure. There is rich backup and restore functionality built into Altinity.Cloud, but some features cannot be directly accessed by users. We’ll start by discussing the use cases for restoring backups, we’ll move on to your options for configuring backups, then we’ll look at the Cluster Restore Wizard and how to Restore an individual table.

Restoring backups - use cases

Please contact Altinity support if you need to restore a backup. The response time for urgent requests is under 4 hours on the Enterprise Support plan, but usually we respond faster.

The most common use cases you’ll encounter are:

  • Partial data corruption in the table - it is possible to restore the table in the following ways:
  • Accidental drop table - The table can be fully restored in place by users; see Restoring an individual table below.
  • Accidental drop cluster - Difficult to do, but the cluster can be fully restored, preserving its configuration. Contact Altinity Support
  • Restoring a backup for testing purposes (for testing upgrades, hardware etc.) Contact Altinity Support Possible use cases include:
    • Restoring a single database to a separate database of an existing cluster
    • Restoring a cluster into a new cluster
  • Restoring a cluster into a new cluster in different region or environment - Contact Altinity Support

Creating a backup manually

Creating a backup of a cluster is straightforward: simply click the Create Backup item on the ACTIONS menu in the cluster view.

Launch New Cluster

You’ll see an informative message like this:

Creating a backup

As the dialog points out, backups are stored separately from the cluster, so you can restore your cluster from a backup even if you delete the cluster. Click OK to create the backup.

Configuring automatic backups

There are a variety of backup settings you can configure for an environment, including when backups should be taken, the cloud provider you’re using for backups, the compression format, and the version of the ClickHouse backup tool to use.

To configure backups for an environment, click the Environments tab on the left to see a list of all your environments:

List of environments

Select an environment, click the vertical dots icon, then choose Edit from the menu:

Editing an environment

The Environment Details dialog will appear; click on the Backups tab to configure your backup infrastructure:

The Backups tab on the Environment Details dialog

There are several options that apply to any environment, we’ll cover those elements first.

Configuring basic settings

No matter which cloud provider you’re using, these fields appear at the top of the dialog:

Common backup configuration elements

Field details

  • Turn On Backups - Turns backups on or off, as you would expect.
  • Backup Schedule - Lets you define days and times when backups should be taken. See Scheduling backups below for the details.
  • Backup Tool Image - The Docker image name and tag that should be used to create backups. The default is altinity/clickhouse-backup:2.4.14, which uses Altinity’s open-source clickhouse-backup utillity.
  • Compression Format - The default is tar; other options are gzip and zstd. Be aware that creating a tar file has the lowest impact on the CPU, but it creates the largest file because a tar file isn’t compressed. On the other hand, the other compression formats take more CPU cycles to create, but have smaller file sizes. Choose accordingly.
  • Enable Objects Labeling - If selected, everything in a backup is labeled with the name of the cluster. This can be useful if you’re working directly with the bucket where the backups are stored.

Configuring an external backup

If you have a Bring Your Own Kubernetes (BYOK) ClickHouse cluster, you can configure external backups at AWS, Azure, or GCP. For Bring Your Own Cloud (BYOC) clusters, Altinity configures backups for you.

Configuring an external backup is different depending on the cloud provider you’re using. If you’d like to jump ahead to the details for a specific provider, feel free:

Backing up to AWS

If you’re backing up your ClickHouse clusters to AWS, you need to provide your AWS credentials and other details:

Common backup configuration elements
Field details
  • Bucket - The name of the S3 bucket you’re using
  • Region - The AWS region where the bucket is stored
  • Access Key - The access key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.
  • Secret Key - The secret key credential for your AWS account. This field is ignored if you have a value in the Assume ARN field below.
  • Assume ARN - The ARN (Amazon Resource Name) for the bucket. If you have a value in this field, ACM ignores any values in the Access Key and Secret Key fields above.
    NOTE - Configuring an ARN that works with your service accounts and the Altinity Cloud Manager is complicated. Contact Altinity support for help in setting up your ARN.
  • Path - The optional path to the directory inside the bucket where your data is stored. The default value is altinity-cloud-managed-clickhouse.

If you’re using an ARN, the TEST CONNECTION button will become active when you enter a value in the Assume ARN field.

If you’re not using an ARN, TEST CONNECTION becomes active the first time you enter an Access Key and Secret Key.

Whatever type of credentials you’re using, clicking the button returns one of these messages:

Successful connection test
Unsuccessful connection test

You’ll of course need to correct any errors before you can continue. Click OK when you’re done.

Note that when you return to this panel, the value of the Secret Key field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your credentials again if you want to re-test the connection.

Backing up to GCP

With GCP, you need a JSON file that contains your credentials:

Common backup configuration elements
Field details
  • Bucket - The name of the bucket
  • Credentials JSON - JSON data that contains credentials associated with a GCP service account. That service account can have access to your entire GCP project, or it may be restricted to a single bucket or even a single folder within a single bucket. See the Google Cloud documentation for details:
  • Path - The optional path to the directory inside the bucket where your data is stored. The default value is altinity-cloud-managed-clickhouse.

When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:

Successful connection test
Unsuccessful connection test

You’ll of course need to correct any errors before you can continue. Click OK when you’re done.

Note that when you return to this panel, the value of the Credentials JSON field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your credentials again if you want to re-test the connection.

Backing up to Azure

Defining a connection to your Azure account is straightforward:

Common backup configuration elements
Field details
  • Container - The name of the storage container
  • Account Name - The name associated with your account
  • Account Key - The security credential for your account
  • Path - The optional path to the directory inside the bucket where your data is stored. The default value is altinity-cloud-managed-clickhouse.

When you’ve defined a complete set of credentials, the TEST CONNECTION button at the bottom of the dialog will become active. Clicking the button will return one of these messages:

Successful connection test
Unsuccessful connection test

You’ll of course need to correct any errors before you can continue. Click OK when you’re done.

Note that when you return to this panel, the value of the Account Key field will be hidden and the TEST CONNECTION button will be disabled. You’ll need to enter your credentials again if you want to re-test the connection.

Scheduling backups

You can create a schedule to create backups automatically at certain times of the day, week, or month. A backup schedule defined on the Environment Details panel applies to every cluster in the environment.

You define a backup schedule with these controls:

Common backup configuration elements

There are five options to define the Period when backups should occur:

  • Monthly - Define the day of the month
  • Weekly - Define the day of the week
  • Daily - Define the time of day
  • Every 6 hours - Backups occur every six hours
  • Every hour - Backups occur every hour.

NOTE: All times are defined in GMT.

In addition to defining the period, you can also define the number of Backups to Keep. The default is seven.

The button lets you define multiple schedules. For example, if you only want backups to occur on Friday and Saturday, create two Weekly schedules, one for Friday and one for Saturday. You can define up to three schedules.

You can override the environment-level backup schedule by defining a different schedule for a particular cluster. Select Backup Settings from the CONFIGURE menu in a cluster view:

You’ll see the same controls above inside a greatly simplified dialog.

A cluster can have a different backup schedule, but none of the other environment-level backup settings (cloud provider, backup tool image, compression format, etc.) can be changed.

The Cluster Restore Wizard

Launch New Cluster

WARNING: FOR ADVANCED USERS ONLY.

To restore a backup, begin by selecting Restore a Backup on the ACTIONS menu. The Cluster Restore Wizard lets you restore a cluster from a backup. We’ll go through all of the steps and options next, but if you’re looking for help on a particular section of the wizard, you can skip ahead to any of the tabs:

1. Backup Location tab

The first step is to specify the location of the backup you’re restoring.

Option 1A - Backup is in Altinity.Cloud

The simplest case is a backup stored in your Altinity.Cloud environment:

Backup location is Altinity.Cloud
Field details
  • Source Environment - The name of the Altinity.Cloud environment that holds the backup. Click the down arrow to see a list of all of your environments.

Click NEXT to continue.

Option 1B - Backup is in your AWS account

Another alternative, of course, is that the backup is stored in your AWS or GCP account. The details you need to provide are different in each case, as you would expect. You’ll see this panel if your backup is at AWS:

Backup location is AWS
Field details
  • Access Key - The access key for your AWS account.
  • Secret Key - The secret key for your AWS account.
  • Region - The AWS region where your backup is stored.
  • Bucket - The name of the bucket where your backup is stored.
  • ACM-Compatible Folder Structure - Check this box if the backup was created by ACM or if you know the backup has a fully ACM-compatible structure.

Click NEXT to continue.

NOTE: When you click NEXT, the ACM takes your credentials and attempts to access the bucket you named in the region you selected. If that fails, you’ll get an error message with details on what went wrong:

Bucket name or credentials is wrong

You’ll have to fix the error before you can continue.

Option 1C - Backup is in your GCP account

Finally, if you’re on GCP, you’ll see this instead:

Backup location is GCP
Field details
  • Credentials JSON - JSON data that contains credentials associated with a GCP service account. That service account can have access to your entire GCP project, or it may be restricted to a single bucket or even a single folder within a single bucket. See the Google Cloud documentation for details:
  • Region - The GCP region where your backup is stored.
  • Bucket - The name of the bucket where your backup is stored.
  • ACM-Compatible Folder Structure - Check this box if the backup was created by ACM or if you know the backup has a fully ACM-compatible structure.

Click NEXT to continue.

NOTE: When you click NEXT, the ACM takes the credentials JSON you entered and attempts to access the bucket you named in the region you selected. If that fails, you’ll get an error message:

Bucket name or credentials is wrong

You’ll have to fix the error before you can continue.

2. Source Cluster tab

Next we need to select the source cluster for the backup we’re restoring:

Source cluster information

The available backups are listed in the Cluster columne. The Namespace is the Kubernetes namespace that contains your ClickHouse installation. Finally, a checkmark indicates that the backup includes cluster configuration information.

Select a cluster and click NEXT to continue.

3. Source Backup tab

Once you’ve selected a cluster to restore, you’ll see a list of all of the backups for that cluster:

Backup information

Select a backup (in the example above there’s only one) and click NEXT to continue.

4. Tables tab

At this point you’ve specified where the backup is stored, selected the cluster you want to restore, and selected the particular backup of that cluster you want to restore. Next, you need to decide which tables you want to restore.

Option 4A - Restore all tables

The simplest option, of course, is All tables:

Select all tables

Click NEXT to continue.

Option 4B - Restore some tables

You can also specify patterns for the table names and engine types you want to include or exclude:

Specify a table include pattern

Separate multiple table or engine patterns with commas.

Patterns can contain splat [*] and question mark [?] wildcards:

  • The splat matches any sequence of characters before or after a separator. For example, default.* matches all tables in the default database.
  • The question mark matches a single character. For example, db.??_table matches db.ab_table and db.cd_table.

Click NEXT to continue.

5. Destination Cluster tab

The final step is to specify where to put the restored cluster.

Option 5A - Launch in a new cluster

One option is to simply launch a new cluster:

Destination new cluster

Enter a name for the destination cluster.

NOTE: If you choose to launch a new cluster, at the end of the Cluster Restore Wizard you’ll be taken to the Launch Cluster Wizard to define all the details of the new cluster.

Click NEXT to continue.

Option 5B - Launch a new cluster based on a source cluster

Another possibility is to use the configuration and settings of the source cluster to create a new cluster:

Destination new cluster with settings

There are some settings you can change, such as the version of ClickHouse the new cluster should run or how much storage the new cluster should have. Beyond the fields shown on this tab, everything else will be the same.

Field details
  • Name - The name of the restored cluster.

  • ClickHouse Version - Select the version of ClickHouse you want your cluster to use. Click the down arrow icon to see a list of available versions. ALTINITY BUILDS is selected by default; that lets you choose which Altinity Stable Build you want to use. You can also click COMMUNITY BUILDS to see other versions of ClickHouse.

    Beneath the ClickHouse version is a link to the release notes for the build you’ve selected. The release notes have extensive details of what is new and changed and fixed in each release. Click the link to open the release notes in a new browser tab.

  • Node Type - Click the down arrow to see what machine types are available. Each item in the list will tell you how many CPUs and how much RAM that machine type has.

  • Volume Type - Click the down arrow to see what classes of storage are available for your restored ClickHouse cluster.

  • Node Storage - The amount of storage in GB that each node in the restored cluster will have.

  • Number of Volumes - The number of storage volumes your restored cluster will have.

Click NEXT to continue.

Option 5C - Launch in an existing cluster

The third option is to restore the backup into an existing cluster:

Select all tables

Click the down arrow icon to see the list of available clusters. Select a cluster and click NEXT to contiue.

6. Review and Confirm tab

The Review and Confirm tab lets you go over the choices you’ve made before restoring the cluster:

Review and confirm details

If everything looks good, click CONTINUE.

If you selected Launch a new Cluster on the Destination Cluster tab (option 5A), you’ll be taken to the Launch Cluster Wizard to specify all of the configuration details and settings for the new cluster. When you’ve completed the Launch Cluster Wizard, the ACM will create the new cluster and restore the backup to it.

If you selected anything else on the Destination Cluster tab, the ACM will start restoring the cluster. As you would expect, this may take several minutes. When the cluster is restored, you’ll get an alert at the top of the ACM UI:

Cluster restored alert

Restoring an individual table

In addition to the cluster-level backup and restore features, you can restore an individual table to an existing database or a different database. To get started, click the EXPLORE button in the Clusters view:

The EXPLORE button

Figure 12 – The EXPLORE button

In the Explorer view, switch to the Schema tab:

The Schema tab

Figure 13 – The Schema tab

Click the vertical dots icon next to the table you want to restore and select Restore from Backup:

The Restore from Backup menu

Figure 14 – The Restore from Backup menu item

Select the backup you want to restore from. By default, the table will be restored from the selected backup into the same database.

NOTE: If you’re restoring a table to the same database, the table is first removed from the database. Once the table is removed, it is restored from the selected backup.

The Restore Table menu

Figure 15 – Restoring the table to the same database

You also have the option of restoring the table from the selected backup to a different database. Select the Different database radio button, then enter the name of the other database:

The Restore Table menu

Figure 16 – Restoring the table to a different database

When you’re ready to restore the table, click CONFIRM. When the table is restored, you’ll see a message in the ACM:

Table restored successfully message

Cloning a database

Cloning a database creates a copy of that database inside the same ClickHouse cluster. This can be useful for testing applications against realistic data.

NOTE: This operation creates a clone of the database, not a replica. Any changes to the original database will not be reflected in the clone, and any changes to the clone will not be reflected in the original database.

To get started, click the EXPLORE button in the Clusters view:

The EXPLORE button

Figure 12 – The EXPLORE button

In the Explorer view, switch to the Schema tab:

The Schema tab

Figure 13 – The Schema tab

Click the vertical dots icon next to the table you want to clone and select Clone Database:

The Clone Database menu

Figure 14 – The Clone Database menu item

You’ll be asked for a database name for the clone. The database name for the clone must begin with an upper- or lowercase letter or an underscore. It can contain letters, numbers, and underscores:

  • weather2024 - Valid
  • 2024weather - Not valid - names must start with a letter or underscore
  • Weather_2024 - Valid
  • WEATHER_2024 - Valid
  • wEaThEr_2024 - Valid
  • WEATHER-2024 - Not valid - only letters, numbers, and underscores are allowed
  • _WEATHER_2024 - Valid - names must start with a letter or underscore
Cloning a database

If the name you enter isn’t valid, you’ll see this message:

Incorrect database name error message

Click CONFIRM to clone the database. The cloned database and its tables will show up in the display after a few minutes.

Converting a table’s engine to ReplicatedMergeTree

To support replication, you can convert a table’s engine from MergeTree to ReplicatedMergeTree. (A table with a MergeTree engine can’t be replicated.)

To get started, click the EXPLORE button in the Clusters view:

The EXPLORE button

Figure 12 – The EXPLORE button

In the Explorer view, switch to the Schema tab:

The Schema tab

Figure 13 – The Schema tab

Click the vertical dots icon next to the table whose engine you want to change and select Convert to ReplicatedMergeTree:

The Convert to ReplicatedMergeTree menu

Figure 14 – The Convert to ReplicatedMergeTree menu item

As you would expect, this menu item is only available if a table’s engine is MergeTree.

When you click the menu item, uou’ll see a confirmation message:

The Convert to ReplicatedMergeTree confirmation message

Click OK to start the conversion.

5 - Billing

Managing billing for Altinity.Cloud.

Accounts with the role orgadmin are able to access the Billing page for their organizations. To access it, click on your account name in the upper right corner and select Billing from the menu:

Access Billing

From the billing page, the following Usage Summary and the Billing Summary are available for the environments connected to the account.

Billing page

Figure 1 - The Billing page

Usage Summary

The Usage Summary displays the following for the current period:

  • Current Spend: The value of charges for Altinity.Cloud services to this point in the current billing period.
  • Avg. Daily Spend: The average cost of Altinity.Cloud services per day.
  • Est. Monthly BIll: The total estimated cost for the current billing period. The estimate includes your costs so far (Current Spend) plus what your charges will be if your current level of usage continues until the end of the billing period.

You can also click the dropdown menus next to Usage for Period: and Environment: to look at historical usage data.

Billing Summary

The Billing Summary section displays the payment method, service address, and email address used for billing purposes.

There are four different payment methods:

Payment method dialog

Figure 2 - The Payment Method dialog

  • Payment against invoice (with prepayments): Prepay your charges and Altinity will invoice you against that prepayment.
  • Payment against invoice (monthly invoice): Altinity will invoice you for your charges each month.
  • Payment with Amazon Marketplace Account: Subscribe to Altinity.Cloud through the AWS Marketplace. Your Altinity charges are paid through your AWS account. (See the AWS Marketplace documentation for all the details.)
  • Payment by credit card: Enter your credit card, expiry date, and CVV number. Altinity will bill that card for your charges each month.