This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Integrating other tools with Altinity.Cloud

Connecting Altinity.Cloud to other tools

1 - Integrating Datadog

Connecting Datadog to your Altinity.Cloud environment

Datadog is a popular observability platform for monitoring applications and infrastructure. Datadog’s access to Altinity.Cloud is controlled at two levels:

  • The Environment level - To use Datadog, you must enable it for your environment.
  • The Cluster level - Once Datadog is enabled at the environment level, you can enable Datadog monitoring for individual clusters.

Enabling Datadog at the environment level

To enable Datadog at the environment level, click the Environments tab on the left. You’ll see the list of environments in your account.

Environment list

Figure 1 - Environment list

Click the radio button next to the environment you want to enable, then click the vertical dots icon next to it. Select Edit from the menu.

The Edit Environment menu

Figure 2 - The Edit Environment menu

In the Environment Details panel, click the Logs tab. Click the Turn On Datadog slider to turn Datadog on or off. When Datadog is turned on, you need to enter the API key from your Datadog account. (The Datadog API and Application Keys page has complete details on how to create and manage API keys.) Click the down arrow to select your nearest Datadog region.

Turning on Datadog

Figure 3 - Turning on Datadog

Once Datadog is turned on, the three options below the region allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.

See Datadog's ClickHouse integration documentation for the details on configuring Datadog and the ClickHouse metrics that Datadog can monitor.

Finally, click OK to save your changes.

Enabling Datadog at the cluster level

If you’re creating a new ClickHouse cluster and your environment is enabled for Datadog, the Datadog integration options are enabled on the Connection Configuration tab of the Launch Cluster Wizard. Simply click the checkboxes to send logs and/or metrics to Datadog. (If the environment isn’t enabled for Datadog, the Datadog section of the Connection Configuration tab is disabled.)

If your ClickHouse cluster has already been created, you can use the Cluster Settings dialog to enable Datadog. From the Clusters tab of the ACM, click the Configure button, then choose Connections from the menu:

Editing a cluster's connections

Figure 4 - Editing a cluster’s connections

In the Connection Configuration dialog, select the checkboxes in the Datadog integration section:

Enabling Datadog at the cluster level

Figure 5 - Enabling Datadog at the cluster level

The three options here allow you to send logs, metrics on the ClickHouse cluster, and/or table-level metrics in your ClickHouse cluster.

Click CONFIRM to save your changes.

2 - Integrating Grafana

Connecting Grafana to your Altinity.Cloud environment

Grafana is a popular open-source observability platform for monitoring applications and infrastructure. We’ll look at connecting Altinity.Cloud to Grafana Cloud, although connecting to any Grafana instance works basically the same.

To connect your ClickHouse cluster to Grafana, you’ll need the connection details for your cluster. In the Clusters view, click the Connection Details link:

Getting connection details for the cluster

Figure 1 - A cluster panel

You’ll need the host name and the HTTP port:

Getting connection details for the cluster

Figure 2 - Connection details for the ClickHouse cluster

With those details, go ahead and log in to your Grafana Cloud account. Click the Grafana menu in the upper left corner, then select the Data sources item in the Connections section:

Connect a data source

Figure 3 - The create new data source menu

On the Data sources page, click the + Add new data source button in the upper right:

Connect a data source

Figur 4 - The Add new data source button

On the Add data source page, type click in the search bar, then click on the Altinity plugin for ClickHouse item:

Connect a data source

Figure 5 - Selecting the Altinity plugin for ClickHouse

To start, give your new data source a name at the top of the panel:

Connect a data source

Figure 6 - Naming your new data source

In the HTTP section of the page, enter https:// followed by the host and HTTP port from the ACM Connection Details page:

HTTP details for the new data source

Figure 7 - Entering the connection data for your ClickHouse cluster

Scroll down to the Auth section, select Basic auth, then enter your user ID and password:

Credentials for the new data source

Figure 8 - Entering the credentials for your ClickHouse cluster

NOTE: The user ID is the ID of your ClickHouse cluster, which is probably not the same as the ID of your Altinity.Cloud account. (admin is the default.)

To complete the connection, scroll to the bottom of the page and click the Save & test button:

Save and test the new data source

Figure 9 - The Save & Test button

If all goes well, you’ll get a Data source is working message from Grafana:

New data source is working

Figure 10 - A successfully created data source

If anything goes wrong, the dialog will contain an error message:

Connection to new data source failed

Figure 11 - An unsuccessfully created data source

From the Data source is working message, you can take either the Building a dashboard or the Explore view link. We’ll look at the Explore view; if you want to build a dashboard, you’ll use the Explore view to define a query against your ClickHouse cluster.

The Explore view starts with an empty query:

Connection to new data source failed

Figure 12 - The initial data source explorer view

The data source you just created is displayed at the top of the panel. Click the –database– and –table– fields to see a list of all of the tables and database available:

Connection to new data source failed

Figure 13 - Selecting a database

Here we’re selecting the system database. Clicking the –table– field displays all of the tables in the system database:

Connection to new data source failed

Figure 14 - Selecting a table within a database

We’ll select the query_log table. With the database and table selected, click the Go to Query button:

Connection to new data source failed

Figure 15 - The Go to Query button

You’ll see the query source and a graph of the results:

Connection to new data source failed

Figure 16 - The syntax of the query and its results

3 - Integrating Loki

Connecting Loki to your Altinity.Cloud environment

Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure.

4 - Integrating Prometheus

Connecting Prometheus to your Altinity.Cloud environment

Prometheus is popular open-source library used for event monitoring and alerting.

5 - Using an external S3 bucket for ClickHouse data

Control where your ClickHouse data is stored

For data sovereignty or other reasons, you might want to store your ClickHouse data in an external S3 bucket.

6 - Replicating Altinity.Cloud backups to S3

Configuring external S3 buckets for Altinity.Cloud backups

You can use external S3 buckets to store replicas of Altinity.Cloud backups. To configure them, you’ll need to contact us to get an Amazon ARN (Amazon Resource Name) that identifies the S3 bucket you’ll use.

Here are the steps to set everything up:

  1. Create the S3 bucket you want to use.
  2. Contact Altinity support and give us the bucket name. Support will use that bucket name to create the ARN.
  3. Set the ARN and bucket name in the following JSON, then use kubectl to apply it:
{
  "Id": "altinity-clickhouse-backup-for-s3-dougco-prod",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "altinity-clickhouse-backup-for-s3-dougco-prod",
      "Action": [
        "s3:GetBucketVersioning",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::dougco-clickhouse-production",
      "Principal": {
        "AWS": [
          "arn:aws:iam::313342380333:user/clickhouse-backup-dougco-prod"
        ]
      }
    },
    {
      "Sid": "altinity-clickhouse-backup-for-s3-dougco-prod",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::dougco-clickhouse-production/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::313342380333:user/clickhouse-backup-dougco-prod"
        ]
      }
    }
  ]
}