Accessing an AWS MSK cluster from Altinity.Cloud

How to connect Altinity.Cloud to Amazon MSK within your VPC

Altinity.Cloud users can connect to Kafka clusters hosted by Amazon MSK (Managed Streaming for Apache Kafka). That makes it easy to use ClickHouse® for real-time analytics on data in Kafka topics. Connecting to MSK from Altinity.Cloud used to be much more complicated, with multiple VPC endpoints and load balancers. With AWS’s multi-VPC connectivity, configuration is much simpler, and scaling your MSK cluster doesn’t require any reconfiguration.

These instructions assume you already have an AWS account with a running MKS cluster. From that starting point, there are four steps to get everything configured and running:

  1. Contact Altinity support to get the ARN (Amazon Resource Name) for Altinity’s service account
  2. Enable multi-VPC connectivity for your MSK cluster
  3. Edit your MSK cluster’s security policy to authorize the Altinity service account to access the cluster
  4. Get back to Altinity support with the ARN of your newly configured MSK cluster.

Step 1. Contact Altinity support

To get started, you’ll need to contact us with your AWS account ID. You can find that by clicking your username in the upper right corner of the AWS console:

Getting your AWS account ID

Figure 1 - Getting your AWS account ID

In exchange for your AWS account ID, support will give you the ARN of Altinity’s service account. You’ll use that to give Altinity access to your MSK cluster. The ARN will be something like arn:aws:iam::111122223333:root.

Step 2. Enable your MSK cluster for multi-VPC communication

In the AWS console, go to your MSK cluster, click the Actions button, and select Turn on multi-VPC connectivity:

The turn on multi-VPC connectivity menu item

Figure 2 - The Turn on multi-VPC connectivity menu item

In the Turn on multi-VPC connectivity dialog, select the authentication type(s) you’re using, then click Turn on selection to enable multi-VPC connectivity:

Turning on multi-VPC connectivity

Figure 3 - Turning on multi-VPC connectivity

It will take several minutes to change your cluster’s configuration, particularly if your MSK cluster has several broker nodes. You won’t be able to make any other changes to your MSK cluster until that is complete.

BTW, this example uses SASL/SCRAM as the authentication method. If you’re using IAM, there are additional steps.

Step 3. Edit your MSK cluster’s policy

Once your cluster is reconfigured, you’ll be able to click the Actions button and select Edit cluster policy:

The edit cluster policy menu item

Figure 4 - The Edit cluster policy menu item

Click the Advanced radio button to edit the cluster policy. At the top of the policy (line 7 in Figure 5), add the Altinity ARN as a principal to allow access to your MSK cluster:

The edit cluster policy menu item

Figure 5 - Editing the cluster’s security policy

While you’re here, copy the ARN of your MSK cluster (line 15). You’ll need to give that to Altinity support.

The policy will look something like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:root"
      },
      "Action": [
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
        "kafka:DescribeCluster",
        "kafka:DescribeClusterV2"
      ],
      "Resource": "arn:aws:kafka:us-west-2:444455556666:cluster/cluster-maddie-2/ef918d5d-e458-b40e-8aea-52d7179363b5"
    }
  ]
}

Step 4. Contact Altinity support

To complete the setup of Altinity access to your MSK cluster, contact support and give your support person the ARN of your MSK cluster. From there, Altinity will complete the connection and you can access Kafka topics in your MSK cluster from your ClickHouse clusters. See Connecting ClickHouse to Apache Kafka for all the details.

For more information

VPC connections can be quite complicated. You can find lots more information on the AWS site;