AWS remote provisioning

Altinity.Cloud Anywhere can operate inside AWS’s Elastic Kubernetes Service (WKS). You need several things before you can create ClickHouse® clusters in an EKS environment in your account:

• An Altinity.Cloud Anywhere environment
• A connection token to connect your Altinity.Cloud Anywhere environment once the EKS cluster that will host ClickHouse is provisioned
• Credentials that give Altinity specific permissions in your AWS account
• An EKS cluster running inside an EC2 instance
• A connection between your EKS cluster and your Altinity.Cloud Anywhere environment.

There are three ways to do this:

• Method 1. Use our Terraform module to create the environment - Everything is created for you.
• Method 2. Use an AWS CloudFormation template - You create the Altinity.Cloud environment and token, then you use CloudFormation to create an EC2 instance and an EKS cluster inside it. With the EKS cluster created, you use the Altinity Cloud Manager (ACM) to connect your EKS cluster and your Altinity.Cloud environment.
• Method 3. Do everything by hand - You create the Altinity.Cloud environment and token, then you create the credentials that give Altinity access to your AWS account. With that done, you contact Altinity support, who creates the EKS cluster and the infrastructure it requires, then finally connects the EKS cluster with your Altinity.Cloud Anywhere environment.

We strongly recommend the first method. If you feel more comfortable with AWS CloudFormation templates, you can go with the second method, But we strongly discourage the third.

Method 1. Using our Terraform provider

The Altinity.Cloud Terraform provider is the easiest way to provision your AWS environment.

Prerequisites

• Terraform version >= 1.5, installed from the official Terraform website
• Authentication with your AWS account
• An Anywhere API access token from your Altinity.Cloud account

As with all Terraform components, the official documentation for the Altinity.Cloud Terraform provider is on the Terraform registry site. We’re going to keep our discussion of the Terraform scripts as high-level as possible; if you have any questions, the official documentation is always the final (and latest) word.

To authenticate with your AWS account, set the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. You must also ensure your AWS account has sufficient permissions for EKS and related services.

Getting an Anywhere API access token

The AWS remote provisioning process needs an Anywhere API Access token. To get the token, log in to your Altinity.Cloud account and click your account name in the upper right. Click the My Account menu item:

Click the Anywhere API Access tab to generate a new token:

It’s recommended that you click the copy icon to copy the text of the token; the token is too wide to appear in the dialog. You won’t be able to see the token after you leave this panel, so be sure to store it in a secure place. See the Altinity API guide for complete details on creating and managing API tokens.

Now create the environment variable ALTINITYCLOUD_API_TOKEN:

export ALTINITYCLOUD_API_TOKEN=eJhbGciJFZERTiSIsInR5cCI6IkpyXQOttVCJ9.eyhdWQiOiJkZXJYuYWx0aW5pdHkuY2xvdWQiLCJpYXQiOjE3MTYyMTE0NjgsnN1YiI6ItImR0aWR3ZWxsQGFdGluaXR5LmNvbSJs9.LPb5KQjF_SLn86dGAzUP1fAlTmYItfdH-SCMTsngGB4EUPVZHM9axW3B7OtKvupgNxsBRfIOkmyKPmJUuG1dsAw

Setting up Terraform

To get started, you need to tell Terraform where to find the Altinity.Cloud provider. Create a new directory for your Terraform project and create a file named version.tf. Paste this text into it:

terraform {
  required_providers {
    altinitycloud = {
      source = "altinity/altinitycloud"
      # Look at https://registry.terraform.io/providers/Altinity/altinitycloud/latest/docs
      # to get the latest version number.
      version = "0.4.11"
    }
  }
}
provider "altinitycloud" {
  # Set this value in the env var ALTINITYCLOUD_API_TOKEN.
  # api_token = “ABCDEFGHI” 
}

Be sure to visit the documentation for the Altinity.Cloud Terraform provider and update the version number above to the latest version number. Check the dropdown menu at the top of the page:

Figure 3 - Finding the latest version number for the Altinity.Cloud Terraform provider

The api_token in the version.tf file is the API token mentioned above. You can uncomment this line and put your token in the version.tf file if you want, but it’s not recommended. Use the ALTINTIYCLOUD_API_TOKEN environment variable instead.

Now it’s time to put together the main Terraform script. You’ll need to choose or find the following details:

• A name for the Altinity.Cloud environment. This must start with your Altinity account name.
• Your AWS account number
• A region, such as us-east-1
• One or more availability zones, such as us-east-1a, us-east-1b, and us-east-1c.
• The node types you need for your Altinity.Cloud environment, such as t4g.large or m6i.large.
• A CIDR block from your private IP addresses, such as 10.67.0.0/21. At least /21 is required.

Now download the AWS environment with public load balancer script into a file named main.tf:

curl -o main.tf https://raw.githubusercontent.com/Altinity/terraform-provider-altinitycloud/refs/heads/master/examples/resources/altinitycloud_env_aws/public/main.tf

(The examples section of the Terraform docs may have other scripts that better meet your needs.)

Here’s where you need to add your specific values:

• Your environment name:

resource "altinitycloud_env_certificate" "this" {
  env_name = "acme-staging"
}

Once again, be sure your env_name starts with your Altinity account name.

• The AWS region where your environment should be hosted:

provider "aws" {
  region = "us-east-1"
}

• Your account number, region, availability zones, and CIDR range (you already set the environment name):

resource "altinitycloud_env_aws" "this" {
  name           = altinitycloud_env_certificate.this.env_name
  aws_account_id = "123456789012"
  region         = "us-east-1"
  zones          = ["us-east-1a", "us-east-1b"]
  cidr           = "10.67.0.0/21"

• For node types, you can assign different node types for ClickHouse, Zookeeper, and System nodes:

  node_groups = [
    {
      node_type         = "t4g.large"
      capacity_per_zone = 10
      reservations      = ["SYSTEM", "ZOOKEEPER"]
    },
    {
      node_type         = "m6i.large"
      capacity_per_zone = 10
      reservations      = ["CLICKHOUSE"]
    }
  ]

Applying the configuration

From the command line, run the following commands to initialize the Terraform project and apply it:

# initialize the terraform project
terraform init

# apply module changes
# btw, did you remember to authenticate with your AWS account? 
terraform apply

This operation will take several minutes to complete. When it completes, you’ll have a new environment in your Altinity.Cloud account. Log in to your account and click the Environments menu in the upper right. The env_name from the Terraform file should be in the list:

If you go to that environment and provisioning is not complete on the AWS side, you’ll see this panel:

Figure 5 - Provisioning status

As provisioning progresses, you may see errors (line 8 in Figure 5 above, for example). Most of the time those messages resolve themselves as resources become available. If provisioning stalls, contact Altinity support for help.

When provisioning is done, you’ll see the ACM Environments dashboard with the details of your environment;

Figure 6 - ACM Environments dashboard

Click the button to see your ClickHouse clusters. You won’t have any, of course, so use the Launch Cluster Wizard to create new ClickHouse clusters.

Deleting the configuration

To delete the AWS environment, first delete the Altinity.Cloud environment in your account. See Deleting an Environment for details on deleting an Altinity.Cloud environment. Once the environment is deleted, run the following command to delete the resources created by Terraform:

# Delete your Altinity.Cloud environment in the Altinity
# Cloud Manager before running this command!
terraform destroy

When this command finishes, all the resources associated with the AWS environment are deleted.

Method 2. Using an AWS CloudFormation template

Another way to create an Altinity.Cloud Anywhere environment in your AWS account is to use a CloudFormation template.

Creating a new Altinity.Cloud Anywhere environment

In the ACM, create a new environment. Depending on your role, you may have an Environments tab on the left side of the console; if so, click that tab.

At this point you’ll have a button to create a new environment; click it. In the Environment Setup dialog, give your environment a name, choose AWS as your cloud vendor, select a region, then choose Bring your own cloud account:

Click OK to continue to the Connection Setup panel.

Creating an Altinity.Cloud connection token

The Connection Setup panel contains the token you need to connect your Altinity.Cloud environment with the EKS cluster you’ll create with the CloudFormation template. Click the icon to copy the token to the clipboard; you’ll pass this to the CloudFormation template.

Figure 8 - The connection token for your Altinity.Cloud Anywhere environment

There is a PROCEED button at the bottom of the screen, but don’t click it until your CloudFormation stack is provisioned.

Creating a stack with AWS CloudFormation

Login to the AWS console, navigate to CloudFormation, click the Create Stack button, and choose With new resources (standard):

Figure 9 - Creating a new stack

Go to the altinitycloud-connect releases page and download the latest Cloud Formation YAML file, altinitycloud-connect-x.xx.x.aws-cloudformation.yaml. The link to the YAML file is at the bottom of the page.

In the Create Stack panel, select Choose an existing template in the Prerequisite section, then select Upload a template file and select the Altinity Cloud Formation Template YAML file as shown in Figure 10. Click Next to continue:

Now fill in any missing fields on the Specify Stack Details page (Figure 11):

• In the Stack Name field, give your stack a name.

• Select the VPC and Subnets where the altinitycloud-connect EC2 instance(s) should be launched. At least one VPC and at least one subnet is required. In addition, the following requirements apply:

  • You must have an internet gateway attached to your VPC.
  • Your subnet(s) must have a route table that assigns 0.0.0.0/0 to the internet gateway.
  • The subnet(s) must have the Auto-assign public IPv4 address property set to yes.

• Set the Token presented by https://acm.altinity.cloud/ to the token value from you copied in the ACM earlier.

• Click Next to continue.

Next, take the defaults on the Configure stack options panel (Figure 12).

At the bottom, check the box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names, then click Next to continue.

Finally, on the Review and create panel, scroll to the bottom and click Submit to start provisioning the new stack.

You’ll see the main CloudFormation panel (Figure 14), which will include your new stack. You can follow its progress in the Events list.

EC2 background processing explained

The EC2 instance is processed in the background as follows:

• An EC2 instance gets started from the cloud formation template
• The EC2 instance gets connected to Altinity.Cloud using altinitycloud-connect and the connection token
• An EKS cluster gets provisioned
• Tne EKS cluster gets connected to Altinity.Cloud using altinitycloud-connect and the connection token

Completing the connection between Altinity.Cloud Anywhere and EKS

Once your new stack is created, go back to the ACM and click the ‘PROCEED’ button in the connection wizard. The ACM will connect to the stack named [Accountname]-$ENV_NAME where AccountName is the name of your Altinity.Cloud account and $ENV_NAME` is the name of your environment. The only choice you have to make is to decide where your ClickHouse clusters will run:

Here we’ve defined us-east-1 as the region. You can take the defaults for everything else. You can of course define more storage classes or node pools. For node pools, at least one node pool must be configured for ClickHouse, at least one for Zookeeper, and at least one for System processes. Once everything is configured, the button will be active; click it to start provisioning. Click VIEW LOG to see status messages as the ACM finishes configuring your environment:

When the connection is complete, the ACM will switch to the Overview page for your environment. Click the button at the top of the page to see the list of your ClickHouse clusters. You won’t have any, of course, so click the button to create one. See the documentation for the Launch Cluster Wizard for all the details.

Method 3. Manual provisioning of the EC2 instance

It’s not recommended, but you can provision the EC2 instance yourself. The AWS EC2 instance you create should meet the following requirements:

• CPU: t2.micro minimum
• OS: Ubuntu Server v20.04

In addition, if you’re provisioning resources with the AWS command line interface, version 2.0 or higher of the AWS CLI is required.

Creating a Role with AIM policies

Set up a role with IAM policies to access IAM, EC2, VPC, EKS, S3 & Lambda as follows:

  arn:aws:iam::aws:policy/IAMFullAccess
  arn:aws:iam::aws:policy/AmazonEC2FullAccess
  arn:aws:iam::aws:policy/AmazonVPCFullAccess
  arn:aws:iam::aws:policy/AmazonS3FullAccess
  arn:aws:iam::aws:policy/AWSLambda_FullAccess
  arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

NOTE:

• AmazonSSMManagedInstanceCore is used for the Break Glass Procedure over AWS SSM.

Creating a policy for EKS full access

Create a standard policy for EKS full access as follows:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "eks  ":"*"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":"iam:PassRole",
         "Resource":"*",
         "Condition":{
            "StringEquals":{
               "iam:PassedToService":"eks.amazonaws.com"
            }
         }
      }
   ]
}

Next, to set this instance to have access to the EC2 metadata and Internet, set the Security group to:

• deny all inbound traffic
• allow all outbound traffic

Creating an Altinity.Cloud Anywhere environment

In the ACM, create a new environment. Depending on your role, you may have an Environments tab on the left side of the console; if so, click that tab.

At this point, you’ll have a button to create a new environment; click it. In the Environment Setup dialog, select Altinity.Cloud Anywhere Bring Your Own Cloud (BYOC) and give your environment a name:

Click OK to continue to the Connection Setup panel.

Creating an Altinity.Cloud connection token

The Connection Setup panel contains the token you need to connect your Altinity.Cloud environment with the EKS cluster you’ll create with the CloudFormation template. The only piece of information you need in this panel is the token, shown in red here:

Figure 18 - The connection token for your Altinity.Cloud Anywhere environment

Make sure Provisioned by User is selected as the Kubernetes setup type. Next, simply copy the value of the token from the text area in the middle. There is a PROCEED button at the bottom of the screen; you’ll click it once your EC2 instance and EKS cluster are provisioned. For now, ignore the instructions on the screen.

Installing altinitycloud-connect

Download the latest release of altinitycloud-connect. Complete installation instructions for MacOS and Linux are on the release page.

Now login to Altinity.Cloud using the connection token you copied from Figure 18 above.

altinitycloud-connect login --token=<registration token>

The altinitycloud-connect command does not have any output; it creates a cloud-connect.pem file in the current directory. The certificate in that file sets up a secure connection between your EKS cluster and your Altinity.Cloud Anywhere environment. Finally, connect to Altinity.Cloud:

altinitycloud-connect --capability aws

Contacting Altinity

At this point, contact Altinity support to set up a VPC and your EKS cluster. You need to provide the following details:

• The CIDR for the Kubernetes VPC (at least /21 recommended, such as 10.1.0.0/21) that does not overlap with existing VPCs
• The number of Availability Zones (3 are recommended)

Your Altinity support representative will start the EKS provisioning process.

Completing the connection

When the connection is complete, the ACM will switch to the Overview page for your environment. Click the button at the top of the page to see the list of your ClickHouse clusters. You won’t have any, of course, so click the button to create one. See the documentation for the Launch Cluster Wizard for all the details.

Break Glass procedure

The “Break Glass” procedure allows Altinity access to EC2 instance with SSH, using AWS SSM in order to troubleshoot altinitycloud-connect that is running on this instance.

1. Create an AnywhereAdmin IAM role with trust policy set:

    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Principal":{
                "AWS":"arn:aws:iam::313342380333:role/AnywhereAdmin"
             },
             "Action":"sts:AssumeRole"
          }
       ]
    }

2. Add a permission policy set:

    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":"ssm:StartSession",
             "Resource":[
                "arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID",
                "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell"
             ]
          }
       ]
    }

3. Send the following ARN string to Altinity: NOTE: This is used to revoke the Break Glass Procedure access change, or remove the permission policy.

    arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID