AWS remote provisioning

Altinity.Cloud Anywhere operates inside your AWS account. You need several things before you can create ClickHouse® clusters in an EKS environment in your account:

• An Altinity.Cloud Anywhere environment
• A connection token to connect your Altinity.Cloud Anywhere environment once the EKS cluster that will host ClickHouse is provisioned
• Credentials that give Altinity specific permissions in your AWS account
• An EKS cluster running inside an EC2 instance
• A connection between your EKS cluster and your Altinity.Cloud Anywhere environment.

There are three ways to do this:

• Method 1. Use our Terraform module to create the environment - Everything is created for you.
• Method 2. Use an AWS CloudFormation template - You create the Altinity.Cloud environment and token, then you use CloudFormation to create an EC2 instance and an EKS cluster inside it. With the EKS cluster created, you use the Altinity Cloud Manager (ACM) to connect your EKS cluster and your Altinity.Cloud environment.
• Method 3. Do everything by hand - You create the Altinity.Cloud environment and token, then you create the credentials that give Altinity access to your AWS account. With that done, you contact Altinity support, who creates the EKS cluster and the infrastructure it requires, then finally connects the EKS cluster with your Altinity.Cloud Anywhere environment.

We strongly recommend the first method. If you feel more comfortable with AWS CloudFormation templates, you can go with the second method, But we strongly discourage the third.

Method 1. Using our Terraform module

This approach is the easiest way to configure your AWS environment.

Prerequisites

• Terraform version >= 1.5
• kubectl
• Authentication with your AWS account
• An Anywhere API access token from your Altinity.Cloud account

To authenticate with your AWS account, set the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. You must also ensure your AWS account has sufficient permissions for EKS and related services.

Getting an Anywhere API access token

The AWS remote provisioning process needs an Anywhere API Access token. To get the token, log in to your Altinity.Cloud account and click your account name in the upper right. Click the My Account menu item:

Click the Anywhere API Access tab to generate a new token:

It’s recommended that you click the copy icon to copy the text of the token; the token is too wide to appear in the dialog. You won’t be able to see the token after you leave this panel, so be sure to store it for later in a secure place. See the Altinity API guide for complete details on creating and managing API tokens.

Now create the environment variable ALTINITYCLOUD_API_TOKEN:

export ALTINITYCLOUD_API_TOKEN=eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJkZXYuYWx0aW5pdHkuY2xvdWQiLCJpYXQiOjE3MTYyMTE0NjgsInN1YiI6ImR0aWR3ZWxsQGFsdGluaXR5LmNvbSJ9.LPb56KQjF_SLn8dGAzUP1fAlTmYItfdH-SCMTsngGB4EUPVZHOM9axW3B7KvupgNxsBuRfIOkmyKPmJUG1dsAw

Creating a new Altinity.Cloud Anywhere environment

Our Terraform script does this for you. (Did we mention this is the easiest method?)

Creating an Altinity.Cloud connection token

You don’t have to do this either. Let’s move on.

Setting up your AWS account

To get started, create a new directory for your Terraform project and create a file named main.tf inside it. Copy and paste the following text into the main.tf file:

terraform {
  required_providers {
    altinitycloud = {
      source  = "altinity/altinitycloud"
      version = "0.2.8"
    }
  }
}

locals {
  region = "us-east-1"
}

# As mentioned above, Terraform will use the 
# ALTINITYCLOUD_API_TOKEN environment variable that you set
# before running the Terraform script. If for some reason
# you didn't set the environment variable, un-comment out
# the api_token line below and hardcode your API token here. 

provider "altinitycloud" {
#  api_token = "the_hardcoded_value_of_my_altinitycloud_api_token"
}

# The environment name MUST begin with your 
# Altinity.Cloud account name. 
resource "altinitycloud_env_certificate" "this" {
  env_name = "YOUR_ENV_NAME"
}

provider "aws" {
  region = local.region
}

module "altinitycloud_connect_aws" {
  source = "altinity/connect-aws/altinitycloud"
  pem    = altinitycloud_env_certificate.this.pem
}

resource "altinitycloud_env_aws" "this" {
  name           = altinitycloud_env_certificate.this.env_name
  aws_account_id = "YOUR_AWS_ACCOUNT"
  region         = local.region
  zones          = ["us-east-1a", "us-east-1b"]
  cidr           = "10.67.0.0/21"
  load_balancers = {
    public = {
      enabled          = true
      source_ip_ranges = ["0.0.0.0/0"]
    }
  }
  node_groups = [
    {
      node_type         = "t4g.large"
      capacity_per_zone = 10
      reservations      = ["SYSTEM", "ZOOKEEPER"]
    },
    {
      node_type         = "m6i.large"
      capacity_per_zone = 10
      reservations      = ["CLICKHOUSE"]
    }
  ]
  cloud_connect = true
  depends_on = [
    # "depends_on" is here to enforce "this resource, then 
    # altinitycloud_connect_aws" order on destroy.
    module.altinitycloud_connect_aws
  ]
}

Other values you need to set

You need to set your aws_account_id in the script. Beyond that, set the values of env_name, region and zones (to define the region and availability zones you’re using), and your CIDR information to meet your needs. If you need further customization, you can always fork the Terraform module and create your personalized version of it.

NOTE: As mentioned above, the env_name must start with your account name.

Applying the configuration

Open the terminal and navigate to the directory you created. Run the following commands to initialize the Terraform project and apply it:

# initialize the terraform project
terraform init

# apply module changes
# btw, did you remember to authenticate with your AWS account? 
terraform apply

This operation will take several minutes to complete. When it completes, you’ll have a new environment in your Altinity.Cloud account. Log in to your account and click the Environments menu in the upper right. The env_name from the Terraform file should be in the list:

Click the name of your new environment. You won’t have any ClickHouse clusters, so use the Launch Cluster Wizard to create new ClickHouse clusters.

Deleting the configuration

To delete the AWS environment, first delete the Altinity.Cloud environment in your account. See Deleting an Environment for details on deleting an Altinity.Cloud environment. Once the environment is deleted, run the following command to delete the resources created by Terraform:

# Delete your Altinity.Cloud environment in the Altinity
# Cloud Manager before running this command!
terraform destroy

When this command finishes, all the resources associated with the AWS environment are deleted.

Method 2. Using an AWS CloudFormation template

Another way to create an Altinity.Cloud Anywhere environment in your AWS account is to use a CloudFormation template.

Creating a new Altinity.Cloud Anywhere environment

In the ACM, create a new environment. Depending on your role, you may have an Environments tab on the left side of the console; if so, click that tab.

At this point you’ll have a button to create a new environment; click it. In the Environment Setup dialog, select Bring Your Own Cloud (BYOC) and give your environment a name:

Click OK to continue to the Connection Setup panel.

Creating an Altinity.Cloud connection token

The Connection Setup panel contains the token you need to connect your Altinity.Cloud environment with the EKS cluster you’ll create with the CloudFormation template. Click the icon to copy the token to the clipboard; you’ll pass this to the CloudFormation template.

Figure 5 - The connection token for your Altinity.Cloud Anywhere environment

There is a PROCEED button at the bottom of the screen, but don’t click it until your CloudFormation stack is provisioned.

Creating a stack with AWS CloudFormation

Login to the AWS console, navigate to CloudFormation, and click the Create Stack button.

Go to the altinitycloud-connect releases page and download the latest Cloud Formation YAML file,altinitycloud-connect-x.xx.x.aws-cloudformation.yaml.(That’s currently version 0.79.0.)The link to the YAML file is at the bottom of the page.

In the Create Stack panel, select Choose an existing template in the Prerequisite section, then select Upload a template file and select the Altinity Cloud Formation Template YAML file as shown in Figure 6. Click Next to continue:

Now fill in any missing fields on the Specify Stack Details page (Figure 7):

• Set Stack Name to altinity-connect-$ENV_NAME (if your environment name is altinity-maddie-byok, the stack name should be altinity-connect-altinity-maddie-byok.)

• Select the Subnets where the altinitycloud-connect EC2 instance(s) should be launched. At least one subnet is required.
  NOTE: The subnet(s) must have the “Auto-assign public IPv4 address” property set to yes.

• Set the Token presented by https://acm.altinity.cloud/ to the token value from you copied in the ACM earlier.

• Click Next to continue.

Next, take the defaults on the Configure stack options panel (Figure 8). Click Next to continue.

Finally, on the Review and create panel, check the box to allow CloudFormation to create resources as shown in Figure 9. The process will not work unless you click this button. Click Submit to start provisioning the new stack.

You’ll see the main CloudFormation panel (Figure 10), which will include your new stack. You can follow its progress in the Events list.

EC2 background processing explained

The EC2 instance is processed in the background as follows:

• EC2 instance gets started from the cloud formation template
• EC2 gets connected to Altinity.Cloud using altinitycloud-connect and the connection token
• EKS cluster gets provisioned
• EKS cluster gets connected to Altinity.Cloud using altinitycloud-connect and the connection token

Completing the connection between Altinity.Cloud Anywhere and EKS

Once your new stack is created, go back to the ACM and click the ‘PROCEED’ button in the connection wizard. The ACM will connect to the stack named altinity-$ENV_NAME where $ENV_NAME is the name of your environment. The only choice you have to make is to decide where your ClickHouse clusters will run:

Here we’ve defined us-east-1 as the region. You can take the defaults for everything else. You can of course define more storage classes or node pools. For node pools, at least one node pool must be configured for ClickHouse, at least one for Zookeeper, and at least one for System processes. Once everything is configured, the button will be active; click it to start provisioning. Click VIEW LOG to see status messages as the ACM finishes configuring your environment:

When the connection is complete, the ACM will switch to the Overview page for your environment. Click the button at the top of the page to see the list of your ClickHouse clusters. You won’t have any, of course, so click the button to create one. See the documentation for the Launch Cluster Wizard for all the details.

Method 3. Manual provisioning of the EC2 instance

It’s not recommended, but you can provision the EC2 instance yourself. The AWS EC2 instance you create should meet the following requirements:

• CPU: t2.micro minimum
• OS: Ubuntu Server v20.04

In addition, if you’re provisioning resources with the AWS command line interface, version 2.0 or higher of the AWS CLI is required.

Creating a Role with AIM policies

Set up a role with IAM policies to access IAM, EC2, VPC, EKS, S3 & Lambda as follows:

  arn:aws:iam::aws:policy/IAMFullAccess
  arn:aws:iam::aws:policy/AmazonEC2FullAccess
  arn:aws:iam::aws:policy/AmazonVPCFullAccess
  arn:aws:iam::aws:policy/AmazonS3FullAccess
  arn:aws:iam::aws:policy/AWSLambda_FullAccess
  arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

NOTE:

• AmazonSSMManagedInstanceCore is used for the Break Glass Procedure over AWS SSM.

Creating a policy for EKS full access

Create a standard policy for EKS full access as follows:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "eks  ":"*"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":"iam:PassRole",
         "Resource":"*",
         "Condition":{
            "StringEquals":{
               "iam:PassedToService":"eks.amazonaws.com"
            }
         }
      }
   ]
}

Next, to set this instance to have access to the EC2 metadata and Internet, set the Security group to:

• deny all inbound traffic
• allow all outbound traffic

Creating an Altinity.Cloud Anywhere environment

In the ACM, create a new environment. Depending on your role, you may have an Environments tab on the left side of the console; if so, click that tab.

At this point, you’ll have a button to create a new environment; click it. In the Environment Setup dialog, select Altinity.Cloud Anywhere Bring Your Own Cloud (BYOC) and give your environment a name:

Click OK to continue to the Connection Setup panel.

Creating an Altinity.Cloud connection token

The Connection Setup panel contains the token you need to connect your Altinity.Cloud environment with the EKS cluster you’ll create with the CloudFormation template. The only piece of information you need in this panel is the token, shown in red here:

Figure 15 - The connection token for your Altinity.Cloud Anywhere environment

Make sure Provisioned by User is selected as the Kubernetes setup type. Next, simply copy the value of the token from the text area in the middle. There is a PROCEED button at the bottom of the screen; you’ll click it once your EC2 instance and EKS cluster are provisioned. For now, ignore the instructions on the screen.

Installing altinitycloud-connect

Download the latest release of altinitycloud-connect. Complete installation instructions for MacOS and Linux are on the release page.

Now login to Altinity.Cloud using the connection token you copied from Figure 15 above.

altinitycloud-connect login --token=<registration token>

The altinitycloud-connect command does not have any output; it creates a cloud-connect.pem file in the current directory. The certificate in that file sets up a secure connection between your EKS cluster and your Altinity.Cloud Anywhere environment. Finally, connect to Altinity.Cloud:

altinitycloud-connect --capability aws

Contacting Altinity

At this point, contact Altinity support to set up a VPC and your EKS cluster. You need to provide the following details:

• The CIDR for the Kubernetes VPC (at least /21 recommended, such as 10.1.0.0/21) that does not overlap with existing VPCs
• The number of Availability Zones (3 are recommended)

Your Altinity support representative will start the EKS provisioning process.

Completing the connection

When the connection is complete, the ACM will switch to the Overview page for your environment. Click the button at the top of the page to see the list of your ClickHouse clusters. You won’t have any, of course, so click the button to create one. See the documentation for the Launch Cluster Wizard for all the details.

Break Glass procedure

The “Break Glass” procedure allows Altinity access to EC2 instance with SSH, using AWS SSM in order to troubleshoot altinitycloud-connect that is running on this instance.

1. Create an AnywhereAdmin IAM role with trust policy set:

    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Principal":{
                "AWS":"arn:aws:iam::313342380333:role/AnywhereAdmin"
             },
             "Action":"sts:AssumeRole"
          }
       ]
    }

2. Add a permission policy set:

    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":"ssm:StartSession",
             "Resource":[
                "arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID",
                "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
             ]
          }
       ]
    }

3. Send the following ARN string to Altinity: NOTE: This is used to revoke the Break Glass Procedure access change, or remove the permission policy.

    arn:aws:ec2:$REGION:$ACCOUNT_ID:instance/$INSTANCE_ID