FIPS-Compatible Altinity Stable® Builds

Altinity Stable® Builds with FIPS 140-3 support

What are Altinity FIPS-Compatible Stable Builds?

FIPS 140-3 is a United States standard for cryptography used in high-security government environments. FIPS 140-3 specifies a number of properties for encryption including handling of keys, permitted versions of TLS, allowed cipher suites, and protections against tampering of builds.

ClickHouse uses OpenSSL libraries for encryption of most application and inter-server traffic. Starting with v25.3, Altinity FIPS-compatible builds use AWS-LC (Amazon’s fork of BoringSSL) libraries for encryption of most application and inter-server traffic. AWS-LC FIPS 2.0.0 has been validated under FIPS 140-3 (CMVP Certificate #4816). Combined with documented procedures this enables ClickHouse to function in a manner that is compatible with the FIPS standard.

FIPS-compatible Altinity Stable Builds are built, tested, and released in the same way as regular Altinity Stable Builds for ClickHouse. FIPS-compatible builds have altinityfips embedded in the release name. They use separate channels for distribution on builds.altinity.cloud/#altinityfips and have separate release notes. FIPS-compatible builds are also available as Docker images.

Release notes

For more technical details, see the release notes for the FIPS releases:

FIPS-Compatible Network Interfaces

The following network connections of ClickHouse can operate in FIPS-compatible mode in Altinity FIPS builds.

Name Type Description Default Server Port
HTTPS Port Server Accepts HTTPS API connections from clients 8443
Secure Native TCP Port Server Accepts native TCP protocol connections from clients (e.g., clickhouse-client) 9440
Interserver HTTPS Port Server & Client Used for communication between ClickHouse replicas 9010
ZooKeeper Client Connection Client Connection from ClickHouse to ZooKeeper or ClickHouse Keeper
Keeper Server Port Server Accepts ZooKeeper protocol connections from clients 9281
Raft Server Port Server & Client Used for synchronization between ClickHouse Keeper servers 9444

Prerequisites for FIPS-Compatible Operation

The minimal requirements for FIPS-compatible operation are:

  • Install FIPS-compatible Altinity Stable Build.
  • Apply FIPS-compatible configuration settings to set allowed ports, TLS version, and ciphers.

Installation

Yum and Apt Packages

FIPS-compatible Altinity Stable releases are distributed from a separate repo from standard Altinity Stable Builds. Follow the directions to set the repo for FIPS-compatible builds at builds.altinity.cloud/#altinityfips.

Important note! FIPS builds use a different repo from standard Altinity Stable Builds. Be sure you’re in the FIPS-compatible build section.

Once the repo is set correctly, you can download and install packages using the same commands as for regular Altinity Stable Builds.

Docker

FIPS-compatible Altinity Stable containers have fips in the container tag. For example:

docker pull altinity/clickhouse-server:24.3.5.8.altinityfips

Configuration of Altinity FIPS 140-3 Builds

The Altinity FIPS build ships with the AWS-LC FIPS cryptographic module enabled at startup. That module must be paired with a FIPS-compatible configuration as described below.

ClickHouse Server Configuration

Required server configuration changes include the following. These settings are by convention stored in /etc/clickhouse-server/config.xml and /etc/clickhouse-server/config.d/.

The FIPS 140-3-compatible configuration for the ClickHouse server includes the following:

  • Explicitly disable non-TLS listeners: HTTP, TCP, interserver HTTP, mysql, and postgresql
  • Enable HTTPS and secure native TCP
  • Enable interserver HTTPS
  • OpenSSL server configuration
  • OpenSSL client configuration (for outbound connections)

⚠️ WARNING: All other non-TLS server connections must be disabled. These are:

  • ZooKeeper default service (2181)
  • Native PROXYv1 protocol (9011)
  • JDBC bridge (9019)
  • gRPC (9100)
  • Prometheus default metrics (9363) - use port 8443 metrics endpoint instead
  • Graphite default (42000)

See ClickHouse Network Ports for more information.

Path: /etc/clickhouse-server/config.d/fips.xml

<clickhouse>
    <!-- Explicitly disable non-TLS listeners: HTTP, TCP, interserver HTTP, mysql, and postgresql -->
    <tcp_port remove="1" />
    <http_port remove="1" />
    <interserver_http_port remove="1" />
    <mysql_port remove="1" />
    <postgresql_port remove="1" />
    <!-- Enable HTTPS and Secure TCP -->
    <https_port>8443</https_port>
    <tcp_port_secure>9440</tcp_port_secure>
    <!-- Enable interserver HTTPS -->
    <interserver_https_port>9010</interserver_https_port>

    <openSSL>
        <server>
            <!-- Replace certificate paths with your layout. -->
            <certificateFile>/etc/clickhouse-server/certs/server.crt</certificateFile>
            <privateKeyFile>/etc/clickhouse-server/certs/server.key</privateKeyFile>
            <!-- For production with a public CA, use <loadDefaultCAFile> true and omit <caConfig> where appropriate. -->
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
            <loadDefaultCAFile>false</loadDefaultCAFile>
            <!-- If you use ECDSA server certificates, add the corresponding ECDHE-ECDSA-* entries to <cipherList>. -->
            <cipherList>ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384</cipherList>
            <cipherSuites>TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384</cipherSuites>
            <disableProtocols>sslv2,sslv3,tlsv1,tlsv1_1</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
            <!-- For lab/self-signed, use <verificationMode> none on the server block and <name> AcceptCertificateHandler under <client><invalidCertificateHandler>. -->
            <verificationMode>relaxed</verificationMode>
        </server>
        <client>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
            <loadDefaultCAFile>false</loadDefaultCAFile>
            <cipherList>ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384</cipherList>
            <cipherSuites>TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384</cipherSuites>
            <disableProtocols>sslv2,sslv3,tlsv1,tlsv1_1</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
            <verificationMode>relaxed</verificationMode>
            <invalidCertificateHandler>
                <name>RejectCertificateHandler</name>
            </invalidCertificateHandler>
        </client>
    </openSSL>
</clickhouse>

Keeper Cluster Configuration

Configure each Keeper cluster node to use secure ports.

Path: /etc/clickhouse-server/config.d/zookeeper.xml

<zookeeper>
    <!-- Each Keeper node is configured to use a secure port -->
    <node>
        <host>keeper1.example.com</host>
        <port>9281</port>
        <secure>1</secure>
    </node>
    <node>
        <host>keeper2.example.com</host>
        <port>9281</port>
        <secure>1</secure>
    </node>
    <node>
        <host>keeper3.example.com</host>
        <port>9281</port>
        <secure>1</secure>
    </node>
</zookeeper>

ClickHouse Keeper

The FIPS 140-3 compatible configuration for ClickHouse Keeper includes the following:

  • Explicitly disable the non-TLS TCP listener (9181)
  • Enable the secure TCP port
  • Configure each Raft server to use a secure port

Path: /etc/clickhouse-server/config.d/keeper.xml

<clickhouse>
    <keeper_server>
        <!-- Explicitly disable the non-TLS TCP listener (9181) -->
        <tcp_port remove="1" />
        <!-- Enable the secure TCP port -->
        <tcp_port_secure>9281</tcp_port_secure>
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <raft_configuration>
            <!-- Each Raft server is configured to use a secure port -->
            <secure>true</secure>
            <server>
                <id>1</id>
                <hostname>keeper1.example.com</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>keeper2.example.com</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>keeper3.example.com</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>

ClickHouse Client Configuration

The FIPS 140-3 compatible configuration for the ClickHouse client is provided below.

Path: /etc/clickhouse-client/config.d/fips.xml

<config>
    <secure>true</secure>
    <openSSL>
        <client>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
            <loadDefaultCAFile>false</loadDefaultCAFile>
            <cipherList>ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384</cipherList>
            <cipherSuites>TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384</cipherSuites>
            <disableProtocols>sslv2,sslv3,tlsv1,tlsv1_1</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
            <!-- For self-signed or lab certificates only, you may set <verificationMode> none and <name> AcceptCertificateHandler. -->
            <verificationMode>relaxed</verificationMode>
            <invalidCertificateHandler>
                <name>RejectCertificateHandler</name>
            </invalidCertificateHandler>
        </client>
    </openSSL>
</config>

Note: With <secure>true</secure>, the client uses TLS by default. Specify --port 9440 when connecting.

Running AWS-LC SSL and ACVP Validation Tests

The cryptographic module in Altinity FIPS builds can be verified by running the AWS-LC SSL conformance tests (8,037 tests) and ACVP known-answer tests (31 algorithm suites).

Prerequisites

  • An installed Altinity FIPS build of ClickHouse (.deb package or Docker image)
  • Go >= 1.13 installed on the test machine
  • The AWS-LC FIPS 2.0.0 source (used as the test harness): 
    curl -L -o /tmp/aws-lc.zip https://github.com/aws/aws-lc/archive/refs/tags/AWS-LC-FIPS-2.0.0.zip
unzip /tmp/aws-lc.zip -d /tmp
export AWSLC_SRC=/tmp/aws-lc-AWS-LC-FIPS-2.0.0

The ClickHouse multi-tool binary exposes the test harness via symlinks:

ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-ssl-shim
ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-ssl-handshaker
ln -sf /usr/bin/clickhouse /usr/bin/clickhouse-acvp-server

SSL Tests (8,037 tests)

These exercise the full TLS stack: handshake flows, cipher negotiation, session resumption, certificate handling, and more.

cd "$AWSLC_SRC/ssl/test/runner"
go test -v . \
  -shim-path /usr/bin/clickhouse-ssl-shim \
  -handshaker-path /usr/bin/clickhouse-ssl-handshaker \
  -num-workers 4

A passing run ends with:

PASS
ok  	boringssl.googlesource.com/boringssl/ssl/test/runner	142.538s

ACVP Tests (31 algorithm suites)

These validate that the FIPS cryptographic module produces correct outputs for known-answer test vectors covering AES, ECDSA, RSA, HMAC, KDF, DRBG, and more.

# Build the test tools
cd "$AWSLC_SRC/util/fipstools/acvp/acvptool"
go build -o /tmp/acvptool .
cd "$AWSLC_SRC/util/fipstools/acvp/acvptool/testmodulewrapper"
go build -o /tmp/testmodulewrapper .

# Run the tests
cd "$AWSLC_SRC/util/fipstools/acvp/acvptool/test"
go run check_expected.go \
  -tool /tmp/acvptool \
  -module-wrappers "modulewrapper:/usr/bin/clickhouse-acvp-server,testmodulewrapper:/tmp/testmodulewrapper" \
  -tests tests.json

A passing run ends with:

31 ACVP tests matched expectations

Verification of FIPS-Compatible Altinity Stable Operation

Verify FIPS library Startup

FIPS-compatible Altinity.Cloud servers will print the following message after a successful start-up test. This ensures that FIPS AWS-LC libraries are present and free from tampering.

$ grep 'FIPS mode' /var/log/clickhouse-server/clickhouse-server.log
2023.05.28 18:19:03.064038 [ 1 ] {} <Information> Application: Starting in FIPS mode, KAT test result: 1

Verify FIPS-Compatible Altinity Stable Version

To verify the software version, run select version() on the running server with any client program. This example confirms the version for both clickhouse-client as well as clickhouse-server.

$ clickhouse-client <options>
ClickHouse client version 23.3.19.34.altinityfips (altinity build).

5f1b329b5fdf :) select version()

SELECT version()

┌─version()───────────────┐
│ 23.3.19.34.altinityfips │
└─────────────────────────┘