Network Hardening

Hardening the network communications for your ClickHouse environment is about reducing exposure of someone listening in on traffic and using that against you. Network hardening falls under the following major steps:

• Reduce Exposure

• Enable TLS

• Encrypt Cluster Communications

• IMPORTANT NOTE: Configuration settings can be stored in the default /etc/clickhouse-server/config.xml file. However, this file can be overwritten during vendor upgrades. To preserve configuration settings it is recommended to store them in /etc/clickhouse-server/config.d as separate XML files with the same root element, typically <yandex>. For this guide, we will only refer to the configuration files in /etc/clickhouse-server/config.d for configuration settings.

Reduce Exposure

It’s easier to prevent entry into your system when there’s less points of access, so unused ports should be disabled.

ClickHouse has native support for MySQL client, PostgreSQL clients, and others. The enabled ports are set in the /etc/clickhouse-server/config.d files.

To reduce exposure to your ClickHouse environment:

1. Review which ports are required for communication. A complete list of the ports and configurations can be found on the ClickHouse documentation site for Server Settings.

2. Comment out any ports not required in the configuration files. For example, if there’s no need for the MySQL client port, then it can be commented out:

   <!-- <mysql_port>9004</mysql_port> -->
   

Enable TLS

ClickHouse allows for both encrypted and unencrypted network communications. To harden network communications, unencrypted ports should be disabled and TLS enabled.

TLS encryption required a Certificate, and whether to use a public or private Certificate Authority (CA) is based on your needs.

• Public CA: Recommended for external services or connections where you can not control where they will be connecting from.
• Private CA: Best used when the ClickHouse services are internal only and you can control where hosts are connecting from.
• Self-signed certificate: Only recommended for testing environments.

Whichever method is used, the following files will be required to enable TLS with CLickHouse:

• Server X509 Certificate: Default name server.crt
• Private Key: Default name server.key
• Diffie-Hellman parameters: Default name dhparam.pem

Generate Files

No matter which approach is used, the Private Key and the Diffie-Hellman parameters file will be required. These instructions may need to be modified based on the Certificate Authority used to match its requirements. The instructions below require the use of openssl, and was tested against version OpenSSL 1.1.1j.

1. Generate the private key, and enter the pass phrase when required:

   openssl genrsa -aes256 -out server.key 2048
   

2. Generate dhparam.pem to create a 4096 encrypted file. This will take some time but only has to be done once:

   openssl dhparam -out dhparam.pem 4096
   

3. Create the Certificate Signing Request (CSR) from the generated private key. Complete the requested information such as Country, etc.

   openssl req -new -key server.key -out server.csr
   

4. Store the files server.key, server.csr, and dhparam.pem in a secure location, typically /etc/clickhouse-server/.

Public CA

Retrieving the certificates from a Public CA or Internal CA performed by registering with a Public CA such as Let’s Encrypt or Verisign or with an internal organizational CA service. This process involves:

1. Submit the CSR to the CA. The CA will sign the certificate and return it, typically as the file server.crt.
2. Store the file server.crt in a secure location, typically /etc/clickhouse-server/.

Create a Private CA

If you do not have an internal CA or do not need a Public CA, a private CA can be generated through the following process:

1. Create the Certificate Private Key:

   openssl genrsa -aes256 -out internalCA.key 2048
   

2. Create the self-signed root certificate from the certificate key:

   openssl req -new -x509 -days 3650 -key internalCA.key \
       -sha256 -extensions v3_ca -out internalCA.crt
   

3. Store the Certificate Private Key and the self-signed root certificate in a secure location.

4. Sign the server.csr file with the self-signed root certificate:

   openssl x509 -sha256 -req -in server.csr -CA internalCA.crt \
       -CAkey internalCA.key -CAcreateserial -out server.crt -days 365
   

5. Store the file server.crt, typically /etc/clickhouse-server/.

Self Signed Certificate

To skip right to making a self-signed certificate, follow these instructions.

• IMPORTANT NOTE: This is not recommended for production systems, only for testing environments.

1. With the server.key file from previous steps, create the self-signed certificate. Replace my.host.name with the actual host name used:

   openssl req -subj "/CN=my.host.name" -new -key server.key -out server.crt
   

2. Store the file server.crt, typically /etc/clickhouse-server/.

3. Each clickhouse-client user that connects to the server with the self-signed certificate will have to allow invalidCertificateHandler by updating theirclickhouse-client configuration files at /etc/clickhouse-server/config.d:

   <config>
   <openSSL>
       <client>
           ...
           <invalidCertificateHandler>
               <name>AcceptCertificateHandler</name>
           </invalidCertificateHandler>
       </client>
   </openSSL>
   

Enable TLS in ClickHouse

Once the files server.crt, server.crt, and dhparam.dem have been generated and stored appropriately, update the ClickHouse Server configuration files located at /etc/clickhouse-server/config.d.

To enable TLS and disable unencrypted ports:

1. Review the /etc/clickhouse-server/config.d files. Comment out unencrypted ports, including http_port and tcp_port:

   <!-- <http_port>8123</http_port> -->
   <!-- <tcp_port>9000</tcp_port> -->
   

2. Enable encrypted ports. A complete list of ports and settings is available on the ClickHouse documentation site for Server Settings. For example:

   <https_port>8443</https_port>
   <tcp_port_secure>9440</tcp_port_secure>
   

3. Specify the certificate files to use:

   <openSSL>
       <server>
           <!-- Used for https server AND secure tcp port -->
           <certificateFile>/etc/clickhouse-server/server.crt</certificateFile>
           <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile>
           <dhParamsFile>/etc/clickhouse-server/dhparams.pem</dhParamsFile>
           ...
       </server>
   ...
   </openSSL>
   

Encrypt Cluster Communications

If your organization runs ClickHouse as a cluster, then cluster-to-cluster communications should be encrypted. This includes distributed queries and interservice replication. To harden cluster communications:

1. Create a user for distributed queries. This user should only be able to connect within the cluster, so restrict it’s IP access to only the subnet or host names used for the network. For example, if the cluster is entirely contained in a subdomain named logos1,logos2, etc. This internal user be set with or without a password:

   CREATE USER IF NOT EXISTS internal ON CLUSTER 'my_cluster'
       IDENTIFIED WITH NO_PASSWORD
       HOST REGEXP '^logos[1234]$'
   

2. Enable TLS for interservice replication and comment out the unencrypted interserver port by updating the /etc/clickhouse-server/config.d files:

   <!-- <interserver_http_port>9009</interserver_http_port> -->
   <interserver_https_port>9010</interserver_https_port> -->
   

3. Set an the interserver_http_credentials in the /etc/clickhouse-server/config.d files, and include the internal username and password:

   <interserver_http_credentials>
       <user>internal</user>
       <password></password>
   </interserver_http_credentials>
   

4. Enable TLS for distributed queries by editing the file /etc/clickhouse-server/config.d/remote_servers.xml

   1. For ClickHouse 20.10 and later versions, set a shared secret text and setting the port to secure for each shard:

   <remote_servers>
       <my_cluster>
       <shard>
           <secret>shared secret text</secret> <!-- Update here -->
           <internal_replication>true</internal_replication>
           <replica>
               <host>logos1</host> <!-- Update here -->
               <port>9440</port> <!-- Secure Port -->
               <secure>1</secure> <!-- Update here, sets port to secure -->
           </replica>
       </shard>
   ...
   

   1. For previous versions of ClickHouse, set the internal user and enable secure communication:

   <remote_servers>
       <my_cluster>
           <shard>
               <internal_replication>true</internal_replication>
               <replica>
                   <host>logos1</host> <!-- Update here -->
                   <port>9440</port> <!-- Secure Port -->
                   <secure>1</secure> <!-- Update here -->
                   <user>internal</port> <!-- Update here -->
               </replica>
           ... 
           </shard>
   ...