Network Hardening
Hardening the network communications for your ClickHouse environment is about reducing exposure of someone listening in on traffic and using that against you. Network hardening falls under the following major steps:
-
IMPORTANT NOTE: Configuration settings can be stored in the default
/etc/clickhouse-server/config.xml
file. However, this file can be overwritten during vendor upgrades. To preserve configuration settings it is recommended to store them in/etc/clickhouse-server/config.d
as separate XML files with the same root element, typically<yandex>
. For this guide, we will only refer to the configuration files in/etc/clickhouse-server/config.d
for configuration settings.
Reduce Exposure
It’s easier to prevent entry into your system when there’s less points of access, so unused ports should be disabled.
ClickHouse has native support for MySQL client, PostgreSQL clients, and others. The enabled ports are set in the /etc/clickhouse-server/config.d
files.
To reduce exposure to your ClickHouse environment:
-
Review which ports are required for communication. A complete list of the ports and configurations can be found on the ClickHouse documentation site for Server Settings.
-
Comment out any ports not required in the configuration files. For example, if there’s no need for the MySQL client port, then it can be commented out:
<!-- <mysql_port>9004</mysql_port> -->
Enable TLS
ClickHouse allows for both encrypted and unencrypted network communications. To harden network communications, unencrypted ports should be disabled and TLS enabled.
TLS encryption required a Certificate, and whether to use a public or private Certificate Authority (CA) is based on your needs.
- Public CA: Recommended for external services or connections where you can not control where they will be connecting from.
- Private CA: Best used when the ClickHouse services are internal only and you can control where hosts are connecting from.
- Self-signed certificate: Only recommended for testing environments.
Whichever method is used, the following files will be required to enable TLS with CLickHouse:
- Server X509 Certificate: Default name
server.crt
- Private Key: Default name
server.key
- Diffie-Hellman parameters: Default name
dhparam.pem
Generate Files
No matter which approach is used, the Private Key and the Diffie-Hellman parameters file will be required. These instructions may need to be modified based on the Certificate Authority used to match its requirements. The instructions below require the use of openssl
, and was tested against version OpenSSL 1.1.1j
.
-
Generate the private key, and enter the pass phrase when required:
openssl genrsa -aes256 -out server.key 2048
-
Generate
dhparam.pem
to create a 4096 encrypted file. This will take some time but only has to be done once:openssl dhparam -out dhparam.pem 4096
-
Create the Certificate Signing Request (CSR) from the generated private key. Complete the requested information such as Country, etc.
openssl req -new -key server.key -out server.csr
-
Store the files
server.key
,server.csr
, anddhparam.pem
in a secure location, typically/etc/clickhouse-server/
.
Public CA
Retrieving the certificates from a Public CA or Internal CA performed by registering with a Public CA such as Let’s Encrypt or Verisign or with an internal organizational CA service. This process involves:
- Submit the CSR to the CA. The CA will sign the certificate and return it, typically as the file
server.crt
. - Store the file
server.crt
in a secure location, typically/etc/clickhouse-server/
.
Create a Private CA
If you do not have an internal CA or do not need a Public CA, a private CA can be generated through the following process:
-
Create the Certificate Private Key:
openssl genrsa -aes256 -out internalCA.key 2048
-
Create the self-signed root certificate from the certificate key:
openssl req -new -x509 -days 3650 -key internalCA.key \ -sha256 -extensions v3_ca -out internalCA.crt
-
Store the Certificate Private Key and the self-signed root certificate in a secure location.
-
Sign the
server.csr
file with the self-signed root certificate:openssl x509 -sha256 -req -in server.csr -CA internalCA.crt \ -CAkey internalCA.key -CAcreateserial -out server.crt -days 365
-
Store the file
server.crt
, typically/etc/clickhouse-server/
.
Self Signed Certificate
To skip right to making a self-signed certificate, follow these instructions.
- IMPORTANT NOTE: This is not recommended for production systems, only for testing environments.
-
With the
server.key
file from previous steps, create the self-signed certificate. Replacemy.host.name
with the actual host name used:openssl req -subj "/CN=my.host.name" -new -key server.key -out server.crt
-
Store the file
server.crt
, typically/etc/clickhouse-server/
. -
Each
clickhouse-client
user that connects to the server with the self-signed certificate will have to allowinvalidCertificateHandler
by updating theirclickhouse-client
configuration files at/etc/clickhouse-server/config.d
:<config> <openSSL> <client> ... <invalidCertificateHandler> <name>AcceptCertificateHandler</name> </invalidCertificateHandler> </client> </openSSL>
Enable TLS in ClickHouse
Once the files server.crt
, server.crt
, and dhparam.dem
have been generated and stored appropriately, update the ClickHouse Server configuration files located at /etc/clickhouse-server/config.d.
To enable TLS and disable unencrypted ports:
-
Review the
/etc/clickhouse-server/config.d
files. Comment out unencrypted ports, includinghttp_port
andtcp_port
:<!-- <http_port>8123</http_port> --> <!-- <tcp_port>9000</tcp_port> -->
-
Enable encrypted ports. A complete list of ports and settings is available on the ClickHouse documentation site for Server Settings. For example:
<https_port>8443</https_port> <tcp_port_secure>9440</tcp_port_secure>
-
Specify the certificate files to use:
<openSSL> <server> <!-- Used for https server AND secure tcp port --> <certificateFile>/etc/clickhouse-server/server.crt</certificateFile> <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile> <dhParamsFile>/etc/clickhouse-server/dhparams.pem</dhParamsFile> ... </server> ... </openSSL>
Encrypt Cluster Communications
If your organization runs ClickHouse as a cluster, then cluster-to-cluster communications should be encrypted. This includes distributed queries and interservice replication. To harden cluster communications:
-
Create a user for distributed queries. This user should only be able to connect within the cluster, so restrict it’s IP access to only the subnet or host names used for the network. For example, if the cluster is entirely contained in a subdomain named
logos1
,logos2
, etc. This internal user be set with or without a password:CREATE USER IF NOT EXISTS internal ON CLUSTER 'my_cluster' IDENTIFIED WITH NO_PASSWORD HOST REGEXP '^logos[1234]$'
-
Enable TLS for interservice replication and comment out the unencrypted interserver port by updating the
/etc/clickhouse-server/config.d
files:<!-- <interserver_http_port>9009</interserver_http_port> --> <interserver_https_port>9010</interserver_https_port> -->
-
Set an the
interserver_http_credentials
in the/etc/clickhouse-server/config.d
files, and include the internal username and password:<interserver_http_credentials> <user>internal</user> <password></password> </interserver_http_credentials>
-
Enable TLS for distributed queries by editing the file
/etc/clickhouse-server/config.d/remote_servers.xml
- For ClickHouse 20.10 and later versions, set a shared secret text and setting the port to secure for each shard:
<remote_servers> <my_cluster> <shard> <secret>shared secret text</secret> <!-- Update here --> <internal_replication>true</internal_replication> <replica> <host>logos1</host> <!-- Update here --> <port>9440</port> <!-- Secure Port --> <secure>1</secure> <!-- Update here, sets port to secure --> </replica> </shard> ...
- For previous versions of ClickHouse, set the internal user and enable secure communication:
<remote_servers> <my_cluster> <shard> <internal_replication>true</internal_replication> <replica> <host>logos1</host> <!-- Update here --> <port>9440</port> <!-- Secure Port --> <secure>1</secure> <!-- Update here --> <user>internal</port> <!-- Update here --> </replica> ... </shard> ...