Setting up a Kafka® connection

To set up the connection between a Kafka server and your ClickHouse® cluster, go to the cluster view and click the Kafka Connections menu item on the CONFIGURE menu:

Figure 1 - The Kafka Connection Setup menu item

This takes you to the Kafka Connections dialog. This dialog lets you configure any number of Kafka configuration files, each of which contains any number of configured connections to Kafka servers.

Figure 2 - The initial Kafka Connections dialog

In Figure 2, there are no Kafka settings files. Click the button to create one. That takes you to this dialog:

Figure 3 - The New Kafka Settings File dialog

The fields in Figure 3 create a new Kafka settings file named kafka_settings.xml in the config.d directory. A settings file contains configuration information for some number of Kafka connections. Here we’re creating a new configuration named localhost_testing that connects to the topic named retail_data on a Kafka server at 2.tcp.ngrok.io:14624. Typically you would also define a number of connection parameters for this connection, but for this example we’re connecting to a Kafka server with no authentication required. (We obviously don’t recommend that; we’ll get to more realistic examples in the section Configuring your Kafka connection below.)

Click CHECK to test the connection. If the ACM successfully connects to the Kafka server and the topic you specified, you’ll see a dialog like this:

Figure 4 - Configuration information for a successful Kafka connection

Click the button to save these settings into the Kafka settings file. You’ll see your new settings file and its one configuration:

Figure 5 - The new Kafka settings file with a single configuration

From this dialog you can click the button to edit a given configuration or click the button to delete one. You’ll be asked to confirm your choice if you ask to delete a configuration. And as you would expect, the button lets you create a new configuration in the current settings file and the button lets you create a new settings file altogether.

Be aware that deleting the last configuration in a settings file will also delete the settings file altogether. You’ll be asked to confirm that choice:

Figure 6 - Deleting the last configuration in a Kafka settings file

Some complications:

First of all, if there’s any problem with the configuration parameters you entered, you’ll see an error message:

Figure 7 - A configuration error

Obviously you’ll need to fix the error before you continue.

Secondly, the value in the Configuration name field in the dialog becomes the name of an XML element in the Kafka settings file. In Figure 3, the configuration is named localhost_testing, so the configuration settings will be stored in the XML element <localhost_testing> as shown in Figure 4. That, of course, means the configuration name must be a valid XML element name. If not (a configuration named 42, for example), you’ll get an error message:

Figure 8 - The error message for a connection name that can’t be used as an XML element name

We won’t go into the details here, but an XML element name should start with a letter or an underscore. We also recommend that you use underscores instead of hyphens; that will make things simpler when you use this configuration to create a table with the Kafka table engine. (In other words, use localhost_testing instead of localhost-testing.)

A Kafka settings file can contain multiple connections to multiple Kafka servers and/or topics:

Figure 9 - A Kafka settings file with multiple connections

Click DONE to save the file and exit the dialog.

Once the settings file is created, it appears in the list of settings in the Server Settings list:

Figure 10 - The Kafka settings file in the list of server settings

Although you can edit the XML file directly, we don’t recommend it. Any changes you make in the Kafka Connections dialog will overwrite any changes you make directly. You’ll get a confirmation message if you try to edit the file directly:

Figure 11 - Warning message against editing XML directly

If you’re sure this is what you want to do, see the section Configuring Settings for all the details on modifying server settings.