Integrating Loki

An open-source project of Grafana Labs, Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure. You can use the Altinity Cloud Manager (ACM) to connect your Altinity.Cloud environment to an external Loki server.

Setting up an external Loki server with Grafana Cloud

As an example, we’ll look at setting up an external Loki server with Grafana Cloud. What we’ll do here can be done with a free Grafana Cloud account, although users with substantial log data can upgrade to a paid account easily. And of course, there are other services that offer a Loki server, but we’ll stay with the creators of Loki.

Typically using an external Loki server means using the promtail agent to send the contents of our local ClickHouse® logs to a Loki server. Fortunately, promtail is deployed by Altinity.Cloud in the altinity-cloud-system namespace:

kubectl get pods --namespace altinity-cloud-system | grep promtail
promtail-4kgcx                        1/1     Running            0                  2d4h
promtail-4lgb7                        1/1     Running            0                  100m
promtail-7wm8b                        1/1     Running            0                  3d22h
promtail-d2n5g                        1/1     Running            0                  3d21h
promtail-mj6qv                        1/1     Running            0                  3d22h

With the logging infrastructure set up inside the ACM, it’s time to set up the Loki server in Grafana Cloud. Log in to your Grafana Cloud account (if you don’t have one already, create a free one). Click the Grafana menu in the upper left corner, then select the Connections menu:

Figure 1 - The Connections menu

On the Add new connection page, enter hosted logs as the connection type. You’ll see the tile for Hosted Loki:

Figure 2 - Select Hosted Loki

Click the Hosted logs tile. On the Hosted logs page, select Send logs from a standalone host in section 1:

Figure 3 - Choose where logs are coming from

Next, create a token for the Loki server in section 2. First, give your token a name; this example uses ch-to-loki-at-grafana:

Figure 4 - Naming the token

With the token name defined, click the button. You’ll see something like this:

Figure 5 - Creating a token for the hosted Loki server

Grafana generates a YAML file to configure promtail. Except you don’t have to configure promtail: the ACM has already done that. The one piece of information you need from the YAML file is the value of the URL as highlighted in Figure 7 above. Enter that value in the External Loki URL field of the Logs tab of the Environment Configuration dialog. (See Figure 7 below.) Once the URL is defined in the ACM, promtail is automatically configured.

Now it’s time to configure our external Loki server in the ACM.

Configuring an external Loki server

Integrating Loki with your ClickHouse clusters is done at the environment level in the ACM. To connect your environment to an external Loki server, go to the list of environments in the ACM and click the vertical dots icon and select Edit from the menu:

Figure 6 - The Edit Environment menu

In the Environment Configuration dialog, go to the Logs tab:

Figure 7 - The Logs tab in the Environment Configuration dialog

In the External Loki URL field, enter the URL of your external Loki server in the format https://username:password@lokiurl.com/api/prom/push. The URL, username, and password come from the YAML file generated by Grafana Cloud above. You need to include your username and password as part of the URL, and use the endpoint /api/prom/push to access the Loki API.

A note about the Logs Bucket and Logs Region fields: These let you define the name of the bucket where your Loki logs should be stored and the region where the bucket is hosted. The bucket is used internally by the ACM; it’s not accessible by users. In addition, this setting is only used for Bring Your Own Kubernetes (BYOK) environments. For Bring Your Own Cloud (BYOC) environments, the bucket is configured automatically and these fields are disabled.

Click OK to save your updated configuration. Loki log entries are now being sent to your external Loki server.

Checking your external Loki server

Now that we have the ClickHouse clusters in our Altinity.Cloud environment sending data to the external Loki server at Grafana Cloud, we’ll take a look at that Loki data as a Grafana data source. Switch back to your Grafana Cloud account and click the Grafana menu in the upper left corner. Click on Data sources:

Figure 8 - The Data Sources menu

When the list of data sources appears, you’ll see a tile for each data source. One or more of them will have the Loki logo.

Figure 9 - A Loki data source

The Loki data source we want will be named grafana-[accountname]-logs. Click the button to open the data source.

Exploring the data in the Loki server

After opening the data source, you’ll see the data source explorer:

Figure 10 - The successful connection to the Loki data source

Log messages in Loki consist of a timestamp, some number of labels (name/value pairs), and the text of the message. Click the Label browser button to see the labels available on your Loki server:

Figure 11 - The label browser

Here we’ve selected a couple of labels: namespace and pod. When selecting a label in section 1, the display in section 2 displays the available values for each label. Furthermore, as you select specific values in section 2, the available values under the other labels change. As an example, clicking on the value altinity-maddie-na under the label namespace removes all of the values under the pod label except for clickhouse-operator-7778d7cfb6-9q8ml. Selecting those two values changes the selector in section 3 to match:

{namespace="altinity-maddie-na",pod="clickhouse-operator-7778d7cfb6-9q8ml"}

To see the logs that match this selector, click the button. You’ll see all the matching messages from the Loki server:

Figure 12 - The log entries matching the selector

The display also includes a graph of how many log messages have been issued over time:

Figure 13 - The volume of log entries over time

In this case the volume of entries has been consistent; if an incident of some kind had happened, there would likely have been a spike in the volume of entries at that point.

It’s straightforward at this point to add data from the Loki service to your Grafana dashboards, including panels that display the log messages themselves and panels that display metric queries based on Loki's built-in aggregation functions. Examples of aggregation functions include rate(), which returns the number of entries per second, and count_over_time(), which counts the number of entries for a given time period.