Altinity Stable for ClickHouse 19.16.12.49

Description

Improved Kafka engine reliability, performance and usability.

Bug Fixes

• When there is a rebalance of partitions between consumers, occasional data duplicates and losses were possible.
• When data was polled from several partitions with one poll and committed partially, occasional data losses were possible.
• Block size threshold (‘kafka_max_block_size’ setting) now triggers block flush correctly that reduces the chance of data loss under high load.
• When new columns are added to the Kafka pipeline (Kafka engine -> Materialized View -> Table) it was previously not possible to alter the destination tables first with default value and then alter MV in order to minimize downtime. The problem was not related directly to Kafka, but general implementation of materialized views. It is fixed now.
• Few other minor problems have been addressed as well.

Since ClickHouse now respects the ‘kafka_max_block_size’ setting that defaults to 65535, we recommend increasing it to the bigger values for high volume streaming. Setting it to 524288 or 1048576 may increase consumer throughput up to 20%.

Known Issues

The ‘kafka_max_block_size’ defaults to 65535. We recommend increasing it to the bigger values for high volume streaming. Setting it to 524288 or 1048576 may increase consumer throughput up to 20%.

References

• Fast Insight From Fast Data: Integrating ClickHouse and Apache Kafka”.