Altinity Stable Release for ClickHouse 19.16.12.49
Details of the Altinity Stable Release 19.16.12.49
Description
Improved Kafka engine reliability, performance and usability.
Bug Fixes
- When there is a rebalance of partitions between consumers, occasional data duplicates and losses were possible.
- When data was polled from several partitions with one poll and committed partially, occasional data losses were possible.
- Block size threshold (‘kafka_max_block_size’ setting) now triggers block flush correctly that reduces the chance of data loss under high load.
- When new columns are added to the Kafka pipeline (Kafka engine -> Materialized View -> Table) it was previously not possible to alter the destination tables first with default value and then alter MV in order to minimize downtime. The problem was not related directly to Kafka, but general implementation of materialized views. It is fixed now.
- Few other minor problems have been addressed as well.
Since ClickHouse now respects the ‘kafka_max_block_size’ setting that defaults to 65535, we recommend increasing it to the bigger values for high volume streaming. Setting it to 524288 or 1048576 may increase consumer throughput up to 20%.
Known Issues
The ‘kafka_max_block_size’ defaults to 65535. We recommend increasing it to the bigger values for high volume streaming. Setting it to 524288 or 1048576 may increase consumer throughput up to 20%.