Altinity Stable® Build for ClickHouse® 24.8

Here are the detailed release notes and upgrade instructions for version 24.8.11.51285. There are a lot of new features, but also many things that changed their behavior. Read these carefully before upgrading. (Also check the Altinity ClickHouse® Upgrade E-book, which provides guidelines for upgrading ClickHouse databases.)

Major new features in 24.8 since the previous Altinity Stable release 24.3

Released 2025-01-31

Build report - 24.8.11.51285 | Release page

A new release introduces a lot of changes and new functions. Please refer to the full list in the following sections. The following major features are worth mentioning on the front page:

Major changes that require attention

  • replace_long_file_name_to_hash now enabled by default. If you need to preserve compatibility with versions before 23.9 - consider disabling that.
  • Analyzer
    • Analyzer is a huge refactoring of the query interpreting engine, enabled by default since 24.3. For new projects you should opt for using analyzer as a default choice.
    • There are a lot of queries which may work better with analyzer enabled.
    • But there are also risks that it can have some negative impact on different legacy queries. So if you upgrade, please give a run of your tests at staging with analyzer enabled first. If you run into some issues, you can disable the flag (enable_analyzer=0) for particular queries or for user profiles (or globally in default user profile).
    • Also check the list of reported issues related to analyzer, including those which are considered important. If you find something which is absent in those lists - please report it.
    • If you decide to disable the analyzer before the upgrade, use the setting allow_experimental_analyzer during a rolling upgrade, since older versions are not aware of enable_analyzer. But please remember that in future versions the old (before analyzer) code will be removed, and eventually the analyzer will be the only choice.
  • text_log & error_log is now enabled by default, consider configuring TTL for it.
  • S3Queue was reworked significantly (setting s3queue_total_shards_num removed,s3queue_buckets introduced, s3queue_allow_experimental_sharded_mode deprecated). In 24.8 the feature is more mature, but some issues still exist.
  • async_load_databases now enabled, it may increase the chances of the issues with the engine=Kafka and other streaming tables to flush data only to some subset of tables, and some of MV may miss the data.
  • JSON datatype is not ready for production on 24.8 (consider newer releases.)

Backward incompatible changes

  • Async Loading: The async_load_databases setting now defaults to true, allowing the server to accept connections before all tables are loaded. Set async_load_databases=false to revert to the previous behavior.
  • Deprecated Functions: Functions neighbor, runningAccumulate, runningDifferenceStartingWithFirstValue, and runningDifference are deprecated. Use window functions instead. Set allow_deprecated_functions=1 to enable them temporarily.
  • Access Changes: Faster system.columns queries, but stricter access rules may skip tables without SHOW TABLES permissions.
  • Index Renaming: “Inverted indexes” are renamed to “full-text indexes,” breaking compatibility with existing indexes. Drop and recreate them after the upgrade.
  • Removed Features:
    • Support for INSERT WATCH queries is removed.
    • The setting optimize_monotonous_functions_in_order_by is no longer available.
  • max_parallel_replicas Restriction: Setting max_parallel_replicas to 0 is no longer allowed to prevent logical errors.
  • Separate Packages: clickhouse-odbc-bridge and clickhouse-library-bridge are now distributed as separate packages.
  • Query Parsing: Invalid queries fail earlier. Experimental KQL expressions in kql table functions now require string literals.
  • Snowflake Functions: New SnowflakeID functions replace older ones. To enable deprecated versions, set allow_deprecated_snowflake_conversion_functions=1.

Known issues in 24.8.11

The following problems are known to exist in 24.8.11:

Let’s Install!

Linux packages can be found at https://packages.clickhouse.com for upstream builds, and at https://builds.altinity.cloud for Altinity Stable builds.

Docker images for the upstream version should be referenced as clickhouse/clickhouse-server:24.8.11.

Altinity Stable build images are available as altinity/clickhouse-server:24.8.11.51285.altinitystable.

For more information on installing ClickHouse from either the Altinity Builds or the upstream builds, see the ClickHouse Altinity Stable Builds Install Guide.

Please contact us at info@altinity.com if you experience any issues with the upgrade.

Appendix: New features

SQL features

  • Recursive CTEs: Enables recursive queries for tree and graph traversal, fully compatible with standard SQL.
  • QUALIFY Clause: Filters data after window functions.
  • JOINS:
    • Enable allow_experimental_join_condition=1 for joins with inequalities, e.g., t1.y < t2.y.
    • Better CROSS (comma) JOIN
    • Faster Parallel Hash Join
    • Merge Join Algorithm For ASOF JOIN (join_algorithm = ‘full_sorting_merge’)
    • Collect hash table sizes statistics in ConcurrentHashJoin (max_entries_for_hash_table_stats)
  • SQL usability:
    • DROP Multiple Tables: Supports dropping multiple tables in one command:
    • TRUNCATE ALL TABLES: New command to truncate all tables in one query.
    • DROP DETACHED PARTITION ALL for removing all detached partitions.
    • trailing commas support
  • Replicated Database Engine: Removed the experimental tag; now in Beta.

Datalakes & object storage

  • S3 Enhancements:
    • Introduces s3_plain_rewritable for S3 disks without metadata storage.
    • Enables reading from archive files in S3 storage.
    • Improved performance for glob selection ({} patterns).
    • a lot of performance optimizations
    • Supports AWS Private Link Interface endpoints.
    • Hive-Style Partitioning: Adds support for Hive-style partitioning in file storage engines like S3 and HDFS.
  • Azure Blob Storage:
    • production ready since 24.6
    • Adds plain_rewritable metadata support.
    • Adds a queue-based storage engine for Azure.
    • use_workload_identity parameter for authentication via Azure Workload Identity.
    • Multicopy support for backups using Azure Blob Storage.
    • Enabled native copy for Azure, including across containers.
    • Added support for plain_rewritable metadata for Azure and local object storages.
  • Hive Table Parsing: Resolves mismatches between the number of input fields and defined table columns.

Experimental features

  • TimeSeries Engine (Experimental): Optimized for time-series data, enabled with allow_experimental_time_series_table.
  • New Kafka Engine (Experimental): Enables exactly-once message processing with kafka_keeper_path and deduplication.
  • Dynamic Data Type and JSON Data Type: both experimental in 24.8.
  • new (experimental) index type vector_similarity (replacing annoy and usearch)
  • Experimental per column statistics allow_experimental_statistics + allow_statistics_optimize
  • Native Parquet Reader: Enabled with input_format_parquet_use_native_reader, directly reads binary Parquet files to ClickHouse columns.

Security and access control

  • GRANT for Table Engines: Allows granting specific permissions on table engines.
  • Certificate Reload: Certificates reload correctly with the entire chain.
  • Quota Key Flexibility: Allows quota keys with different auth schemes in HTTP requests.
  • Role in HTTP Queries: The role parameter in HTTP queries applies roles, e.g., ?role=x&role=y is equivalent to SET ROLE x, y.
  • Migration BoringSSL -> OpenSSL

Operations

  • Named Collections in Keeper:
  • Startup Scripts: Allows running preconfigured queries at startup.
  • Limits on Tables: max_database_num_to_throw, max_table_num_to_throw.
  • Query Cache Tagging: Use query_cache_tag to control cache entries for specific queries.
  • Added limits for MOVE PARTITION TO TABLE queries (e.g., max_parts_in_total).
  • YAML Enhancements: YAML merge key support, Configuration substitutions from YAML files.
  • HTTP Enhancements: Introduced http_response_headers for custom HTTP handler responses.
  • Error Logging: Introduces system.error_log, a persistent log of errors from system.errors.
  • system.detached_tables
  • system.error_log
  • new setting disable_insertion_and_mutation
  • global_profiler_cpu_time_period_ns / global_profiler_real_time_period_ns - to profile all threads of the clickhouse globally.

MergeTree

  • Optimized Table Sorting: Use optimize_row_order=true to improve compression by optimizing data order.
  • Reduce Disk I/O: Skips merging newly created projection blocks during INSERT.
  • Unload Primary Index: Saves memory by unloading outdated parts’ primary indices: SYSTEM UNLOAD PRIMARY KEY
  • ORDER BY Optimization: Buffering improves performance for high-selectivity filters, controlled by read_in_order_use_buffering.
  • New Virtual Columns: Adds _block_offset, _block_number, and _part_data_version.
  • TTL Information: Added to the system.parts_columns table for better tracking.
    | merge_tree_settings | deduplicate_merge_projection_mode | | throw | Whether to allow create projection for the table with non-classic MergeTree, if allowed, what is the action when merge, drop or rebuild. |

Functions

  • date_diff and age and date_trunc to subsecond units.
  • JSONAllPaths, JSONAllPathsWithTypes, JSONDynamicPaths, JSONDynamicPathsWithTypes, JSONMergePatch, JSONSharedDataPaths, JSONSharedDataPathsWithTypes
  • UUIDToNum, UUIDv7ToDateTime, generateUUIDv7
  • base64URLDecode,base64URLEncode, tryBase64URLDecode
  • changeDay, changeHour, changeMinute, changeMonth, changeSecond, changeYear
  • clamp
  • snowflakeIDToDateTime,snowflakeIDToDateTime64,generateSnowflakeID, dateTime64ToSnowflakeID, dateTimeToSnowflakeID
  • denseRank,percentRank (+percent_rank), groupConcat (+group_concat)
  • dynamicElement,dynamicType, isDynamicElementInSharedData,tupleNames
  • editDistanceUTF8,levenshteinDistanceUTF8
  • hilbertDecode,hilbertEncode
  • readWKTLineString,readWKTMultiLineString |
  • parseReadableSize,parseReadableSizeOrNull,parseReadableSizeOrZero
  • partitionID
  • printf

formats

  • new format Form - HTML forms (x-www-form-urlencoded MIME type),
  • Raw handy shortcut for TSVRaw
  • TSV with CRLF Support: TSV files with CRLF line endings are supported via input_format_tsv_crlf_end_of_line.
  • Apache Arrow Schema: Nullable types are now inferred automatically.

Other

  • Four File-Based Storage options: Adds _etag, _path, _size, and _timestamp.
  • Compression Based on STDOUT: Automatically applies compression for STDOUT if the output file has a compression extension.
  • 2 new WKT (Well-known text representation of geometry) types: LineString & MultiLineString
  • Compile Expressions: JIT compiler for expression fragments (compile_expressions) enabled by default.
    ( see also compiled_expression_cache_size compiled_expression_cache_elements_size)

Changes in Altinity Stable build compared to upstream build

Like upstream builds, Altinity Stable builds for ClickHouse are open source and are based on upstream LTS versions.

Fixes backported to Altinity Stable Build 24.8.11

Altinity Stable Build version 24.8.11 is based on upstream 24.8.11.5-lts, but we have additionally backported several fixes:

  • Thread pool: move thread creation out of lock (#68694)
  • Passing external user roles from query originator to other nodes (#70332)
  • Make numactl respect EPERM error, when get_mempolicy is restricted by seccomp (#70900)
  • Fix S3 Express Disk Initialization issue (#73777)
  • Thread pool metrics (#68674)
  • More advanced SSL options for Keeper (#69582)
  • Fix advanced SSL configuration for Keeper’s internal communication (#72730)

Backports of our work on better Parquet performance and compatibility

We have also backported some of our work on better Parquet performance and compatibility:

  • Parquet bloom filter support (#62966)
  • Parquet page header V2 support (#70807)
  • Boolean datatype support (#71055)
  • Compression level for Parquet output journal (#73651)

Build/Testing/Packaging improvements

  • Port CI/CD to Altinity’s infrastructure
  • Fixed tests
  • Changed version number generation