This is the multi-page printable view of this section. Click here to print.
Altinity Stable Build for ClickHouse 21.8
- 1: Altinity Stable Build for ClickHouse 21.8.15 Release Notes
- 2: Altinity Stable Build for ClickHouse 21.8.13 Release Notes
- 3: Altinity Stable Build for ClickHouse 21.8.12 Release Notes
- 4: Altinity Stable Build for ClickHouse 21.8.11 Release Notes
- 5: Altinity Stable Build for ClickHouse 21.8.10 Release Notes
- 6: Altinity Stable Build for ClickHouse 21.8.8 Release Notes
1 - Altinity Stable Build for ClickHouse 21.8.15 Release Notes
Release Notes
Based on upstream/v21.8.15.7-lts.
ClickHouse release v21.8.15.7-altinitystable as compared to community ClickHouse v21.8.15.7:
Changes compared to the Community Build
Bug Fixes
- Bug Fix: Fixed issue with merging configs that have and root XML tags. (backport of ClickHouse@213ecae) (via #114).
- Bug Fix: Backport of ClickHouse#31823 to 21.8: Fix invalid cast of nullable type when nullable primary key is used (via #110)
Features
- Build/Testing/Packaging Improvement: Various macOS compilation and packaging fixes/backporting (via #108)
- Build/Testing/Packaging Improvement: Minor test improvements (via afb4a0a, 4c35386)
Changes compared to Altinity Stable 21.8.13.1
Bug Fixes
- Fix segfault while parsing ORC file with corrupted footer. Closes ClickHouse#33797 via ClickHouse#33984 Kruglov Pavel.
- Fix usage of functions array and tuple with literal arguments in distributed queries. Previously it could lead to Not found columns exception. ClickHouse#33938 (Anton Popov).
- Fix s3 table function reading empty file. Closes ClickHouse#33008 via ClickHouse#33037 (Kseniia Sumarokova).
- Fix hdfs url check that didn’t allow using HA namenode address. Bug was introduced in ClickHouse#31042. ClickHouse#32976 (Kruglov Pavel).
- Server might fail to start if database with MySQL engine cannot connect to MySQL server, it’s fixed. Fixes ClickHouse#14441 via ClickHouse#32802 (tavplubix).
Test Results
Packages
- Deb/RPMs: https://builds.altinity.cloud/
- Docker image:
altinity/clickhouse-server:21.8.15.15.altinitystable
- Homebrew formula:
clickhouse@21.8-altinity-stable-21.8.15.15
2 - Altinity Stable Build for ClickHouse 21.8.13 Release Notes
Release Notes
ClickHouse release v21.8.13.1-altinitystable as compared to v21.8.12.29-altinitystable:
Bug Fixes
-
Fixed Apache Avro Union type index out of boundary issue in Apache Avro binary format. #33022 (Harry Lee).
-
Quota limit was not reached, but the limit was exceeded. This PR fixes #31174. #31656 (sunny).
-
NO CL ENTRY: ‘fix json error after downgrade’. #33166 (bullet1337).
-
Integer overflow to resize the arrays causes heap corrupt. #33024 (varadarajkumar).
-
fix crash when used fuzzBits with multiply same FixedString, Close #32737. #32755 (SuperDJY).
-
Fix possible exception at RabbitMQ storage startup by delaying channel creation. #32584 (Kseniia Sumarokova).
-
Fixed crash with SIGFPE in aggregate function
avgWeighted
withDecimal
argument. Fixes #32053. #32303 (tavplubix). -
Some replication queue entries might hang for
temporary_directories_lifetime
(1 day by default) withDirectory tmp_merge_<part_name>
orPart ... (state Deleting) already exists, but it will be deleted soon
or similar error. It’s fixed. Fixes #29616. #32201 (tavplubix). -
XML dictionaries identifiers, used in table create query, can be qualified to
default_database
during upgrade to newer version. Closes #31963. #32187 (Maksim Kita). -
Number of active replicas might be determined incorrectly when inserting with quorum if setting
replicated_can_become_leader
is disabled on some replicas. It’s fixed. #32157 (tavplubix). -
Fixed
Directory ... already exists and is not empty
error when detaching part. #32063 (tavplubix). -
Some
GET_PART
entry might hang in replication queue if part is lost on all replicas and there are no other parts in the same partition. It’s fixed in cases when partition key contains only columns of integer types orDate[Time]
. Fixes #31485. #31887 (tavplubix). -
Change configuration path from
keeper_server.session_timeout_ms
tokeeper_server.coordination_settings.session_timeout_ms
when constructing aKeeperTCPHandler
- Same withoperation_timeout
. #31859 (JackyWoo). -
Fix a bug about function transform with decimal args. #31839 (李帅).
-
Fix crash when function
dictGet
with type is used for dictionary attribute when type isNullable
. Fixes #30980. #31800 (Maksim Kita). -
Fix crash with empty result on odbc query. Closes #31465. #31766 (Kseniia Sumarokova).
-
Fix possible crash (or incorrect result) in case of
LowCardinality
arguments of window function. Fixes #31114. #31888 (Nikolai Kochetov).
Changes compared to the Community Build
- Fix invalid cast of Nullable type when nullable primary key is used. (Nullable primary key is a discouraged feature - please do not use). This fixes #31075 #31823 Amos Bird).
Test Reports
3 - Altinity Stable Build for ClickHouse 21.8.12 Release Notes
Release Notes
ClickHouse release v21.8.12.29-altinitystable as compared to v21.8.11.1-altinitystable:
Performance Improvements
- Improved performance of JSON and XML output formats. #31673 (alexey-milovidov).
Bug Fixes
Build/Testing/Packaging Improvements
Bug Fixes (user-visible misbehaviour in official stable or prestable release)
- Fixed functions
empty
andnotEmpty
with the arguments ofUUID
type. Fixes #31819. #31883 (Anton Popov). - Fixed possible assertion
../src/IO/ReadBuffer.h:58: bool DB::ReadBuffer::next(): Assertion '!hasPendingData()' failed.
in TSKV format. #31804 (Kruglov Pavel). - Fixed usage of
Buffer
table engine with typeMap
. Fixes #30546. #31742 (Anton Popov). - Fixed race in JSONEachRowWithProgress output format when data and lines with progress are mixed in output. #31736 (Kruglov Pavel).
- Fixed
there are no such cluster here
error on execution ofON CLUSTER
query if specified cluster name is name ofReplicated
database. #31723 (tavplubix). - Settings
input_format_allow_errors_num
andinput_format_allow_errors_ratio
did not work for parsing of domain types, such asIPv4
, it’s fixed. Fixes #31686. #31697 (tavplubix). RENAME TABLE
query worked incorrectly on attempt to rename an DDL dictionary inOrdinary
database, it’s fixed. #31638 (tavplubix).- Fix invalid generated JSON when only column names contain invalid UTF-8 sequences. #31534 (Kevin Michel).
- Resolve
nullptr
in STS credentials provider for S3. #31409 (Vladimir Chebotarev). - Remove not like function into RPNElement. #31169 (sundyli).
- Fixed bug in Keeper which can lead to inability to start when some coordination logs was lost and we have more fresh snapshot than our latest log. #31150 (alesapin).
- Fixed abort in debug server and
DB::Exception: std::out_of_range: basic_string
error in release server in case of bad hdfs url by adding additional check of hdfs url structure. #31042 (Kruglov Pavel).
4 - Altinity Stable Build for ClickHouse 21.8.11 Release Notes
ClickHouse release v21.8.11.1-altinitystable FIXME as compared to v21.8.10.1-altinitystable
New Features
CompiledExpressionCache
limit elements size usingcompiled_expression_cache_elements_size
setting. #30667 (Maksim Kita).
Improvements
- Made query which fetched table structure for PostgreSQL database more reliable. #30477 (Kseniia Sumarokova).
Bug Fixes
- Fixed
SHOW GRANTS
when partial revokes are used. This PR fixes #31138. #31249 (Vitaly Baranov).
Bug Fixes (user-visible misbehaviour in official stable or prestable release)
- Fixed
StorageMerge
with aliases and where (it did not work before at all). Closes #28802. #31044 (Kseniia Sumarokova). - Fixed
JSONValue
/Query
with quoted identifiers. This allows to have spaces in json path. Closes #30971. #31003 (Kseniia Sumarokova). - Using
formatRow
function with not row formats led to segfault. Don’t allow to use this function with such formats (because it doesn’t make sense). #31001 (Kruglov Pavel). - Skip
max_partition_size_to_drop
check in case ofATTACH PARTITION ... FROM
andMOVE PARTITION ...
#30995 (Amr Alaa). - Fixed set index not used in
AND/OR
expressions when there are more than two operands. This fixes #30416 . #30887 (Amos Bird). - Fixed ambiguity when extracting auxiliary ZooKeeper name from ZooKeeper path in
ReplicatedMergeTree
. Previously server might fail to start withUnknown auxiliary ZooKeeper name
if ZooKeeper path contains a colon. Fixes #29052. Also it was allowed to specify ZooKeeper path that does not start with slash, but now it’s deprecated and creation of new tables with such path is not allowed. Slashes and colons in auxiliary ZooKeeper names are not allowed too. #30822 (tavplubix). - Fixed a race condition between
REPLACE/MOVE PARTITION
and background merge in non-replicatedMergeTree
that might cause a part of moved/replaced data to remain in partition. Fixes #29327. #30717 (tavplubix). - Fixed
PREWHERE
withWHERE
in case of always truePREWHERE
. #30668 (Azat Khuzhin). - Functions for case-insensitive search in UTF8 strings like
positionCaseInsensitiveUTF8
andcountSubstringsCaseInsensitiveUTF8
might find substrings that actually does not match is fixed. #30663 (tavplubix). - Limit push down optimization could cause a error
Cannot find column
. Fixes #30438. #30562 (Nikolai Kochetov). - Fixed exception handling in
parallel_view_processing
. This resolves issues / prevents crashes in some rare corner cases when that feature is enabled and exception (likeMemory limit exceeded ...
) happened in the middle of materialized view processing. #30472 (filimonov). - Fixed segfault which might happen if session expired during execution of
REPLACE PARTITION
. #30432 (tavplubix). - Fixed
ComplexKeyHashedDictionary
,ComplexKeySparseHashedDictionary
parsingpreallocate
option from layout config. #30246 (Maksim Kita). - Fixed
[I]LIKE
function. Closes #28661. #30244 (Nikolay Degterinsky). - Support nullable arguments in function
initializeAggregation
. #30177 (Anton Popov). - Fixed data-race between
LogSink::writeMarks()
andLogSource
inStorageLog
. #29946 (Azat Khuzhin). - Fixed hanging DDL queries on Replicated database while adding a new replica. #29328 (Kevin Michel).
- Fixed bad optimizations of
ORDER BY
if it containsWITH FILL
. This closes #28908. This closes #26049. #28910 (alexey-milovidov). - Fixed queries to external databases (i.e. MySQL) with multiple columns in
IN
( i.e.(k,v) IN ((1, 2))
) (but note that this has some backward incompatibility for theclickhouse-copier
since it uses alias for tuple element). #28888 (Azat Khuzhin). - Fixed “Column is not under aggregate function and not in GROUP BY” with
PREWHERE
(Fixes: #28461). #28502 (Azat Khuzhin). - Fixed
NOT-IN
index optimization when not all key columns are used. This fixes #28120. #28315 (Amos Bird).
Bug Fixes (user-visible misbehaviour in official stable or prestable release
- Fixed
ORDER BY ... WITH FILL
with setTO
andFROM
and no rows in result set. #30888 (Anton Popov).
5 - Altinity Stable Build for ClickHouse 21.8.10 Release Notes
Release notes for Altinity Stable Build 21.8.10
ClickHouse release 21.8.10 as compared to v21.8.8
Improvements
- Update zoneinfo files to 2021c. #29925 (alexey-milovidov).
- Allow symlinks to files in user_files directory for file table function. #30309 (Kseniia Sumarokova).
Bug Fixes
- Fix shutdown of
AccessControlManager
. Now there can’t be reloading of the configuration after AccessControlManager has been destroyed. This PR fixes the flaky test test_user_directories/test.py::test_relative_path. #29951 (Vitaly Baranov). - Allow using a materialized column as the sharding key in a distributed table even if
insert_allow_materialized_columns=0
:. #28637 (Vitaly Baranov). - FlatDictionary, HashedDictionary fix bytes_allocated calculation for nullable attributes. #30238 (Maksim Kita).
- Dropped
Memory
database might reappear after server restart, it’s fixed (#29795). Also addedforce_remove_data_recursively_on_drop
setting as a workaround forDirectory not empty
error when droppingOrdinary
database (because it’s not possible to remove data leftovers manually in cloud environment). #30054 (tavplubix). - Fix crash of sample by
tuple()
, closes #30004. #30016 (Flynn). - Fix possible data-race between
FileChecker
andStorageLog
/StorageStripeLog
. #29959 (Azat Khuzhin). - Fix system tables recreation check (fails to detect changes in enum values). #29857 (Azat Khuzhin).
- Avoid
Timeout exceeded: elapsed 18446744073.709553 seconds
error that might happen in extremely rare cases, presumably due to some bug in kernel. Fixes #29154. #29811 (tavplubix). - Fix bad cast in
ATTACH TABLE ... FROM 'path'
query when non-string literal is used instead of path. It may lead to reading of uninitialized memory. #29790 (alexey-milovidov). - Fix concurrent access to
LowCardinality
duringGROUP BY
(leads to SIGSEGV). #29782 (Azat Khuzhin). - Fixed incorrect behaviour of setting
materialized_postgresql_tables_list
at server restart. Found in #28529. #29686 (Kseniia Sumarokova). - Condition in filter predicate could be lost after push-down optimisation. #29625 (Nikolai Kochetov).
- Fix rare segfault in
ALTER MODIFY
query when using incorrect table identifier inDEFAULT
expression likex.y.z...
Fixes #29184. #29573 (alesapin). - Fix bug in check
pathStartsWith
becuase there was bug with the usage ofstd::mismatch
:The behavior is undefined if the second range is shorter than the first range.
. #29531 (Kseniia Sumarokova). - In ODBC bridge add retries for error Invalid cursor state. It is a retriable error. Closes #29473. #29518 (Kseniia Sumarokova).
- Fix possible
Block structure mismatch
for subqueries with pushed-downHAVING
predicate. Fixes #29010. #29475 (Nikolai Kochetov). - Avoid deadlocks when reading and writting on JOIN Engine tables at the same time. #30187 (Raúl Marín).
- Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column. #30189 (Azat Khuzhin).
- Fix null deference for
GROUP BY WITH TOTALS HAVING
(when the column fromHAVING
wasn’t selected). #29553 (Azat Khuzhin).
6 - Altinity Stable Build for ClickHouse 21.8.8 Release Notes
A few months ago we certified ClickHouse 21.3 as an Altinity Stable release. Since then, we have worked on newer releases and run them in-house. We completed several new features, and even more have been added by community contributors. We were running ClickHouse 21.8 to power our public datasets at Altinity.Cloud instance and testing it in our environments. We have also performed a comprehensive QA of ClickHouse 21.8 to make sure upgrades go smoothly. As of 21.8.8 we are confident in certifying 21.8 as an Altinity Stable release.
This release is a significant upgrade since the previous Altinity Stable release. It includes 1472 pull requests from 224 contributors. Please look below for the detailed release notes.
Major new features since the previous stable release 21.3
A new release introduces a lot of changes and new functions. The full list is available in the Appendix, so refer to this section for more detail. The following new features are worth mentioning on the front page:
- SQL features:
DISTINCT ON
a subset of columns- Partial support of SQL/JSON standard
- Arrays in dictionaries are now supported
- Arrays and nested data types are now supported for Parquet and Arrow formats
- DateTime64 extended range, Now dates between 1925 to 2283 years are supported. a)
- Security features:
- Disk level encryption
- Kerberos authentication for HTTP protocol a)
- Active Directory groups mapping for LDAP user directory a)
- Replication and Cluster improvements:
- ClickHouse Keeper (experimental) – in-process ZooKeeper replacement
- SYSTEM RESTORE REPLICA – a handy tool that makes life easier in the unfortunate occasion if ZooKeeper metadata is lost a)
- Support for MySQL/PostgreSQL clusters when using mysql/postgresql table functions
- Zero-copy replication for S3 tables (see Altinity blog post for details)
- Parts movement between shards (experimental)
- MergeTree features:
- Projections (experimental)
_partition_id
and_partition_value
virtual columns were added- Deduplication for non-replicated tables. See non_replicated_deduplication_window merge tree setting
- Integrations:
MaterializedPostgreSQL
database engine for replication from PostgreSQL- HDFS disk support (experimental)
- Allow to catch Kafka errors into a separate stream (see the KB article on this)
- Other:
- YAML configuration format as an alternative to XML
As usual with ClickHouse, there are many performance and operational improvements in different server components.
a) - contributed by Altinity developers.
Backward Incompatible Changes
The following changes are backward incompatible and require user attention during an upgrade:
- Users of
LowCardinality(Nullable(...))
can not safely downgrade to versions older than 21.4. Data in tables with columns of those types inserted / merged by 21.8 will be lost after the downgrade. - Values of
UUID
type cannot be compared with integers. For example, instead of writinguuid != 0
typeuuid != '00000000-0000-0000-0000-000000000000'
- The
toStartOfIntervalFunction
will align hour intervals to midnight (in previous versions they were aligned to the start of unix epoch). For example,toStartOfInterval(x, INTERVAL 11 HOUR)
will split every day into three intervals:00:00:00..10:59:59
,11:00:00..21:59:59
and22:00:00..23:59:59
. - It’s not possible to rollback to the older ClickHouse version after executing
ALTER ... ATTACH
query as the old servers would fail to process the new command entryATTACH_PART
in the replicated log. - The behaviour of
remote_url_allow_hosts
has changed. In previous versions the empty section did nothing, in 21.8 it will block access to all external hosts. Remove this section from the configuration files after an upgrade if you experience issues with url() or s3() functions. - If you will downgrade to version before 21.1 clickhouse will not be able to start automatically – you will need to remove the system.*_log tables manually to downgrade
- There is an issue with uniqueState(UUID) in AggregatingMergeTree tables, and can be corrected by replacing
uniqState(uuid)
in MATERIALIZED VIEWs
with uniqState(sipHash64(uuid))
and change data type for already saved data fromAggregateFunction(uniq, UUID)
toAggregateFunction(uniq, UInt64)
. For more information see the following:
Upgrade Notes
There were several changes between versions that may affect the rolling upgrade of big clusters. Upgrading only part of the cluster is not recommended.
- Distributed queries with explicitly defined large sets are now executed differently. Compatibility setting
legacy_column_name_of_tuple_literal
may be enabled during the rolling upgrade of the cluster. Otherwise distributed queries with explicitly defined sets at IN clause may fail during upgrade. ATTACH PART[ITION]
queries may not work during cluster upgrade
Other Important Changes
ClickHouse embedded monitoring has become a bit more aggressive. It now collects several system stats, and stores them in the table system.asynchronious_metric_log
. This can be visible as an increase of background writes, storage usage, etc. To return to the old rate of metrics refresh / flush, adjust those settings in config.xml
:
<asynchronous_metrics_update_period_s>
60
</asynchronous_metrics_update_period_s>
<asynchronous_metric_log>
<flush_interval_milliseconds>
60000
</flush_interval_milliseconds>
</asynchronous_metric_log>
Alternatively, metric_log and asynchronous_metric_log tables can be completely disabled:
<yandex>
<asynchronous_metric_log remove="1"/>
<metric_log remove="1"/>
</yandex>
Some new ClickHouse features are now enabled by default. It may lead to a change in behaviour, so review those carefully and disable features that may affect your system:
async_socket_for_remote
compile_aggregate_expressions
compile_expressions
cross_to_inner_join_rewrite
insert_null_as_default
optimize_skip_unused_shards_rewrite_in
query_plan_enable_optimizations
query_plan_filter_push_down
In the previous releases we recommended disabling optimize_on_insert
. This recommendation stays for 21.8 as well as inserts into Summing
and AggregatingMergeTree
can slow down.
Known issues in 21.8.8
The development team continues to improve the quality of the 21.8 release. The following issues still exist in the 21.8.8 version and may affect ClickHouse operation. Please inspect them carefully to decide if those are applicable to your applications:
system.events for event = 'Merge'
are overstated. ClickHouse incorrectly increments this counter.Timeout exceeded: elapsed 18446744073.709553 seconds
error that might happen in extremely rare cases, presumably due to some bug in kernel.
You may also look into a GitHub issues using a special v21.8-affected label.
ClickHouse Altinity Stable Releases are based on the community versions. For more information on installing ClickHouse from either the Altinity Stable builds or the community builds, see the ClickHouse Altinity Stable Release Build Install Guide.
Please contact us at info@altinity.com if you experience any issues with the upgrade.
Appendix
New functions
- DateTime functions:
dateName
timeZone, timeZoneOf, timeZoneOffset, timezoneOf, toTimezone
- JSON processing functions:
JSON_EXISTS, JSON_QUERY, JSON_VALUE
simpleJSONExtractBool, simpleJSONExtractFloat, simpleJSONExtractInt, simpleJSONExtractRaw, simpleJSONExtractString, simpleJSONExtractUInt, simpleJSONHas
toJSONString
- String functions:
bin/unbin
leftPad, lpad, leftPadUTF8
rightPad, rpad, rightPadUTF8
splitByRegexp
- Array functions:
arrayProduct
bitPositionsToArray
validateNestedArraySizes
- Dictionary:
dictGetChildren
dictGetDescendants
dictGetOrNull
- Aggregate functions
deltaSumTimestamp
intervalLengthSum
lagInFrame/leadInFram
esequenceNextNode
sumCount
uniqTheta
- Geo:
polygonAreaCartesian, polygonAreaSpherical
polygonConvexHullCartesian
polygonPerimeterCartesian, polygonPerimeterSpherical
polygonsDistanceCartesian, polygonsDistanceSpherical
polygonsEqualsCartesian
polygonsIntersectionCartesian, polygonsIntersectionSpherical
polygonsSymDifferenceCartesian, polygonsSymDifferenceSpherical
polygonsUnionCartesian, polygonsUnionSpherical
polygonsWithinCartesian, polygonsWithinSpherical
readWktMultiPolygon, readWktPoint, readWktPolygon, readWktRing
wkt
- Math/Statistics:
quantileBFloat16, quantilesBFloat16
- Type related:
reinterpretAsUInt128
toUInt128, toUInt128OrNull, toUInt128OrZero
- Other:
indexHint
isIPAddressInRange
partitionId
– that corresponds to_partition_id virtual
column added to MergeTree tables
New table functions
dictionary
s3Cluster
– see our blog article that highlights this one
New table engines
ExternalDistributed
– allows to query MySQL or PostgreSQL clustersMaterializedPostgreSQL
New metrics and events
system.asynchronous_metrics
AsynchronousMetricsCalculationTimeSpent
The table has also got a lot of new metrics for a host monitoring rather than ClickHouse.
system.metrics
BrokenDistributedFilesToInsert
MMappedFileBytes
NetworkReceive
NetworkSend
system.events
MMappedFileCacheHits
MMappedFileCacheMisses
MergeTreeDataProjectionWriterBlocks
MergeTreeDataProjectionWriterBlocksAlreadySorted
MergeTreeDataProjectionWriterCompressedBytes
MergeTreeDataProjectionWriterRows
MergeTreeDataProjectionWriterUncompressedBytes
NetworkReceiveBytes
NetworkSendBytes
StorageBufferLayerLockReadersWaitMilliseconds
StorageBufferLayerLockWritersWaitMilliseconds
StorageBufferPassedBytesFlushThreshold
StorageBufferPassedRowsFlushThreshold
StorageBufferPassedTimeFlushThreshold
system.errors
CANNOT_CREATE_FILE
CANNOT_PARSE_YAML
CANNOT_SET_ROUNDING_MODE
CANNOT_SYSCONF
CONCURRENT_ACCESS_NOT_SUPPORTED
DISTRIBUTED_BROKEN_BATCH_FILES
DISTRIBUTED_BROKEN_BATCH_INFO
ILLEGAL_PROJECTION
INCORRECT_PART_TYPE
INVALID_FORMAT_INSERT_QUERY_WITH_DATA
KERBEROS_ERROR
NO_SUCH_PROJECTION_IN_TABLE
POSTGRESQL_CONNECTION_FAILURE
PROJECTION_NOT_USED
TOO_LARGE_DISTRIBUTED_DEPTH
UNKNOWN_SNAPSHOT
New system tables
data_skipping_indices
:- database, table, name, type, expr, granularity
part_moves_between_shards
:- database, table, task_name, task_uuid, create_time, part_name, part_uuid, to_shard, update_time, state, num_tries, last_exception
projection_parts
– same as parts but with extra ‘parent_*’ columnsprojection_parts_columns
– same as parts_columns but with extra ‘parent_*’ columns
New columns in system tables
- clusters
shutdown_count
- dictionaries:
key
– removedkey.names, key.types, found_rate
- distribution_queue:
broken_data_files, broken_data_compressed_bytes
- errors:
last_error_time, last_error_message, last_error_trace, remote
- metric_log:
- multiple columns
- parts_columns:
uuid, min_time, max_time
- processes:
- Settings (Map replaces Settings.Names/Values arrays), ProfileEvents (Map replaces ProfileEvents.Names/Values arrays), current_database
- query_log:
projections, initial_query_start_time, initial_query_start_time_microseconds
, Settings (Map replaces Settings.Names/Values arrays), ProfileEvents (Map replaces ProfileEvents.Names/Values arrays)
- query_thread_log:
initial_query_start_time_microseconds
, Settings (Map replaces Settings.Names/Values arrays), ProfileEvents (Map replaces ProfileEvents.Names/Values arrays)
- stack_trace:
thread_name
- table_engines:
supports_projections
- tables:
- comment
- users
grantees_any, grantees_list, grantees_except
System.merge_tree_settings added/changed
Name | Old value | New value | Description |
---|---|---|---|
max_parts_to_merge_at_once | 100 | Max amount of parts which can be merged at once (0 - disabled). Doesn’t affect OPTIMIZE FINAL query. | |
max_replicated_fetches_network_bandwidth | 0 | The maximum speed of data exchange over the network in bytes per second for replicated fetches. Zero means unlimited. | |
max_replicated_sends_network_bandwidth | 0 | The maximum speed of data exchange over the network in bytes per second for replicated sends. Zero means unlimited. | |
min_bytes_to_rebalance_partition_over_jbod | 0 | Minimal amount of bytes to enable part rebalance over JBOD array (0 - disabled). | |
non_replicated_deduplication_window | 0 | How many last blocks of hashes should be kept on disk (0 - disabled). | |
part_moves_between_shards_delay_seconds | 30 | Time to wait before/after moving parts between shards. | |
part_moves_between_shards_enable | 0 | Experimental/Incomplete feature to move parts between shards. Does not take into account sharding expressions. | |
replicated_fetches_http_connection_timeout | 0 | HTTP connection timeout for part fetch requests. Inherited from default profile http_connection_timeout if not set explicitly. | |
replicated_fetches_http_receive_timeout | 0 | HTTP receive timeout for fetch part requests. Inherited from default profile http_receive_timeout if not set explicitly. | |
replicated_fetches_http_send_timeout | 0 | HTTP send timeout for part fetch requests. Inherited from default profile http_send_timeout if not set explicitly. | |
remote_fs_execute_merges_on_single_replica_time_threshold | 10800 | When greater than zero only a single replica starts the merge immediately when merged part on shared storage and ‘allow_remote_fs_zero_copy_replication’ is enabled. |
system.settings added/changed
Type | Name | Old value | New value | Description |
---|---|---|---|---|
settings | allow_experimental_bigint_types | 0 | 1 | Obsolete setting, does nothing. |
settings | allow_experimental_codecs | 0 | If it is set to true, allow to specify experimental compression codecs (but we don't have those yet and this option does nothing). | |
settings | allow_experimental_database_materialized_postgresql | 0 | Allow to create database with Engine=MaterializedPostgreSQL(…). | |
settings | allow_experimental_funnel_functions | 0 | Enable experimental functions for funnel analysis. | |
settings | allow_experimental_map_type | 0 | 1 | Obsolete setting, does nothing. |
settings | allow_experimental_projection_optimization | 0 | Enable projection optimization when processing SELECT queries | |
settings | async_socket_for_remote | 0 | 1 | Asynchronously read from socket executing remote query |
settings | background_schedule_pool_size | 16 | 128 | Number of threads performing background tasks for replicated tables, dns cache updates. Only has meaning at server startup. |
settings | compile_aggregate_expressions | 1 | Compile aggregate functions to native code. | |
settings | compile_expressions | 0 | 1 | Compile some scalar functions and operators to native code. |
settings | cross_to_inner_join_rewrite | 1 | Use inner join instead of comma/cross join if possible | |
settings | database_replicated_always_detach_permanently | 0 | Execute DETACH TABLE as DETACH TABLE PERMANENTLY if database engine is Replicated | |
settings | distributed_ddl_entry_format_version | 1 | Version of DDL entry to write into ZooKeeper | |
settings | distributed_ddl_output_mode | throw | Format of distributed DDL query result | |
settings | distributed_directory_monitor_split_batch_on_failure | 0 | Should StorageDistributed DirectoryMonitors try to split batch into smaller in case of failures. | |
settings | distributed_push_down_limit | 0 | If 1, LIMIT will be applied on each shard separatelly. Usually you don't need to use it, since this will be done automatically if it is possible, i.e. for simple query SELECT FROM LIMIT. | |
settings | experimental_query_deduplication_send_all_part_uuids | 0 | If false only part UUIDs for currently moving parts are sent. If true all read part UUIDs are sent (useful only for testing). | |
settings | external_storage_max_read_bytes | 0 | Limit maximum number of bytes when table with external engine should flush history data. Now supported only for MySQL table engine, database engine, dictionary and MaterializeMySQL. If equal to 0, this setting is disabled | |
settings | external_storage_max_read_rows | 0 | Limit maximum number of rows when table with external engine should flush history data. Now supported only for MySQL table engine, database engine, dictionary and MaterializeMySQL. If equal to 0, this setting is disabled | |
settings | force_optimize_projection | 0 | If projection optimization is enabled, SELECT queries need to use projection | |
settings | glob_expansion_max_elements | 1000 | Maximum number of allowed addresses (For external storages, table functions, etc). | |
settings | group_by_two_level_threshold_bytes | 100000000 | 50000000 | From what size of the aggregation state in bytes, a two-level aggregation begins to be used. 0 - the threshold is not set. Two-level aggregation is used when at least one of the thresholds is triggered. |
settings | handle_kafka_error_mode | default | Obsolete setting, does nothing. | |
settings | http_max_field_name_size | 1048576 | Maximum length of field name in HTTP header | |
settings | http_max_field_value_size | 1048576 | Maximum length of field value in HTTP header | |
settings | http_max_fields | 1000000 | Maximum number of fields in HTTP header | |
settings | http_max_uri_size | 1048576 | Maximum URI length of HTTP request | |
settings | insert_null_as_default | 1 | Insert DEFAULT values instead of NULL in INSERT SELECT (UNION ALL) | |
settings | legacy_column_name_of_tuple_literal | 0 | List all names of element of large tuple literals in their column names instead of hash. This settings exists only for compatibility reasons. It makes sense to set to 'true', while doing rolling update of cluster from version lower than 21.7 to higher. | |
settings | max_distributed_depth | 5 | Maximum distributed query depth | |
settings | max_replicated_fetches_network_bandwidth_for_server | 0 | The maximum speed of data exchange over the network in bytes per second for replicated fetches. Zero means unlimited. Only has meaning at server startup. | |
settings | max_replicated_sends_network_bandwidth_for_server | 0 | The maximum speed of data exchange over the network in bytes per second for replicated sends. Zero means unlimited. Only has meaning at server startup. | |
settings | min_count_to_compile_aggregate_expression | 3 | The number of identical aggregate expressions before they are JIT-compiled | |
settings | normalize_function_names | 1 | 0 | Normalize function names to their canonical names |
settings | odbc_bridge_connection_pool_size | 16 | Connection pool size for each connection settings string in ODBC bridge. | |
settings | optimize_functions_to_subcolumns | 0 | Transform functions to subcolumns, if possible, to reduce amount of read data. E.g. 'length(arr)' -> 'arr.size0', 'col IS NULL' -> 'col.null' | |
settings | optimize_fuse_sum_count_avg | 0 | Fuse aggregate functions sum(), avg(), count() with identical arguments into one sumCount() call, if the query has at least two different functions | |
settings | optimize_move_to_prewhere_if_final | 0 | If query has FINAL , the optimization move_to_prewhere is not always correct and it is enabled only if both settings optimize_move_to_prewhere and optimize_move_to_prewhere_if_final are turned on |
|
settings | optimize_skip_unused_shards_limit | 1000 | Limit for number of sharding key values, turns off optimize_skip_unused_shards if the limit is reached | |
settings | optimize_skip_unused_shards_rewrite_in | 1 | Rewrite IN in query for remote shards to exclude values that does not belong to the shard (requires optimize_skip_unused_shards) | |
settings | output_format_arrow_low_cardinality_as_dictionary | 0 | Enable output LowCardinality type as Dictionary Arrow type | |
settings | postgresql_connection_pool_size | 16 | Connection pool size for PostgreSQL table engine and database engine. | |
settings | postgresql_connection_pool_wait_timeout | 5000 | Connection pool push/pop timeout on empty pool for PostgreSQL table engine and database engine. By default it will block on empty pool. | |
settings | prefer_column_name_to_alias | 0 | Prefer using column names instead of aliases if possible. | |
settings | prefer_global_in_and_join | 0 | If enabled, all IN/JOIN operators will be rewritten as GLOBAL IN/JOIN. It's useful when the to-be-joined tables are only available on the initiator and we need to always scatter their data on-the-fly during distributed processing with the GLOBAL keyword. It's also useful to reduce the need to access the external sources joining external tables. | |
settings | query_plan_enable_optimizations | 1 | Apply optimizations to query plan | |
settings | query_plan_filter_push_down | 1 | Allow to push down filter by predicate query plan step | |
settings | s3_max_single_read_retries | 4 | The maximum number of retries during single S3 read. | |
settings | sleep_in_send_data | 0 | ||
settings | sleep_in_send_data_ms | 0 | Time to sleep in sending data in TCPHandler | |
settings | sleep_in_send_tables_status | 0 | ||
settings | sleep_in_send_tables_status_ms | 0 | Time to sleep in sending tables status response in TCPHandler | |
settings | use_antlr_parser | 0 |
Also, please refer to the release notes from the development team available at the following URLs:
- 21.4 release notes: https://clickhouse.com/docs/en/whats-new/changelog/#clickhouse-release-21-4
- 21.5 release notes: https://clickhouse.com/docs/en/whats-new/changelog/#clickhouse-release-21-5-2021-05-20
- 21.6 release notes: https://clickhouse.com/docs/en/whats-new/changelog/#clickhouse-release-21-6-2021-06-05
- 21.7 release notes: https://clickhouse.com/docs/en/whats-new/changelog/#clickhouse-release-v21-7-2021-07-09
- 21.8 release notes: https://clickhouse.com/docs/en/whats-new/changelog/#clickhouse-release-v21-8-2021-08-12