Release Notes - 7.4
7.4.10
AVX enabled release.
Update to RocksDB 8.11.5 again. This contains a disk format change (originally made in RocksDB 9.0) such that it cannot be downgraded back to RocksDB 8.11.4 or lower, and no other changes. This new disk format is fully forwards and backwards compatible with RocksDB 9+ (for how FDB uses it). Note that RocksDB-9.7.3 was used in FoundationDB-7.4.0, downgraded to RocksDB-8.11.5 in FoundationDB-7.4.6, and then to RocksDB-8.11.4 in FoundationDB-7.4.7. All 7.4 releases before FoundationDB-7.4.7 were considered pre-stable-release, apologies for the confusion. (PR #14042), (PR #14554)
If you use the RocksDB storage engine, and care about the possibility of rolling-back a deployment, this FoundationDB release is meant to be useful as a step to the new RocksDB disk format. Once you have upgraded to FDB 7.4.9 and have confidence in that version, this FDB 7.4.10 release should have identical behavior in all respects, but cannot be downgraded back.
FoundationDB |
RocksDB |
Can Downgrade FDB |
|---|---|---|
7.3.40+ |
8.11.4 |
YES |
7.4.0 |
9.7.3 |
NO |
7.4.6 |
8.11.5 |
TO 7.4.0 - 7.4.5 |
7.4.7 |
8.11.4 |
TO 7.3.x |
7.4.9 |
8.11.4 |
TO 7.3.x, 7.4.7, 7.4.8 |
7.4.10 |
8.11.5 |
TO 7.4.0 - 7.4.6 |
8.0.0 |
11.1.2 |
TO 7.4.10, 7.4.0 - 7.4.6 |
See also note about encrypted backups in release 7.4.7 below.
7.4.9
AVX enabled release.
Rejected file and object names that are absolute or contain a
..component in places where a name is joined onto a base directory: local directory backup containers, backup file listing, S3 copy-down, bulk load job and task manifests, the profiler output file, and SST file names in fetched checkpoints. (PR #14334), (PR #14433)On rhel9 based distros, avoid linking GLIBC_2.35 symbols. This fixes compatibility with RHEL 9.6 (and equivalent), and other linux distributions with glibc-2.34 (which lack some symbol backports added in RHEL 9.7). This issue affected FoundationDB 7.4.7 and 7.4.8. (PR #14274)
Fixed a client crash (SIGSEGV) when opening a tenant on a database whose cluster file could not be used (missing, or not a valid connection string). Operations on the tenant now fail with the database’s error instead. (PR #14308)
Fixed clients hanging, with every transaction timing out (error 1031) under the multi-version client, when the connection string still listed a coordinator hostname that no longer resolves in DNS. The client’s protocol version check kept retrying that hostname instead of following the switch to a working coordinator. (PR #14338)
Fixed global configuration permanently missing the latest committed update after a protocol upgrade, when a refresh accepted an older cached GRV proxy snapshot. (PR #14363)
Fixed data distribution picking its primary DC from the highest-priority configured region instead of the active master’s DC, which after a region failover made it treat the old primary region as active. (PR #14387)
Fixed a memory leak in fdbmonitor on Linux each time a watched configuration file symlink is re-resolved. (PR #14379)
Fixed
can_clean_bouncealways being false instatus json. (PR #14272)Fixed storage server
finished_queriesdrifting belowtotal_querieswhen a read was cancelled or failed, which madequery_queue_maxin status json grow permanently. (PR #13908)Fixed one backup with a missing or invalid container URL dropping every backup from the backup layer status in
status json(layers.backup). The other backups are still reported, and the failing one gets anerrorfield. (PR #14491)Mutation log tasks of a backup whose snapshot has finished now run ahead of other backups’ initial snapshot tasks, so a new backup’s first snapshot no longer starves other backups’ logs and increases their backup lag. This raises the
TASKBUCKET_MAX_PRIORITYclient knob default from 1 to 2. Backup agents older than 7.4.9 only run tasks up to priority 1, so keep at least one 7.4.9 or newer backup agent running for backups that ran under 7.4.9. (PR #14390)Added per-URL blobstore parameters for backups:
server_side_encryption(orsse) overrides theBLOBSTORE_ENCRYPTION_TYPEknob (sse=nonesends no server-side encryption header),proxyoverrides the--proxyoption and proxy environment variables (proxy=noneconnects directly), andaws_v4_header(orv4) overrides theHTTP_REQUEST_AWS_V4_HEADERknob. The URL is saved with the backup, so each backup on a cluster can use its own values. (PR #14420), (PR #14471)fdbbackup statusnow shows the backup’s mutation log type (defaultorpartitioned-log-experimental). The same value is infdbbackup status --json(MutationLogType), instatus jsonon each backup tag underlayers.backup(mutation_log_type), and infdbrestore status(Mutation Log Type:). (PR #14335)Added a bounded-time rollback for shard-encoded location metadata (
SHARD_ENCODE_LOCATION_METADATA). Newconfigureoptionsshard_metadata_format={original|encoded}andshard_metadata_migration={enabled|disabled}(defaultdisabled) let data distribution actively rewrite shard-encoded metadata back to the original format when it starts, so the binary can be downgraded (moving toencodedstill happens through normal data moves), andaudit_storage metadata_encodingin fdbcli reports the encoding state of the cluster. Nothing changes until an operator sets these options. (PR #13965)Bulk dump now picks its target DC from the region configuration: the primary region by default, or the remote region when the new
DD_BULKDUMP_USE_REMOTE_REGIONknob is set and the cluster has more than one usable region. Previously the target was decided by the lexicographic order of DC ids. (PR #14276), (PR #14385)Added opt-in audit logging to fdbcli. With
--audit-log(orFDB_CLI_AUDIT_LOG_ENABLE), every key returned byget,getrange,getrangekeysandgetallis logged as anFdbCliAudittrace event.--audit-log-values(none,systemorall, defaultsystem) picks which values are logged. Audit events go to their owntrace_audit.*files, in theFDB_CLI_AUDIT_LOG_ENABLEvalue or--audit-log-dirif set, else next to the trace files, else the current directory. (PR #14485)Added opt-in JSON audit logging to
transaction_profiling_analyzer.py. With--audit-log(orFDB_TRANSACTION_PROFILER_AUDIT_LOG_ENABLE), every key, range and mutation it prints is appended totransaction_profiling_analyzer_audit.json, in theFDB_TRANSACTION_PROFILER_AUDIT_LOG_ENABLEvalue or--audit-log-dirif set, else theFDB_NETWORK_OPTION_TRACE_ENABLEdirectory, else the current directory. (PR #14485)
7.4.8
AVX enabled release.
Reconciled uncertain S3 multipart upload completions: a 200 response that embeds an error is now treated as a failure, and a completion that returns NoSuchUpload or a transport error is verified against the committed object with a HEAD request before being reported as successful. (PR #14089), (PR #14093)
Fixed RPM build-id link conflicts that prevented installing versioned client packages alongside the main client package. (PR #14103)
Changed debug level to “-g1” and enabled compression for “*.debug” release artifacts, to make them much smaller. Also changed to include the main executable in the “*.debug” release artifact so it can optionally be run directly, on its own. (PR #14103), (PR #14131)
7.4.7
AVX enabled release.
Avoided restarting transaction-system recovery when an old-epoch backup worker fails. (PR #13939)
Moved
waitForShardReadyoutside the transaction used to finish data moves, preventing slow destination servers from exhausting the transaction lifetime and stalling data distribution. (PR #13642)Fixed a use-after-free when cancelling S3 backup file copies. (PR #13916)
Fixed server knob overrides being silently skipped when a knob name is shared with client knobs. (PR #13932)
Added opt-in eviction of stale client connections to failed storage servers and proxies. (PR #13912)
Disabled data distribution global admission control by default. (PR #13917)
Fixed crashes and incorrect behavior in snapshot options, cancelled file reads, data distribution shutdown, and TLog minimum popped-version tracking. (PR #13871)
Fixed an incorrect shard assignment after an excluded server failed. (PR #13869)
Added a retry limit for repeated
startMoveKeysfailures to prevent infinite retries. (PR #13782)Fixed exclusion tracking for data distribution. (PR #13766)
Fixed read and write overflow handling for snapshot manifests larger than 2 GB. (PR #13820)
Added GCM authentication tags to encrypted backup files and fixed encrypted restore block-size initialization. (PR #13777)
Dropped stale backup workers during recovery. (PR #13655)
Fixed successful automatic-idempotency replays not populating the committed version. (PR #13294)
Fixed read-ahead cache reads extending past the end of small or final blocks. (PR #13292)
Improved RocksDB memory accounting by charging write buffers and selected memory usage to the block cache. (PR #13371), (PR #13346)
Improved data distribution relocation backoff and admission control to reduce retry contention and pipeline pressure. (PR #13145), (PR #13280)
Added mutation-log-type configuration for backups. (PR #13127)
Fixed encrypted restore failures, including a
platform_errorcrash. (PR #13118)Made cluster status requests asynchronous so status retrieval can return before the timeout. (PR #13099)
Hardened S3 error handling for non-XML and HTTP 4xx responses. (PR #12996), (PR #13021)
Fixed
minRestorableVersionandmaxRestorableVersionupdates when mutation logs are missing. (PR #12710)Restricted
backup_worker_enabledconfiguration throughfdbcli. (PR #12711)Enabled TLS hostname validation against certificate CN and SAN values. (PR #12730)
Downgraded RocksDB to 8.11.4. (PR #13830)
Encrypted Backups are production-ready starting with 7.4.7. Encrypted backups created with earlier versions are not compatible with 7.4.7+ and cannot be restored due to a format change. If using this feature, create a new encrypted backup with 7.4.7+.
7.4.6
AVX enabled release.
Fixed a restore stuck bug where readLogData() errors were not propagated correctly. (PR #12433)
Fixed FdbDecode memory issues. (PR #12495)
Fixed decode range file keys and values interpretation bug. (PR #12420)
Fixed the flags used for opening the backup encryption key file. (PR #12424)
Fixed backup encryption on S3. (PR #12289)
Added support for backup encryption via fdbbackup modify command. (PR #12591)
Added support for backup worker to use proxy from command line to upload to S3. (PR #12566)
Added metadata for encrypted backups. (PR #12354)
Added FileLevelEncryption field to BackupDescription JSON output. (PR #12626)
Added encryption key info to status JSON. (PR #12657)
Added SNI in TLS handshake. (PR #12385)
Added ARM support for Docker images. (PR #12425)
Added restore validation feature allowing backup/restore validation in a single cluster. (PR #12648)
Downgraded RocksDB to 8.11.5. (PR #12651)
7.4.5
Same as 7.4.4 release with AVX enabled.
7.4.4
Fixed a race that could cause recovery to be stuck when purging old generations. (PR #12214)
Fixed TSS mismatch handling crashes. (PR #12330)
Fixed potential DatabaseContext leaks in multi-version client. (PR #12308)
Fixed double-encoding of URIs in requests to S3BlobStore. (PR #12302)
Fixed “Unknown error” when configuring regions. (PR #12314)
Improved TLS handshake process to avoid blocking main thread. (PR #12346)
Fixed an issue in TLog server code that was causing OOMs. (PR #12298)
7.4.3
Same as 7.4.2 release with AVX enabled.
7.4.2
Fixed a backup restore speed regression. (PR #12098)
Added backup URL validation for directory traversals. (PR #12143)
Fixed an issue where backup status update transaction would not retry in case of failure. (PR #12175)
Added various improvements to bulk dump/load. (PR #12135)
Turned off replica consistency check knobs by default which were overloading storage servers. (PR #12133), (PR #12136)
Fixed an issue where source storage server could be overloaded by data movements with replica consistency check. (PR #12176)
Added support for checkall command with DCID. (PR #12100)
Optimized keyInfo lookup in commit proxy. (PR #12106)
7.4.1
Same as 7.4.0 release with AVX enabled.
7.4.0
Features (Supported)
Added support to restore from new backup’s partitioned log files. (PR #11901)
Added LRU-like cache replacement for in-memory page checksums to save memory usage. (PR #11194)
Added gray failure features to track remote processes and allow complaints from storage servers. (PR #11717), (PR #11753)
Added network option to disable non-TLS connections. (PR #9984)
Features (Experimental)
Added support to bulk load TBs’ snapshot of key-values from S3 to an empty cluster. (Bulk Load User Guide), (PR #11369)
Added support to bulk dump TBs’ snapshot of key-values to S3 from an idle cluster. (Bulk Dump User Guide), (PR #11780)
Added support to upload/download to/from S3 (for bulk dump and bulk load). (PR #11899)
Added support to perform exclusive read range lock that blocks user write traffic to a specific range. (Range Lock User Guide), (PR #11693)
Added multiple improvements to the version vector feature, so that commits are sent only to tlogs buddied with the storage server that will receive the mutations.
Added support to compute mutation and accumulative checksums to conduct real-time detection of mutation corruptions on write path. (PR #11255)
Added support to detect hot shards and throttle commits to them. (PR #10970)
Added support to synthesize test data on a cluster. (PR #11107)
Added support to compare storage replicas on reads. (PR #11235)
Added gRPC integration with Flow. (PR #11782), (PR #11892), (PR #12023)
Performance
Improved storage performance with ptree optimizations. (PR #11435)
Added yields to backup agents to avoid slow tasks. (PR #10878)
Fixes
Fixed a fdbmonitor issue on FreeBSD where child processes continued to run after fdbmonitor termination. (PR #11361)
Fixed issues where backup workers missed mutations and caused assertion failures. (PR #11908), (PR #12026), (PR #12046)
Fixed AuditStorage empty range read error. (PR #12043)
Fixed an issue where failover was triggered even though remote storage servers were lagging behind. (PR #11054)
Fixed an issue where fdbserver was not being able to join the cluster during an upgrade. (PR #9814)
Fixed an assert in GetMappedRange that depends on the range read returning once it has at least one result. (PR #10522)
Fixed an issue where clients connecting to coordination server were never getting a response. (PR #10363)
Fixed an issue where describeBackup() never updated continousLogEnd property. (PR #10488)
Fixed an issue where watch was stuck due to races. (PR #11112)
Fixed an issue where ConnectionMonitor would incorrectly close connections. (PR #10495)
Fixed an issue where TSS conversion can be stuck sometimes. (PR #10711)
Fixed an issue where status timeout error on Cluster Controller was incorrectly triggering recovery. (PR #10791)
Status
Added RocksDB version to status JSON. (PR #11868)
Added support to fetch a specific group of status JSON fields. (PR #10927)
Added gray failure excluded processes to status JSON. (PR #11758)
Bindings
Fixed an issue where not calling Close() on the db object would result in memory leak in Go bindings. This is a breaking change since clients are now expected to close their db to avoid memory leak. (PR #11394)
Fixed an issue causing SIGSEGV when network routine was started multiple times concurrently in Go bindings. (PR #11104)
Fixed an issue where user’s transaction function FoundationDB error was getting reset in Go bindings. (PR #11810)
Fixed an issue causing panic when connecting to database from multiple threads in Go bindings. (PR #10702)
Added support to cancel snapshots and R/O transactions in Go bindings. (PR #11614)
Added GetClientStatus method to database in Go bindings. (PR #11627)
Other Changes
Removed upgrade support from 6.2 and earlier TLogs and made xxhash checksum the default for TLog. (PR #11667)
Added rate keeper logs for zones with lowest tps. (PR #11067)
Added LOG_CONNECTION_ATTEMPTS_ENABLED and CONNECTION_LOG_DIRECTORY to log all incoming connections to an external file. (PR #11704)
Added exclude in progress signal to fdbcli. (PR #11569)
Added a sidecar container that refreshes S3 credentials. (PR #11945)
Fixed an issue where storage and tlog store types were not valid as part of configure command. (PR #10876)
Improved BytesWritten in MovingData trace event to account for non-overlapped server lists. (PR #10076)
Dependencies
Upgraded boost to version 1.86. (PR #11788)
Upgraded awssdk to version 1.11.473. (PR #11853)
Upgraded RocksDB to 9.7.3. (PR #11735)
Added support for GCC 13 and Clang 19 compilers.