Release Notes
8.0.1
AVX enabled release.
Raised the protocol versions that 8.0 binaries treat as the next release from 8.1 to 9.0 (future protocol version 9.0, minimum invalid protocol version 9.1). 8.0.0 rejects data stamped with protocol 8.2 or later, so a future 9.0 cluster can be downgraded only to 8.0.1 or later 8.0.x releases, not to 8.0.0. The current protocol version, minimum compatible version and API version are unchanged. (PR #14423)
Rejected file and object names that are absolute or contain a
..component in places where a name is joined onto a base directory: local directory backup containers, backup file listing, S3 copy-down, bulk load job and task manifests, the profiler output file, and SST file names in fetched checkpoints. (PR #14421)Avoided linking newer glibc symbols into RHEL 9 builds that prevented binaries built on RHEL 9.7 from starting on RHEL 9.6 systems and containers. (PR #14278)
Fixed network setup failures when older multiversion clients load the 8.0 client, even when the application does not use blob granules. Restored the required blob granule C symbols as stubs that abort the process when called. (PR #14239)
Removed the unused blob granule types
FDBBGMutationTypeand theFDBBG*file description structs fromfdb_c.h. They were only used by the blob granule read API removed in 8.0.0.FDBGranuleSummaryandFDBReadBlobGranuleContextare kept for the compatibility stubs. (PR #14247)Outbound TLS connections that fdbserver opens by hostname (S3 and other HTTPS blob stores, REST and KMS servers, including connections through an HTTP proxy) are now verified by CA chain and hostname instead of the cluster’s
--tls-verify-peersrules. Previously, on clusters that set peer rules, every S3 server certificate was rejected, which broke bulk dump and bulk load uploads. Cluster traffic keeps using the peer rules. (PR #14469)Fixed an interrupted remote-region recovery possibly starting the next generation one version too late and leaving a storage server behind, because a new log router reported its synthetic startup pop floor as committed progress. (PR #14295)
Added the
SS_SKIP_DELIVERED_TO_NOT_ASSIGNED_VERSIONserver knob (default0, disabled) to recover a storage server that crash-loops on a mutation delivered to a shard it is not assigned. Setting it to that one version makes the storage server drop such mutations at that version instead of asserting. (PR #14480)Fixed DR (
fdbdr) mutation log copying possibly skipping mutations when a copy step timed out. Copying now finishes the current version and resumes at the first version it did not copy. (PR #14296)Fixed a race in the client library where an application thread could destroy a result while the network thread was still completing it, which could crash the client. (PR #14373)
Fixed a client memory leak when setting watches on many distinct keys. Watch reference-count entries are now removed once no watch uses them. (PR #14490)
Retried
fallocate()andftruncate()when they fail withEINTRin the Linux KAIO file truncate path, instead of failing the operation. (PR #14326)Fixed Redwood page-cache memory accounting when a reused page ID changes size, so cache eviction uses the current page size. (PR #14291)
Fixed a Linux
fdbmonitormemory leak when changes to a configuration-file symlink caused its target path to be resolved again. (PR #14253)Remote-region TLogs reuse recent disk-space samples in the low-disk admission check instead of calling
statvfs()for every version they pull. On a two-region cluster under write load, the per-version disk checks made spilling fall behind, and remote TLogs ran out of memory during a machine exclude. (PR #14345)Reduced CPU spent on every completed RPC reply. Cancelling the per-request failure monitor wait no longer throws C++ exceptions. (PR #14352)
Cached the network interface name used for network-traffic metrics to reduce periodic network-thread stalls on hosts with many interfaces. (PR #14283)
Fixed one backup with a missing or invalid container URL dropping every backup from the backup layer status in
status json(layers.backup). The other backups are still reported, and the failing one gets anerrorfield. (PR #14484)Mutation log tasks of a backup whose snapshot has finished now run ahead of other backups’ initial snapshot tasks, so a new backup’s first snapshot no longer starves other backups’ logs and increases their backup lag. This raises the
TASKBUCKET_MAX_PRIORITYclient knob default from 1 to 2. Backup agents older than 8.0.1 (and older than 7.4.9 on 7.4) only run tasks up to priority 1, so keep at least one 8.0.1 or newer backup agent running for backups that ran under 8.0.1, and do not downgrade all backup agents while a backup is running. (PR #14499)Added bounded retries for disk errors when opening or reading backup encryption-key files, allowing transient failures to recover before the backup container retains a failed initialization result. (PR #14294)
Added per-URL blobstore parameters for backups:
server_side_encryption(orsse) overrides theBLOBSTORE_ENCRYPTION_TYPEknob (sse=nonesends no server-side encryption header),proxyoverrides the--proxyoption and proxy environment variables (proxy=noneconnects directly), andaws_v4_header(orv4) overrides theHTTP_REQUEST_AWS_V4_HEADERknob. The URL is saved with the backup, so each backup on a cluster can use its own values. (PR #14519), (PR #14521)Range file restores now send one clear per restore transaction instead of one clear per gap between restored keys, and skip that clear when the range has no existing keys. In tests on 20-storage-server clusters this made restores about 2x faster and roughly halved storage server CPU. Setting the client knobs
RESTORE_RANGE_DISABLE_RYWorRESTORE_RANGE_READ_BEFORE_CLEAR(both defaulttrue) tofalseon the backup agents restores the previous behavior. (PR #14513), (PR #14525)Bulk dump now picks its target DC from the region configuration: the primary region by default, or the remote region when the new
DD_BULKDUMP_USE_REMOTE_REGIONknob is set and the cluster has more than one usable region. Previously the target was decided by the lexicographic order of DC ids. (PR #14275), (PR #14384)Added opt-in audit logging to fdbcli. With
--audit-log(orFDB_CLI_AUDIT_LOG_ENABLE), every key returned byget,getrange,getrangekeysandgetallis logged as anFdbCliAudittrace event.--audit-log-values(none,systemorall, defaultsystem) picks which values are logged. Audit events go to their owntrace_audit.*files, in theFDB_CLI_AUDIT_LOG_ENABLEvalue or--audit-log-dirif set, else next to the trace files, else the current directory. (PR #14486)Added opt-in JSON audit logging to
transaction_profiling_analyzer.py. With--audit-log(orFDB_TRANSACTION_PROFILER_AUDIT_LOG_ENABLE), every key, range and mutation it prints is appended totransaction_profiling_analyzer_audit.json, in theFDB_TRANSACTION_PROFILER_AUDIT_LOG_ENABLEvalue or--audit-log-dirif set, else theFDB_NETWORK_OPTION_TRACE_ENABLEdirectory, else the current directory. (PR #14486)Fixed truth testing of an absent value in the Python bindings on Python 3.
if not tr[key]:raisedTypeErrorfor a missing key and now evaluates as false. (PR #14517)Fixed
FDB::Tuple.compareandFDB::Tuple::SingleFloatcomparisons raisingArgumentErroron Ruby 3.3 and later when comparing floating point values. (PR #14518)Changed the Kubernetes monitor image to declare its existing non-root user numerically, allowing it to start with
runAsNonRoot: truewhenrunAsUseris not specified. (PR #14248)Updated
fdb-kubernetes-monitor: it retries loading its configuration while thefdbserverbinary is missing, empty or not executable, can be configured to always use the shared binaries (useful for compatible version upgrades without recreating pods), and exposes the staged and runningfdbserverversions in a pod annotation for the operator. Its Go toolchain and dependencies now matchmain. (PR #14556)
8.0.0
AVX enabled release.
Compatibility and removed features
The API version is now 800. Native CDC’s C API requires applications to select API version 800. (PR #12510), (PR #13674)
Removed the experimental multitenancy and metacluster features, including their administration commands. The tenant C symbols retained for older bindings to load are stubs that abort the process when called; they do not preserve tenant functionality. Applications using tenants require a migration plan before adopting 8.0. (PR #12583), (PR #12593)
Removed experimental encryption at rest and its key-management roles. This removal does not remove file-level backup encryption. Deployments using encryption at rest require a migration plan before adopting 8.0. (PR #12667)
Removed experimental blob granules, the storage-server ChangeFeed feature, and storage cache servers, together with their associated APIs and commands. Native CDC is a separate interface, not a compatible replacement for the removed ChangeFeed API. (PR #12435), (PR #12470), (PR #12486)
Removed the experimental configuration database and dynamic knobs, including the
use_config_databasedatabase option. (PR #12683)Removed quota-based global tag throttling and its proxy-side machinery. Manual and automatic tag throttling remain available. (PR #13181)
Removed the experimental parallel restore implementation and its restore worker roles. This does not remove the regular
fdbrestoretool. (PR #12903)Removed the synthetic-data generation feature and the orphaned Azure backup integration. (PR #12707), (PR #13963)
Features (Experimental)
Added Native CDC: durable named streams over one or more disjoint user-key ranges, commit-version-grouped mutations, resumable consumers, and durable acknowledgements controlling log retention. Native C++, C, and Python APIs support registration, listing, removal, consumption, and acknowledgement; ordered fixed-partition streams can merge mutations into commit-version order. Operator commands report stream status and guard removal while consumers lag. New stream admission is disabled by default through
ENABLE_NATIVE_CDC. Consumers must handle redelivery and acknowledge only durably processed data; CDC does not provide exactly-once application side effects. See C API, Python API, and the CDC design. (PR #13287), (PR #13674), (PR #13925), (PR #13926), (PR #13971), (PR #14116)Extended bulk dump and bulk load, introduced in 7.4, to operate on selected key ranges. Bulk-load restore can target a non-empty database; data in the destination range is replaced. See BulkDump (Dev) and BulkLoad User Guide. (PR #12288), (PR #12252), (PR #13340)
Integrated bulk dump/load with backup and restore.
fdbbackup startaccepts--mode bulkdumpor--mode bothfor snapshots;fdbrestore startaccepts--mode bulkloadto load an available BulkDump dataset. Traditionalrangefilemode remains the default. An incomplete bulk dataset produces an error directing the operator to retry with--mode rangefile. (PR #12608), (PR #13873)Added range-partitioned mutation logging (Backup V3), including partition maps, range backup worker recruitment, and partitioned log uploads. It is selected with
--mutation-log-type range-partitioned-log-experimental. This is under active development and is not ready for general or critical workloads; the on-disk layout, knobs, and command-line interface remain subject to change. See the Backup V3 design. (PR #12671), (PR #13286), (PR #13304)Added a gRPC ControlService and the
fdbctllibrary for cluster administration, including configuration, status, coordinator management, worker inclusion/exclusion, and maintenance. This extends the gRPC integration already present in 7.4; it requires a build with gRPC enabled. (PR #12540), (PR #12555), (PR #12603)Added the
range_digeststorage audit to compute a partition-independent content fingerprint on a quiescent cluster. This can validate a backup and restore without reading all data through an external client, and exposes per-range progress for localizing mismatches. See the range-digest design. (PR #13866)
Client APIs and bindings
Added the
MAX_GRV_QUEUE_DELAYtransaction option. A GRV proxy can reject a request withtransaction_grv_queue_rejectedwhen its estimated queue delay from ratekeeper throttling exceeds the supplied limit in milliseconds. The estimate is advisory, not an end-to-end transaction deadline. (PR #13085)Added
fdb_transaction_get_range_split_points_with_limitand GoGetRangeSplitPointsWithLimitto bound the number of interior split points, including shard boundaries. The endpoints are always included; a negative limit preserves unlimited behavior. (PR #13693)Added the
TRACE_IPnetwork option to explicitly select the IPv4 or IPv6 address recorded in traces. (PR #12645)Added
Database.GetMainThreadBusynessto the Go binding. (PR #12594)Fixed read-your-writes transactions leaving watch futures unresolved when a commit is interrupted by a transaction error such as a timeout. (PR #13395)
Removed deprecated Java finalizer-based cleanup. Java applications must close databases, transactions, and other native-backed objects explicitly. (PR #13997)
Backup, restore, and object storage
Added Google Cloud Storage access through its S3-compatible XML API with OAuth2 bearer-token authentication. (PR #12975)
Added
prefix=to blobstore backup URLs, placing both backup data and index objects under a chosen object-key prefix. All tools accessing such a backup must use the same prefix. Upgrade agents and client tools before using this option: older versions reject URLs containing it. See Backup, Restore, and Replication for Disaster Recovery. (PR #13915)Added SHA-256 integrity checking for multipart S3 uploads and improved S3/REST request handling and connection reuse. (PR #12246), (PR #12447)
Coalesced encrypted blobstore backup reads and issued encrypted block reads concurrently, reducing small object-store read requests. (PR #13750), (PR #13992)
Fixed backup agents that reopen an existing encrypted backup writing unreadable data before the encryption key finished loading, which left restores retrying authentication failures without progress. (PR #14072)
Fixed reopening the same backup URL with a different proxy, encryption key file, or encryption block size silently reusing the first configuration. (PR #14065)
Added progress-based timeouts for backup/restore bulk jobs. Failed or incomplete bulk dumps and unverifiable bulk-load restores now fail instead of appearing successful or waiting indefinitely. (PR #13945), (PR #13936), (PR #13873)
Split bulk-load tasks that cannot be placed on a destination team and preserved tasks across relocation failures, improving restore progress at scale. (PR #13923), (PR #13873)
Rejected path traversal in bulk-load manifest file paths. (PR #13665)
Fixed
fdbbackup status --jsonomitting recorded errors, and madefdbdecodeapply version filters before inspecting snapshot manifests. (PR #14124), (PR #14122)
Cluster operations and reliability
Added
fdbcli rangelockcommands to register and list owners, inspect locks, and take or release exclusive read locks. These locks reject writes only whenENABLE_READ_LOCK_ON_RANGEis enabled on commit proxies. Hardened range and owner validation, and prevented unregistering owners with active locks. See FoundationDB Range Locks (WiP). (PR #13323), (PR #13922), (PR #13942)Added
--tlog-spill-datadirto place TLog spill data in a separate directory, and--tlog-spill-filesystemfor Linux mount validation. Disk snapshot procedures must include the spill directory when it is configured. (PR #13314)Added log-router replacement after failures without requiring full transaction-system recovery. (PR #12558)
Changed low-disk TLog handling to keep existing logs available while recruiting replacements and to avoid repeatedly selecting low-disk workers. (PR #13781)
Hardened worker registration checks before recruitment to avoid using stale worker interfaces. (PR #13646)
Bounded retries for degraded storage teams across data-distribution pipeline transitions and avoided publishing transient pipeline capacity, allowing stranded shards to make progress without repeated duplicate submissions. (PR #13838), (PR #14012)
Fixed Sharded RocksDB resource lifetime and compaction shutdown handling during storage-server rollback. (PR #13726)
Enabled stale client-peer eviction and eager proxy refresh by default (
LOCATION_CACHE_PEER_EVICTOR_ENABLED,DBCONTEXT_EAGER_PROXY_UPDATE,SHRINK_PROXY_LIST_CLEAR_CACHE_BELOW_THRESHOLD). These were available but disabled by default in 7.3 and 7.4. This reduces connection-timeout churn after storage-server or proxy failures. (PR #14050)Preserved old TLog history until remote replicas have durably copied it, and retried lost old-log-router initialization replies during recovery. (PR #13913), (PR #14047)
Fixed commit-proxy recruitment in small configurations and enforced admission control on key-location requests after a cluster bounce. (PR #13400), (PR #14090)
Fixed a RocksDB checkpoint-reader initialization race. (PR #14084)
Performance and observability
Reduced conflict-detection work with word-level bitmap operations and reusable buffers, and reduced arena allocations when materializing RocksDB range reads. (PR #12365), (PR #12367), (PR #13273)
Added watch and version-vector metrics to status JSON, and system-keyspace size reporting to
fdbcli status. (PR #13814), (PR #13191), (PR #12443)Added commit statistics and transaction-size histograms, data-distribution maintenance-duration reporting, commit-batch flush reasons, and TLog disk-queue write/commit size histograms. (PR #13416), (PR #13403), (PR #13767)
Build and packaging
Migrated Flow actors to standard C++20 coroutines and removed the actor compiler and its source-generation step. Updated developer documentation and tutorials to use
co_await,co_return, and the Flow coroutine runtime. (PR #13961), (PR #13958)Added optional build integration for the separately maintained Swift bindings. It requires Swift 6.1 or newer and Clang; Linux builds also require libc++. (PR #12428), (PR #12500)
Upgraded RocksDB to 11.1.2. Recent 7.4.x releases use RocksDB 8.11.x. (PR #14189)
Updated the main Docker image base to Rocky Linux 10.2. (PR #12549)
Upgraded Boost from 1.86.0 to 1.89.0. (PR #14137)
Removed Flow’s unused
CompressionUtilsabstraction and its zstd support. (PR #13708)