Client Testing

Testing Error Handling with Buggify

FoundationDB clients need to handle errors correctly. Wrong error handling can lead to many bugs - in the worst case it can lead to a corrupted database. Because of this it is important that an application or layer author tests properly their application during failure scenarios. But this is non-trivial. In a development environment cluster failures are very unlikely and it is therefore possible that certain types of exceptions are never tested in a controlled environment.

The simplest way of testing for these kind of errors is a simple mechanism called Buggify. If this option is enabled in the client, the client will randomly throw errors that an application might see in a production environment. Enable this option in testing will greatly improve the probability that error handling is tested properly.

Options to Control Buggify

There are four network options to control the buggify behavior. By default, buggify is disabled (as it will behave in a way that is not desirable in a production environment). The options to control buggify are:

  • buggify_enable This option takes no argument and will enable buggify.

  • buggify_disable This can be used to disable buggify again.

  • client_buggify_section_activated_probability (default 25) A number between 0 and 100.

  • client_buggify_section_fired_probability (default 25) A number between 0 and 100.

The way buggify works is by enabling sections in the code first that get only executed with a certain probability. Generally these code sections will simply introduce a synthetic error.

When a section is passed for the first time, the client library will decide randomly whether that code section will be enabled or not. It will be enabled with a probability of client_buggify_section_activated_probability.

Whenever the client executes a buggify-enabled code-block, it will randomly execute it. This is to make sure that a certain exception doesn’t always fire. The probably for executing such a section is client_buggify_section_fired_probability.

Simulation and Cluster Workloads

FoundationDB comes with its own testing framework. Tests are implemented as workloads. A workload is nothing more than a class that gets called by server processes running the tester role. Additionally, a fdbserver process can run a simulator that simulates a full fdb cluster with several machines and different configurations in one process. This simulator can run the same workloads you can run on a real cluster. It will also inject random failures like network partitions and disk failures.

This tutorial explains how one can implement a workload, how one can orchestrate a workload on a cluster with multiple clients, and how one can run a workload within a simulator. Running in a simulator is also useful as it does not require any setup: you can simply run one command that will provide you with a fully functional FoundationDB cluster.

General Overview

Workloads in FoundationDB are generally compiled into the binary. However, FoundationDB also provides the ability to load workloads dynamically. This is done through dlopen (on Unix like operating systems) or LoadLibrary (on Windows).

Parallelism and Determinism

A workload can run either in a simulation or on a real cluster. In simulation, fdbserver will simulate a whole cluster and will use a deterministic random number generator to simulate random behavior and random failures. This random number generator is initialized with a random seed. In case of a test failure, the user can reuse the given seed and rerun the same test in order to further observe and debug the behavior.

However, this will only work as long as the workload doesn’t introduce any non-deterministic behavior. One example of non-deterministic behavior is the running multiple threads.

The workload is created in the main network thread and it will run in the main network thread. Because of this, using any blocking function (for example blockUntilReady on a future object) will result in a deadlock. Using the callback API is therefore required if one wants to keep the simulator’s deterministic behavior.

For existing applications and layers, however, not using the blocking API might not be an option. For these use-cases, a user can chose to start new threads and use the blocking API from within these threads. This will mean that test failures will be non-deterministic and might be hard to reproduce.

To start a new thread, one has to “bind” operating system threads to their simulated processes. This can be done by setting the ProcessId in the child threads when they get created. In Java this is done by only starting new threads through the provided Executor. In the C++ API one can use the FDBWorkloadContext to do that. For example:

template<class Fun>
std::thread startThread(FDBWorkloadContext* context, Fun fun) {
    auto processId = context->getProcessID();
    return std::thread([context, processID, fun](
        context->setProcessID(processID);
        fun();
    ));
}

Finding the Shared Object

When the test starts, fdbserver needs to find the shared object to load. The name of this shared object has to be provided.

For Java, we provide an implementation in libjava_workloads.so which can be built out of the sources. The tester will look for the key libraryName in the test file which should be the name of the library without extension and without the lib prefix (so java_workloads if you want to write a Java workload).

By default, the process will look for the library in the directory ../shared/foundationdb/ relative to the location of the fdbserver binary. If the library is somewhere else on the system, one can provide the absolute path to the library (only the folder, not the file name) in the test file with the libraryPath option.

Implementing a C++ Workload

In order to implement a workload, one has to build a shared library that links against the fdb client library. This library has to expose a function (with C linkage) called workloadFactory which needs to return a pointer to an object of type FDBWorkloadFactory. This mechanism allows the author to implement as many workloads within one library as she wants. To do this the pure virtual classes FDBWorkloadFactory and FDBWorkload have to be implemented.

FDBWorkloadFactory *workloadFactory(FDBLogger*)

This function has to be defined within the shared library and will be called by fdbserver for looking up a specific workload. FDBLogger will be passed and is guaranteed to survive for the lifetime of the process. This class can be used to write to the FoundationDB traces. Logging anything with severity FDBSeverity::Error will result in a hard test failure. This function needs to have c-linkage, so define it in a extern "C" block.

std::shared_ptr<FDBWorkload> FDBWorkload::create(const std::string &name)

This is the only method to be implemented in FDBWorkloadFactory. If the test file contains a key-value pair workloadName the value will be passed to this method (empty string otherwise). This way, a library author can implement many workloads in one library and use the test file to chose which one to run (or run multiple workloads either concurrently or serially).

std::string FDBWorkload::description() const

This method has to return the name of the workload. This can be a static name and is primarily used for tracing.

bool FDBWorkload::init(FDBWorkloadContext *context)

Right after initialization

void FDBWorkload::setup(FDBDatabase *db, GenericPromise<bool> done)

This method will be called by the tester during the setup phase. It should be used to populate the database.

void FDBWorkload::start(FDBDatabase *db, GenericPromise<bool> done)

This method should run the actual test.

void FDBWorkload::check(FDBDatabase *db, GenericPromise<bool> done)

When the tester completes, this method will be called. A workload should run any consistency/correctness tests during this phase.

void FDBWorkload::getMetrics(std::vector<FDBPerfMetric> &out) const

If a workload collects metrics (like latencies or throughput numbers), these should be reported back here. The multitester (or test orchestrator) will collect all metrics from all test clients and it will aggregate them.

Implementing a Java Workload

In order to implement your own workload in Java you can simply create an implementation of the abstract class AbstractWorkload. A minimal implementation will look like this:

package my.package;
import com.apple.foundationdb.testing.Promise;
import com.apple.foundationdb.testing.AbstractWorkload;
import com.apple.foundationdb.testing.WorkloadContext;

class MinimalWorkload extends AbstractWorkload {
    public MinimalWorkload(WorkloadContext ctx) {
        super(ctx);
    }

    @Override
    public void setup(Database db, Promise promise) {
        log(20, "WorkloadSetup", null);
        promise.send(true);
    }

    @Override
    public void start(Database db) {
        log(20, "WorkloadStarted", null);
        promise.send(true);
    }

    @Override
    public boolean check(Database db) {
        log(20, "WorkloadFailureCheck", null);
        promise.send(true);
    }
}

The lifecycle of a test will look like this:

  1. All testers will create an instance of the AbstractWorkload implementation.

  2. All testers will (in parallel but not guaranteed exactly at the same time) call setup and they will wait for all of them to finish. This phase can be used to pre-populate data.

  3. All tester will then call start (again, in parallel) and wait for all of them to finish.

  4. All testers will then call check on all testers and use the returned boolean to determine whether the test succeeded.

All these methods take a Database object as an argument. This object can be used to create and execute transactions against the cluster.

When implementing workloads, an author has to follow these rules:

  • To write tracing to the trace-files one should use AbstractWorkload.log. This Method takes three arguments: an integer for severity (5 means debug, 10 means log, 20 means warning, 30 means warn always, and 40 is a severe error). If any tester logs something of severity 40, the test run is considered to have failed.

  • In order to increase throughput on the cluster, an author might want to spawn several threads. However, threads MUST only be spawn through the Executor instance one can get from AbstractWorkload.getExecutor(). Otherwise, a simulation test will probably segfault. The reason for this is that we need to keep track of which simulated machine a thread corresponds to internally.

Within a workload you have access to the WorkloadContext which provides additional information about the current execution environment. The context can be accessed through this.context and provides the following methods:

  • String getOption(String name, String defaultValue). A user can provide parameters to workloads through a configuration file (explained further down). These parameters are provided to all clients through the context and can be accessed with this method.

  • int getClientId() and int getClientCount(). An author can determine how many clients are running in the cluster and each of those will get a globally unique ID (a number between 0 and clientCount - 1). This is useful for example if you want to generate transactions that are guaranteed to not conflict with transactions from other clients.

  • int getSharedRandomNumber(). At startup a random number will be generated. This will allow for generating the same random numbers across several machines if this number is used as a seed.

Running a Workload in the Simulator

We’ll first walk how one can run a workload in a simulator. FoundationDB comes already with a large number of workloads. But some of them can’t be run in simulation while other don’t work on a real cluster. Most will work on both though. To look for examples how these can be ran, you can find configuration files in the tests directory in the FoundationDB source tree.

We will now go through an example how you can write a relatively complex test and run it in the simulator. Writing and running tests in the simulator is a simple two-step process.

  1. Write the test.

  2. Run fdbserver in simulation mode and provide it with the test file.

Write the Test

A workload is not a test. A test is a simple test file that tells the test orchestrator which workloads it should run and in which order. Additionally one can provide parameters to workloads through this file.

A test file might look like this:

testTitle=MyTest
  testName=External
  libraryName=java_workloads
  workloadName=my.package.MinimalWorkload
  classPath=PATH_TO_JAR_OR_DIR_CONTAINING_WORKLOAD,OTHER_DEPENDENCIES

  testName=Attrition
  testDuration=5.0
  reboot=true
  machinesToKill=3

testTitle=AnotherTest
  testName=External
  libraryName=java_workloads
  workloadName=my.package.MinimalWorkload
  classPath=PATH_TO_JAR_OR_DIR_CONTAINING_WORKLOAD,OTHER_DEPENDENCIES
  someOption=foo

  testName=External
  libraryName=java_workloads
  workloadName=my.package.AnotherWorkload
  classPath=PATH_TO_JAR_OR_DIR_CONTAINING_WORKLOAD,OTHER_DEPENDENCIES
  anotherOption=foo

This test will do the following:

  1. First it will run MinimalWorkload without any parameter.

  2. After 5.0 seconds the simulator will reboot 3 random machines (this is what Attrition does and this workload is provided by FoundationDB. This is one of the few workloads that only work in the simulator).

  3. When all workloads are finished, it will run MinimalWorkload again. This time it will have the option someOption set to foo. Additionally it will run AnotherWorkload in parallel.

How to set the Class Path correctly

As you can see from above example, we can set the classpath through two different mechanisms. However, one has to be careful as they can’t be used interchangeably.

  • You can set a class path through the JVM argument -Djava.class.path=.... This is how you have to pass the path to the FoundationDB client library (as the client library is needed during the initialization phase). However, only the first specified section will have any effect as the other Workloads will run in the same VM (and arguments, by nature, can only be passed once).

  • The classPath option. This option will add all paths (directories or JAR-files) to the classPath of the JVM while it is running. Not being able to add the path will result in a test failure. This is useful to add different dependencies to different workloads. A path can appear more than once across sections. However, they must not conflict with each other as we never remove something from the classpath.

Run the simulator

This step is very simple. You can simply run fdbserver with role simulator and pass the test with -f:

fdbserver -r simulation -f testfile.txt

Running a Workload on an actual Cluster

Running a workload on a cluster works basically the same way. However, one must actually setup a cluster first. This cluster must run between one and many server processes with the class test. So above 2-step process becomes a bit more complex:

  1. Write the test (same as above).

  2. Set up a cluster with as many test clients as you want.

  3. Run the orchestrator to actually execute the test.

Step 1. is explained further up. For step 2., please refer to the general FoundationDB configuration. The main difference to a normal FoundationDB cluster is that some processes must have a test class assigned to them. This can be done in the foundationdb.conf. For example this file would create a server with 8 processes of which 4 would act as test clients.

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
restart-delay = 60
cluster-file = /etc/foundationdb/fdb.cluster

## Default parameters for individual fdbserver processes
[fdbserver]
command = /usr/sbin/fdbserver
public-address = auto:$ID
listen-address = public
datadir = /var/lib/foundationdb/data/$ID
logdir = /var/log/foundationdb

[fdbserver.4500]
[fdbserver.4501]
[fdbserver.4502]
[fdbserver.4503]
[fdbserver.4510]
class = test
[fdbserver.4511]
class = test
[fdbserver.4512]
class = test
[fdbserver.4513]
class = test

Running the actual test can be done with fdbserver as well. For this you can call the process with the multitest role:

fdbserver -r multitest -f testfile.txt

This command will block until all tests are completed.

API Tester

Introduction

API tester is a framework for implementing end-to-end tests of FDB C API, i.e. testing the API on a real FDB cluster through all layers of the FDB client. Its executable is fdb_c_api_tester, and the source code is located in bindings/c/test/apitester. The structure of API Tests is similar to that of the Simulation Tests. The tests are implemented as workloads using FDB API, which are all built into the fdb_c_api_tester. A concrete test configuration is defined as a TOML file, which specifies the combination of workloads to be executed by the test together with their parameters. The test can be then executed by passing the TOML file as a parameter to fdb_c_api_tester.

Since simulation tests rely on the actor model to execute the tests deterministically in single-threaded mode, they are not suitable for testing various multi-threaded aspects of the FDB client. End-to-end API tests complement the simulation tests by testing the FDB Client layers above the single-threaded Native Client.

  • The specific testing goals of the end-to-end tests are:

  • Check functional correctness of the Multi-Version Client (MVC) and Thread-Safe Client

  • Detecting race conditions. They can be caused by accessing the state of the Native Client from wrong threads or introducing other shared state without proper synchronization

  • Detecting memory management errors. Thread-safe reference counting must be used where necessary. MVC works with multiple client libraries. Memory allocated by one client library must be also deallocated by the same library.

  • Maintaining interoperability with other client versions. The client functionality is made available depending on the selected API version. The API changes are correctly adapted.

  • Client API behaves correctly in case of cluster upgrades. Database and transaction state is correctly migrated to the upgraded connections. Pending operations are canceled and successfully retried on the upgraded connections.

Implementing a Workload

Each workload is declared as a direct or indirect subclass of WorkloadBase implementing a constructor with WorkloadConfig as a parameter and the method start(), which defines the entry point of the workload.

WorkloadBase provides a set of methods that serve as building blocks for implementation of a workload:

execTransaction(start, cont, failOnError = true)

creates and executes an FDB transaction. Here start is a function that takes a transaction context as parameter and implements the starting point of the transaction, and cont is a function implementing a continuation to be executed after finishing the transaction execution. Transactions are automatically retried on retryable errors. Transactions are retried by calling the start function again. In case of a fatal error, the entire workload is considered as failed unless failOnError is set to false.

schedule(task)

schedules a task for asynchronous execution. It is usually used in the continuations to schedule the next step of the workload.

info(msg)
error(msg)

are used for logging a message with a tag identifying the workload. Issuing an error message marks the workload as failed.

The transaction context provides methods for implementation of the transaction logics:

tx()

the reference to the FDB transaction object

continueAfter(future, cont, retryOnError = true)

set a continuation to be executed when the future is ready. The retryOnError flag controls whether the transaction should be automatically retried in case the future results in a retriable error.

continueAfterAll(futures, cont)

takes a vector of futures and sets a continuation to be executed when all of the futures get ready. The transaction is retried if at least one of the futures results in an error. This method is useful for handling multiple concurrent reads.

commit()

commit and finish the transaction. If the commit is successful, the execution proceeds to the continuation of execTransaction(). In case of a retriable error the transaction is automatically retried. A fatal error results in a failure of the workoad.

done()

finish the transaction without committing. This method should be used to finish read transactions. The transaction gets destroyed and execution proceeds to the continuation of execTransaction(). Each transaction must be finished either by commit() or done(), because otherwise the framework considers that the transaction is still being executed, so it won’t destroy it and won’t call the continuation.

onError(err)

Handle an error: restart the transaction in case of a retriable error, otherwise fail the workload. This method is typically used in the continuation of continueAfter called with retryOnError=false as a fallback to the default error handling.

A workload execution ends automatically when it is marked as failed or its last continuation does not schedule any new task or transaction.

The workload class should be defined in the namespace FdbApiTester. The file name convention is Tester{Name}Workload.cpp so that we distinguish them from the source files of simulation workloads.

Basic Workload Example

The code below implements a workload that consists of only two transactions. The first one sets a randomly generated key to a randomly generated value, and the second one reads the key and checks if the returned value matches the written one.

#include "TesterWorkload.h"
#include "TesterUtil.h"

namespace FdbApiTester {

class SetAndGetWorkload : public WorkloadBase {
public:
	fdb::Key keyPrefix;
	Random random;

	SetAndGetWorkload(const WorkloadConfig& config) : WorkloadBase(config) {
		keyPrefix = fdb::toBytesRef(fmt::format("{}/", workloadId));
	}

	void start() override { setAndGet(NO_OP_TASK); }

	void setAndGet(TTaskFct cont) {
		fdb::Key key = keyPrefix + random.randomByteStringLowerCase(10, 100);
		fdb::Value value = random.randomByteStringLowerCase(10, 1000);
		execTransaction(
		    [key, value](auto ctx) {
			    ctx->tx().set(key, value);
			    ctx->commit();
		    },
		    [this, key, value, cont]() {
			    execTransaction(
			        [this, key, value](auto ctx) {
				        auto future = ctx->tx().get(key, false);
				        ctx->continueAfter(future, [this, ctx, future, value]() {
					        std::optional<fdb::Value> res = copyValueRef(future.get());
					        if (res != value) {
						        error(fmt::format(
						            "expected: {} actual: {}", fdb::toCharsRef(value), fdb::toCharsRef(res)));
					        }
					        ctx->done();
				        });
			        },
			        cont);
		    });
	}
};

WorkloadFactory<SetAndGetWorkload> SetAndGetWorkloadFactory("SetAndGet");

} // namespace FdbApiTester

The workload is implemented in the method setAndGet. It generates a random key and a random value and executes a transaction that writes that key-value pair and commits. In the continuation of the first execTransaction call, we execute the second transaction that reads the same key. The read operation returns a future. So we call continueAfter to set a continuation for that future. In the continuation we check if the returned value matches the written one and finish the transaction by calling ctx->done(). After completing the second transaction we execute the continuation passed as parameter to the setAndGet method by the start method. In this case it is NO_OP_TASK, which does nothing and so finishes the workload.

Finally, we declare an instance WorkloadFactory to register this workload with the name SetAndGet.

Note that we use workloadId as a key prefix. This is necessary for isolating the key space of this workload, because the framework may be instructed to create multiple instances of the SetAndGet workload. If we do not isolate the key space, another workload can write a different value for the same key and so break the assumption of the test.

The workload is implemented using the internal C++ API, implemented in fdb_api.hpp. It introduces a set of classes representing the FDB objects (transactions, futures, etc.). These classes provide C++-style methods wrapping FDB C API calls and automate memory management by means of reference counting.

Implementing Control Structures

Our basic workload executes just 2 transactions, but in practice we want to have workloads that generate multiple transactions. The following code demonstrates how we can modify our basic workload to generate multiple transactions in a loop.

class SetAndGetWorkload : public WorkloadBase {
public:
   ...
   int numIterations;
   int iterationsLeft;

   SetAndGetWorkload(const WorkloadConfig& config) : WorkloadBase(config) {
      keyPrefix = fdb::toBytesRef(fmt::format("{}/", workloadId));
      numIterations = config.getIntOption("numIterations", 1000);
   }

   void start() override {
      iterationsLeft = numIterations;
      setAndGetLoop();
   }

   void setAndGetLoop() {
      if (iterationsLeft == 0) {
         return;
      }
      iterationsLeft--;
      setAndGet([this]() { setAndGetLoop(); });
   }
   ...
}

We introduce a workload parameter numIterations to specify the number of iterations. If not specified in the test configuration it defaults to 1000.

The method setAndGetLoop implements the loop that decrements iterationsLeft counter until it reaches 0 and each iteration calls setAndGet with a continuation that returns the execution to the loop. As you can see we don’t need any change in setAndGet, just call it with another continuation.

The pattern of passing a continuation as a parameter also can be used to decompose the workload into a sequence of steps. For example, we can introduce setup and cleanUp steps to our workload and modify the setAndGetLoop to make it composable with an arbitrary continuation:

 void start() override {
    setup([this](){
        iterationsLeft = numIterations;
        setAndGetLoop([this](){
            cleanup(NO_OP_TASK);
        });
    });
 }

 void setAndGetLoop(TTaskFct cont) {
    if (iterationsLeft == 0) {
        schedule(cont);
    }
    iterationsLeft--;
    setAndGet([this, cont]() { setAndGetLoop(cont); });
}

void setup(TTaskFct cont) { ... }

void cleanup(TTaskFct cont) {  ... }

Note that we call schedule(cont) in setAndGetLoop instead of calling the continuation directly. In this way we avoid keeping setAndGetLoop in the call stack, when executing the next step.

Subclassing ApiWorkload

ApiWorkload is an abstract subclass of WorkloadBase that provides a framework for a typical implementation of API test workloads. It implements a workflow consisting of cleaning up the key space of the workload, populating it with newly generated data and then running a loop consisting of random database operations. The concrete subclasses of ApiWorkload are expected to override the method randomOperation with an implementation of concrete random operations.

The ApiWorkload maintains a local key-value store that mirrors the part of the database state relevant to the workload. A successful database write operation should be followed by a continuation that performs equivalent changes in the local store, and the results of a database read operation should be validated against the values from the local store.

Test Configuration

A concrete test configuration is specified by a TOML file. The file must contain one [[test]] section specifying the general settings for test execution followed by one or more [[test.workload]] configuration sessions, specifying the workloads to be executed and their parameters. The specified workloads are started all at once and executed concurrently.

The [[test]] section can contain the following options:

  • title: descriptive title of the test

  • multiThreaded: enable multi-threading (default: false)

  • minFdbThreads and maxFdbThreads: the number of FDB (network) threads to be randomly selected from the given range (default: 1-1). Used only if multiThreaded=true. It is also important to use multiple database instances to make use of the multithreading.

  • minDatabases and maxDatabases: the number of database instances to be randomly selected from the given range (default 1-1). The transactions of all workloads are randomly load-balanced over the pool of database instances.

  • minClients and maxClients: the number of clients, i.e. instances of each workload, to be randomly selected from the given range (default 1-8).

  • minClientThreads and maxClientThreads: the number of client threads, i.e. the threads used for execution of the workload, to be randomly selected from the given range (default 1-1).

  • blockOnFutures: use blocking waits on futures instead of scheduling future callbacks asynchronously (default: false)

  • buggify: Enable client-side failure injection (default: false)

  • databasePerTransaction: Create a separate database instance for each transaction (default: false). It is a special mode useful for testing bugs related to creation and destruction of database instances.

  • fdbCallbacksOnExternalThreads: Enables the option FDB_NET_OPTION_CALLBACKS_ON_EXTERNAL_THREADS causting the callbacks of futures to be executed directly on the threads of the external FDB clients rather than on the thread of the local FDB client.

The workload section [[test.workload]] must contain the attribute name matching the registered name of the workload to be executed. Other options are workload-specific.

The subclasses of the ApiWorkload inherit the following configuration options:

  • minKeyLength and maxKeyLength: the size range of randomly generated keys (default: 1-64)

  • minValueLength and maxValueLength: the size range of randomly generated values (default: 1-1000)

  • maxKeysPerTransaction: the maximum number of keys per transaction (default: 50)

  • initialSize: the number of key-value pairs in the initially populated database (default: 1000)

  • readExistingKeysRatio: the probability of choosing an existing key for read operations (default: 0.9)

  • numRandomOperations: the number of random operations to be executed per workload (default: 1000)

  • runUntilStop: run the workload indefinitely until the stop command is received (default: false).

    This execution mode in upgrade tests and other scripted tests, where the workload needs to be generated continously until completion of the scripted test.

  • numOperationsForProgressCheck: the number of operations to be performed to confirm a progress

    check (default: 10). This option is used in combination with runUntilStop. Progress checks are initiated by a test script to check if the client workload is successfully progressing after a cluster change.

The FDB server configuration can be specialized in the section [[server]]:

  • tenants_enabled: enable multitenancy (default: true)

  • blob_granules_enabled: enable support for blob granules (default: false)

  • tls_enabled: enable TLS (default: false)

  • tls_client_chain_len: the length of the client-side TLS chain (default: 2)

  • tls_server_chain_len: the length of the server-side TLS chain (default: 3)

  • min_num_processes and max_num_processes: the number of FDB server processes to be randomly selected from the given range (default 1-3)

Executing the Tests

The fdb_c_api_tester executable takes a single TOML file as a parameter and executes the test according to its specification. Before that we must create a FDB cluster and pass its cluster file as a parameter to fdb_c_api_tester. Note that multithreaded tests also need to be provided with an external client library.

The run_c_api_tests.py script automates execution of the API tests on a local cluster. The cluster is created according to the options specified in the [[server]] section of the given test file.

${srcDir}/bindings/c/test/apitester/run_c_api_tests.py
   --build-dir ${buildDir}
   --api-tester-bin ${buildDir}/bin/fdb_c_api_tester
   --external-client-library ${buildDir}/bindings/c/libfdb_c_external.so
   --test-file ${srcDir}/bindings/c/test/apitester/tests/CApiCorrectnessMultiThr.toml

The test specifications added to the bindings/c/test/apitester/tests/ directory are executed as a part of the regression test suite as ctest targets with names fdb_c_api_test_{file_name}.

The ctest targets provide a more convenient way for executing the API tests. We can execute a single test:

ctest -R fdb_c_api_test_CApiCorrectnessMultiThr -VV

or execute all of them in parallel (here -j20 specifies the parallelization level):

ctest -R fdb_c_api_test_ -j20 --output-on-failure

More sophisticated filters can be applied to execute a selected set of tests, e.g. the tests using TLS:

ctest -R 'fdb_c_api_test_.*TLS' -j20 --output_on_failure