Building a Cluster
This guide walks through the steps to build an externally accessible FoundationDB cluster of one or more machines. Before setting up a cluster for performance testing or production use, you should also read the reference material in Configuration and Administration.
The macOS version of the FoundationDB server is intended for single-machine development use only; its use in multi-machine clusters is not supported. In the present release, the Linux version is the best-tested and most performant platform for multi-machine clusters.
To build an externally accessible FoundationDB cluster, perform the following steps:
Follow the steps in Getting Started on Linux to install FoundationDB locally on each of the Linux machines that you wish to use in your cluster.
When building a cluster, do not simply copy the FoundationDB installation, and in particular the data files, from one machine to another, whether by direct copying or by cloning a VM.
Optimize for your hardware
The default installation runs only one FoundationDB server process per machine (which will use only one CPU core). Most users of multi-machine configurations will want to maximize performance by running one FoundationDB server process per core. This is accomplished by modifying the configuration file (located at
/etc/foundationdb/foundationdb.conf) to have
[fdbserver.<ID>] sections for each core. Note that 4GiB ECC RAM are required per FoundationDB server process (see System requirements).
We recommend changing the configuration file once and copying to other machines in the cluster.
Make FoundationDB externally accessible
By default, FoundationDB installs on a single server in a locally accessible mode suitable for development — only clients on the same machine will be able to access the database. To allow external access, you will have to make your cluster file public.
Choose a machine to be the starting machine for your cluster. The database on this machine will be the one that we grow to span the cluster. Use the
/usr/lib/foundationdb/make_public.py script on that server to update your cluster file to use a public interface. For example:
user@host1$ sudo /usr/lib/foundationdb/make_public.py /etc/foundationdb/fdb.cluster is now using address 10.0.1.1
A FoundationDB cluster has the option of supporting Transport Layer Security (TLS) for all connections (between server processes and between clients and servers). To enable TLS on a new cluster, see Enabling TLS.
By default, the script will pick a local network interface that can access the internet. To specify the address manually, use the
-a flag and choose an address that is accessible by all machines in the cluster as well as by all intended clients.:
user@host1$ sudo /usr/lib/foundationdb/make_public.py -a 10.0.1.1 /etc/foundationdb/fdb.cluster is now using address 10.0.1.1
Test the database
At this point and after each subsequent step, it is a good idea to test the database to make sure it is operating normally. Run
fdbcli on the starting machine:
user@host1$ fdbcli Using cluster file `/etc/foundationdb/fdb.cluster'. The database is available. Welcome to the fdbcli. For help, type `help'. fdb> status Configuration: Redundancy mode - single Storage engine - ssd Coordinators - 1 Cluster: FoundationDB processes - 1 Machines - 1 Memory availability - 4.1 GB per process on machine with least available Fault Tolerance - 0 machines Server time - Thu Mar 15 14:41:34 2018 Data: Replication health - Healthy Moving data - 0.000 GB Sum of key-value sizes - 8 MB Disk space used - 103 MB Operating space: Storage server - 1.0 GB free on most full server Transaction log - 1.0 GB free on most full server Workload: Read rate - 2 Hz Write rate - 0 Hz Transactions started - 2 Hz Transactions committed - 0 Hz Conflict rate - 0 Hz Backup and DR: Running backups - 0 Running DRs - 0 Client time: Thu Mar 15 14:41:34 2018
If the database is not operational the status command will provide diagnostic information to help you resolve the issue. For more help, please post a question (and the results of the status command) on the community forums.
Add machines to the cluster
To add the rest of your machines to the cluster, perform the following steps on each one:
Copy the cluster file from a server already in the cluster (located at
/etc/foundationdb/fdb.cluster) to the new machine, overwriting the existing
Restart FoundationDB on the new machine so that it uses the new cluster file:
user@host2$ sudo service foundationdb restart
Change redundancy mode and storage engine
By default, the database will be in
single redundancy mode and use the
memory storage engine. You should change the redundancy mode (see Choosing a redundancy mode) and storage engine (see Storage engines) to appropriate values for your cluster.
For example, to use a triple-replicated database with the SSD storage engine, use the
configure command in the
user@host1$ fdbcli Using cluster file `/etc/foundationdb/fdb.cluster'. The database is available. Welcome to the fdbcli. For help, type `help'. fdb> configure triple ssd Configuration changed
If the configure command hangs or returns an error message, see Test the database.
Change coordination servers
At this point, your cluster will be using the starting machine as the only coordination server, leaving that as a single point of failure. You should therefore select a fault-tolerant set of coordinators according to the criteria in Choosing coordination servers. To switch the cluster to your chosen coordinators, run the
fdbcli command on one of the servers and use the
coordinators command to set the coordinators.
user@host$ fdbcli Using cluster file `/etc/foundationdb/fdb.cluster'. The database is available. Welcome to the fdbcli. For help, type `help'. fdb> coordinators 10.0.4.1:4500 10.0.4.2:4500 10.0.4.3:4500 Coordinators changed
There is also a convenience option,
coordinators auto, that will automatically select a set of coordinators based on your redundancy mode.
coordinators auto selects processes based on IP address. If your cluster has processes on the same machine with different IP addresses,
coordinators auto may select a set of coordinators that are not fault tolerant. To ensure maximal fault tolerance, we recommend selecting coordinators according to the criteria in Choosing coordination servers and setting them manually.
You can also change the cluster
description, as described in Changing the cluster description.
To add or remove machines from the cluster or perform other administrative tasks, see Administration.