FoundationDB: The Distributed Database That Apple Built iCloud On

Most distributed databases make you choose between consistency and scale. Cassandra gives you scale but eventual consistency. MongoDB gives you flexible documents but unreliable transactions. FoundationDB is built on the premise that this trade-off is wrong. It delivers strict serializability — the strongest form of transactional isolation — across a distributed cluster at the performance levels Apple needs to run iCloud.

What FoundationDB is

FoundationDB is a distributed, transactional, ordered key-value store. Apache 2.0 licensed, owned and maintained by Apple (open sourced 2018), with 16,400+ GitHub stars. Latest stable: 7.3.77 (April 2026). Linux for production, macOS/Windows for local development.

Who uses it in production: Apple runs iCloud metadata and CloudKit on it (hundreds of millions of devices). Snowflake used it for metadata storage as documented in their SIGMOD paper. CouchDB is rebuilding its storage layer on FDB. These are not toy workloads.

The unbundled architecture

Most databases bundle a storage engine, data model, and query language. FDB separates these completely. The core does one thing: ordered, transactional key-value storage with strict serializability. Everything else is a layer — a stateless application that maps a higher-level data model to key-value transactions.

One FDB cluster can simultaneously run a document store layer, a graph layer, and a relational records layer, with full ACID across all of them. Official layers include the Record Layer (relational semantics for CloudKit), Document Layer (MongoDB wire protocol), and JanusGraph backend.

ACID across distributed machines

FDB achieves strict serializability using optimistic concurrency control (OCC) combined with MVCC. Each transaction gets a read version — a consistent snapshot. Reads acquire no locks and never block writes. At commit, the system checks for concurrent changes to data the transaction touched. Conflict: retry. No conflict: atomic commit across all machines involved.

Deadlocks are impossible. Slow clients cannot block others. FDB tolerates f failures with only f+1 replicas (vs 2f+1 for quorum systems) — a meaningful efficiency gain at cluster scale.

The simulation testing framework

This is what makes FoundationDB remarkable — worth understanding even if you never deploy it.

Before writing database code, the team built a deterministic simulation framework. It runs an entire cluster — networking, disk I/O, clocks, random number generators — as a single-threaded process. Every source of nondeterminism is controlled. Same random seed means same sequence of events, making test runs fully reproducible.

The simulator injects arbitrary failure combinations: five machines failing simultaneously during a commit, network partitions between specific components, disk failures in sequence. It fast-forwards the clock between events, so one test machine simulates years of cluster operation in hours. Distributed systems bugs that only appear after specific failure sequences are found in simulation before production.

This is why Apple trusted FDB with iCloud and why Snowflake's engineers described it as the only database they trusted enough to build Snowflake on. The SIGMOD paper documenting this approach is required reading for anyone working on distributed systems.

Architecture overview

FDB separates control plane (cluster management) from data plane (reads/writes). The data plane components: Sequencer (assigns read and commit versions), Proxies (transaction entry point, batch commits), Resolvers (detect conflicts between concurrent transactions), Log servers (write-ahead log — durability before storage is updated), and Storage servers (serve reads, hold the actual data).

The key architectural decision: logging is decoupled from storage. Log servers and storage servers scale independently. Transaction processing (Sequencer, Proxies, Resolvers) is stateless and easily replaceable. Recovery is simplified — a failed node is replaced by a fresh one that replays from the log.

Getting started

Install on Ubuntu (production-ready single-node for development):

wget https://github.com/apple/foundationdb/releases/download/7.3.77/foundationdb-clients_7.3.77-1_amd64.deb
wget https://github.com/apple/foundationdb/releases/download/7.3.77/foundationdb-server_7.3.77-1_amd64.deb
sudo dpkg -i foundationdb-clients_7.3.77-1_amd64.deb
sudo dpkg -i foundationdb-server_7.3.77-1_amd64.deb

Python client example — ACID transfer between two accounts:

import fdb
fdb.api_version(730)
db = fdb.open()

@fdb.transactional
def transfer(tr, from_key, to_key, amount):
    balance_a = int(tr[from_key])
    balance_b = int(tr[to_key])
    if balance_a < amount:
        raise Exception("Insufficient funds")
    tr[from_key] = str(balance_a - amount).encode()
    tr[to_key] = str(balance_b + amount).encode()

transfer(db, b'alice', b'bob', 100)

The @fdb.transactional decorator automatically retries the transaction on conflict. The application code sees a simple function call; conflict resolution is transparent.

Honest limitations

5-second transaction limit: Transactions cannot run longer than five seconds. OLAP, analytics, and long-running bulk operations that need to hold read snapshots don't fit this model. FDB is designed for OLTP, not OLAP.

No built-in query language: The core has no SQL, no query optimizer, no secondary indexes. These are provided by layers. If you need them without building a layer, you use the Record Layer or another layer.

Operational complexity: A production cluster requires understanding all five server roles, capacity planning, and careful configuration. This is not a docker compose up deployment for production.

Apple-governed project: Development happens primarily inside Apple, with community contributions accepted but the roadmap driven by Apple's internal needs. Less community-governed than PostgreSQL or etcd.

FoundationDB vs alternatives

vs etcd: etcd targets small cluster configuration data. FDB scales to petabytes and millions of transactions per second. They serve fundamentally different scale ranges.

vs Cassandra / DynamoDB: These sacrifice strict consistency for availability. FDB wins on correctness; Cassandra wins on write availability under network partitions. Pick based on whether your application can tolerate eventual consistency.

vs CockroachDB / Spanner: The closest peers. CockroachDB is a complete distributed SQL database built on a Paxos-replicated key-value layer. FDB provides the transactional foundation without a query language — more primitive and more flexible. Use CockroachDB for distributed PostgreSQL-compatible SQL; use FDB when building a new system that needs to choose its own data model.

vs PostgreSQL: When you have outgrown single-node PostgreSQL and need truly distributed transactions across multiple machines, FDB is the open source answer.

Who it is for

Good fit: teams building distributed platform infrastructure where correctness is non-negotiable; organizations that have outgrown single-node databases but want transactions rather than eventual consistency; platform teams building higher-level data services on a reliable transactional foundation; multi-model data systems that want one transactional store backing multiple data models.

Not the right fit: teams needing a ready-to-use SQL database (use PostgreSQL or CockroachDB); OLAP and analytics workloads; most web application stacks where the operational overhead is not justified; small teams without distributed systems expertise.

My take

FoundationDB is the most intellectually interesting database in this blog series. The simulation testing framework is worth studying even if you never deploy it — it is the most rigorous distributed systems testing approach that any production database has shipped, and the reason Apple trusted it with iCloud at scale.

The unbundled architecture reframes the question that most database debates ask. Instead of which database, FDB asks what is the correct transactional primitive to build everything else on. One small, correct, well-tested core. All data models as layers on top. This is compelling if you are building the infrastructure that other applications run on, not an application itself.

Operational complexity and the 5-second limit rule it out for most web stacks. But for platform teams at the scale where these trade-offs matter, FoundationDB is the most principled answer to how you get ACID guarantees at distributed scale that exists in open source today.