Clustering

Atoll clustering is an optional runtime subsystem that orders durable write proposals through Raft and applies committed commands to configured databases. It is separate from process-local Unit placement: a replicated Unit creates execution capacity, while Raft creates an ordered durable command history.

The cluster feature is off by default and enables atoll-cluster with the embedded Turso executor. PostgreSQL and MySQL integrations are separately feature-gated.

Status

The current implementation includes the Raft writer, segmented command log, database applier, proposal acknowledgements, subscriptions, follower forwarding, and workflow data structures.

Two boundaries are especially important:

• Raft snapshot data transfer is not implemented. Snapshot metadata is built with an empty payload, install is a no-op, and recovery relies on full log replay.
• workflow event-await execution is not integrated. Workflow definitions, deduplication, and compensation substrate exist, but an awaiting workflow is not yet driven end to end by committed events.

The rest of this chapter distinguishes implemented data paths from those limits.

Components

client or follower
  → RaftWriter
  → OpenRaft consensus
  → state machine commit watermark
  → mmap segmented log
  → Applier
     ├─ Turso
     ├─ PostgreSQL
     └─ MySQL
  → ACK or NACK
  → ProposalTracker

committed events
  → datasource watermark
  → subscriptions

The cluster runtime combines:

• an OpenRaft node and writer;
• proposal and acknowledgement tracking;
• the database applier;
• subscription and workflow registries;
• commit, datasource, and purge watermarks;
• backlog and retention accounting;
• follower forwarding and leader tracking.

Proposals

A proposal is a transaction-shaped command stream:

BEGIN
  SQL, prepared statements, row operations, or events
COMMIT or ABORT
ACK or NACK

The RaftWriter hashes a transaction ID to a worker. Commands for one proposal therefore preserve order while independent proposals can be prepared in parallel. Workers greedily combine up to 64 subcommands into one Raft entry.

Current defaults are eight writer workers, a queue capacity of 256 per worker, and 256 MiB of byte-accounted backlog. A byte semaphore applies pressure before unbounded command data enters consensus. Applying an entry releases its backlog budget.

These are operational defaults, not protocol constants.

Consensus

OpenRaft orders entries and advances the state machine. The state machine does not execute SQL. Its current responsibilities are to:

• release writer backlog for applied entries;
• advance the Raft commit high-water mark;
• update purge-floor, pin-ceiling, prefetch, and retention state.

Database effects occur later in the applier. Consequently, “committed by Raft” and “applied to a datasource” are separate observable positions.

The state machine deliberately reports no durable applied state on restart so the log is replayed. Snapshot construction currently emits metadata and empty data, snapshot installation makes no data change, and no current snapshot is served. A deployment must retain sufficient log history for recovery; it must not assume a new node can bootstrap database/apply state from a Raft snapshot.

Log

Committed commands are stored in memory-mapped segments. A record has the following physical shape:

[length:4]
[record type:1]
[group id:3]
[term:4]
[node tag:4]
[index:8]
[command bytes]
[CRC:8]
[padding to 8-byte alignment]

Readers iterate raw records without allocating and can explode batched entries into their leaf commands.

A physical CommandPosition combines a segment_id: u32 and segment_offset: u32 into a packed u64. It identifies one leaf command, not merely the enclosing Raft entry.

Applier

The applier consumes committed records and reconstructs complete proposals. It:

1. scans and explodes batched log records;
2. assembles commands from BEGIN through terminal state;
3. derives conflict keys;
4. partitions nonconflicting work for parallel database writes;
5. uses group commit where the executor supports it;
6. retries proposals individually when a batch fails;
7. records ACK, result-bearing ACK, or NACK;
8. advances each datasource’s command watermark.

Recovery checks the _raft_applied marker for a proposal that committed in the database but lost its acknowledgement. This closes the common committed-without-ACK gap without blindly executing the write twice.

The ProposalTracker exposes one-shot completion as Ack, AckWithResults, or Nack.

Write modes

Embedded

For Turso, the runtime holds an interactive local transaction while it executes statements, preserving read-your-writes. Change-data capture records final row images as SQL_ROWOPS. Followers apply those images instead of re-evaluating nondeterministic SQL.

The held transaction writes its _raft_applied marker, proposes the captured payload, and waits at a gate so the applier commits in log order. A nontransactional write is wrapped in an implicit held transaction.

If the client times out around commit, the outcome can be indeterminate. The local path may roll back its held transaction while the already accepted log entry later applies successfully. Callers need stable transaction or idempotency identifiers when retrying.

Server

PostgreSQL and MySQL executors record a statement-manifest identifier with encoded parameters. A shared server database lets the _raft_applied marker short-circuit reapplication on nodes observing the same durable state.

The manifest is an ABI shared by compiler and cluster runtime. An unknown or incompatible statement ID is a deployment error, not text to execute optimistically.

Forwarding

A follower uses ForwardClient to send SQL work to the current leader’s ForwardAcceptor over a bidirectional TCP protocol, with optional TLS.

The protocol includes:

• a node-ID handshake;
• monotonically sequenced command batches;
• client and request correlation;
• routing and transaction identifiers;
• result and error frames.

Follower deduplication tracks a high-water mark so reconnect replay does not blindly duplicate accepted batches. A leader tracker observes OpenRaft metrics, retargets the connection, and reconnects with backoff when leadership changes.

The replicated SQL executor’s direct local path still assumes it is on the leader; forwarding is the separate follower path. Membership bootstrap, certificate distribution, and production deployment orchestration are outside this library-level contract.

Watermarks

Three positions answer different questions:

Waiting for the commit HWM does not prove that SQL effects are visible. A reader requiring database visibility waits for the relevant datasource watermark.

Subscription cursors pin the purge floor. Retention may delete a segment only when it lies below the purge floor, satisfies age or disk policy, and is not active.

Subscriptions

Proposals can contain typed events. Subscriptions filter by topic, event, and key, but delivery is held until the associated datasource watermark proves that preceding database changes are applied.

Each subscriber cursor contributes to the purge floor. A stalled subscriber can therefore retain log segments and must be monitored or expired by explicit policy.

The runtime event machinery exists, but guest-language event production does not yet have a complete source-level surface. Applications must use the currently exposed host/runtime integration rather than assume undocumented syntax.

Workflows

The cluster crate defines workflow records, deduplication, compensation, and event-wait concepts. The event-await client path is currently a placeholder and does not connect committed subscriptions to end-to-end workflow resumption.

It is accurate to build on the proposal and subscription primitives. It is not yet accurate to claim a complete durable-workflow engine solely because those data structures exist.

Guarantees

Within the implemented path:

• Raft establishes one order for accepted write commands;
• worker hashing preserves command order within a proposal;
• byte-accounted queues apply bounded backpressure;
• database apply is tracked separately from consensus commit;
• _raft_applied detects already committed proposals during recovery;
• follower forwarding correlates and deduplicates sequenced batches;
• subscription delivery waits for the datasource apply watermark;
• purge respects active reader cursors.

Limits

Current clustering does not by itself provide:

• nonempty Raft snapshots or snapshot-based node recovery;
• complete workflow event-await execution;
• a full guest-language event API;
• automatic membership bootstrap or cross-node deployment orchestration;
• automatic reconciliation of arbitrary external side effects;
• a cluster-wide singleton interpretation of local Unit placement;
• proof that a timed-out commit did not later apply.

These are implementation boundaries, not merely missing examples. Production designs must account for log retention, idempotent retries, leader forwarding, and operational membership explicitly.

Implementation

The main implementation is in atoll-cluster, with runtime integration behind the WebAssembly runtime’s cluster, cluster-postgres, and cluster-mysql features. The statement manifest is shared with compiler-generated ABI data.