Skip to content
LoomCache

Persistence Design

Persistence makes committed Raft and state-machine data survive crashes, node replacement, and controlled backup/restore operations. It is local to each member and depends on a stable nodeId and durable dataDir. This page covers the persistence components, durability levels, corruption handling, and the recovery path.

The animation below traces the full durability lifecycle — writes, snapshot compaction, crash, and recovery:

Durability & Crash Recovery

Write ops append entries to the WAL — every commit is fsynced.

writesnapshotwritecrashrecover
cache nodelocal durable dataDirrunning
WAL · append-only log fsync
e1
e2
e3
e4
full snapshotsretains 2 recent
emptystaging
→
emptyfinal
state machine
in-memory state · volatile
on restart○A · Load the newest valid full snapshot○B · Replay the WAL tail○C · Serve traffic again

Components

Section titled “Components”
  • The write-ahead log (WAL) owns append-only records, integrity checks, and fsync. Each node stores a single append-only log per group. When a maximum file size is configured, the write path enforces that ceiling and throws if the limit would be exceeded — it does not rotate the file.
  • The in-memory Raft index is paired with durable command records when persistence is enabled. Automatic compaction creates a full snapshot and compacts WAL entries from the committed-entry threshold.
  • Raft metadata persists term, voted-for, and commit index through atomic replacement; a file lock guards against concurrent writers.
  • Snapshot persistence manages full snapshots. Each snapshot is written to a staging file first, then atomically promoted to the final full-snapshot file. The store retains two recent snapshots by default.
  • Registered data structures contribute state to each full snapshot.
  • Hot backup jobs produce group snapshots and manifests under the backup directory. Production hot backups require HMAC-SHA256 manifest/snapshot-set integrity fields.

WAL recovery boundary

Section titled “WAL recovery boundary”

Persisted index hints are non-authoritative on recovery. WAL recovery re-derives the actual index range by scanning records and validating their integrity metadata before replay.

Durability levels

Section titled “Durability levels”

The WAL write path operates at one of three durability levels:

LevelBehavior
FSYNCFull FileChannel.force() after every commit (default when syncOnCommit=true).
WRITEData is written to the OS page cache but not forced to disk.
NONEWrites may be coalesced; no durability guarantee beyond in-process memory.

The default level is FSYNC. Disabling fsync (-Dloomcache.wal.disableFsync=true) drops the level to WRITE. Production startup rejects NONE and WRITE with a hard preflight failure.

Committed Raft WALs require unbatched FSYNC durability. Deferred-fsync batching is rejected when a WAL is attached to Raft because it would create a durability window for committed entries.

Corruption handling

Section titled “Corruption handling”

WAL recovery distinguishes between two corruption positions:

  • Torn tail (corruption at the very end of the log): the default torn-tail recovery mode silently truncates the partial record and proceeds. This is the normal case for a crash mid-write.
  • Mid-log corruption (corruption before the current tail): startup aborts with an IOException unless -Dloomcache.wal.allow-truncate-on-mid-log-corruption=true is set, which also requires a byte-range authorization property. Mid-log truncation is a destructive recovery mode and discards all entries beyond the corruption point.

Write path

Section titled “Write path”
  1. The Raft leader appends a command to its log.
  2. Durable log and metadata updates use the configured DurabilityGuarantee level.
  3. The committed command applies to the in-memory state machine.
  4. WAL compaction waits until snapshots cover the compacted range.

Recovery path

Section titled “Recovery path”
  1. Create and validate the group WAL directory, WAL file/header, metadata file, and snapshot directory.
  2. Recover term/voted-for/commit-index from durable Raft metadata.
  3. Load the newest valid full snapshot when one exists.
  4. Replay records newer than the snapshot index, or the whole WAL when no snapshot exists yet.
  5. Reject startup when the selected recovery policy forbids the local data shape.

Invariants

Section titled “Invariants”
  • WAL record integrity validation and snapshot checksum validation run before replay/apply.
  • Snapshot install cannot silently skip registered data structures.
  • WAL compaction cannot remove records not covered by a durable snapshot.
  • Quorum-loss restore must be explicit.

Failure behavior

Section titled “Failure behavior”

A single failed machine can be replaced from its durable dataDir or documented backup path if quorum survived. Corrupt local files fail closed unless the selected recovery policy allows partial local recovery. Hot Backup is point-in-time; it does not replace per-node WAL durability.

Verification

Section titled “Verification”

WAL durability, crash recovery, CRC validation, compaction, disk-fault, graceful restart, snapshot store, and Hot Backup tests cover this layer. Operators watch fsync latency, WAL file size and entry count, snapshot duration, validation errors, backup age, and startup recovery logs.

Section titled “Related”
  • Raft Design — the log replication and commit path that persistence makes durable.
  • Persistence & Durability — the operator-facing guide to WAL, snapshot, and backup behavior.

LoomCache is an independent open-source project. It is not affiliated with, endorsed by, or sponsored by Hazelcast, Inc. or by any other company whose products are named in this documentation. “Hazelcast” is a trademark of Hazelcast, Inc.; references to it are nominative and describe only migration and comparison. All other product and company names are trademarks of their respective owners and are used for identification purposes only.