ZeroLock Architecture

ZeroLock is a concurrency-correctness substrate for multi-agent AI systems. It solves the problem of agents contradicting each other, redoing work, or losing shared state when multiple agents concurrently touch the same state.

Why this shape

Multi-agent systems fail in a small number of recurring ways once more than one agent can touch the same piece of state: two agents write the same value and the second write silently erases the first (a lost update); two agents grab the same queued task because nothing enforced an exclusive claim (a duplicate claim); two agents each commit to a decision — "use Postgres," "use DynamoDB" — and nothing arbitrates a single winner (a contradiction); or an agent sends the email, executes the trade, or ships the infrastructure change before anyone had a chance to say no (an irreversible mistake).

Each of ZeroLock's five layers exists to close exactly one of these failure modes, and they're deliberately kept separable rather than folded into one opinionated "coordination framework": Layer 1 stops lost updates and duplicate claims structurally (the gateway simply refuses a stale write); Layer 2 makes that structural safety cheap at scale instead of trading correctness for chattiness; Layer 3 gives contradictions somewhere to go once they're detected, instead of just rejecting and stopping; and Layer 4 is the one layer that isn't about resolving a conflict at all — it's about not letting an unresolvable action happen in the first place. Layer 0 is what makes all of this usable without asking you to abandon whatever orchestration you already run agents on.

Design Philosophy

• Gateway is the authority: The gateway is the single stateful service; SDKs are stateless clients. This is a deliberate constraint, not an accident of the first implementation — if correctness logic existed in the SDKs too, a TypeScript agent and a Python agent could disagree about what's valid, and the two SDKs could drift out of sync with each other over time. Putting every correctness decision in one place means "is this commit valid" always has exactly one authority to ask.
• Schema as contract: Both SDKs validate against a single shared schema package (packages/schema) to prevent drift. A shard's shape, its value_schema, and the wire messages both languages send are generated from — or checked against — the same source, so a change to the protocol can't quietly diverge between the TypeScript and Python clients.
• Correctness over speed: Structural correctness is never compromised for performance. If a choice comes down to "reject this write and make the caller retry" versus "accept it and hope it's fine," ZeroLock always rejects. Layer 2's caching exists specifically so that this stance doesn't have to cost you throughput.
• Byzantine calibration: Fault tolerance is an explicit, calibrated parameter, not an unconditional guarantee. ZeroLock does not claim to defend against an unbounded fraction of arbitrarily malicious agents in every configuration — see Benchmarking for what's actually been tested (ZL-007) and at what scale, and treat "how much Byzantine tolerance do I need" as a question to answer for your own deployment, not an assumption to inherit.

Five-Layer Architecture

Layers 0–1 are load-bearing: nothing above them works without a substrate to run in and a structural correctness guarantee to build on. Layers 2–4 are each independently optional — you can run ZeroLock with just Layer 1 and get correct, non-cached, non-semantic, no-approval-gate coordination; Layers 2 through 4 are refinements you add as your latency, conflict-frequency, or irreversibility requirements demand them.

Layer 0: Substrate Adapters

Integration points for existing agent runtimes. ZeroLock plugs into these systems rather than replacing them — the design assumption is that you already have an orchestration layer deciding which agent runs when, and ZeroLock's only job is to be the thing all of them talk to when they touch shared state.

Supported Substrates:

• Temporal workflows
• Restate virtual objects
• Inngest functions
• LangGraph state machines (see examples/langgraph-demo/)
• CrewAI crews (see examples/crewai-demo/)
• Vercel AI SDK tool-calling agents (see examples/vercel-ai-demo/)
• Bare deployments (direct SDK usage, no orchestration framework at all)

For LangGraph specifically, the adapter provides a ZeroLockSaver checkpointer (so graph state is durably persisted as shards, not just held in memory) and create_zerolock_tools() (so a graph's tool-calling nodes can read/commit shards without hand-wiring HTTP calls). This was independently verified by actually compiling and running a real StateGraph through it — see Benchmarking §2.2.C — including three integration bugs the verification surfaced and fixed in zerolock_langgraph/checkpointer.py (a wrong create_shard() call signature, a serializer API mismatch, and a missing aput_writes() implementation that LangGraph's Pregel loop requires for any graph with more than one node).

The CrewAI and Vercel AI SDK adapters expose the same primitives (read_shard/readShard, commit_shard/commitShard, claim_task/claimTask) as CrewAI BaseTools and Vercel AI SDK CoreTools respectively, so a crew or a generateText() tool-calling loop can coordinate through ZeroLock the same way a LangGraph node does.

Layer 1: Structural Plane

State is split into two shard types based on coordination needs. This split is the single most important design decision in ZeroLock: it exists because "how do I merge two concurrent writes" has two genuinely different correct answers depending on what the data means, and pretending there's one universal merge strategy is where most naive coordination layers go wrong.

Additive Shards (CRDT)

For data that can merge automatically without conflicts — where "both agents' contributions survive" is always the right outcome:

• Task queues
• Logs and events
• Artifacts and metadata

Implementation: Yjs CRDTs, gateway-side, with three crdt_types:

• map: Key-value storage
• array: Ordered collections
• text: Collaborative text (creatable via the schema; commit-side support is not yet wired — see the note below)

Coordination: Lock-free convergence via CRDT merge semantics. Two agents can append to the same task queue simultaneously and neither has to wait for the other — the CRDT guarantees both appends survive and every replica eventually agrees on the same final state, without any agent needing to know the other was writing at the same time.

Committing to an additive shard is not a single generic operation — the gateway routes it to one of two distinct paths depending on crdt_type, and getting this distinction right was itself one of the more consequential fixes made to the gateway:

• map shards go through CRDTStore.commitMapPatch(), which applies your new_value as a grow/update-only patch: every key present in the object you send is set (or deleted, if you send it as exactly null); a key that's simply absent from your payload is left untouched. This is deliberate — a client's new_value reflects only what it had read, which can already be stale by the time the commit lands if another agent added a key of its own in the meantime. Treating "absent" as "delete" was tried and reproducibly clobbered concurrent additions; treating "absent" as "leave alone" is what makes two agents writing different keys converge with zero conflict. Two agents writing the same key resolve via Yjs's own last-write-wins register.
• array shards go through CRDTStore.appendItem(), which pushes your item as a genuine Yjs array operation rather than diffing a full array against server state. This distinction matters under real concurrency: a "read the array, append one item, write the whole thing back" pattern can't reliably tell a real append from a structural rewrite once two commits race — by the time the second commit is processed, the array may already contain the first commit's item, making the second client's locally-computed array look like a rewrite and silently dropping the first item. Sending append-intent explicitly (rather than a full array) sidesteps this: every concurrent appender's item survives regardless of commit order. This was verified live at up to 80 concurrent appenders with zero loss (see Benchmarking).
• text shards can be created via the schema, but the gateway's commit route does not yet have a text-specific patch path — attempting to commit to one today returns 400 UNSUPPORTED_DIFF. Treat text as forward-looking schema support, not a currently usable CRDT type, until this is implemented.

Decision Shards (OCC)

For mutually exclusive facts that cannot merge — where exactly one value has to win, because averaging or combining the two candidate values would be meaningless or actively wrong:

• Configuration choices (e.g., "use Postgres" vs "use DynamoDB" — there is no sensible "merge" of these two answers)
• Approval states
• Ownership assignments
• Task queues, in practice — see the note below; this contradicts what you might expect from "task queues" being listed under additive shards above, and the contradiction is deliberate and load-bearing.

Implementation: Optimistic Concurrency Control with:

• Version checking on commit — every commit names the version it read; the gateway only applies it if that version is still current
• DeliveryLog for read tracking — the gateway records every read, so it can tell whether a value an agent based a decision on has since changed
• Read-set validation for cross-shard staleness — a commit can name other shards it depended on, so the gateway can reject it if any of those dependencies changed since they were read, even if the shard being written hasn't

Coordination: Write-verify pattern ensures exactly one winner. This is intentionally "optimistic": nothing blocks a second writer from attempting a commit while a first is in flight. Instead, both attempts race to the gateway, and the loser gets a 409 back rather than a lock timeout — which is why the standard client pattern is read → attempt commit → on 409, re-read and retry, rather than acquiring anything up front.

Task Claiming Protocol

A specific, named pattern for claiming work from a shared queue without two agents ever believing they both own the same task:

1. Write claimant ID to the queue
2. Wait for the commit round-trip
3. Re-read to confirm ownership
4. If confirmed, proceed; else retry

Steps 2–3 are the part that's easy to skip and shouldn't be: only a re-read after the round-trip actually tells you whether your write "won" or was overwritten by a concurrent claim. Both SDKs' claimTask / claim_task helpers implement this for you — see TypeScript SDK and Python SDK.

Why the task queue itself is a decision shard, not additive, despite being the textbook "queue" example. This project's own earlier design docs described task-claiming as naturally additive — write-wait-reread on a Y.Map, no locking needed. Building and stress-testing that at real scale (50+ concurrent claimants) found otherwise: an additive queue's last-write-wins register lets two agents' claim-writes race in a way that both re-reads can independently see their own write "confirmed," producing genuine duplicate claims under load — 113 claim-results for 100 unique tasks in one measured run. Switching the queue itself to a decision (OCC) shard — while still keeping shards like a findings log genuinely additive — closes this: OCC's version check means only one of two racing claim-commits can ever succeed, full stop, regardless of what either agent's local re-read believes. This is documented in detail, with the isolating experiment that found it, in examples/research-desk-demo/demo.py's docstring and README. If you're modeling your own task queue, use a decision shard for the claim state, and reserve additive shards for state where every contribution is genuinely meant to survive (a results/findings log fed by claimed tasks, for instance).

Request Lifecycle: How a Commit Is Routed

The diagram above says what each layer is for; this one shows what actually happens, mechanically, inside the gateway when POST /shards/:id/commit arrives — the single request every guarantee in this document ultimately reduces to.

The one thing this diagram makes explicit that's easy to miss reading the wire protocol alone: an additive shard's commit path never touches OCCStore at all, and a decision shard's commit path never touches CRDTStore. They are two structurally separate code paths from the moment the gateway looks at getCrdtType(id), not one generic "commit" function with a flag — which is exactly why "can this additive shard's write ever be rejected as stale" isn't a meaningful question to ask (it can't; there is no version check on that path) and "can this decision shard's write silently merge with a concurrent one" isn't either (it can't; OCC never merges, it only accepts-or-rejects).

Layer 2: Efficiency Plane

MESI-cache-coherence-inspired invalidation reduces message overhead. This layer exists purely for cost and latency — it changes nothing about correctness, and everything above and below it behaves identically whether Layer 2 is enabled or not.

Subscriber-Scoped Push

Instead of broadcasting all changes to all agents — which is the naive approach, and the one that scales worst:

• Track which agents are subscribed to which shards
• Send invalidations only to relevant subscribers
• Achieve a large reduction in message traffic compared to naive rebroadcast, measured live per-deployment via GET /metrics/ws's costSavingsPercent field (see ZL-008 in Benchmarking) rather than a fixed constant

Cache Coherence States

┌──────────┐     invalidate     ┌─────────┐
│  Fresh   │ ←───────────────── │ Invalid │
└────┬─────┘                    └────┬────┘
     │                               │
     │ read from                     │ re-read from
     │ gateway                       │ gateway
     │                               │
     └───────────────────────────────┘

A cached client's local copy of a shard starts Fresh the moment it's read. If any other agent commits a change to that shard, the gateway pushes an invalidate message (over WebSocket) only to agents subscribed to it, flipping their local copy to Invalid. The next read triggers a fresh fetch from the gateway, which brings it back to Fresh. Nothing about a shard's correctness depends on this cache — a stale read still can't produce a stale write, because commits are always version-checked against the gateway regardless of what the local cache believes.

On the client SDKs specifically: CachedZeroLockClient only gets this automatic invalidation if you actually give it a WebSocket URL (wsUrl) and either call connectWs() yourself or pass autoConnectWs: true — without one, the cache still works (it serves reads from memory and respects ttlMs if you set one), but it will never receive a push telling it another agent changed the shard. See TypeScript SDK — Cached Client / Python SDK — Cached Client for the exact configuration.

Benefits:

• Reduced network traffic
• Lower latency for cached reads
• Automatic cache invalidation — you never have to manually decide when a cached value might be wrong, provided the WebSocket connection above is actually wired up

Layer 3: Semantic Plane

When structural conflicts occur, enable semantic resolution. This is the layer where "the gateway rejected your commit" stops being a dead end and becomes an input to a decision.

Conflict Notification

When a commit fails due to OCC conflict:

1. Gateway sends conflict_notify via WebSocket
2. Message includes winning value, rejected value, and diff
3. Agent's onConflict callback receives the data

Resolution Strategies

client.setOnConflict((shardId, version, winning, rejected, diff) => {
  // Option 1: Accept winning value
  // (default if no callback)
 
  // Option 2: Merge values
  const merged = merge(winning, rejected);
  client.commitShard(shardId, merged, version);
 
  // Option 3: Escalate to LLM judge
  const resolution = await llm.judge(winning, rejected, context);
  client.commitShard(shardId, resolution.value, version);
});

The three options above form a spectrum from cheapest to most expensive: accepting the winning value is free and correct by construction, but throws away the rejected agent's work; a programmatic merge function is cheap and preserves both sides, but only works when the value type actually has a sensible merge (numeric sums, set unions); escalating to an LLM judge is the most expensive option but the only one that can make a genuinely contextual call — for example, deciding which of two conflicting natural-language decisions better reflects the task's actual goal. Which strategy is appropriate is a property of what the shard represents, not something ZeroLock can decide for you — this is deliberately left as a hook rather than a built-in policy.

Static Check Variant

For code shards where linters/type checkers provide better evidence than an LLM's judgment call:

const validation = await linter.check(rejectedCode);
if (validation.errors.length === 0) {
  // Rejected code is valid, may need human review
}

The insight behind staticCheck is that for some conflict types, a fast deterministic tool (a type checker, a linter, a schema validator) is strictly better evidence than an LLM judgment — if the rejected code fails to type-check, there's no ambiguity to resolve; if it passes, that's real signal that the conflict may be superficial (a race, not a genuine disagreement) and worth a human's attention rather than an automatic pick. As of this writing this remains a documented pattern for you to implement on top of onConflict, not a built-in SDK method.

Layer 4: Irreversibility Policy

Approval gates for operations that cannot be undone. Every other layer in this architecture is about resolving conflicts after the fact — Layer 4 is the one exception, because some actions don't have an "after the fact." You can't un-send an email or unwind a trade the way you can retry a stale commit.

Irreversible Operations

Operations on shards marked reversible: false:

• Payment transactions
• Sent messages/emails
• Infrastructure changes
• Database migrations

Approval Flow

┌──────────┐     request     ┌─────────┐     approve     ┌──────────┐
│  Agent   │ ───────────────→│ Pending │ ───────────────→│ Approved │
└──────────┘                 └────┬────┘                 └────┬─────┘
                                  │                           │
                                  │ reject                    │ auto-commit
                                  ↓                           ↓
                            ┌──────────┐               ┌──────────┐
                            │ Rejected │               │ Committed│
                            └──────────┘               └──────────┘

An agent attempting to commit to a reversible: false shard without a valid approval_id gets rejected outright with 403 APPROVAL_REQUIRED — its intended write is never held or queued anywhere; the agent must call POST /approvals/request itself first, then retry the commit carrying the resulting approval_id once that approval is approved. This is why Layer 4 is described as a gate rather than a check: the operation genuinely cannot occur without a human (or another authorized approver) actively saying yes, and the commit-with-approval_id path is verified against the approval's shard, agent, and expected version before the underlying commit is even attempted (see API Reference — Commit Shard for the exact verification steps).

Approving is what actually causes the commit to apply — POST /approvals/:id/approve calls the same OCCStore.commit() the direct commit path uses, against the shard's version at approval time. If the shard changed between the request and the approval, the approval can still fail with a version conflict; the approval record itself stays approved, but no commit happened, so check the response for a version/audit_entry_id before assuming the operation actually took effect.

TTL Expiration

Pending approvals have a TTL (default: 1 hour, configurable via ttl_seconds / ttlSeconds on the request). Expired approvals are treated as resolved-negative to prevent stale operations — an approval request that nobody looked at for a day shouldn't suddenly execute against current state that's since moved on.

Wire Protocol

HTTP (OCC/DeliveryLog)

WebSocket (CRDT sync + push)

Full request/response shapes, headers, and error codes for every entry above live in the API Reference.

Storage Architecture

Shard Schema

{ "id": "task_queue", "type": "decision", "owner_policy": "single-writer", "reversible": true }
{ "id": "findings_log", "type": "additive", "crdt_type": "array", "reversible": true }
{ "id": "db_schema_choice", "type": "decision", "owner_policy": "single-writer", "value_schema": { "enum": ["postgres", "dynamodb"] }, "reversible": false }

Note the third example: a decision shard marked reversible: false — this is what a Layer 4 approval gate actually looks like at the schema level. Nothing special happens at commit time to make it irreversible; it's simply that the gateway checks this flag before applying any commit and returns 403 APPROVAL_REQUIRED instead when it's false and no valid approval is attached.

Storage Adapter Interface

interface StorageAdapter {
  getShard(id: string): Promise<StoredShard>;
  createShard(id, type, value, schema): Promise<StoredShard>;
  commitShard(id, version, value): Promise<number>;
  appendAuditLog(entry: AuditEntry): Promise<void>;
  recordDelivery(agentId, shardId, version): Promise<void>;
  subscribe(shardId, callback): () => void;
  close(): Promise<void>;
}

This interface is what makes the storage backend swappable without touching any layer above it: OCC validation, CRDT sync, and every SDK call all go through this interface, never through adapter-specific code.

Available Adapters

• SQLiteAdapter: Default, WAL-mode local storage. Good for development and single-process deployments; see Deployment — Storage.
• PostgresAdapter: Selected with --postgres / ZEROLOCK_POSTGRES_URL. Proper OCC via SELECT ... FOR UPDATE, connection pooling, LISTEN/NOTIFY, and its own migration runner. A shared parity test suite keeps its behaviour identical to SQLite, and the harness scenarios pass against it. Several gateway processes can share one database: CRDT caches are evicted and invalidate pushes are relayed through LISTEN/NOTIFY, covered by multi-instance.test.ts — see Deployment — Storage.

Scalability

Tested Envelope

• Up to 64 concurrent agents (per the coordination-systems literature this design draws on — see Benchmarking §6 References)
• Independently measured up to 80 concurrent agents on a single contended decision shard during this project's own applied benchmarking (see Benchmarking §2.2.A), where task-claim attempt counts were observed to scale non-linearly — roughly 18 attempts at 5 agents, ~2,000 at 50 agents, ~6,700 at 80 agents
• Hundreds of shards per gateway; thousands of operations per second in typical mixed workloads

State this envelope to anyone evaluating ZeroLock rather than assuming it: it describes what's been tested, not a hard ceiling, and the coordination-overhead curve above is exactly the kind of thing that's cheap to re-measure against your own workload shape using the harness in packages/harness (see Benchmarking).

Scaling Strategies

1. Horizontal sharding: Multiple gateway instances, each owning shard ranges
2. Read replicas: For read-heavy workloads
3. Event sourcing: Audit log as source of truth for replay

Both strategies above assume a shared, multi-process-safe storage backend (Postgres) — see the storage adapter note above before planning either around today's default SQLite setup.

Security Model

Authentication

• API key-based authentication (x-api-key header), enabled via --auth-required / an apiKeysPath
• Per-key permissions: shard:read, shard:write, shard:create, approval:*, admin
• Key expiration support (expiresAt), plus admin-only endpoints to list keys and hot-reload the keys file without restarting the gateway

Rate Limiting

• Per-key and per-IP sliding-window limits (default: 1000 requests/minute)
• Configurable windows and per-key overrides
• Automatic Retry-After / X-RateLimit-* headers

Data Integrity

• OCC prevents lost updates
• Audit log tracks all operations
• CRDT convergence guarantees eventual consistency for additive shards

Full setup steps — generating keys, writing the permissions file, enabling --auth-required and --rate-limit — live in Deployment §Authentication Setup.

Design Trade-offs and Non-Goals

Being clear about what ZeroLock is not trying to be matters as much as describing what it does:

• It is not a general-purpose database. Shards are a coordination primitive, not a query engine — there's no equivalent of a SQL WHERE clause across shards, and it isn't meant to replace your application's primary datastore.
• It does not replace your orchestration framework. Layer 0 exists precisely because ZeroLock assumes Temporal, LangGraph, CrewAI, or something else already decides when an agent runs; ZeroLock only governs what happens when two of them touch the same state at once.
• It is not unconditionally Byzantine fault-tolerant. The design philosophy calls this out explicitly: Byzantine tolerance is a calibrated, testable parameter (see ZL-007 in Benchmarking), not a blanket guarantee against arbitrarily malicious agents at any scale.
• Correctness is prioritized over raw throughput. If your workload has almost no write contention, the OCC overhead on decision shards is close to free; if it has heavy contention on a small number of hot shards, expect the coordination-overhead curve described above, and consider whether that state genuinely needs to be a decision shard versus an additive one.

Frequently Asked Questions

Why two shard types instead of one universal CRDT? Because some state genuinely has no correct "merge." A CRDT can merge two agents both appending to a log; it cannot meaningfully merge "agent A chose Postgres" with "agent B chose DynamoDB" into a value that means anything. Decision shards exist for exactly that case.

Why is a task queue a decision shard when it's described right above as the canonical additive-shard example? See Task Claiming Protocol above — the claim state (who owns which task) needs mutual exclusion and is a decision shard in practice; a downstream log fed by completed tasks (findings, results, events) is genuinely additive. "Task queue" as a single concept splits into these two pieces once you look closely at what needs to merge versus what needs a single winner.

What happens if the gateway process dies mid-commit? This is the single biggest open item in the project's own benchmarking gap analysis — see Benchmarking §3.1. Today's adversarial coverage tests application-layer races and Byzantine agents, not gateway process kills; treat gateway durability under a hard crash as unverified until that scenario (proposed ZL-010) exists.

Do I need Layer 3 or Layer 4 if I'm not doing anything irreversible? No — both are optional. A read-heavy, low-stakes agent system can run correctly on Layer 1 alone, add Layer 2 if message costs matter, and skip 3 and 4 entirely if nothing it does is either genuinely ambiguous on conflict or genuinely irreversible.