Example: A Multi-Agent Support Desk (Gemini)
Time: 30 minutes to read and run | Languages: TypeScript and Python | Needs: a ZeroLock gateway URL, two API keys, a Gemini API key
This is the full, working example for the private beta. You will build a small customer-support desk in which three LLM agents work one shared queue of tickets at the same time, on a live ZeroLock gateway, using only the published SDKs (@zlock/sdk-ts and zerolock-sdk). Nothing here is mocked: the agents really claim tickets, really call Gemini, really compete for money, and a human really has to approve every reply before it goes out.
The complete source is the examples/support-desk folder of the ZeroLock repository (ask your contact for access). Every important piece of it is shown and explained below, in both languages.
What you will learn
| Problem when several agents share state | What ZeroLock does | Where you will see it |
|---|---|---|
| Two agents grab the same task | An ordered claim log decides exactly one winner | Step 3 |
| Two agents update the same number and one update is lost | Version-checked commits (optimistic concurrency): the loser gets a conflict, re-reads and decides again | Step 5 |
| An agent does something that cannot be undone (sends a message) | An irreversible shard that the gateway will not write until someone else approves | Step 6 |
| An agent approves its own action | The gateway rejects an approval from the same key that requested it | Step 7 |
| "Did it really work?" | A verifier reads the gateway state and checks the guarantees | Step 9 |
The scenario
A shop has 6 open tickets (tickets.json). Four are refund requests that together ask for $305, one is a how-to question and one is a login problem. The desk has a $150 refund budget for the day. Three agents, agent-a, agent-b and agent-c, each repeat this loop until no ticket is left:
- Claim an open ticket (only one agent may work any given ticket).
- Ask Gemini to triage it: category, priority and how much money the customer is asking for.
- Take that amount from the shared refund budget. Whatever is left in the budget is all there is, no matter how many agents ask at once.
- Ask Gemini to write the reply, telling it exactly how much refund was granted.
- Ask for approval to send the reply. A separate supervisor approves or rejects it.
Because the requests total more than the budget, the agents have to compete, and some customers get a partial or no refund. That makes the example a real test of coordination: if ZeroLock did not do its job, you would see tickets answered twice, money granted twice, or messages sent without approval.
Architecture
agent-a ─┐ ┌─ queue (additive map) the 6 tickets, read-only
agent-b ─┼─► ZeroLock gateway (your environment) ├─ claims (additive array) who claimed what, in order
agent-c ─┘ ▲ one API key: "worker" ├─ budget (decision shard) $150, ledger of every draw
│ ├─ results (additive array) one row per handled ticket
supervisor ──┘ other API key: "supervisor" └─ reply-t1 … reply-t6 (decision shards, reversible: false)Each agent also talks to Gemini over plain HTTPS. The agents never talk to each other. Everything they need to agree on lives in shards on the gateway.
The shards
Every shard of one run starts with desk-<run id>-, so you can run the example many times without collisions and delete a run with one command.
| Shard | Type | Why this type |
|---|---|---|
…-queue | additive, map | The tickets. Written once at the start, only read afterwards |
…-claims | additive, array | An append-only log. Many agents append at once; nothing is ever overwritten |
…-results | additive, array | Each agent appends its result row; no coordination needed |
…-budget | decision | One shared number. Must never be lost or double-spent, so every change is version-checked |
…-reply-<ticket> | decision, irreversible | The outgoing message. Cannot be written without an approval |
The rule of thumb: additive shards are for data that many agents add to (merged automatically, never a conflict); decision shards are for facts that exactly one version of the truth may exist (checked, one winner).
Before you start
You need three things.
-
A gateway and two keys. Your operator creates a private environment and two keys on the gateway. They are different on purpose:
Key Permissions Used by workerdefault (read and write shards, create shards, request approvals) the three agents. Not bound to one agent name, because three agents share it supervisorapproval:resolve,approval:read,shard:read, bound to the agent namesupervisorthe human in the loop Keys look like
zl_…. They are shown once. Treat them like passwords. -
A Gemini API key from Google AI Studio.
-
Node 18 or newer for TypeScript, or Python 3.10 or newer for Python.
Put the secrets in your shell, never in code:
export ZEROLOCK_GATEWAY="https://<your gateway>.cloudfront.net"
export ZEROLOCK_API_KEY="zl_...worker key..."
export ZEROLOCK_SUPERVISOR_KEY="zl_...supervisor key..."
export GEMINI_API_KEY="...your Gemini key..."If you leave GEMINI_API_KEY out, the example uses a small offline stand-in for the model so you can check the gateway side first.
Step 1: Set up the project
TypeScript:
mkdir support-desk && cd support-desk
npm init -y
npm install @zlock/sdk-ts@beta
npm install -D tsx typescript @types/nodeSet "type": "module" in package.json. The SDK needs no extra packages for this example. (The ws package is only needed for live WebSocket subscriptions on Node 18 or 20.)
Python:
mkdir support-desk && cd support-desk
python3 -m venv .venv && source .venv/bin/activate
pip install --pre zerolock-sdk--pre is needed because the SDK is a pre-release (beta). The SDK brings httpx, which we also use to call Gemini.
Step 2: Describe the shared state
The first job is to create the shards the agents will share. A small seed function does it. Run it once per run, before the agents start.
TypeScript:
import { ShardExistsError, ZeroLockClient } from '@zlock/sdk-ts';
const client = new ZeroLockClient({
gateway: process.env.ZEROLOCK_GATEWAY!,
apiKey: process.env.ZEROLOCK_API_KEY!,
agentId: 'seeder',
runId, // groups everything this run does in the gateway's trace
});
// The 6 tickets, as an additive map: { t1: { id, status, payload }, … }
await client.createShard(`desk-${runId}-queue`, {
type: 'additive', crdtType: 'map',
initialValue: Object.fromEntries(tickets.map((t) => [t.id, { id: t.id, status: 'pending', payload: t }])),
});
// Append-only logs: who claimed what, and one result row per ticket
await client.createShard(`desk-${runId}-claims`, { type: 'additive', crdtType: 'array', initialValue: [] });
await client.createShard(`desk-${runId}-results`, { type: 'additive', crdtType: 'array', initialValue: [] });
// The shared refund budget: a decision shard, so every change is version-checked
await client.createShard(`desk-${runId}-budget`, {
type: 'decision',
initialValue: { limit: 150, remaining: 150, ledger: [] },
});Python:
from zerolock_sdk import CreateShardOptions, ZeroLockClient
async with ZeroLockClient(gateway=GATEWAY, api_key=WORKER_KEY, agent_id="seeder", run_id=run_id) as client:
await client.create_shard(f"desk-{run_id}-queue", CreateShardOptions(
type="additive", crdt_type="map",
initial_value={t["id"]: {"id": t["id"], "status": "pending", "payload": t} for t in tickets},
))
await client.create_shard(f"desk-{run_id}-claims", CreateShardOptions(type="additive", crdt_type="array", initial_value=[]))
await client.create_shard(f"desk-{run_id}-results", CreateShardOptions(type="additive", crdt_type="array", initial_value=[]))
await client.create_shard(f"desk-{run_id}-budget", CreateShardOptions(
type="decision", initial_value={"limit": 150, "remaining": 150, "ledger": []},
))Things to notice:
runId/run_idon the client is optional metadata. The gateway stores it in every audit event, so afterwards you can filter one run's whole history.- The budget keeps its own ledger (every draw: ticket, agent, requested, granted) inside the same shard as the number. Because they change in one version-checked commit, the total and the ledger can never disagree. The verifier relies on this.
- The reply shards are created later, by the agent that owns the ticket (Step 6).
Step 3: Claim a ticket (exactly one winner)
Every agent runs the same loop. First it looks for tickets nobody has claimed yet, then it tries to claim one. This is the part where three agents can reach for the same ticket in the same millisecond.
TypeScript:
import { ZeroLockClient, claimTask } from '@zlock/sdk-ts';
const client = new ZeroLockClient({
gateway, apiKey: workerKey, agentId: 'agent-a', runId,
metadata: { example: 'support-desk', lang: 'ts' },
});
for (;;) {
const claims = await client.getShard<{ task: string; agent: string }[]>(`desk-${runId}-claims`);
const taken = new Set((claims.value ?? []).map((c) => c.task));
const open = ticketIds.filter((id) => !taken.has(id));
if (open.length === 0) break; // nothing left: this agent is done
let mine: string | undefined;
for (const id of shuffle(open)) { // random order, so agents do not all fight over t1
const r = await claimTask(client, queueShard, id, 'agent-a', {
claimLog: `desk-${runId}-claims`, // decide the winner with the ordered log
reportEvents: true, // show claim_won / claim_lost in the gateway trace
});
if (r.claimed) { mine = id; break; }
console.log(`${id}: lost the claim to ${r.owner}`);
}
if (!mine) continue;
// … work on ticket `mine` (Steps 4 to 6) …
}Python:
from zerolock_sdk import ZeroLockClient, claim_task
async with ZeroLockClient(gateway=GATEWAY, api_key=WORKER_KEY, agent_id="agent-a", run_id=run_id) as client:
while True:
claims = (await client.get_shard(claims_shard)).value or []
taken = {c["task"] for c in claims}
open_ids = [i for i in ticket_ids if i not in taken]
if not open_ids:
break
random.shuffle(open_ids)
mine = None
for tid in open_ids:
r = await claim_task(client, queue_shard, tid, "agent-a", claim_log=claims_shard, report_events=True)
if r.claimed:
mine = tid
break
print(f"{tid}: lost the claim to {r.owner}")
if mine is None:
continue
# … work on ticket `mine` (Steps 4 to 6) …How it works: claimTask appends { task, agent } to the claim log and then re-reads the log. The gateway keeps appends in a strict order, so the first entry for a ticket is its owner. Agents that appended later see someone else's name first and get claimed: false. No locks, no waiting, and still exactly one winner.
Why a log and not a flag on the ticket? If agents simply wrote
claimedByonto the ticket in the shared map, two agents writing at once would both believe they won: the last write wins. The log makes the order explicit. This is the single most common mistake in multi-agent queues.
In a live run you can see it happen: agent-c: t4: lost the claim to agent-b.
Step 4: Triage and draft with Gemini
The LLM calls are plain HTTPS requests to the Gemini API, with the JSON response mode turned on so the model answers in a parseable format. ZeroLock is not involved here: it only needs the results of the thinking to be coordinated safely.
TypeScript:
async function gemini(system: string, user: string): Promise<unknown> {
const url = `https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent`;
for (let attempt = 0; attempt < 4; attempt++) {
const res = await fetch(url, {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-goog-api-key': process.env.GEMINI_API_KEY! },
body: JSON.stringify({
systemInstruction: { parts: [{ text: system }] },
contents: [{ role: 'user', parts: [{ text: user }] }],
generationConfig: { temperature: 0.2, responseMimeType: 'application/json' },
}),
});
if (res.status === 429 || res.status >= 500) { // overloaded: wait and retry
await new Promise((r) => setTimeout(r, 1000 * 2 ** attempt));
continue;
}
if (!res.ok) throw new Error(`Gemini ${res.status}`);
const data = await res.json();
const text = data.candidates?.[0]?.content?.parts?.map((p: any) => p.text ?? '').join('') ?? '';
return JSON.parse(text);
}
throw new Error('Gemini kept failing');
}The triage prompt asks for a strict shape:
You are a support triage agent. Read the ticket and answer ONLY with JSON:
{"category":"billing|product|how-to|technical|other","priority":"low|normal|high",
"refund_usd":<number, the dollar amount the customer explicitly asks to be refunded, else 0>,
"summary":"<one sentence>"}Two habits worth copying:
- Never trust model output blindly. The code clamps
refund_usdto a positive number and falls back to defaults for missing fields. The model suggests; your code decides. - Keep the model's decisions small and typed. The model returns a category and an amount. The rules (how much can actually be refunded) are enforced by shared state in the next step, not by the prompt.
The model name defaults to gemini-flash-latest, which always points at Google's current Flash model. Pin an exact model with the GEMINI_MODEL environment variable if you want repeatable behaviour.
Python uses the same request with httpx:
async def gemini(cfg, system: str, user: str) -> dict:
url = f"https://generativelanguage.googleapis.com/v1beta/models/{cfg.model}:generateContent"
body = {
"systemInstruction": {"parts": [{"text": system}]},
"contents": [{"role": "user", "parts": [{"text": user}]}],
"generationConfig": {"temperature": 0.2, "responseMimeType": "application/json"},
}
async with httpx.AsyncClient(timeout=60) as http:
for attempt in range(4):
res = await http.post(url, json=body, headers={"x-goog-api-key": cfg.gemini_key})
if res.status_code == 429 or res.status_code >= 500:
await asyncio.sleep(2 ** attempt)
continue
res.raise_for_status()
parts = res.json()["candidates"][0]["content"]["parts"]
return json.loads("".join(p.get("text", "") for p in parts))
raise RuntimeError("Gemini kept failing")Step 5: Spend from the shared budget (no lost updates)
This is the heart of the example. All three agents may want money at the same moment, and there is only $150. The budget lives in one decision shard, and every change is a read, decide, write-with-the-version-you-read cycle.
TypeScript:
import { StaleCommitError } from '@zlock/sdk-ts';
async function reserveRefund(client, budgetShard, ticket, agent, want) {
let conflicts = 0;
for (let attempt = 0; attempt < 25; attempt++) {
const cur = await client.getShard(budgetShard); // value AND version
const already = cur.value.ledger.find((l) => l.ticket === ticket);
if (already) return { granted: already.granted, conflicts }; // safe to retry after a crash
const granted = Math.max(0, Math.min(want, cur.value.remaining));
const next = {
...cur.value,
remaining: cur.value.remaining - granted,
ledger: [...cur.value.ledger, { ticket, agent, requested: want, granted }],
};
try {
await client.commitShard(budgetShard, next, cur.version); // "apply this on top of version N"
return { granted, conflicts };
} catch (err) {
if (!(err instanceof StaleCommitError)) throw err;
conflicts++; // someone else spent first: look again
await sleep(20 + Math.random() * 80);
}
}
throw new Error('too many conflicts');
}Python:
from zerolock_sdk import StaleCommitError
async def reserve_refund(client, budget_shard, ticket, agent, want):
conflicts = 0
for _ in range(25):
cur = await client.get_shard(budget_shard)
v = cur.value
done = next((l for l in v["ledger"] if l["ticket"] == ticket), None)
if done:
return done["granted"], conflicts
granted = max(0, min(want, v["remaining"]))
nxt = {**v, "remaining": v["remaining"] - granted,
"ledger": [*v["ledger"], {"ticket": ticket, "agent": agent, "requested": want, "granted": granted}]}
try:
await client.commit_shard(budget_shard, nxt, cur.version)
return granted, conflicts
except StaleCommitError:
conflicts += 1
await asyncio.sleep(0.02 + random.random() * 0.08)
raise RuntimeError("too many conflicts")What happens when two agents collide:
budget = $90 left, version 7
agent-a reads (remaining 90, v7) agent-b reads (remaining 90, v7)
agent-a wants $60 → writes remaining 30 on top of v7 ✓ accepted → version 8
agent-b wants $60 → writes remaining 30 on top of v7
✗ rejected: 409, the budget is already at v8
agent-b reads again (remaining 30, v8)
agent-b is granted $30 → writes on top of v8 ✓Without the version check both agents would have written "30 left", and the desk would have handed out $120 of a $90 budget. With it, the second writer is told it is stale, looks at the new reality and decides again. That is called optimistic concurrency control. The SDK raises StaleCommitError; your code only has to retry the decision, not the whole task.
Three details that make this production-grade:
- The agent re-decides after a conflict (
Math.min(want, remaining)runs again on fresh data). It does not blindly resend its old number. - The ledger check at the top makes the function idempotent: if an agent crashes after the commit and restarts, it finds its own ledger entry and does not spend twice.
- The ledger and the remaining amount are in one shard, so they change together or not at all.
In the output you may see a line like t1: refund granted $0 of $80 (1 OCC conflicts retried). That is this mechanism doing its job.
Step 6: Send the reply only after approval
Sending a message to a customer cannot be undone, so it goes through an irreversible shard. A shard created with reversible: false cannot be written by an agent on its own; the gateway demands an approval first.
TypeScript:
import { ApprovalClient, ShardExistsError } from '@zlock/sdk-ts';
const approvals = new ApprovalClient({ gateway, apiKey: workerKey, agentId: 'agent-a', runId });
const replyShard = `desk-${runId}-reply-${ticket.id}`;
try {
await client.createShard(replyShard, { type: 'decision', reversible: false, initialValue: { status: 'draft' } });
} catch (e) { if (!(e instanceof ShardExistsError)) throw e; }
const cur = await client.getShard(replyShard);
const approval = await approvals.requestApproval({
shardId: replyShard,
expectedVersion: cur.version,
proposedValue: { status: 'sent', ticket: ticket.id, to: ticket.customer, by: 'agent-a', refund_usd: granted, text },
description: `Send reply to ${ticket.customer} (${ticket.id}, refund $${granted}): ${text.slice(0, 140)}`,
ttlSeconds: 900, // an unanswered request expires after 15 minutes
});
const final = await approvals.waitForApproval(approval.id, { pollIntervalMs: 1000, timeoutMs: 10 * 60_000 });
const sent = final.status === 'approved'; // 'rejected' or 'expired' means: do not sendPython:
from zerolock_sdk import ApprovalClient, CreateShardOptions, ShardExistsError
async with ApprovalClient(gateway=GATEWAY, agent_id="agent-a", api_key=WORKER_KEY, run_id=run_id) as approvals:
try:
await client.create_shard(reply_shard, CreateShardOptions(type="decision", reversible=False, initial_value={"status": "draft"}))
except ShardExistsError:
pass
cur = await client.get_shard(reply_shard)
approval = await approvals.request_approval(
shard_id=reply_shard,
proposed_value={"status": "sent", "ticket": ticket["id"], "to": ticket["customer"], "by": "agent-a", "refund_usd": granted, "text": text},
expected_version=cur.version,
description=f"Send reply to {ticket['customer']} ...",
ttl_seconds=900,
)
final = await approvals.wait_for_approval(approval.id, poll_interval_seconds=1.0, timeout_seconds=600)
sent = final.status == "approved"The agent does not write the message itself. It proposes the final value (status: 'sent' plus the text) and waits. When the supervisor approves, the gateway writes the proposed value to the shard. That means the message and the approval are one atomic fact: the shard can only ever say "sent" if an approval exists, and the audit log shows who proposed it and who approved it.
If the agent tried to skip the approval and commit directly, the gateway would answer 403 APPROVAL_REQUIRED. Try it (see Experiments).
Step 7: The supervisor (the human in the loop)
The supervisor is a separate process with a separate key. It lists pending approvals and approves or rejects them. In this example the SDK has no helper for listing, so it calls the gateway's HTTP API directly with the supervisor key.
TypeScript:
const headers = { 'x-api-key': process.env.ZEROLOCK_SUPERVISOR_KEY!, 'content-type': 'application/json' };
// 1. What is waiting?
const res = await fetch(`${gateway}/approvals?status=pending&limit=100`, { headers });
const { approvals } = await res.json();
for (const a of approvals.filter((x) => x.shard_id.startsWith(`desk-${runId}-reply-`))) {
const text = a.proposed_value?.text ?? '';
// 2. Decide. Here: ask a person on the terminal (or approve automatically with --auto).
const yes = (await ask(`Approve? ${a.agent_id} wants to send:\n ${text}\n[y/N] `)).startsWith('y');
// 3. Resolve. The gateway commits the proposed value when you approve.
await fetch(`${gateway}/approvals/${a.id}/${yes ? 'approve' : 'reject'}`, {
method: 'POST', headers, body: JSON.stringify({ agent_id: 'supervisor' }),
});
}Python:
async with httpx.AsyncClient(headers={"x-api-key": SUPERVISOR_KEY}) as http:
pending = (await http.get(f"{GATEWAY}/approvals", params={"status": "pending", "limit": 100})).json()["approvals"]
for a in pending:
text = (a.get("proposed_value") or {}).get("text", "")
yes = input(f"Approve? {a['agent_id']} wants to send:\n {text}\n[y/N] ").lower().startswith("y")
await http.post(f"{GATEWAY}/approvals/{a['id']}/{'approve' if yes else 'reject'}", json={"agent_id": "supervisor"})Why two keys matter: the gateway remembers which key requested an approval and refuses to let the same key resolve it (403 SELF_APPROVAL). Agents share the worker key, which has no permission to resolve approvals at all, and the supervisor key can resolve but cannot write shards. A buggy or manipulated agent therefore cannot approve its own message, even if it knows the approval endpoint.
Step 8: Run it
Everything above is wired together in the example's command line. From the folder of your language:
TypeScript:
npm run demo # seed → 3 agents + supervisor (auto-approve) → verifyPython:
python -m support_desk allAdd --manual to approve each reply yourself, and --cleanup to delete the run's shards afterwards. A real run looks like this (shortened):
Run rmv5fr054, LLM: Gemini (gemini-flash-latest)
seeded run rmv5fr054: 6 tickets, refund budget $150
[agent-a] t5: claimed "Login keeps failing after password reset"
[agent-b] t4: claimed "Cancel and refund the last 3 months"
[agent-c] t4: lost the claim to agent-b ← Step 3
[agent-b] t4: billing/normal, refund requested $120 ← Step 4
[agent-b] t4: refund granted $120 of $120 ← Step 5
[agent-c] t2: refund granted $30 of $60 ← the budget runs dry
[agent-b] t1: refund granted $0 of $80 (1 OCC conflicts retried) ← Step 5, a collision
[agent-a] t5: reply waiting for approval 24f2707d ← Step 6
[supervisor] approved t5 for agent-a -> HTTP 200 ← Step 7
[agent-a] t5: reply APPROVED and sentWhich agent gets which ticket, and who gets the money, is different every run. What must be the same every run is the result of the verifier in the next step.
Step 9: Verify the guarantees
After the run, verify reads the final state straight from the gateway and checks the promises ZeroLock makes. It exits with code 1 if any fails, so you can use it as a test in your own pipeline.
| Check | What it proves |
|---|---|
| every ticket was claimed | No ticket was dropped |
| every ticket answered exactly once | No duplicate work, even though agents raced |
| the claim winner handled the ticket | The agent that won the claim log is the one that replied |
| budget never overspent | The total granted never exceeded $150 |
| budget arithmetic is exact (no lost updates) | limit - granted - remaining = 0: not a single dollar appeared or vanished |
| ledger has exactly one entry per ticket | No ticket was charged twice |
| results agree with the ledger | What each agent reports matches what the gateway recorded |
| every reply was sent through an approval, by the claiming agent | Nothing irreversible happened without a human |
PASS every ticket was claimed (6/6 claimed (1 losing claim attempts were recorded and ignored))
PASS budget arithmetic is exact (no lost updates) (limit - granted - remaining = 0.00)
PASS every reply was sent through an approval, by the claiming agent (all reply shards are "sent")
ALL CHECKS PASSEDThe verifier uses the same SDK you used for the agents; there is no back door. Anything you can verify here you can also verify in your own system.
Mixed languages: TypeScript and Python agents together
The two implementations use the same shard layout and the same result format, so they can work the same queue at the same time. Open two terminals with the same environment variables:
export RUN_ID=mixed1
# once: create the shards
(cd ts && npm run seed -- --run $RUN_ID)
# two TypeScript agents and two Python agents, plus one supervisor
(cd ts && npx tsx src/cli.ts agents --run $RUN_ID --ids ts-1,ts-2) &
(cd py && python -m support_desk agents --run $RUN_ID --ids py-1,py-2) &
(cd py && python -m support_desk supervisor --run $RUN_ID --auto)
wait
# either language can verify
(cd ts && npm run verify -- --run $RUN_ID)Four agents in two languages race for the same tickets and the same $150, and the same eight checks must still pass. The gateway is the only place where correctness is decided, which is exactly why the SDKs can be so thin.
Experiments: break it on purpose
The fastest way to understand what ZeroLock gives you is to take a guarantee away and watch what breaks.
- Remove the claim log. Delete
claimLogfrom theclaimTaskcall. Under load two agents can both think they own a ticket, and the verifier fails on "answered exactly once". - Ignore conflicts. In
reserveRefund, after aStaleCommitError, reuse the oldcurinstead of reading again. The gateway never accepts a write based on an old version, so the agent keeps getting409until its attempts run out and it stops with "too many conflicts". A stale decision can never overwrite a newer one; the only way forward is to look at the new state and decide again. - Bypass the approval. Add a direct
client.commitShard(replyShard, { status: 'sent' }, version)in place of the approval. The gateway answers403 APPROVAL_REQUIRED. - Approve your own message. Run the supervisor with the worker key. The gateway answers
403and the approval stays pending. - Reject a reply. Run with
--manualand answern. The agent seesrejected, does not send, and the verifier flags the unsent reply. This is the correct behaviour: a human said no. - Revoke a key mid-run. Ask your operator to revoke the worker key while agents are running. Within a few seconds every call fails with
401, and nothing further is written.
Troubleshooting
| You see | Cause and fix |
|---|---|
401 | A key is missing or wrong. Check ZEROLOCK_API_KEY and ZEROLOCK_SUPERVISOR_KEY |
403 AGENT_ID_MISMATCH | The worker key was created bound to one agent name. Ask for one created without a bound name |
403 when approving | The supervisor key lacks approval:resolve, or it is the same key as the worker |
Gemini 404 | GEMINI_MODEL is not available to your key. Unset it to use gemini-flash-latest |
Gemini 429 | Rate limit on your Gemini plan. The agents retry on their own; run fewer agents if it persists |
| The agents wait forever at "waiting for approval" | The supervisor is not running. Start it, or use the all command |
413 or 429 from the gateway | A beta limit was hit (request size or request rate). Back off and retry, or ask for higher limits |
Where to go next
- Replace the support tickets with your own work items: any queue of tasks, any shared number, any irreversible action fits the same three patterns (claim log, version-checked decision, approval gate).
- Read the TypeScript SDK and Python SDK guides for live subscriptions, caching and conflict hooks.
- Read the Architecture guide to see how the structural, efficiency, semantic and irreversibility layers fit together.
- Back to the Private Beta guide for limits and what to expect from the beta.
