ZeroLock Deployment Guide

Overview

This guide covers deploying ZeroLock in production environments. It follows the natural progression from a local, single-instance evaluation setup to a hardened, horizontally-scaled production deployment — each section builds on assumptions established by the one before it (Docker → manual build → Postgres → auth → monitoring → scaling), so read in order the first time through even if you only need one section later.

Storage: SQLite or PostgreSQL

The gateway runs on either backend, chosen at startup:

• SQLite (default, --db ./zerolock.db): zero setup, one file. Right for development and a single gateway process.
• PostgreSQL (--postgres postgres://... or ZEROLOCK_POSTGRES_URL): serve connects, runs migrations automatically, and uses the database for shards, the audit log, the delivery log, and approvals. If the connection fails the gateway exits with an error instead of falling back to SQLite.

Both backends are held to the same behaviour by a shared test suite (packages/gateway/test/adapter-parity.test.ts) that runs against SQLite always and against a real Postgres when ZEROLOCK_TEST_POSTGRES_URL is set. The nine harness scenarios (ZL-001 to ZL-009) pass against a gateway running on each.

What is and isn't validated for Postgres

• Validated: a single gateway process on Postgres, including concurrent commits (exactly one winner), concurrent appends to additive shards, and the approval workflow.
• Validated (packages/gateway/test/multi-instance.test.ts): several gateway processes on one database. Decision-shard commits use SELECT ... FOR UPDATE, so exactly one concurrent commit wins. Additive-shard caches are evicted when another process changes the shard (via LISTEN/NOTIFY), concurrent appends through different gateways are all kept, and WebSocket invalidate pushes reach clients on every process.
• Approximate across processes: per-environment usage totals (each process flushes its own counters, so quota checks can overshoot slightly) and API-key revocation, which takes effect within the key cache TTL (5 seconds) on other processes.

Existing SQLite data is not migrated to Postgres automatically.

Pre-Deployment Checklist

Before pointing real agent traffic at a gateway instance, confirm:

• Storage is chosen deliberately: SQLite for a single process, or Postgres via --postgres. Multiple gateway processes sharing one Postgres database are validated by the multi-process tests (see Storage and Scaling).
• --auth-required is enabled and every API key in your keys file has the minimum permissions it actually needs, not admin by default (see Authentication Setup).
• --rate-limit is enabled with limits sized to your expected agent concurrency (see Rate Limiting).
• Postgres connection string uses ?ssl=true for any connection that leaves a private network (see PostgreSQL Configuration).
• /health is wired into your orchestrator's liveness/readiness checks (see Monitoring).
• Prometheus is scraping /metrics and you have at least one alert on zerolock_shard_commits_total{status="conflict"} climbing sharply, which usually indicates unexpected write contention (see High Conflict Rates).
• A backup schedule exists for Postgres (pg_dump or continuous WAL archiving) — see Backup & Recovery.
• You've read Architecture — Scalability and know what agent-concurrency envelope your deployment has actually been tested at, versus assumed.

Docker Deployment

Quick Start

# Build and start all services
docker-compose up -d
 
# Check status
docker-compose ps
 
# View logs
docker-compose logs -f gateway

Services

Configuration

Edit docker-compose.yml environment variables:

environment:
  - PORT=8080
  - ZEROLOCK_POSTGRES_URL=postgres://zerolock:zerolock@postgres:5432/zerolock
  - ZEROLOCK_API_KEYS_PATH=/config/api-keys.json

The default docker-compose.yml credentials (zerolock/zerolock) are meant for local evaluation only — change the Postgres password and Grafana admin password before this stack is reachable from anywhere but your own machine.

Deploying on AWS (ECS Fargate)

The repository ships OpenTofu code (infra/) for a managed deployment: CloudFront with WAF in front of an internal load balancer, the gateway on ECS Fargate (2 to 6 tasks), and RDS PostgreSQL with TLS verification. It is what the private beta runs on.

agents ─HTTPS/WSS─► CloudFront + WAF ─► internal ALB ─► ECS Fargate (gateway) ─TLS─► RDS PostgreSQL

• Two stacks, infra/envs/staging and infra/envs/prod, built from the same modules. Remote state lives in S3 with a lock table, created once by infra/bootstrap.
• GitHub Actions deploys with short-lived AWS credentials (OIDC), so no AWS keys are stored in GitHub. A push to main builds the image once, deploys it to staging, and (once the production stack exists and its variables are set) waits for approval and promotes the same image to production.
• Database credentials come from an RDS-managed secret and reach the gateway as PGUSER/PGPASSWORD; see Database settings from PG variables.
• The gateway runs with ZEROLOCK_TRUST_PROXY_HOPS=2 (CloudFront and the load balancer), JSON logs, a 256 KiB body limit and the WebSocket heartbeat at 25 s, which keeps connections alive through CloudFront.
• /admin and /metrics return 404 at the load balancer. Manage environments and keys with the operator CLI through ECS Exec: infra/scripts/admin.sh <staging|prod> create-env ....

Setup, day-to-day operations, rollback and the list of things to verify are in infra/README.md in the repository. Partners joining the private beta start with Private Beta: Getting Started.

Manual Deployment

Prerequisites

• Node.js 20+
• PostgreSQL 14+ (recommended for production)
• pnpm

Build

# Install dependencies
pnpm install
 
# Build all packages
pnpm build

Configure

Create config/api-keys.json:

{
  "sk_live_your_admin_key": {
    "name": "Admin",
    "permissions": ["admin"]
  },
  "sk_live_your_agent_key": {
    "name": "Agents",
    "permissions": ["shard:read", "shard:write", "shard:create"]
  }
}

Treat this file the same way you'd treat any secrets file: it should never be committed to version control, and access to it should be as narrow as access to the keys themselves.

Database Setup

# Run PostgreSQL migrations
pnpm --filter @zerolock/gateway start migrate --postgres "postgres://user:pass@host:5432/zerolock"

Run migrations as a separate step before starting the gateway, especially in any automated deploy pipeline — never rely on the gateway process applying migrations implicitly on boot, since that makes rollbacks and multi-instance deploys harder to reason about.

Start Gateway

# Production start
cd packages/gateway
NODE_ENV=production node dist/cli.js serve \
  --postgres "postgres://user:pass@host:5432/zerolock" \
  --api-keys ./config/api-keys.json \
  --auth-required \
  --rate-limit

Of these four flags, --api-keys, --auth-required, and --rate-limit are worth treating as the production baseline, not optional extras — the local Quick Start deliberately omits them because they add friction that isn't useful when you're just trying a shard commit for the first time, but each addresses a real production concern covered later in this guide. --postgres switches the gateway from the local SQLite file to PostgreSQL; see Storage for what is validated.

Environment Variables

PostgreSQL Configuration

Connection URL Format

postgres://user:password@host:port/database?ssl=true

Always include ?ssl=true for any connection that crosses a network boundary you don't fully control (i.e., anything other than a private VPC link to a colocated database) — plaintext Postgres connections over the public internet expose both credentials and shard contents in transit.

Recommended Settings

-- Increase connection limit
ALTER SYSTEM SET max_connections = 200;
 
-- Configure shared buffers (25% of RAM)
ALTER SYSTEM SET shared_buffers = '4GB';
 
-- Write-ahead log
ALTER SYSTEM SET wal_level = 'replica';
ALTER SYSTEM SET max_wal_size = '2GB';

Connection Pooling

PostgresAdapter supports connection pooling natively:

const adapter = new PostgresAdapter({
  connectionUrl: process.env.ZEROLOCK_POSTGRES_URL,
  maxConnections: 20,
  idleTimeout: 30000,
});

This is what --postgres constructs for you; use it directly only if you embed the gateway in your own process via createGateway({ storage }). Size maxConnections per gateway instance against your Postgres max_connections setting and how many gateway instances you're running — three gateway instances each configured for 20 connections need at least 60 available Postgres connections, plus headroom for migrations, backups, and any direct psql access.

Database settings from PG* variables

If ZEROLOCK_POSTGRES_URL (or --postgres-url) is not set, the gateway builds the connection URL from PGHOST, PGPORT, PGDATABASE, PGUSER and PGPASSWORD. This lets platforms such as ECS inject credentials from a secrets store without composing a URL. Values are URL-encoded, so passwords with special characters work. Set ZEROLOCK_POSTGRES_CA_FILE to verify the server certificate.

Authentication Setup

Generate API Keys

# Using CLI
pnpm --filter @zerolock/gateway start keys generate

Operational settings

On SIGTERM the gateway drains, then stops; event streams still open after 5 seconds are cut.

Operator CLI (no HTTP admin API needed)

For production you can leave /admin unreachable and manage environments and keys from a one-off task that uses the same image and database credentials as the gateway:

node dist/cli.js admin create-env acme --name "Acme" --max-shards 50 --max-ops-per-day 100000 --max-ws 40
node dist/cli.js admin create-key acme --name planner --agent-id planner-1   # prints the key once
node dist/cli.js admin list-keys acme
node dist/cli.js admin set-limits acme --max-shards 100
node dist/cli.js admin revoke-key key_abc123def456
node dist/cli.js admin usage acme
node dist/cli.js admin delete-env acme --yes

It reads ZEROLOCK_POSTGRES_URL (or DB_PATH for SQLite). With Postgres, running gateways pick changes up through the usual notifications: deleted environments are shut down at once and revoked keys stop working within the key cache TTL (5 seconds). A SQLite database is only safe to administer this way while the gateway is stopped or from the same host.

Environments and Database-Backed Keys

An environment is the isolation boundary: shards, audit events, approvals, WebSocket subscriptions and usage are all scoped to the environment of the key that made the request, so the same shard id can exist in two environments without interacting. Requests without a key (auth off) use the default environment.

Start the gateway with a bootstrap platform admin key, then create environments and keys through the API:

ZEROLOCK_ADMIN_KEY=change-me zerolock serve --auth-required   # or ZEROLOCK_AUTH_REQUIRED=true
 
curl -X POST localhost:8080/admin/environments -H 'x-api-key: change-me' \
  -H 'content-type: application/json' \
  -d '{"id":"staging","name":"Staging","limits":{"max_shards":100,"max_operations_per_day":100000}}'
 
curl -X POST localhost:8080/admin/keys -H 'x-api-key: change-me' \
  -H 'content-type: application/json' \
  -d '{"environment":"staging","name":"planner","permissions":["shard:read","shard:write","shard:create"],"agent_id":"planner-1"}'

The key (zl_...) is returned once and only its SHA-256 hash is stored. agent_id binds the key to one agent identity. Revoke with DELETE /admin/keys/:id; a revoked key stops working within 5 seconds.

API Keys File

{
  "sk_live_abc123...": {
    "name": "Production Agent Pool",
    "permissions": ["shard:read", "shard:write", "shard:create"],
    "rateLimit": 5000
  },
  "sk_live_xyz789...": {
    "name": "Admin Dashboard",
    "permissions": ["admin"],
    "expiresAt": "2026-01-01T00:00:00Z"
  }
}

Permission Model

Security Hardening Checklist

• Principle of least privilege per key. An agent pool that only ever reads and commits shards should hold shard:read + shard:write, not admin. Reserve admin keys for genuinely administrative tooling (dashboards, migration scripts), and keep as few of them in circulation as possible.
• Set expiresAt on every key you can. A key with no expiration is a key someone has to remember to rotate manually; one with expiresAt fails safe.
• Rotate keys on a schedule, not just on suspected compromise. Generate the replacement key, roll it out to callers, then remove the old key from the keys file — the file can hold both during the transition window.
• Never commit config/api-keys.json. Treat it exactly like any other secrets file; if your deployment tooling supports mounting secrets separately from application config (Docker secrets, Kubernetes Secrets, a vault), prefer that over baking the file into an image.
• Use approval:* sparingly. A key that can approve or reject Layer 4 approvals is, functionally, a key that can authorize irreversible operations — hold it to the same standard as admin.

An admin-permissioned key can also hit POST /admin/auth/reload (re-read the keys file from disk without restarting the gateway — the fastest way to roll out a key rotation) and GET /admin/auth/keys (list configured keys by name/permissions/expiry; never returns the key values themselves, only their first 8 characters). Both bypass rate limiting the same way /health and /metrics do — see API Reference — Authentication.

Rate Limiting

Default Limits

• 1000 requests per minute per API key
• Per-IP fallback when no API key provided

Custom Limits

Set per-key limits in API keys file:

{
  "sk_live_limited": {
    "name": "Limited Key",
    "permissions": ["shard:read"],
    "rateLimit": 100
  }
}

Size per-key rateLimit values against actual expected traffic rather than leaving every key at the global default — a key used by a single low-frequency admin tool benefits from a tighter limit (it caps blast radius if the key leaks), while a key shared across a large agent pool may need a higher one than the 1000/minute default to avoid spurious 429s under normal load.

Monitoring

Health Check

curl http://localhost:8080/health

Response:

{
  "status": "ok",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "websocket": {
    "connections": 5,
    "subscriptions": 12
  }
}

Wire this endpoint into your orchestrator's liveness and readiness probes. A gateway that responds here but whose websocket.connections count is unexpectedly zero when agents should be connected is a useful early signal of a networking or proxy misconfiguration (see Load Balancer Configuration — WebSocket upgrade headers are the most common culprit).

Prometheus Metrics

curl http://localhost:8080/metrics

Key metrics:

• zerolock_http_requests_total: Request counts
• zerolock_shard_commits_total: Commit counts by status
• zerolock_ws_connections: Active WebSocket connections
• zerolock_ws_cost_savings: Message savings ratio

Grafana Dashboard

1. Open http://localhost:3000
2. Login: admin / zerolock
3. Add Prometheus data source: http://prometheus:9090
4. Import ZeroLock dashboard

Change the default Grafana admin credentials immediately in any environment reachable outside your own machine — see the note under Docker Deployment.

Scaling

Horizontal Scaling

For high availability, run multiple gateway instances behind a load balancer:

# docker-compose.scale.yml
services:
  gateway:
    deploy:
      replicas: 3

Important: Use PostgreSQL (not SQLite) for multi-instance deployments. SQLite's storage model is a single local file with a single-writer assumption — it has no mechanism for two separate gateway processes to coordinate writes to the same file safely, so running multiple SQLite-backed instances behind a load balancer will produce exactly the kind of lost-update and staleness bugs ZeroLock exists to prevent, just one layer lower in the stack.

Load Balancer Configuration

upstream zerolock {
    least_conn;
    server gateway1:8080;
    server gateway2:8080;
    server gateway3:8080;
}
 
server {
    listen 80;
 
    location / {
        proxy_pass http://zerolock;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The Upgrade/Connection "upgrade" headers are not optional — omitting them is the most common cause of WebSocket connections silently falling back to failing or polling behavior behind a reverse proxy. If Layer 2 invalidation or Layer 3 conflict notifications appear to never fire in a load-balanced deployment, check this configuration first.

Rolling Restarts

When restarting gateway instances (for a deploy, a config change, or a Postgres failover), restart one instance at a time and let the load balancer's own health check against /health drain traffic away from an instance before it goes down and back to it once it's healthy again — rather than restarting all replicas simultaneously, which briefly drops every agent's connection at once and forces a synchronized reconnect/re-subscribe storm.

Capacity Planning

Use Architecture — Scalability and the coordination-overhead data in Benchmarking §2.2.A as your starting reference point, not a guarantee: task-claim attempts on a single heavily-contended decision shard were observed to scale non-linearly (roughly 18 attempts at 5 concurrent agents, ~2,000 at 50, ~6,700 at 80). If your workload concentrates writes on a small number of hot decision shards, plan capacity — and consider whether that state should be a decision shard at all versus an additive one — around that curve rather than around raw agent count alone.

Backup & Recovery

PostgreSQL Backup

# Full backup
pg_dump -h host -U zerolock -d zerolock > backup.sql
 
# Restore
psql -h host -U zerolock -d zerolock < backup.sql

SQLite Backup

# Online backup
sqlite3 zerolock.db ".backup backup.db"

Point-in-Time Recovery

The audit log enables reconstruction of state at any point:

SELECT * FROM audit_log
WHERE timestamp <= '2025-01-15T10:00:00Z'
ORDER BY timestamp DESC;

For production Postgres deployments, pair scheduled pg_dump snapshots with continuous WAL archiving if your recovery point objective is tighter than your backup interval — a nightly pg_dump alone means up to 24 hours of potential data loss in a worst-case failure.

Troubleshooting

Connection Issues

# Check gateway status
curl http://localhost:8080/health
 
# Test PostgreSQL connection
psql $ZEROLOCK_POSTGRES_URL -c "SELECT 1"
 
# Check logs
docker-compose logs gateway

High Conflict Rates

If seeing many 409 conflicts:

1. Check if multiple agents are racing on same shards
2. Consider using additive (CRDT) shards for mergeable data
3. Implement exponential backoff in clients

A 409 rate that's high but stable is often expected behavior for a genuinely hot decision shard, not a bug — the gateway is doing its job of rejecting stale writes. What's worth investigating is a 409 rate that's climbing over time relative to overall commit volume, which usually means either more agents were added without a corresponding change to how they coordinate, or a shard that should have been modeled as additive is being treated as a decision shard.

Memory Usage

For SQLite, ensure WAL checkpoint is running:

PRAGMA wal_checkpoint(TRUNCATE);

For PostgreSQL, monitor connection pool:

curl http://localhost:8080/metrics | grep pool

WebSocket Connections Dropping Under a Load Balancer

If zerolock_ws_connections stays near zero despite agents attempting to connect, revisit Load Balancer Configuration first — a missing Upgrade/Connection "upgrade" header pair is the most common root cause, followed by a load balancer or proxy timeout shorter than the WebSocket client's idle/reconnect interval.