Disaster Recovery

Kazma Disaster Recovery Runbook

Version: 0.6.x / Phase 4.5
Audience: Operators deploying single-node or multi-replica Kazma
Related: kazma_core.backup.restore, kazma_core.backup.restore_drill, SECURITY.md

1. What to protect

Asset	Location	Criticality
Settings + secrets pointers	`kazma-data/settings.db` (+ vault)	Critical
Vault encryption key	`KAZMA_VAULT_KEY` env / `.env`	Critical — without it secrets are unreadable
Shared operator secret	`KAZMA_SECRET`	Critical
LangGraph checkpoints	`kazma-data/checkpoints.db`	High — conversation continuity
Chat sessions	`kazma-data/chat_sessions.db` (or configured path)	High
Swarm tasks	`kazma-data/swarm_tasks.db`	Medium–High
Memory / vectors / graph	`kazma-data/vector_memory/`, `memory.db`, `vector.db`, `knowledge_graph.db`	Medium
Forget journal	`kazma-data/memory_forget_journal.db`	High — every forget, kept outside the memory database so a restore of it does not undo them
Cron jobs	`kazma-data/cron.db`	Medium
Document Intelligence metadata	`{documents.storage_root}/documents.db` on SQLite (catalog, ACL, audit, and the jobs and intake reservations when they are on SQLite); Postgres tables when the install uses Postgres, unless `KAZMA_DOCUMENTS_METADATA_BACKEND=sqlite`	High — library + jobs
Document content-addressed blobs	`{documents.storage_root}/` (`quarantine`/`originals`/`artifacts` + manifests)	Critical — irrecoverable without backup. The 6-hour document backup captures metadata and blobs as one point in time: `kazma-data/backups/document-store-` when metadata and jobs are on SQLite (restore: copy into an empty storage root), `document-pg-` when both are on Postgres (restore with `scripts/document_pg_backup.py`; after the shared Postgres dump, with `--replace-document-tables`). SQLite metadata beside Postgres jobs is snapshotted like a SQLite store, its job queue being in the shared Postgres dump; Postgres metadata beside SQLite jobs (a fallback) is refused by name. The next start re-publishes indexed documents to their knowledge libraries (see Document processing, which records the measured restore drills)
Graph memory	Neo4j (`bolt://…`) — Docker volume, NOT in the data dir	Medium–High
MCP + connector config	`kazma.yaml` at the install root, NOT in the data dir	Critical — a restore without it boots with no tools
Opaque web sessions	ConfigStore / Postgres	Low (users re-login)

Out of band (never only on the app disk):

KAZMA_SECRET
KAZMA_VAULT_KEY
Provider API keys if not vaulted
OIDC client secret
Postgres credentials (KAZMA_DATABASE_URL)

2. Backup procedure (single-node SQLite)

Frequency

Environment	RPO target	Action
Lab / personal	Best effort	Daily or before upgrades
Production single-node	≤ 24h	Daily automated zip + offsite copy
Production multi-replica	≤ 1h	Continuous Postgres backups + the 6-hourly app snapshot

How it works now (restic, since 2026-08-29)

Backups are automatic. Nothing needs running by hand.

Each cycle produces a staging generation under kazma-data/backups/universal/<epoch>/ containing every SQLite database (WAL-safe via the Online Backup API), assets, .env, kazma.yaml, research/, and a JSONL export of the Neo4j graph. Postgres is dumped separately to kazma-data/backups/pg/ with pg_dump -Fc.

That generation is then snapshotted into two independent restic repositories:

Repository	Location	Credential
Local	`kazma-data/backups/restic`	passphrase at `<install>/.kazma/restic.pass` (see below)
Offsite (recommended)	`s3:https://<account>.r2.cloudflarestorage.com/<bucket>` (or B2)	append-only host key (`PutObject`/`GetObject`/`ListBucket` + `DeleteObject` on `locks/*` only)
Offsite (legacy)	`rclone:<remote>/restic`	rclone OAuth — do not use Google Drive / a service account. Service accounts have no Drive quota; `rclone:` write probes can look healthy while every upload 403s.

An rclone destination must name an existing configured remote, such as backup:restic. For a restic repository use rclone:backup:restic. Inline backend specifications (:s3:…), option-like names and control characters are refused before executing rclone. Paths may contain spaces and Arabic text. The command places destinations after --.

Prefer S3-native restic (Cloudflare R2 or Backblaze B2). The host key must not be able to restic forget --prune. Keep a full-access prune key off this machine. remote_writable() probes s3: with a real SigV4 PUT+DELETE under locks/ — an rclone: remote that can list but cannot write is no longer treated as healthy.

The two repositories use different credentials on purpose: the offsite path must not fail when a connector token is revoked. Drive+rclone did exactly that on 2026-08-27, and 29 consecutive backups went local-only without anyone noticing.

Operator still to do (not code): create the R2/B2 bucket and the append-only host key, set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, point backups.restic.remote at s3:…, restic init, then run python -m kazma_core.backup.restore_drill against a restored generation before retiring Drive. Escrow the restic passphrase off this host.

Why restic rather than the zip archives this runbook used to describe: deduplication, encryption, and a restore that upstream tests harder than we can. On this install four generations cost 131 MB where the zip scheme cost 3.8 GB, and each additional generation adds roughly 2 MB.

Retention

Time-based, not count-based: 24 hourly, 30 daily, 8 weekly, 12 monthly, applied by the restic_maintenance task, which also clears stale locks and runs restic check. “Keep the last 30” is thirty days or thirty hours depending on how often the loop ran — not a recovery guarantee.

The policy is applied per kind of backup (forget --group-by host,tags: universal, pg, legacy, probe). restic’s default groups by host + paths, and every universal backup is a new backups/universal/<epoch> directory — so each snapshot was its own group of one and, until 2026-09-23, retention never deleted anything (199 snapshots, 199 groups). Thirty dailies, not seven, is the operator’s choice from the same day: 69 chat memories emptied between 08-30 and 09-21 were recoverable only from a three-week-old snapshot.

A clean run logs [restic] maintenance ok: <local|remote> (forget --prune, check); before that line existed, a maintenance that never ran was indistinguishable from one that always succeeded.

On-disk staging keeps only 2 generations; the history lives in restic.

The cloud zip copy has its own count. Settings → Backup can also send each generation, zipped and without .env, to a cloud provider’s kazma-backups folder (Google Drive, OneDrive, WebDAV, FTP, S3, or an rclone remote). Since 2026-11-03 Kazma keeps the newest backups.offsite.keep copies there (Settings → Backup → Keep cloud copies; default 7, 0 keeps every copy) and deletes the rest after each upload that succeeds, oldest first — for good, never to a trash. Nothing is deleted from a listing that failed, is not whole, or does not hold the copy just uploaded; a copy that could not be deleted raises the alert backup.offsite_prune_failed and is tried again at the next backup. Before this every copy stayed: a 344 MB zip every six hours on the live install, until its owner deleted them by hand.

The passphrase

<install>/.kazma/restic.pass (a non-empty legacy ~/.kazma/restic.pass still wins) decrypts every snapshot, local and offsite. It is deliberately NOT KAZMA_SECRET — the defect being fixed was the vault key travelling with the data it protects.

Keep a copy off this machine. Encryption moves the single point of failure from the archive to the key. Without it, every backup is permanently unreadable.

Verifying without restoring

python -m kazma_core.backup.restore_drill

Exit codes distinguish evidence: 0 / PASS means every applicable check passed; 1 / FAIL means a check found a failure; 2 / UNVERIFIED means required checks could not complete, without evidence of corruption. Missing pg_restore, unavailable readback, and a repository locked by a live backup cannot certify recovery. Unverified runs issue a warning through the existing ops channel; failed checks issue a critical alert. Install the missing client tools or resolve the reported dependency and rerun the read-only drill.

It also runs on its own, daily, five minutes after boot and then once every 24 hours — counted from the last completed run, stored in the config store, so a host that restarts often cannot keep resetting the clock. (It used to be weekly, and the loop slept a full interval before its first pass, which on a frequently-restarting host meant it never ran at all. Three days of live logs: 34 scheduler starts, zero results.)

What the daily pass checks. Integrity alone proves the bytes are not corrupt; it does not prove you could get your data back. So, in order of consequence:

Check	What it would catch
`vault:decrypt` — the backup’s own `.env` key opens the backup’s own vault	a stale or rotated `KAZMA_VAULT_KEY`. The file is present and exactly the right size either way, and every encrypted secret behind a wrong key is lost
completeness against the manifest’s own `databases.items`	a database the backup claimed to take and did not
`PRAGMA integrity_check` on every SQLite file in a scratch copy	corruption in the copied pages
zero tables is a failure	an empty file, which passes `integrity_check` happily
`pg_restore --list` on the Postgres archive	a truncated or unreadable dump header

What the weekly pass adds, reading the bytes rather than the headers:

Check	What it would catch
the Postgres dump streamed through `pg_restore --file=-`	a dump whose data sections are damaged behind an intact TOC. Every block is read — but nothing is loaded, so this proves the archive reads, not that it restores
the dump restored into a scratch database, checked, dropped	a dump that reads back but does not restore: a schema that will not recreate, rows that will not load, Kazma’s tables missing, an empty `kazma_settings`. On by default with Postgres — see below
`restic check --read-data-subset=5%`	bit rot inside the repository’s packs
a `HEAD` read-back of the offsite object, comparing stored size to uploaded size	a truncated upload. A short object and a complete one look identical from the sending side, which returns 200 for both

The restore rehearsal (kazma_core.backup.restore_rehearsal) is the only backup check that writes to the database server. It is on by default with Postgres (since 2026-09-27; it was opt-in, so “the dump restores” had only ever been inferred from “the dump reads”). Set backups.pg.restore_rehearsal to false, or KAZMA_PG_RESTORE_REHEARSAL=0, to turn it off (=1 turns it on over the setting). The weekly pass creates kazma_restore_rehearsal_<epoch> on the same server, restores the newest dump into it with pg_restore, checks that Kazma’s tables came back and kazma_settings is not empty, and drops it. Every CREATE/DROP re-checks the name against that exact pattern and refuses the live database’s own name; a crash between the two leaves a scratch database the next run removes (older than a day, same pattern — nothing else). The database user needs CREATEDB (ALTER ROLE <user> CREATEDB); without it the check reports UNVERIFIED with that grant, never a failed backup. Size the server for a second copy of the Kazma tables while it runs.

To see it work without waiting for the weekly pass, run the deep drill now (it includes the rehearsal when it is on):

& '.venv\Scripts\python.exe' -m kazma_core.backup.restore_drill --deep

A drill that has never run proves nothing. Do not describe backups as verified until a drill result appears in the log — the presence of a scheduler in the code is not evidence. The resilience manifest marks this mechanism proven_in_production=False for exactly that reason, and it goes back to True when a live result exists.

The legacy scripts

scripts/backup_kazma.py and scripts/restore_kazma.py still work and still produce zips. They are no longer the primary path and do not include the graph export or kazma.yaml.

3. Restore procedure (single-node)

RTO target: < 1 hour for single-node with known secrets. Rehearsed: 2026-08-29, from both the local and the offsite repository.

Step 1 — see what you can restore to

python -m kazma_core.backup.restore --list

Lists every recoverable generation with its snapshot and the Postgres dump that accompanies it.

Never use restic restore latest. It selects the newest snapshot by the time the SNAPSHOT was taken, which is not the newest DATA. Generations ingested out of order — a bulk import, a re-upload — carry recent timestamps and old content, so latest can hand you a backup missing kazma.yaml and the graph export while looking like a clean success. The command above selects by generation; use it rather than restic directly.

Step 2 — restore the files

python -m kazma_core.backup.restore --target D:\fresh-kazma

Add --generation <epoch> to pick a specific point, or --repo <path> to restore from the offsite repository when the machine is gone.

The target must be empty — a restore over an existing tree interleaves two states into one that looks plausible and is neither.

You get an install layout: .env, kazma.yaml, kazma-data/, neo4j_graph.jsonl, and the paired Postgres dump under pg/. The backup’s dbs/ and assets/ are laid out AS the data dir: kazma-data/settings.db, kazma-data/vault.db, kazma-data/attachments/ and so on, where Kazma reads them. A database that sat in an asset folder is taken from the consistent copy in dbs/, never the raw one, with no -wal/-shm beside it. Every restored SQLite database is integrity-checked (read immutable, so the check adds no file), and the last step asks Kazma itself, pointed at the restored kazma-data/, where each store lives: a store the backup lists that is not there fails the restore. Until 2026-11-02 the restore copied dbs/ and assets/ in as two folders, one level below every path Kazma reads, and reported success (FA-268).

Step 3 — load the databases

Not automatic, and deliberately so: these overwrite live data. The exact commands are printed at the end of step 2 with paths filled in.

# Postgres
pg_restore --clean --if-exists -d "$env:KAZMA_DATABASE_URL" "D:\fresh-kazma\pg\…\pg_shared_….dump"

# Graph memory, into an EMPTY Neo4j (it refuses a populated one)
python -c "from kazma_core.backup.neo4j_backup import restore_graph; print(restore_graph(r'D:\fresh-kazma\neo4j_graph.jsonl'))"

Step 4 — point Kazma at it and start

For a restored pre-V2 generation, keep its runtime stopped and take a copy before conversion. Use an explicit existing data directory:

python -m kazma_core.memory.backfill_v2 --data-dir D:\fresh-kazma\kazma-data --dry-run
python -m kazma_core.memory.backfill_v2 --data-dir D:\fresh-kazma\kazma-data --apply

The CLI converts legacy memory.db and knowledge_graph.db into memory_state.db, without provider calls. It preserves episode text, tenant IDs, timestamps and available embeddings, converts graph entities/fact edges, and skips structural edges. Stable IDs make reruns idempotent; reported insert counts include only new rows. Inspect the returned seen/inserted/skipped counts before starting the restored runtime. All store paths use this directory even when the shell inherited live-install path overrides. This does not convert a Postgres generation or qualify semantic belief extraction.

Copy .env, kazma.yaml and kazma-data/ into the install root, or point KAZMA_DATA_DIR at the restored tree. Then:

GET /health/ready → ready, all checks ok
Forgets made after the backup stay forgotten. Restoring an older memory_state.db brings back the turns forgotten since; the forget journal (memory_forget_journal.db, beside it, NOT restored from the older copy) names them, and Kazma applies them again at start and every 15 minutes (“forget convergence”). Keep the journal you have when you restore the memory database; the drill reports how many forgets a backup would replay.
Memory catches up to the chat store. Turn reconcile’s cursor lives in memory_state.db, so a restored copy reads every chat changed since its backup again and writes the turns since then back as memories. Restoring from the Memory page (POST /api/system/memory/restore) applies the forget journal and runs that catch-up at once; a restore by hand gets both at the next start. With Postgres as memory’s primary store, the next maintenance pass sees a mirror that went back and compares it in full. Measured RPO and RTO: the guide’s “Restore drills” (Memory and RAG).
Login works
A prior chat session appears
Settings → providers still configured (the vault unlocks — this proves .env came back)
Tools are listed (this proves kazma.yaml came back)

Failure modes

Symptom	Likely cause	Fix
Settings empty / keys missing	Wrong or missing `KAZMA_VAULT_KEY`	Restore vault key from password manager
401 everywhere	Wrong `KAZMA_SECRET`	Restore secret; clear old cookies

Repository password rotation

kazma_core.backup.restic_repo.rotate_password(repos, old, new, persist=...) is the maintained operator Python API. Pass the active repositories and a durable persistence callback for the configured passphrase. It adds and verifies the new key on every repository, persists it, then removes old keys; a failed add, verification or persistence refuses revocation. Read passphrases through a password manager or getpass, never command arguments or printed output. Back up the stored passphrase, schedule rotation while backups are idle, and verify repository access and a restore drill afterwards. | SQLite “database is locked” | Process still running | Kill uvicorn/python; retry | | Vector search empty | Vector volume not in backup path | Restore vector_memory / Chroma path; re-index if needed |

4. Postgres and the complete state volume

When KAZMA_DATABASE_URL=postgresql://… is set:

Component	Backend	Notes
Settings / sessions / platform users schema	Postgres (`kazma_core.db`)	These backends do not move every runtime store to Postgres
SQLite stores, Chroma and persistent files	Complete fenced state volume	Preserve alongside Postgres; do not share SQLite files over NFS
Checkpoints	Prefer Postgres checkpointer when configured	See env below

The HA preparation profile permits one active runtime owner. Restore Postgres and the complete state volume together; receipts and approval decisions remain on that volume. See Multi-Region & HA for ownership and drills.

Postgres environment

KAZMA_DATABASE_URL=postgresql://kazma:…@db:5432/kazma
KAZMA_DB_BACKEND=postgres
KAZMA_PG_POOL_MIN=1
KAZMA_PG_POOL_MAX=10
KAZMA_PRODUCTION=1
KAZMA_VAULT_KEY=…
KAZMA_SECRET=…          # or IdP-only with multi-user
KAZMA_PUBLIC_URL=https://kazma.example.com

Optional packages:

pip install -e ".[postgres]"     # psycopg[binary,pool]>=3.3.6, langgraph-checkpoint-postgres>=3.1.3

Postgres backup

Use your platform standard:

Managed: enable automated backups + PITR (RDS, Cloud SQL, Azure).

Self-hosted:

pg_dump -Fc "$KAZMA_DATABASE_URL" -f kazma-$(date -u +%Y%m%d).dump

Restore:

pg_restore -d "$KAZMA_DATABASE_URL" --clean --if-exists kazma-YYYYMMDD.dump

Rule: Never run multiple replicas against a shared SQLite file.

A whole-database dump also holds the document tables, but not as of the moment the document content files were copied. Restore a Postgres document store from its coordinated backup (scripts/document_pg_backup.py, see Document processing → Backup), in this order (drilled, tests/test_document_pg_restore_order.py): the whole-database dump first, then the coordinated backup with --replace-document-tables, which drops the dump’s older document tables once the backup has verified and the storage root is found empty. Without the option the coordinated restore refuses the database; the other order lets the dump’s --clean put older document rows back over the coordinated copy.

Compose example: docker-compose.postgres.yml.

5. Multi-user / IdP (Phase 4.4)

Mode	How
Single operator	`KAZMA_SECRET` + opaque session (default)
Local multi-user	Create users via `platform.users` / `create_local_user()`; login with username+password
OIDC	`KAZMA_OIDC_ISSUER`, `CLIENT_ID`, `CLIENT_SECRET`, `KAZMA_PUBLIC_URL` → `/api/auth/oidc/start`

Roles: viewer < operator < admin (see platform_rbac.py).

After DR restore, re-test:

Admin login
Operator can chat/approve
Viewer cannot hit /api/settings

6. Drill checklist (run quarterly)

Backups run themselves; the drill exists to prove the RESTORE still works, which is the half that rots unnoticed.

7. Incident contacts

Interrupted tools and approval resumes

After a crash, a recorded approval proves only the human’s decision. Startup preserves that decision and actor, marks an interrupted resume as execution unconfirmed, and withholds the old approval buttons. A question that was still unanswered keeps its checkpoint and approval card.

Graph mutating tools reserve a receipt before dispatch in tool_effects.db. The receipt identity binds the tenant, thread, turn, iteration, tool call and tool; its request hash also binds arguments and the workspace. Completed receipts supply their saved result on checkpoint replay. Started receipts hold an unknown outcome and stop the turn without synthesis or another dispatch. A saved error with result_uncertain also holds the turn: completed means the receipt was committed, not that the action succeeded. The local registry invokes mutating and unknown tools once; only classified reads receive automatic retries. A post-invocation error is uncertain because an action may have completed partially. A refusal a tool makes before it writes, sends or starts anything (an argument, a policy, an account not connected, a refused URL, a browser the host cannot launch) is an ordinary error, not an uncertain one: the model may correct the call, and the turn goes on. So is a failure whose effect the tool names exactly and that is safe to repeat (generate_pdf’s check that the text reached the PDF). A send, a post or a payment is never treated that way. Reads are not cached. Failed MCP mutator dispatches also carry uncertainty, including when an output hook rewrites the error. Direct IDE tool calls and in-process swarm tool loops also write receipts. An uncertain swarm tool stops without a synthesis call. An unsuccessful worker that invoked a local/MCP mutator cannot retry or fall back to a replacement worker. Custom workers’ effects outside these shared executors and legacy graph checkpoints without turn identity remain uncovered.

IDE mutation requests support Idempotency-Key. Use one new key for each deliberate operation and retain it if the response is lost. A repeated request with the same principal, tenant, payload and workspace returns the saved response; changed requests and unresolved effects return 409 without another dispatch or approval. Authentication and operator role are checked before a saved response is returned. Rate admission precedes receipt admission and is applied once per request, including saved-response reads. HTTP 429 does not admit an effect; the same operation key can be submitted again after the limiter window expires. The web IDE retains unresolved operation IDs across reload within the same browser tab; session storage holds only a payload digest and ID. Clients that omit the header get a generated response key, which cannot rescue a lost response they never received. Do not start a new keyed operation merely to bypass an uncertain result. Direct non-HTTP IDE calls have per-call receipts, not an automatic retry protocol.

Inspect metadata from the install’s environment with its thread ID:

python -m kazma_core.agent.effect_journal --thread THREAD_ID

The command shows receipt identities, tool names, state and timestamps. It does not print arguments or returned content, change a receipt, or replay anything. Inspect the actual target and its external audit trail before starting a new request. Do not delete or reset an unknown receipt to force a retry. Keep receipts, checkpoints and approval decisions together during backup and restore; restoring an older receipt store can repeat newer effects.

External APIs are not atomic with this local store. Whole-agent Temporal activities still have one attempt; automatic activity retries remain unqualified. The receipt uses SQLite WAL with full commit durability and must stay on the complete fenced state volume.

Event	Action
Suspected compromise	Rotate `KAZMA_SECRET`, `KAZMA_VAULT_KEY`, provider keys; invalidate sessions; restore from last known-good backup if needed
Data corruption	Stop writers → restore from last good zip/dump → smoke test
Lost vault key	Unrecoverable for vaulted secrets — restore key from offline store or re-enter secrets

Maintain this runbook with every production architecture change.