Multi-replica & SaaS residual hardening
Multi-replica & SaaS residual hardening
Section titled “Multi-replica & SaaS residual hardening”Kazma’s default profile remains single-operator / single-node. The work below closes residual gaps for multi-user and multi-replica where the architecture allows — it does not claim full multi-tenant SaaS.
Post-HITL host power
Section titled “Post-HITL host power”After a human approves a danger tool, the host is still powerful by design. Hardening (2026-07 residual batch):
| Control | Behavior |
|---|---|
| Allowlist binaries | No interpreters; prod drops ps/kazma |
| Archive tools | tar/zip off in production strict mode unless KAZMA_SHELL_ALLOW_ARCHIVE=1 |
| Restricted PATH | KAZMA_SHELL_STRICT defaults on in production — system/build dirs only |
| Binary resolve | shutil.which under restricted PATH; no /tmp/evil/git |
| Scrubbed env | No parent API keys |
| Git denylist | push/force/credential/reset/rebase/clean -fd blocked |
| Workspace paths | Absolute args must stay inside workspace root |
| Docker code_exec | Force in production / KAZMA_CODE_EXEC_DOCKER=force |
Not removed: approved shell_exec / file_write / python_exec still
mutate the host for the trusted operator. Treat YOLO and long HITL grants
as elevated.
Document Intelligence (jobs + metadata)
Section titled “Document Intelligence (jobs + metadata)”| Layer | Multi-replica status |
|---|---|
| Job claim/lease | Yes on Postgres (documents/jobs_pg.py) on a Postgres install (KAZMA_DB_BACKEND=postgres or a postgresql:// KAZMA_DATABASE_URL), unless KAZMA_DOCUMENTS_JOBS_BACKEND=sqlite |
| Intake admission | Yes with the Postgres job queue: durable reservations, an intake lease and a takeover fence, so a stalled replica’s intake is taken over and can never publish (documents/admission.py) |
| Metadata CRUD | Yes when KAZMA_DOCUMENTS_METADATA_BACKEND=postgres, or auto/unset on a Postgres install (repository_pg.py); the tenant storage quota is enforced in the publishing transaction |
| Content blobs | Shared filesystem/volume on documents.storage_root required |
| Upload spool slots | Per process (documents.capacity.max_concurrent_uploads) |
| GC | Backend-agnostic — retention._mark dispatches to repository.gc_mark on both SQLite and Postgres |
These are capabilities of the document backends, not a qualified deployment:
scripts/certify_documents.py reports postgres_multi_replica and
metadata_multi_replica as NOT RUN, and the runtime as a whole keeps one
active owner (below). Always check GET /api/documents/ops/readiness. See Document processing ops and Document Intelligence.
Multi-tenant isolation
Section titled “Multi-tenant isolation”| Layer | Status |
|---|---|
| Tenant ContextVar | tenant_context + middleware |
Client X-Tenant-ID | Ignored when production or multi-user; JWT / principal only |
| Opaque sessions | Forced when multi-user |
| Document ACL | Per-tenant document + principal ACL on repository |
| Vault secrets | Tenant column + ContextVar filter |
| Vector memory | Metadata tenant_id filter on query |
| Swarm tasks | metadata.tenant_id stamped on dispatch |
| Knowledge / cron / MCP | Still largely global (not per-tenant productized) |
Env: KAZMA_MULTI_USER=1, KAZMA_PRODUCTION=1, KAZMA_DATABASE_URL,
KAZMA_VAULT_KEY, KAZMA_JWT_SECRET (if JWT tenancy).
Multi-replica process state
Section titled “Multi-replica process state”| State | Was | Now |
|---|---|---|
| HITL bus approvals | In-process Event only | ConfigStore dual-write via swarm/shared_approvals.py + local Event |
| Swarm task results | TaskStore (SQLite/PG) | Unchanged — use Postgres for multi-node |
| Task list tenancy | Global | Auto metadata.tenant_id filter when multi-user/prod |
| Circuit breakers | Process memory | Shared via ConfigStore when multi-user/prod (KAZMA_SHARED_BREAKERS) |
| MCP servers (user-added) | Global ConfigStore | Tenant-scoped key tenant.<id>.mcp.servers |
| Cron job list | Global | tenant_id column; UI list filtered |
_active_tasks | Process memory | Still process-local; persist via TaskStore for history |
| KB crawl jobs | Process + ConfigStore | The crawl journal (web_acquire/crawl_jobs.py): one row per job, owners scoped to this install |
| Knowledge libraries | Global | Own knowledge.db; refused on an install with platform users (single-operator) |
| Swarm bus adapters | Process singleton | Fan-out multi-platform; still one process |
| Browser affinity | None | Cookie kazma-replica (KAZMA_REPLICA_ID, LB sticky) |
These shared subsystems do not make the whole runtime safe for concurrent active replicas. The supported HA preparation profile has one active owner, a dedicated Postgres ownership session and a complete fenced state volume. Sticky routing or networked SQLite storage cannot replace ownership and fencing. See Multi-Region & HA for setup and qualification drills.
Host code_exec sandbox
Section titled “Host code_exec sandbox”| Mode | Behavior |
|---|---|
| Production or multi-user | No host-local python_exec — Docker required |
KAZMA_CODE_EXEC_DOCKER=force | Same |
| Lab escape | KAZMA_CODE_EXEC_ALLOW_LOCAL=1 (not recommended) |
Discord / Slack vs Telegram
Section titled “Discord / Slack vs Telegram”| Feature | Telegram | Discord | Slack |
|---|---|---|---|
| Text chat | Full | Full | Full |
| Typing indicator | Full | Yes | Yes |
| Voice STT/TTS | Deep path | Same depth via voice_helpers + discord_stt / slack_stt | |
| Graph HITL buttons | telegram_keyboards | discord_keyboards | slack_blocks |
| Swarm HITL buttons | Inline keyboard | Components v2 | Block Kit |
| Shared callback IDs | platform_callbacks | same schemes | same schemes |
| Modular parse/send | telegram_parse/send | discord_parse/send | slack_parse/send |
| Shared multi-replica approvals | Yes | Yes | Yes |
| Message reactions | setMessageReaction | set_reaction API | set_reaction API |
Layout (2026-07 UX modules):
adapters/ platform_callbacks.py # hitl: / swarm_ / model_ schemes platform_keyboards.py # Discord components + Slack blocks telegram_{keyboards,callbacks,parse,send,stt}.py discord_{keyboards,callbacks,parse,send}.py slack_{blocks,callbacks,parse,send}.py voice_helpers.py