Troubleshooting

Practical fixes for the issues you will actually hit: provider limits, Windows/Docker specifics, SQLite concurrency, Arabic tokenization edge cases, and the known codebase gaps the audit uncovered. Every item is source-referenced.

Multi-path bugs (same feature, several code paths): start with the Diagnosis map — “symptom → first files” and “X is related to Y” tables. Then return here for operational workarounds.

---

1. Provider & LLM issues

1.1 “Function not found” / 404 on tool calls (NVIDIA NIM)

Cause: Some providers (notably NVIDIA NIM) reject tool definitions.

What Kazma does: llm_provider.py:285-300 detects status_code == 404 and "function" in detail (and 400/422 with tool/function language) and retries once without tools. The caller still gets a text response.

Workaround if it persists: Disable tools for that provider, or route through an OpenAI-compatible proxy that translates tool definitions. Do not remove the fallback branch — it’s a documented invariant.

1.1b Tool-schema 400 after enabling KAZMA_STRICT_TOOLS

Cause: OpenAI function.strict: true (all properties required, optionals as T | null) is rejected by many local servers, Anthropic Messages, and Gemini.

Fix: Leave KAZMA_STRICT_TOOLS unset. Closed schemas (additionalProperties: false) stay on. Invented extra arguments are still dropped at execute(). Use response_format only on calls that need a JSON reply, not on every chat turn.

1.1d Agent will not write files / run shell

Cause: Plan mode is on (/plan on or the Plan pill). Write/exec tools are removed from the schema until you approve.

Fix: /plan go (or Proceed) to execute the plan, or /plan off to leave without executing. KAZMA_PLAN_MODE=0 disables the feature.

1.1e Plan checklist drawn, then no reply (memory save / any tool turn)

Cause: The supervisor asks the model to open tool turns with a plan fence for the workbench. Some providers (DeepSeek v4 flash) then (1) emit the fence and stop with no `tool_calls`, or (2) glue the real answer onto the closing ticks (` Saved.`). CommonMark never closes that fence, so the answer is hidden inside a code block. If the SSE client also drops after the plan tokens, the tab shows only the checklist.

What Kazma does (2026-08-26): plan_fence.py is the SoT. Supervisor auto-continues a plan-only hop once (not in /plan mode). respond_node + SSE/WS done.content persist the un-glued payload. The chat bubble strips the fence and always replace-paints done.content.

Fix if you still see it on an old process: refresh the tab (the reply is in the session). Restart the server to pick up this code. /plan go only if the Plan pill is on.

1.1c Every tool returns [hook] blocked

Cause: A PreToolUse command or in-process hook is denying calls (agent.hooks.pre_tool, or a skill that called register_pre_tool_hook).

Fix: Set KAZMA_TOOL_HOOKS=0 and restart to disable all hooks. Then fix or empty agent.hooks.pre_tool. Hooks are not HITL — they cannot auto-approve; a deny is an extra restriction.

1.2 Provider/model mismatch (wrong endpoint)

Symptom: Requests go to the wrong API endpoint after switching models.

Cause: Changing the model without switching the provider.

Fix: Always use set_active_model() (it auto-switches the provider via find_provider_for_model()) or set_active_provider() — never set one without the other. get_client() auto-corrects (model_registry.py:275-303) but only persists the provider change when model is None.

Verify:

kazma status          # shows active provider/model

1.3 Anthropic native path

Cause (historical): the generic OpenAI-compatible client always sent Authorization: Bearer.

Now: Anthropic traffic goes through AnthropicProvider (/messages, x-api-key). Do not send Anthropic-native models through the generic LLMProvider — get_client() / find_provider_for_model() must keep provider and model paired (AGENTS.md §1).

1.4 Rate-limit (429) handling

Kazma does retry 429 with exponential backoff + Retry-After in LLMProvider.chat() and native Anthropic /messages. Exhausted 429 is LLMError(transient=True, kind=rate_limit_exhausted) — the supervisor retry loop honors transient. If you still burn the budget:

• Lower concurrency (BoundedConcurrency, default 5), or
• Put a rate-limiting proxy in front of the provider.

1.5 Cost breaker not halting runaway spend

Cause (historical): CostCircuitBreaker existed as a dataclass and was not on the live loop.

Now: it is instantiated per-agent (agent_runner.py) and driven on the supervisor: record_user_interaction() on each inbound turn, should_halt() before the LLM call, record_cost() after (graph_supervisor.py). Halt forces RESPOND with a ⚠️ budget message — it does not synthesize a fake answer. Tune KAZMA_MAX_COST / KAZMA_SILENCE_WINDOW or Settings → cost.

1.6 ModelRegistry “not initialized” (RuntimeError)

Symptom: RuntimeError: ModelRegistry not initialized. Call initialize_model_registry() first. when importing or calling get_model_registry().

Cause: initialize_model_registry(config_store) must run once before any consumer uses the registry. The singleton (model_registry.py) is created lazily.

Fix:

from kazma_core.config_store import get_config_store
from kazma_core.model_registry import initialize_model_registry

cs = get_config_store()
initialize_model_registry(cs)

In tests, use an autouse fixture that re-initializes per test:

@pytest.fixture(autouse=True)
def _init_model_registry(tmp_path):
    from kazma_core.config_store import ConfigStore
    from kazma_core.model_registry import initialize_model_registry, reset_model_registry
    db_path = str(tmp_path / "test_registry.db")
    cs = ConfigStore(db_path=db_path)   # isolated per-test store
    initialize_model_registry(cs)
    yield
    reset_model_registry()

Do not call reset_model_registry() mid-run unless you immediately re-initialize.

1.7 Failed model discovery / empty model list

Symptom: ModelRegistry.discover_models(provider_name) returns [], or GET /api/swarm/models is empty despite a configured provider.

Cause: the provider has no base_url, no api_key (when auth is required), is not enabled, doesn’t expose a /models-compatible endpoint, or a prior discovery failure is cached in _discovered_models.

Fix:

from kazma_core.model_registry import get_model_registry
registry = get_model_registry()
print(registry.get_provider("openai"))   # check base_url / api_key / enabled
print(registry.list_providers())
print(asyncio.run(registry.discover_models("openai")))  # explicit rediscovery

Discovery logs the failure reason — check startup logs.

1.8 Active profile returns empty / default values

Symptom: get_active_profile() returns \{"provider": "custom", "base_url": "", "model": "", "api_key": ""\} despite saved settings.

Cause: no active provider is set; the registry is falling back to legacy llm.* keys, which are also empty.

Fix:

registry.set_active_provider(
    provider="deepseek",
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-chat",
)
# or, within the active provider only:
registry.set_active_model("deepseek-reasoner")

Verify persistence via get_config_store().get("registry.active_provider") / ...active_model.

1.9 Settings not persisting after a model/provider switch

Symptom: you switch model/provider in the UI or CLI, but the next request uses the old value.

Cause: a different ConfigStore instance/db_path was used to save; the mutation bypassed the registry (so the cached client wasn’t invalidated); or a stale client is still in use.

Fix:

1. All code paths must share the singleton — get_config_store() (kazma-data/settings.db). Never construct a second ConfigStore() for the same DB (SQLite lock contention).
2. Confirm stored values: get_config_store().get_all() vs cs.export_yaml().
3. Mutate through the registry (set_active_provider / set_active_model), not via a bare cs.set() on unrelated keys — the registry owns the active profile and client-cache invalidation.
4. To revert a key to its YAML default: get_config_store().delete("registry.active_model"). To force a full reset: delete kazma-data/settings.db (loses all runtime settings) and restart.

---

2. Memory & RAG issues

Updated July 2026 (post-V2 cutover) — Kazma now uses the V2 cognitive engine (bi-temporal beliefs, PPR recall) as the single memory stack. The legacy ChromaDB / VectorMemory / 4-layer RRF stack was removed. Items below reflect the V2 architecture.

2.1 “My agent doesn’t seem to remember anything”

V2 memory works in two ways:

1. LLM tools — the model can call memory_search / memory_store at any time. Both route through V2: recall() for reads, mutate_belief() for writes.
2. Compaction injection — when the conversation hits 80% of the context window, the CompactionEngine automatically retrieves the top-5 relevant memories via V2 recall.search() and injects them into the fresh system message as ## Relevant Memories.

If memories still aren’t surfacing:

• V2 recall needs the shared embedder (get_embedder()). Verify pip install -e ".[rag]" (sentence-transformers + sqlite-vec). Without it, recall degrades to FTS5-only keyword match. On Postgres, also CREATE EXTENSION vector or dense search stays on sqlite-vec / empty (KAZMA_PGVECTOR=0 forces local).
• Check /api/memory/v2/health — if beliefs.active is 0, no beliefs have been extracted yet (post-turn extraction runs on a background thread + queue; give it a few turns).
• Check the dashboard V2 board (/api/memory/v2/graph) — confirms the belief graph has content.
• Instruct the model (via system_prompt) to proactively call memory_store for important facts.

2.2 V2 recall returns empty / embedder issues

Cause: the [rag] extra is not installed, or the embedder failed to load.

What happens: V2 recall() catches all failures and returns an empty RecallResult — tools return "No relevant memories found.", compaction gets []. The system never crashes; it just has no semantic recall (FTS5 keyword match may still surface hits).

Fix: pip install -e ".[rag]". The embedder (local MiniLM by default) loads on first use; check the boot log for [Memory] V2 embedder ready. If remote embeddings are configured (memory.embedding.provider: openai-compatible), verify the API key + base_url.

2.3 Embedding dimension mismatch (cosmetic)

kazma.yaml may declare storage.vector_dim: 1024, but the default model (BAAI/bge-m3) is 1024-d. This value is informational; V2’s vector_engine.py reads the actual dimension from the embedder at runtime. Don’t rely on the config value for dimension logic.

2.4 Previously-documented V1 bugs (historical)

These were fixed in the July 2026 memory overhaul and are now moot (the V1 code was removed in the V1→V2 cutover):

• distance() SQL function error → V2 uses vector_engine.py (sqlite-vec / numpy).
• _vec_available false-positive → V2 vector_engine.py smoke-tests vec_version().
• 4-layer adapter L1 dead / callers crash on tuple unpack → V1 adapter removed; V2 recall.search() returns a consistent list[dict] shape.

---

3. HITL / safety issues

3.1 Danger tools execute without approval

Check 1 — is the graph gate active? All production build sites pass hitl_config (see Security & Safety §2.2). If you wrote a custom build via create_supervisor_graph() without hitl_config, the gate is dormant.

Check 2 — is the tool on the right list? There are three lists:

• Graph path: kazma.yaml safety.hitl.require_approval_for.
• Swarm bus: _EXTENDED_DANGER (swarm/safety.py:23).
• MCP: pattern-based classify_mcp_tool. Adding to one does not add to the others.

Check 3 — is allow_headless_danger=True set? That disables fail-closed in headless/test mode. Ensure it’s False in production.

3.2 HITL pauses never resume

Cause: the resume endpoint is POST /api/approve/\{thread_id\} in routes_direct.py:454 (not app.py). It requires KAZMA_SECRET if set, and enforces ownership (403 on cross-user).

Fix: ensure the approving caller has the matching identity fields and the correct secret.

3.3 Pipeline tasks stay “paused” after restart

Cause: restore_paused_tasks() re-arms auto-reject timeouts on startup (checkpoint_manager.py:222-230). If the approval never arrives within checkpoint_timeout, the task is auto-rejected.

Fix: approve before the timeout, or raise checkpoint_timeout in task metadata.

---

4. SQLite concurrency

4.1 “database is locked”

All Kazma stores use PRAGMA busy_timeout=5000 + WAL. If you still see lock errors:

• One writer at a time. WAL allows concurrent readers but a single writer. Long write transactions block others for up to 5 s.
• Use batch_set() for multi-key writes (atomic, single transaction) rather than looping set().
• Always use get_config_store() — constructing ConfigStore() directly bypasses the singleton and can open a second connection.
• Don’t share a connection across processes. Kazma is single-process; if you fork workers, each gets its own connection and they’ll contend on the same DB file.

4.2 In-memory fallback silently active

If SQLite init fails, get_config_store() returns an _InMemoryStore (TTL eviction, 1-hour, 10k entries). Settings then don’t survive a restart. Check startup logs for SQLite init errors.

---

5. Windows specifics

5.1 PowerShell command chaining

Never use && or || in PowerShell. Use ; and check $LASTEXITCODE (per AGENTS):

& '.venv\Scripts\python.exe' -m pytest kazma-core/tests/ -v
if ($LASTEXITCODE -ne 0) { Write-Error "tests failed" }

5.2 Path length / non-ASCII paths

The repo path G:\GitHubRepos\kazma and Arabic content in kazma.yaml/templates are fine, but some Windows tools choke on long paths or non-ASCII. If you hit FileNotFoundError on deeply nested test temp dirs, enable long paths (HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled = 1).

5.3 The .pytest_tmp_* directories

The repo root contains many .pytest_tmp_* directories (artifacts from test runs). They are git-ignored clutter; safe to delete: Remove-Item -Recurse -Force .pytest_tmp_*.

---

6. Docker specifics

6.1 Vector volume path mismatch

docker-compose.yml mounts kazma_vectors:/root/.kazma/vector_memory, but the container runs as user kazma (home /home/kazma). Set KAZMA_VECTOR_PATH=/app/kazma-data/vector_memory (aligned with the data volume) to avoid writes to an unmounted path.

6.2 OOM with RAG extras

ChromaDB + sentence-transformers can exceed 512 Mi. The K8s manifest’s 512Mi limit is too small for the main agent with RAG. Use ≥1 Gi for the main agent container.

6.3 Health check path

Use /api/gateway/status (as docker-compose.yml does) or /health/live — not /api/v1/health (which belongs to the separate Hub API).

---

7. Arabic tokenization edge cases

7.1 Conflicting hamza rules

ؤ and ئ are normalized by two overlapping rules in arabic_tokenizer.py (Yeh normalization at lines 220-232 and the Waw/Ya-Hamza rules at lines 154-157). This is a known minor conflict; in practice the later rule wins. If you see inconsistent search results for hamza-bearing words, this is why.

7.2 Stemmer is basic

The stemmer (_init_stemmer, lines 104-130) does regex suffix/prefix stripping only — it’s not a lemmatizer. Plural/gender variants may not collapse to a common stem. For higher recall, consider pre-normalizing queries or adding domain synonyms.

7.3 Tatweel (ـ) in stored text

Tatweel is stripped on tokenize, so stored content_arabic won’t contain it — but if you query with raw text containing ـ, the same normalization applies, so matches still work.

---

8. Framework / build quirks

8.1 Version drift

Three independent version strings:

• pyproject.toml: 0.3.0
• kazma.yaml agent.version: 0.2.0
• CLI --help text: v0.2.0

They are not synced. Don’t rely on any single one for “the Kazma version.”

8.2 models.router does not import LiteLLM

models.router: kazma is a label. Kazma never import litellm. To use a LiteLLM proxy for OpenAI-compatible providers, set KAZMA_LITELLM_URL (or llm.gateway.url). Native Anthropic/Azure/Bedrock/Gemini and local Ollama/LM Studio stay direct. KAZMA_LITELLM=0 disables it. You can also point one provider’s base_url at http://host:4000/v1.

8.3 .env.example lists unused env vars

DEEPSEEK_API_KEY and ANTHROPIC_API_KEY appear in .env.example but no code reads them. Key those providers via the ConfigStore provider list instead. Don’t waste time wondering why setting them has no effect.

8.4 mcp.servers[].trust is a no-op

The trust: trusted string in kazma.yaml MCP config is not read by any code. It’s documentation-only. “Trust tiers” are not a code feature.

8.5 /undo and /edit are stubs

Both slash commands exist in the help system but are explicitly “not yet implemented” (slash_commands.py:257, 267). Don’t advertise them to users.

8.6 UnifiedModelRegistry is just ModelRegistry

The alias (model_registry.py:950) exists for backward compatibility. They are the same class.

---

9. Diagnostics checklist

When something is wrong, run through these:

# 1. Is the server up? Versions sane?
kazma status

# 2. Health
curl -s http://127.0.0.1:9090/health/details | jq

# 3. Active provider/model
curl -s http://127.0.0.1:9090/api/provider/active | jq

# 4. Gateway adapters
curl -s http://127.0.0.1:9090/api/gateway/status | jq

# 5. Swarm status + breaker states
curl -s http://127.0.0.1:9090/api/swarm/status | jq
curl -s http://127.0.0.1:9090/api/swarm/circuit-breakers | jq

# 6. Pending HITL approvals
curl -s http://127.0.0.1:9090/api/pending-approvals | jq

Enable JSON logging for structured diagnostics:

logging:
  level: DEBUG
  format: json

Or enable debug loggers in code:

for name in ("kazma_core", "kazma_gateway", "kazma_ui"):
    logging.getLogger(name).setLevel(logging.DEBUG)

9.1 Where the data lives

• Settings ConfigStore → kazma-data/settings.db
• Swarm TaskStore → kazma-data/swarm_tasks.db
• Gateway sessions → kazma-data/sessions.db
• LangGraph checkpoints → kazma-data/checkpoints.db
• Console tracing → KazmaTracer (backend="console") writes to stdout via kazma_core/tracing.py

9.2 Inspect the TaskStore directly

sqlite3 kazma-data/swarm_tasks.db \
  "SELECT id, type, status, workers, created_at FROM swarm_tasks ORDER BY created_at DESC LIMIT 10;"

sqlite3 kazma-data/swarm_tasks.db \
  "SELECT worker, date, tasks_completed, tasks_failed, avg_latency FROM swarm_worker_metrics ORDER BY date DESC LIMIT 10;"

9.3 Check GPU / VRAM

nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits

On a system without an NVIDIA GPU or driver, this errors and kazma_core/telemetry.py falls back to zeros.

9.4 Reset singletons (test or recovery)

from kazma_core.model_registry import reset_model_registry
from kazma_core.swarm.engine import set_swarm_engine
from kazma_core.tracing import get_trace_store

reset_model_registry()
set_swarm_engine(None)
ts = get_trace_store()
ts._traces.clear(); ts._total_cost = 0.0; ts._total_tokens = 0
# or fully replace the global TraceStore:
_tracing._trace_store = TraceStore()

---

10. Gateway & Telegram

10.1 “Connected” but no replies

Symptom: GET /api/gateway/status shows the Telegram adapter as connected, but the bot doesn’t reply and the queue depth is rising.

Cause: the message handler wasn’t registered (GatewayManager.on_message() never called); the adapter’s listen task crashed and the BaseAdapter.start() done callback left _running stale; invalid/missing bot token; a webhook is still registered (getUpdates → 409); allowed_users is filtering out the sender; or the handler is throwing on every message.

Fix:

1. Confirm the handler is registered: gateway_manager.on_message(create_graph_handler(agent)).
2. Check adapter logs for getMe validation — adapters/telegram.py calls getMe on startup and sets _running = False on a bad token.
3. deleteWebhook is called automatically in TelegramAdapter.listen() — verify it ran.
4. If allowed_users is non-empty, only those Telegram user IDs are processed (TelegramAdapter.set_allowed_users([...])).
5. Inspect gateway.stats for handler_registered and queue_depth.

10.2 Telegram 401 / 403 / 409 / 400 errors

Cause:

• 401 → bot token wrong/expired/revoked.
• 403 → bot isn’t a member of the group/channel, or the user blocked it.
• 409 on getUpdates → a webhook is still registered.
• 400 on sendMessage → Markdown parse error from unescaped characters.

Fix:

# validate the token directly
curl https://api.telegram.org/bot<TOKEN>/getMe
# clear a stale webhook
curl https://api.telegram.org/bot<TOKEN>/deleteWebhook

For outbound 400, the adapter auto-retries once without parse_mode. If you call sendMessage manually, escape Markdown specials (_, *, `, [) or set parse_mode="".

For LLM 401/403, the provider key is the problem — update it in Settings → Models/Providers. retry.py maps these to the friendly message “The model request was rejected due to an invalid or missing API key.”

10.3 Gateway bus queue full / adapter offline

Symptom: GET /api/gateway/status reports queue_depth at 100 (the default maxsize), or an adapter is offline; messages dropped with asyncio.QueueFull.

Cause: the handler is too slow/blocked; the adapter’s listen() task raised an unhandled exception and exited; or no handler is registered.

Fix:

1. Check gateway.stats (handler_registered, queue_depth, per-adapter running).
2. Move heavy work off the consumer path. The queue default is maxsize=100 (GatewayManager.__init__) — don’t raise it unless you’ve also improved consumer throughput.
3. If an adapter is offline, find the original exception in the logs (the BaseAdapter done callback in gateway.py logs it). Fix the root cause, then restart the gateway.
4. Always call GatewayManager.on_message(handler) before GatewayManager.start().

10.4 Tracing a message by correlation_id

Every IncomingMessage gets a correlation_id at ingress — field(default_factory=lambda: f"cid-\{uuid.uuid4().hex[:12]\}") (gateway.py). It is carried through the bus but not auto-injected into downstream loggers.

async def handler(msg: IncomingMessage):
    logger.info("[%s] Processing message from %s", msg.correlation_id, msg.sender_id)

grep "cid-abc123def456" /var/log/kazma.log

To trace into the graph/swarm, add it to the graph state or SwarmTask.metadata in your handler.

---

11. Providers & Connectors hub (UI)

11.1 Provider shows “down”/“degraded” / Test Connection fails

Symptom: a provider card in Settings → Providers & Connectors shows down/degraded, or Test Connection returns Cannot connect to <base_url> / Cannot connect to <base_url> (connection failed) / HTTP 401.

Cause: Base URL wrong/unreachable; API key missing/expired/invalid; provider disabled; or the provider’s /models endpoint isn’t OpenAI-compatible. Cannot connect to <base_url> (connection failed) (previously the unhelpful HTTP None:) means every candidate endpoint threw — typically a wrong host/port, or the server runs in WSL but the local provider (Ollama/LM Studio) runs on the Windows host: inside WSL, 127.0.0.1 is the WSL VM, not Windows. Point the provider’s Base URL at the host running the daemon (the WSL gateway IP — ip route show default).

Fix: Edit the provider, verify the Base URL, paste the current key, click Test Connection (backend hits /models and reports latency or the exact HTTP error). The Save button is disabled until the test passes. On 401, regenerate the key from the provider dashboard. For local servers (Ollama, LM Studio), ensure the server is running and the URL has the /v1 suffix (e.g., http://127.0.0.1:11434/v1).

11.2 Platform connector test fails (Telegram / Discord / Slack)

Cause: token not saved; token revoked; Slack needs a Bot Token (xoxb-...) for the polling Web API adapter (no Socket Mode); Discord expects Authorization: Bot <TOKEN>.

Fix: the backend runs a non-destructive health check:

• Telegram → GET https://api.telegram.org/bot<TOKEN>/getMe
• Discord → GET https://discord.com/api/v10/users/@me with Authorization: Bot <TOKEN>
• Slack → POST https://slack.com/api/auth.test with the bot token

Save stays disabled until the test passes. After saving, click Refresh Gateway (or restart the server) so the new token is picked up.

11.3 Masked-secret placeholder overwrites the real key

Symptom: you edit a provider, leave the API key field as ****XXXX, save, and the provider stops authenticating.

Cause: the backend preserves the existing secret only when it recognizes the value as a masked placeholder (*** or containing ****). An unexpected format can be saved as the real key.

Fix: leave the masked value unchanged to keep the existing secret, or clear the field and paste the full new secret. If you suspect the placeholder was saved, clear + repaste + Test + Save. Verify:

from kazma_core.model_registry import get_model_registry
provider = get_model_registry().get_provider("openai")
print(provider.get("api_key", "")[:4])   # should NOT be "****"

11.4 Test-before-save behavior

Save is disabled for any provider/connector until Test Connection succeeds (new entries and edits). For existing entries, opening the modal pre-fills the masked secret and marks the entry tested, so you can save without re-testing if the secret is unchanged.

---

12. Web research & dashboard metrics

12.1 Long page only shows the beginning

Cause: read_url returns a window (default 16k). The graph used to also hard-cap research tools at 4k; research tools now use KAZMA_TOOL_RESULT_RESEARCH_MAX_CHARS (default 16k).

Fix: Page with offset / max_chars, or read_url_to_file then digest_research_file / read_research_chunk. See Web research.

12.2 Search returns nothing / flaky results

Cause: DuckDuckGo rate limits; Bing HTML fallback is fragile.

Fix: Run SearXNG and set KAZMA_SEARXNG_URL. Optionally raise reliability with KAZMA_JINA_READER=1 or KAZMA_FIRECRAWL_API_KEY for page fetch, not search.

12.3 Bot wall / empty extract

Cause: CDN challenge or JS SPA shell.

Fix: Install Playwright (pip install 'kazma[web]' + playwright install chromium). Or enable Jina/Firecrawl backends. Not all sites can be opened automatically.

12.4 Dashboard Total Cost / Tokens show $0 / 0 after a chat turn

Cause (historical): API/WS sent strings like "$0.0043" / "27,127" and JS Number(...) became NaN. Fixed to numeric metrics + safe parsing.

Still true: totals are process-local TraceStore — restart resets them. Cost breaker budget is separate from durable billing. Dashboard “Circuit Breaker” is the cost breaker, not swarm worker breakers. Hard-refresh /dashboard after upgrade.

12.5 There is no /research command

Expected. Use natural-language chat or /swarm research …. See Web research and FAQ.

---

13. TUI (kazma-tui)

13.1 Dashboard freezes / metrics stop updating

Cause: a synchronous call in the 2 s refresh path is blocking the Textual event loop. MetricsDashboard._do_refresh() (kazma-tui/kazma_tui/dashboard.py) calls TraceStore.recent(), MetricsCollector.get_all_metrics(), and SwarmEngine._workers synchronously — any block freezes the UI. An exception every refresh leaves metrics stuck at their last good value.

Fix: check logs for Dashboard refresh failed; ensure psutil is installed and kazma-core importable. To isolate a blocking custom source, inject it from the constructor (MetricsDashboard(hardware_monitor=..., trace_store=..., ...)).

13.2 Dashboard shows “N/A” everywhere

Cause: the dashboard is read-only and falls back to N/A when any source is missing: psutil absent (HardwareMonitor import fails); TraceStore empty (RPM None); MetricsCollector/SwarmEngine unavailable; or running outside an event loop.

Fix:

uv pip install -e ".[tui]"     # or: pip install textual psutil

Verify the sources import, then generate some trace activity (RPM is N/A when idle).

13.3 VRAM card shows N/A / 0.0

Cause: kazma_core/telemetry.py shells out to nvidia-smi. If it’s not in PATH, the driver is missing, the subprocess times out, or the GPU is non-NVIDIA, it falls back to zeros (debug-level log).

Fix:

nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits

If that fails, the TUI can’t show VRAM — install the NVIDIA driver / add nvidia-smi to PATH. On non-NVIDIA systems, N/A is expected (graceful degradation).

13.4 TUI crashes on startup (import error / Python < 3.11)

Cause: textual not installed; Python older than 3.11; ModelRegistry not initialized before the TUI header composes (header.py calls get_model_registry()); or a kazma-core import failing.

Fix:

uv pip install -e ".[tui]"
python --version        # must be ≥ 3.11

from kazma_core.config_store import get_config_store
from kazma_core.model_registry import initialize_model_registry
from kazma_tui.app import main
initialize_model_registry(get_config_store())   # before the TUI reads the profile
main()

---

13. CLI

13.1 kazma.yaml disagrees with the running config

Cause: ConfigStore (config_store.py) overrides kazma.yaml at runtime. The SQLite DB (kazma-data/settings.db) is the source of truth for any key written via the UI/CLI/code; env vars override both.

Fix:

from kazma_core.config_store import get_config_store
cs = get_config_store()
print(cs.get("llm.model"), cs.get("registry.active_model"))   # active values
print(cs.export_yaml())                                         # YAML base
cs.delete("registry.active_model")                              # revert one key to YAML default

For a full reset: rm kazma-data/settings.db (loses all runtime settings) and restart.

13.2 kazma command not found / entry point missing

Cause: not installed, or installed in a venv that isn’t activated; the pyproject.toml entry point (kazma = "kazma_cli.main:main") wasn’t wired (incompatible editable mode); on Windows the script isn’t on PATH.

Fix:

uv sync --all-extras
uv pip install -e .
which kazma && kazma --help          # PowerShell: Get-Command kazma
# fallback if the entry point won't work:
python -m kazma_cli --help

13.3 Shell tab-completion

Cause: completion script not installed for the active shell; --list-models/--list-providers need a ModelRegistry (they fall back if uninitialized); zsh fpath / bash-PowerShell source missing.

Fix:

kazma completion install {bash|zsh|powershell}
kazma completion --list-models      # verify dynamic lists; init registry first if this errors
kazma completion --list-providers

Restart the shell (or source the printed path) after install.

Model profiles: save_model_profile(name, profile), get_model_profile(name), and list_model_profiles() are valid registry methods (model_registry.py) for named \{provider, base_url, model, api_key\} snapshots. Note: profiles are stored under models.saved.\{name\} in ConfigStore — they are not dispatched by any worker.

---

14. Swarm panel & workers

14.1 OutputValidator rejects the worker’s plan

Cause: the dispatch payload carried a validation_schema (JSON Schema or Pydantic model) in metadata, and the worker output didn’t match. OutputValidator (swarm/reliability.py) rejects before the result is accepted.

Fix:

1. Inspect the validation_schema in your payload and align the worker prompt to it.
2. Ensure the worker returns valid JSON without Markdown fences (stringified JSON is parsed).
3. Remove validation_schema from the payload if validation isn’t needed.

14.2 Worker error / timeout / circuit breaker open

Cause: the worker’s LLM call failed; the task exceeded timeout (default 300 s, swarm/worker.py); or the circuit breaker (swarm/reliability.py) has opened after repeated failures.

Fix:

kazma swarm circuit-breaker worker-1              # inspect
kazma swarm circuit-breaker worker-1 --reset      # reset once the root cause is fixed
curl -X POST http://localhost:9090/api/swarm/workers/worker-1/circuit-breaker/reset

Raise the per-task timeout for genuinely long work: \{"workers": ["worker-1"], "task": "...", "timeout": 600\}.

14.3 Task not appearing in Active Tasks

Cause: the SSE bus isn’t wired to the engine — kazma-ui/kazma_ui/swarm_panel.py calls wire_engine_events(engine, _sse_bus) in _current_engine(); if the engine predates the bus, events are lost. Or the dispatch didn’t return a task_id; or a frontend JS error in swarm.js dispatchTask().

Fix:

1. Confirm wire_engine_events() ran and the engine was obtained after the SSE bus was available.
2. Check the dispatch response for task_id:

   curl -X POST http://localhost:9090/api/swarm/dispatch -H "Content-Type: application/json" \
     -d '{"workers":["worker-1"],"task":"test"}'

3. Open the browser console for swarm.js errors; confirm the SSE stream responds:

   curl -N http://localhost:9090/api/swarm/tasks/<task_id>/stream

14.4 Results Dashboard empty / task-detail modal blank

Cause: SwarmTask.to_dict() nests result fields under result, but the UI expects them top-level — _flatten_swarm_task() in swarm_panel.py is the bridge. If it misses a field, the UI renders nothing. Or the modal element is missing from the template / viewTaskDetail() reads the wrong key.

Fix:

curl http://localhost:9090/api/swarm/tasks/<task_id>

The response should carry task_id, status, worker_results, aggregated_output, synthesized_output, duration_seconds, total_cost, total_tokens at the top level. If a new field is missing, promote it from the nested result dict in _flatten_swarm_task(). Confirm the task-detail-modal element exists in templates/swarm.html and that swarm.js renderTaskDetailHTML() reads task.worker_results / task.individual_opinions / task.synthesized_output.

14.5 Task history lost on restart

Cause: SwarmEngine was created without a shared TaskStore, so tasks live only in-memory and are lost on restart. The default TaskStore writes to kazma-data/swarm_tasks.db.

Fix:

from kazma_core.swarm.engine import get_swarm_engine
engine = get_swarm_engine()
print(engine.task_store)   # must not be None

ls -la kazma-data/swarm_tasks.db
sqlite3 kazma-data/swarm_tasks.db ".tables"

If you construct a SwarmEngine manually, always pass the same TaskStore:

from kazma_core.swarm.engine import SwarmEngine
from kazma_core.swarm.config import SwarmConfig
from kazma_core.swarm.task_store import TaskStore
engine = SwarmEngine(SwarmConfig(enabled=True, workers=[]), task_store=TaskStore())

14.6 Pipeline HITL checkpoint missing (swarm/pipeline)

Two HITL mechanisms, don’t confuse them. This section is about swarm/pipeline checkpoints (POST /api/swarm/tasks/\{id\}/approve|reject). The agent tool-call approval gate is a different endpoint — POST /api/approve/\{thread_id\} (routes_direct.py:454) — covered in §3.2.

Symptom: a pipeline task should pause at a checkpoint, but no HITL card appears, or approve/reject does nothing.

Cause: the task wasn’t dispatched with metadata.hitl_checkpoints (1-based step indices where the pause occurs); the SSE bus wasn’t wired so the checkpoint event didn’t reach the frontend; HITLCheckpointHandler wasn’t restored from TaskStore after restart (SwarmEngine.restore_paused_tasks() must run on startup); or approve/reject targets the wrong task_id.

Fix:

payload = {
    "workers": ["worker-1", "worker-2"],
    "task": "...",
    "type": "pipeline",
    "metadata": {"hitl_checkpoints": [1]},   # pause before step 1
}

curl -X POST http://localhost:9090/api/swarm/tasks/<task_id>/approve
curl -X POST http://localhost:9090/api/swarm/tasks/<task_id>/reject

Ensure restore_paused_tasks() runs on startup (else paused tasks auto-reject on their checkpoint_timeout — see §3.3).

---

15. Production bind & auth (v0.9+)

15.0 Browser ERR_CONNECTION_RESET on http://127.0.0.1:9090 (Windows)

Symptom: Edge/Chrome shows “can’t reach this page” / ERR_CONNECTION_RESET on port 9090. Health curl also fails. You may think Kazma is broken after a pull.

Cause (common): Windows IP Helper / portproxy (WSL2, Docker Desktop, Hyper-V) is listening on 127.0.0.1:9090 and forwarding to a dead WSL IP. That is not the Kazma Python process. Confirm:

netsh interface portproxy show all
# typical bad row:
# 127.0.0.1  9090  →  172.x.x.x  9090

Get-NetTCPConnection -LocalPort 9090 -State Listen |
  Select-Object OwningProcess
# if OwningProcess is svchost (iphlpsvc), Kazma is not serving that port

Fix:

# Easiest: run Kazma on a free port
kazma serve 9091
# open http://127.0.0.1:9091/

# Optional: remove stale proxy (Admin PowerShell) if you own that rule:
# netsh interface portproxy delete v4tov4 listenaddress=127.0.0.1 listenport=9090

Verify Kazma (not the proxy):

curl http://127.0.0.1:9091/health
# expect JSON with "status":"ok"

15.1 Server won’t start on public bind without secret

Cause: CLI/serve refuse non-loopback bind when KAZMA_SECRET is missing, or refuse the known-bad historical default secret.

Fix:

# set a strong secret, then:
export KAZMA_SECRET='…'          # PowerShell: $env:KAZMA_SECRET='…'
export KAZMA_HOST=0.0.0.0
kazma serve 9090

Default local port is 9090 (not 8000). Prefer loopback for single-operator: KAZMA_HOST=127.0.0.1.

15.2 YOLO / danger tools in production

With KAZMA_PRODUCTION=1, YOLO is hard-blocked unless KAZMA_ALLOW_YOLO=1 (avoid in real prod). NullBus is fail-closed. See Production checklist.

15.3 API returns 401 on every /api/* call

Cause: Auth default-deny when a secret is configured; cookie/session missing; wrong secret header.

Fix: Log in via /login, send the session cookie, or use the documented secret header for single-operator automation. Open routes: /health, /health/live, /health/ready.

---

16. Email (Gmail / Microsoft)

Full guide: Email integration.

16.1 Google: “has not completed the Google verification process” / access_denied

Cause: OAuth consent app is in Testing; only listed test users can sign in.

Fix: Cloud Console → OAuth consent screen → Test users → add your Gmail. Continue past the unverified-app warning (expected for self-hosted).

16.2 Gmail API 403 insufficient authentication scopes / insufficientPermissions

Cause: Token has profile/email only — Gmail mail scopes were not granted (or not listed on the consent screen).

Fix: Enable Gmail API. Consent screen → scopes → add https://www.googleapis.com/auth/gmail.modify and …/gmail.send. Settings → Email → Disconnect → Connect with Google → approve Gmail access (not only “See your email address”).

16.3 IMAP/POP rejects Gmail password

Cause: Google does not accept normal account passwords for IMAP/POP.

Fix: Use a Google App Password (2FA required) or prefer OAuth.

16.4 Microsoft IMAP/POP fails on M365

Cause: Many tenants disable basic auth.

Fix: Use Microsoft Graph OAuth (Settings → Email → OAuth).

---

Documentation Audit Notes

This file consolidates audits plus the former operator guide (archive/docs-loose/TROUBLESHOOTING.md — folded into §1.6–1.9 and §9–14). Gotchas:

1. Memory is opt-in (§2.1) — don’t assume recall.
2. Three HITL lists (§3.2) — adding a danger tool in one place isn’t enough.
3. KAZMA_SECRET gates approval auth (§3.2) — set it always.
4. Default port 9090 (§15.1) — old samples may say 8000.
5. No 429 handling (§1.4) — proxy or throttle externally.
6. Two HITL mechanisms (§3.2 vs §14.6) — /api/approve/{thread_id} vs swarm task approve.
7. Always get_config_store() (§4.1, §1.9) — never construct ConfigStore() for the shared DB.