Chaos Testing & Fault Injection

Chaos Testing & Fault Injection

Kazma includes a built-in, production-grade Chaos Testing & Fault Injection Framework (kazma_core/chaos) designed to validate system resilience, supervisor recovery, circuit breaker failovers, and graceful degradation under real-world failures.

---

Key Capabilities

• Fail-Closed Safety: Inactive by default. Requires KAZMA_CHAOS_ENABLED=true in the environment to execute any injection.
• 10 Predefined Experiments: Latency spikes, intermittent errors, database slowdowns, message bus partitions, tool execution failures, and swarm engine degradation.
• Granular Target Scoping: Target specific components such as LLM_PROVIDER, DATABASE, MESSAGE_BUS, TOOL_EXECUTOR, or SWARM_ENGINE.
• Flexible Interfaces: Python decorator @chaos_injection, chaos_experiment() context manager, or REST APIs at /api/chaos/*.

---

Architecture & Failure Types

  ┌────────────────────────────────────────────────────────┐
  │                 Chaos Engine (_chaos_enabled)          │
  └───────────────────────────┬────────────────────────────┘
                              │
       ┌──────────────────────┼──────────────────────┐
       ▼                      ▼                      ▼
  [LLM_PROVIDER]         [DATABASE]            [MESSAGE_BUS]
  • LATENCY (ms)         • TIMEOUT             • NETWORK_PARTITION
  • ERROR (500/429)      • DATA_CORRUPTION     • CIRCUIT_BREAKER_OPEN

Supported Failure Types

Failure Type	Enum Name	Description
Latency	FailureType.LATENCY	Injects sleep delay (latency_ms) before operation completes.
Error	FailureType.ERROR	Raises simulated exceptions or HTTP status errors (error_code).
Timeout	FailureType.TIMEOUT	Blocks execution past the configured timeout threshold.
Circuit Breaker Open	FailureType.CIRCUIT_BREAKER_OPEN	Forces circuit breaker state to open immediately.
Resource Exhaustion	FailureType.RESOURCE_EXHAUSTION	Simulates memory or CPU saturation.
Network Partition	FailureType.NETWORK_PARTITION	Drops RPCs and messages between nodes or swarm agents.
Data Corruption	FailureType.DATA_CORRUPTION	Simulates corrupted payload responses.
Partial Degradation	FailureType.PARTIAL_DEGRADATION	Triggers intermittent degradation under load.

---

10 Predefined Experiments

Kazma ships with 10 out-of-the-box experiments ready to run against test environments:

1. llm_high_latency — Simulates 5000ms latency on LLM calls to test agent timeout handling.
2. llm_intermittent_errors — Injects random 500 errors (30% probability) on model calls.
3. llm_timeout — Injects hard timeouts on model completions.
4. database_slow — 2000ms delay on memory retrieval and task ledger writes.
5. database_errors — Simulates SQLite / Postgres connection dropouts.
6. message_bus_partition — Splits swarm communication bus to test autonomous recovery.
7. tool_executor_failures — Randomly fails danger and native tool executions.
8. swarm_engine_degradation — Triggers slow worker responses to test supervisor task delegation.
9. circuit_breaker_force_open — Forces breaker open to verify fallback model failovers.
10. resource_exhaustion — Simulates host memory pressure.

---

Python API Usage

1. Using the Context Manager

from kazma_core.chaos import FailureType, InjectionTarget, chaos_experiment

# Inject 1500ms latency to LLM providers during this block
async with chaos_experiment(
    target=InjectionTarget.LLM_PROVIDER,
    failure_type=FailureType.LATENCY,
    latency_ms=1500,
    probability=1.0,
):
    result = await agent.run("Summarize the latest findings")

2. Running Predefined Experiments

from kazma_core.chaos import run_predefined_experiment

# Run LLM intermittent error experiment for 30 seconds
async with run_predefined_experiment("llm_intermittent_errors", duration_seconds=30):
    await test_suite.run_all()

3. Function Decorator

from kazma_core.chaos import FailureType, InjectionTarget, chaos_injection

@chaos_injection(
    target=InjectionTarget.DATABASE,
    failure_type=FailureType.ERROR,
    probability=0.2,
    error_message="Simulated DB pool failure",
)
async def query_knowledge_base(query: str):
    # This call will fail 20% of the time when KAZMA_CHAOS_ENABLED=true
    return await db.search(query)

---

REST API Endpoints

When KAZMA_CHAOS_ENABLED=true, the following management endpoints are active on the Web gateway:

Method	Endpoint	Description
GET	/api/chaos/status	Current chaos engine status and active injection count.
GET	/api/chaos/injections	List all currently active failure injections.
POST	/api/chaos/injections	Create and activate a new failure injection dynamically.
DELETE	/api/chaos/injections/{id}	Remove a specific active injection.
POST	/api/chaos/experiments/predefined	Trigger one of the 10 predefined experiments by name.
POST	/api/chaos/reset	Clear all active injections and restore normal operation.
GET	/api/chaos/metrics	View metrics on injected failures and recovery success rates.

---

Production Safeguards

[!CAUTION] Chaos testing should only be enabled in staging, staging-mirror, or controlled resilience testing pipelines.

To prevent accidental failure injection in live production environments:

1. Double Gate: The framework evaluates _chaos_enabled() at registration and at the exact moment of execution.
2. Auto-Expiration: All dynamic injections can include duration_seconds to automatically expire and clean up.
3. No-Op in Default Builds: If KAZMA_CHAOS_ENABLED is missing or false, all decorators and context managers execute as zero-overhead passthroughs.