Scenarios Runtime

​

What it enables

• Decouple agent flow shape from worker code. New scenarios — code, writing, research, security, Q&A — ship as YAML + prompt builders + (optional) rubric markdown. No worker deploy required for a new flow once the relevant runners exist.
• Per-job scenario dispatch. Each scenario YAML declares the intent_route values it serves (code_build → project, document_writing → document/writing, quick_qa → task/qa, …). The worker compiles every loaded scenario once at startup and picks one per job from (in priority): payload.metadata["scenario_id"] → payload.metadata["intent_route"] mapped via the loader → CEREBRUM_DEFAULT_SCENARIO. The command widget’s task type sets intent_route; the future universal-input classifier will set it (or pin scenario_id directly) by classifying the prompt.
• Per-level model pinning. Each level resolves its own ModelConstraint through worker/worker/llm/resolver.py, so an L4 executor can be pinned to Claude Opus while a Q&A scenario runs the whole thing on the free tier.
• Pluggable verifiers (subprocess + LLM judge + schema + MCP). Verifiers can run on prompt-kind levels (e.g., an llm_judge rubric on l3_accept) or on a dedicated verifier-kind level that gates the next branch.
• First-class DAG branching. Edges carry a typed boolean DSL (verdict == 'PASS' && !escalate_now); the compiler evaluates them at routing time. fan_out parents run children in parallel; aggregators (majority_pass, weighted_score, all_pass) merge results.
• Agentic file-tool loop. kind: tool_loop levels (l4_execute, l2_escalate) mutate the project workspace through tool calls the worker executes — surgical edit_files on the scaffolded template, not a full-file Markdown blob. The workspace filesystem is the shared state; levels exchange paths + summaries.
• Human approval gates. A kind: human_approval level pauses the graph via LangGraph’s interrupt(), packages a question for the approver, and resumes with their reply as the gate-readable verdict.
• Budget kill switch + user Stop. Scenario-wide caps (cost / tokens / duration) and a user-initiated Stop both set aborted_reason, which short-circuits subsequent levels with synthetic skipped states and routes the gate straight to END. The run ends scenario_finished status="aborted" (a cap) or "cancelled" (a Stop); a cancelled run discards partial work — no commit/push. Stop is a Redis flag (cerebrum:job:{id}:cancel) the worker polls between levels and per tool-turn via EventBus.is_cancelled().
• Per-tenant overrides. A partial YAML stored in Postgres deep-merges over the base scenario at orchestrator construction time, so an org can tweak budgets / constraints / verifier weights without forking the base file.

​

Module layout

• schemas.py — Pydantic models with strict (extra="forbid") validation. Scenario, Level, Edge, Gate, Budget, ModelConstraint, VerifierRef, FanInRef. Composite primary-key validation (unique level ids, edges target existing levels or the __end__ sentinel, mixed-tier allows rejected).
• state.py — ScenarioState TypedDict threaded through the LangGraph. Explicit top-level fields for every value the gate DSL or runners read: verdict, escalate_now, confidence, aborted_reason, budget_exceeded, per-level entries under levels: dict[str, LevelState].
• scratchpad.py — Redis-backed scenario-scoped KV (cerebrum:job:{job_id}:scratchpad). Available to every level via the scratchpad.{get,put,list} intra-tools.
• expr.py — Recursive-descent parser for the edge DSL. Supports paths (verifier.build.passed), string/number/bool literals, ==/!=/<,<=,>,>=, !, &&, ||, parens. Missing paths resolve to a fail-closed UNDEFINED sentinel. Loader compiles every Edge.when at load time.
• loader.py — ScenarioLoader reads *.yaml from knowledge-hub/scenarios/. refresh() swaps the registry atomically and preserves the old version on any validation error. with_overrides({tenant_id: partial_yaml}) deep-merges tenant tweaks. Scenario.checksum is the sha256 of the canonical YAML, recorded on each ScenarioResult for reproducibility.
• compiler.py — ScenarioCompiler turns a Scenario into a LangGraph StateGraph. One node per level; non-conditional edges via add_edge; multi-branch edges via add_conditional_edges with a router that evaluates the edge DSL and emits gate_decision events.
• executor.py — ScenarioExecutor dispatches each level to its kind-specific runner. Wraps every invocation with level_started / level_finished events alongside the legacy node_started / node_finished pair (back-compat through one release).
• runners.py — PromptLevelRunner, ToolLoopLevelRunner, VerifierLevelRunner, FanOutLevelRunner, HumanApprovalLevelRunner. See Scenario execution for the per-runner semantics. ToolLoopLevelRunner drives the kind: tool_loop levels (l4_execute, l2_escalate): an agentic file-tool loop where the model mutates the workspace through read_file / edit_file / write_file / list_dir / delete_file / finish tools — the filesystem is the artifact, downstream levels receive paths, not file bodies.
• prompts.py — Registered prompt builders for every scenario level (l1_team_lead_brief, writing_outline, security_triage, qa_answer, etc.). Builders return a BuiltPrompt { body, response_kind, handoff_* } and read prior-level outputs from state["levels"]. code_build’s L1 emits a Markdown brief and L3 emits Markdown prose + a trailing ```control block carrying the verdict (worker/runtime/control_block.py) — not a JSON AgentMessage envelope; the runner runs a bounded repair-retry when the verdict won’t parse.
• worker/runtime/file_tools.py — the six sandboxed file tools + the native (Anthropic) and textual transports the tool loop dispatches.
• aggregators.py — MajorityPassAggregator, WeightedScoreAggregator, AllPassAggregator. Aggregator name is referenced from the parent Level.fan_in.aggregator field.
• tenant_overrides.py — TenantOverrideProvider Protocol with two impls: InMemoryTenantOverrideProvider (tests) and PostgresTenantOverrideProvider (production). DB schema is on worker/worker/storage/postgres_repo.py under tenant_scenario_overrides.
• replay.py — CassetteProviders + ReplayHarness. Drives a scenario through recorded model responses (no live calls) and asserts no verifier or rubric-score regression beyond the scenario’s regression_thresholds. Runs in CI via .github/workflows/scenario-replay.yml.
• orchestrator.py — ScenarioOrchestrator is the public entry the worker calls. Same run(payload, routing_decision, event_bus) -> ScenarioResult contract the legacy HierarchyOrchestrator exposed.

​

Lifecycle events

​

Bundled scenarios

​

CI gate

​

Impact scope

• Worker: hot path for every paid job goes through ScenarioOrchestrator. Legacy HierarchyOrchestrator is deleted; the wrapping WorkerOrchestrator is unchanged.
• Frontend: JobDetailPanel mounts ScenarioRunTimeline for every job. Renders nothing on legacy backlog events for backwards compat.
• Postgres: one new table tenant_scenario_overrides (composite PK on tenant_id, scenario_id).
• Defaults: code_build is the default scenario; override via the CEREBRUM_DEFAULT_SCENARIO env var or per-job scenario_id metadata.
• Backwards compat: legacy node_started / node_finished events still fire alongside the new level_* events for one release.

​

Tests

• test_scenarios.py (28) — schemas, loader, scratchpad.
• test_scenario_expr.py (25) — gate DSL.
• test_verifiers.py + test_verifiers_phase2.py (55) — verifier framework + built-ins.
• test_scenario_compiler.py (10) — compiler topology + end-to-end with mock runners.
• test_scenario_prompts.py (10) — prompt-builder parity.
• test_scenario_orchestrator.py (9) — production orchestrator with fake providers.
• test_scenario_events.py (13) — Phase-4 lifecycle event payloads.
• test_scenario_phase5.py (12) — fan_out + aggregators + budget kill switch.
• test_scenario_phase6.py (16) — checksum, hot-reload, tenant overrides, replay harness.
• test_tenant_overrides.py (12) — DB-backed tenant overrides.
• test_model_resolver.py (15) — per-level model pinning.
• test_document_writing.py (15), test_quick_qa_and_research_brief.py (12), test_security_review.py (15) — bundled scenarios end-to-end.

/Users/azat/labs/cerebrum/worker/.venv/bin/python -m pytest tests/ -v