Runbook — incident copilot

When a production system breaks, the person on call loses the first fifteen or twenty minutes to orientation rather than repair: which runbook covers this, has it happened before, what the dashboards are saying. Runbook does that part. Given an alert, it sorts the alert into a lane, retrieves the relevant runbook and any similar past incident, gathers signal from metrics, logs, recent deploys and service dependencies through read-only tools, and proposes a diagnosis with ordered remediation steps, each one grounded in a specific runbook line. It pauses for a human before anything that changes system state. The How it works page walks the loop stage by stage.

The orchestration is deliberately thin. There is no agent framework: the loop is a few hundred lines you can read top to bottom, because a framework's tool-runner hides the exact point where each safety decision is made, and that point is what I most needed to keep visible. Every real design choice, from retrieval to the approval gate to whether to fine-tune, is a short written record on the Decisions page.

Retrieval is more than a vector lookup. Incident text is full of exact tokens, alert names, error strings, config keys, that a semantic embedding blurs together, so it runs dense vector search and Postgres full-text in parallel, fuses the two rankings, then reranks the shortlist with a cross-encoder. The matching runbook is pulled in whole rather than as the fragment that matched the symptoms, so the remediation section is in context. The embedding and rerank models are hosted rather than bundled in the container, which kept the deployment small and raised retrieval quality at the same time.

The safety layer is structural rather than prompt text. Any step that would change system state moves the run into an awaiting-approval state that only a human can clear, and the loop has no code path to approve its own work. A separate classifier decides whether each action is read-only or state-changing and defaults to state-changing when unsure, so the model that wrote the diagnosis is not the one certifying its fix. A remediation step that does not quote a real runbook line is dropped, and a proposal with nothing left becomes an escalation. Secrets and PII are removed before any text reaches a model or a trace.

Knowing whether the output is any good took the most effort. A golden set of labelled incidents runs on every change: hard pass/fail checks on the invariants that must never regress, quality scores measured against a committed baseline, and a model judge for the diagnosis itself. A reference-free slice of the same checks runs continuously on a sample of real traffic. The evals page carries the current numbers, and a separate report sets out what they do and do not establish; it lands on advisory use with a human approving every action, not autonomous remediation. Fine-tuning was on the plan and skipped, because better prompting and retrieval got there first.

The security work targets indirect prompt injection: someone who can get a string into a log line, not the prompt, writing that an incident is pre-approved and waiting for the copilot to read it back during an unrelated incident weeks later. A red-team harness runs these attacks against the real system and measures how often they land with the prompt-level defences on and off. Through a log line, the realistic path, the success rate is zero, held by the structural checks alone; the security page has the full breakdown, including the attacks that are not fully closed. That harness was briefly a release gate, then went back to a manual check: one run against non-deterministic models says too little.

The system is live and redeploys on every push. The most direct way in is a worked run, or the incident console, where every past run keeps its full audit trail.