Audits — agentic improvement detection

Audits are recurring jobs that mine your agent logs across sessions to find things worth improving. Where an alert watches one metric you already know about in near-real-time, an audit investigates: on a schedule you set, it runs a deterministic policy pass over the window, then turns an AI reliability agent loose on your sessions — the agent queries the data itself, reads suspicious transcripts, and (when it helps) runs small analysis scripts, then writes up improvement recommendations with the evidence behind each one. Use audits to answer “what should I fix or improve in my agents?” — and alerts to be paged the moment a specific threshold is crossed. Every improvement links to the exact sessions and queries behind it, and one click creates a prefilled alert to catch recurrences. The dashboard surface is /<org-slug>/audits (sidebar → analyze → audits), gated by the audits:read / audits:write permissions.

---

​

How a run works

Each run has two layers — a deterministic floor and an agentic investigation.

​

1. The policy pass (deterministic)

Before any model runs, the audit executes a small catalog of SQL policy checks over the window: bounded aggregate queries that flag known-bad patterns and report how many events / which sessions matched — never the matched text itself. The catalog includes:

• Secret / credential leakage in event payloads — AWS access keys, sk-… API keys, PEM private keys, JWT / bearer tokens, and KEY=… credential assignments.
• Prompt-injection markers — “ignore previous instructions”, “reveal your system prompt”, and similar.
• PII — SSN-shaped numbers (heuristic).
• Tool-permission denials and runaway tool-call loops.

Policy hits are persisted as findings (kind policy) that always surface (they are never trimmed by the per-run cap), and they are handed to the AI agent as starting leads. Because this layer needs no model, an audit still produces its most important security signals even if the AI agent is unavailable.

​

2. The agentic investigation (AI)

The audit then runs an autonomous reliability agent (the same Claude Agent SDK service that powers the dashboard assistant, with an audit-specific prompt). Given the audit’s scope (selected agents × environments) and time window, the agent:

• runs read-only SQL queries against your analytics tables,
• reads a handful of representative session transcripts,
• optionally writes and runs short Python scripts in a locked-down in-pod sandbox (no network, no filesystem access, secrets scrubbed) for analysis SQL can’t express — clustering errors, computing distributions, sweeping payloads it already fetched,
• and records each well-evidenced improvement it finds.

It works through several investigative lines — error clustering, drift vs. a baseline, goal failure in transcripts, tool misuse, quality/cost trade-offs, and coverage gaps — at the audit’s sensitivity (low / medium / high). Every improvement must cite evidence: the session ids the agent actually inspected and/or the SQL it ran. The server validates that cited sessions exist and discards any improvement with no surviving evidence, so the agent investigates but never invents. Each improvement carries:

• a recommendation (the concrete change to make — a prompt tweak, a tool-schema fix, a retry policy, a guardrail, more eval coverage),
• an expected impact and an effort estimate (low / medium / high),
• a magnitude — big (an operator should be paged), medium (belongs in the run report), or small (dashboard context),
• a stable fingerprint (from the issue’s category + scope, not this run’s sessions) so the same issue is tracked run-over-run even as the evidence changes,
• and, where a simple deterministic watcher could catch recurrence, a suggested alert you can create in one click.

The AI layer is optional-but-recommended. If no AI agent is configured for the audit pipeline, runs still execute, persist the policy findings, and honestly report “analysis unavailable” for the agentic layer rather than silently passing.

​

Failure modes

Improvements classify into your org’s durable failure-mode catalog (or propose a new mode). Modes give patterns a stable identity across runs and long-horizon recurrence tracking.

​

Triage lifecycle

On a finding page (/audits/<id>/findings/<finding-id>): Low-signal noise is controlled per audit with a per-run findings cap (top_k) on the agentic improvements. Policy findings bypass the cap (they are security-relevant and always shown). Anything cut by the cap is counted in the run’s stats — nothing is silently dropped.

​

Scheduling

• Cadence (schedule_interval_secs): hourly to weekly; daily is the default. Audits are deliberately coarser than alerts — an agentic investigation scans whole windows and runs for minutes.
• Window: either a fixed rolling lookback (e.g. “each run scans the last 7 days”) or since-last-run (the default) — each run picks up where the previous successful one ended, with a small overlap so boundary events are never missed.
• The next run is scheduled a full interval after the previous one completes, so a slow run never stacks a second concurrent run of the same audit.
• Run now on the audit page makes it due immediately.

​

Model selection

When creating an audit you can pick which model the investigation uses, from the list of models your operator has configured for the agent service. With a single model configured, the picker shows it as a caption; with several, you choose. Leaving it unset uses the configured default.

​

Notifications

When a run surfaces new findings, the audit notifies your organization’s configured channels — the same alerts.enabled_channels gate and settings the alerts pipeline uses:

• Slack — a summary of the significant (big) new items with a deep link.
• Email — a designed audit report listing the new improvements (top severity, per-item recommendations, deep link), sent when the audit has an email channel attached and there is at least one new finding.

Recurring-but-known findings do not re-notify.

​

Configuration reference

Audit definitions are managed in the dashboard (/audits/new) or via the API. Per-audit settings include the schedule cadence and window, the scope ({"environments": [...], "agent_ids": [...]}), the sensitivity (low / medium / high), the notification channels, the per-run findings cap (top_k), and the model (via llm_budget.model). Operator-level server settings (timeouts, sandbox, the agent-service URL) are documented in deployment.md.

​

API

All endpoints are org-scoped and follow the standard bearer-key auth (see api-keys.md). For “audit ran but found nothing”, “the code sandbox is disabled”, and “audit email not delivered”, see troubleshooting.md.