# Alerts
Source: https://docs.befailproof.ai/agenteye/alerts
Find out the moment something crosses your line, on the channel your team already watches, instead of hearing about it from a customer.
Find out the moment something crosses your line, on the channel your team already watches, instead of hearing about it from a customer. Set a rule once and Failproof AI Observability checks it on a schedule, then pages you by email, Slack, webhook, or right in the dashboard.
*Every alert rule at a glance: what it watches, how often, where it pages, and how urgent.*
## Hear about problems before your users do
Stop refreshing a dashboard hoping to catch a regression. Reach for an alert whenever there is a signal you would want to hear about even when nobody is looking, and have it land where you already are:
* **Email**, to whoever should know.
* **Slack**, a rich message with a button that jumps straight to the incident.
* **Webhook**, a JSON POST for PagerDuty, Opsgenie, or your own endpoint, with an optional signature so the receiver can trust it.
* **In-dashboard**, quiet by design, for when you are tuning a rule and do not want to page anyone yet.
Attach any combination to a single rule, and its severity (info, warning, or critical) rides along so the urgent ones look urgent.
## Build the rule in a form, not JSON
You describe what "broken" means in a form, and Failproof AI Observability writes the underlying rule for you. The JSON spec is just what that form produces under the hood, so you can read it to understand a rule but you rarely type it.
*Pick a trigger and the form swaps in the right fields; Save writes the rule.*
The happy path is quick: name it, pick a **trigger** (what to watch), set the **threshold and window** (how bad, over how long), attach at least one **channel**, then **Save** and hit **Test** to fire a synthetic notification and confirm every destination is wired up. Under the hood that produces a small spec like:
```json theme={null}
{ "metric": "p95_latency_ms", "op": ">", "value": 5000, "window_secs": 900 }
```
You are not limited to one kind of signal. Pick the trigger that matches how you think about the failure:
| Trigger | Fires when |
| -------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Metric threshold** | a preset metric (error rate, p95 or p99 latency, event or error counts, token spend) crosses your line over a window |
| **Custom SQL** | your own read-only query returns a row, or a value it computes crosses a threshold |
| **Evaluation score** | an evaluator score's average (say, hallucination) crosses a threshold |
| **Compound eval** | several score checks combine with any, all, or at-least-N logic, to catch a regression that only shows across scores |
| **Per event** | a single matching event lands: a specific agent, a specific error type, or a message substring |
Already staring at a failure on the [Errors page](/agenteye/error-tracking)? Every row there has a **+ alert** button that opens this same form prefilled to catch that exact failure again, so the incident you just triaged becomes the one that pages you next time.
**Where to find it:** Alerts live at `//alerts`. Creating, editing, deleting, and testing rules needs **`alerts:write`**; `alerts:read` is enough to look. The recipient picker lists your org's members by name, so you can page a person without leaving the form.
## Page me only when it is real
One bad measurement should not wake you. The **M of N** noise filter controls how many of the last few checks must fail before the alert actually pages you. Set it to **3 of 5** and the rule fires only after it has breached three of its last five checks, so a jittery signal stops crying wolf; leave it at the default **1 of 1** to fire on the first breach. You also choose how often the rule runs, from presets of 1m, 5m, 15m, and 1h, matched to how fast the signal really moves.
## What happens when an alert fires
A breach opens an **incident** and pages your channels once. From there your team acknowledges it, assigns an owner, talks it through, and resolves it, all against a clean, attributed record. That triage workflow has its own home: see [Incidents](/agenteye/incidents).
## Related
* [Incidents](/agenteye/incidents): track a firing alert from open to acknowledged to resolved.
* [Error tracking](/agenteye/error-tracking): group agent failures and promote one to an alert in a click.
* [Dashboards](/agenteye/dashboards): watch the shared boards the thresholds you alert on come from.
* [CLI and agents](/agenteye/cli-and-agents): create alerts and ack incidents from your terminal, or script them into CI.
# API Keys
Source: https://docs.befailproof.ai/agenteye/api-keys
API keys control who and what can reach your Failproof AI Observability server, so a collector can send events without ever gaining read or admin powers.
API keys control who and what can reach your Failproof AI Observability server, so a collector can send events without ever gaining read or admin powers. Each key carries one or more permissions, and each permission gates specific server routes; you grant only the few a job needs. Most deployments create just three kinds of key.
## The 3 keys most deployments need
| Key | Permissions | Who uses it |
| ------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Collector key | `events:add` | The `agenteye-collector` on each agent machine, to send events. |
| Dashboard read key | `events:read`, `keys:read` | A read-only operator or integration that queries data without changing it. |
| Bootstrap admin key | all permissions | The operator who first brings the instance up (and the dashboard). Seeded from the `ADMIN_KEY` environment variable. See [Bootstrap admin key](#bootstrap-admin-key). |
Start here. Reach for the full permission catalogue below only when you need a narrower, custom-scoped key. See also [Recommended key layout](#recommended-key-layout) and [Creating keys](#creating-keys).
***
## Permissions
The server enforces a fixed catalogue of permissions; each one gates specific HTTP routes. An **admin key** holds all of them; a scoped key holds the subset you grant on creation. Unknown permission strings are rejected when a key is created.
> **Note:** Two valid permissions are human/dashboard-only and cannot be granted to an API key: `orgs:admin` (instance administration, which is operator-only) and `keys:update`. A request to `POST /keys` or `PATCH /keys/:id` that tries to grant either one is rejected with HTTP 422. See the `keys:update` row below for why a bearer key may create keys but never edit them.
### Events ingest & query
| Permission | HTTP routes | What it allows |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events:add` | `POST /events` | Ingest batches of events from a collector. The only permission a collector needs. |
| `events:read` | `GET /events`, `GET /events/latency_aggregate`, `GET /events/environments`, `GET /events/models`, `GET /sessions/:session_id/export` | Query events, list the known environments, list the model identifiers seen in the data (used by the Models view and model filters), compute the latency aggregate that powers the heat-map / percentile band, and export a session as JSONL. The shared filter-bar facet endpoints `GET /events/environments` and `GET /events/agent_ids` are reachable with **either** `events:read` **or** `evaluations:read`, so the sessions page (gated `evaluations:read`) reuses the same per-org facet. `GET /events/models` is not one of them: it requires `events:read`, so a principal holding only `evaluations:read` gets a 403 from it. |
### Sessions & evaluations
| Permission | HTTP routes | What it allows |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `evaluations:read` | `GET /sessions`, `GET /evaluations`, `GET /evaluations/aggregate`, `GET /evaluations/environments`, `GET /evaluation-jobs` | List sessions, read evaluation results, the rolled-up eval health used by dashboards, and the evaluation-job worker queue state. |
| `evaluations:trigger` | `POST /sessions/:session_id/re-evaluate` | Manually enqueue a re-evaluation for a finished session. |
### Dashboards
| Permission | HTTP routes | What it allows |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| `dashboards:read` | `GET /dashboards`, `GET /dashboards/:id`, `GET /dashboards/:id/tiles` | List dashboards, load one, and read its tiles. |
| `dashboards:write` | `POST /dashboards`, `PUT /dashboards/:id`, `POST /dashboards/:id/tiles`, `PUT /dashboards/:id/tiles/:tile_id`, `DELETE /dashboards/:id/tiles/:tile_id`, `PUT /dashboards/:id/tiles/layout` | Create and edit dashboards, add / edit / remove tiles, and reorder the tile grid. |
| `dashboards:delete` | `DELETE /dashboards/:id` | Delete an entire dashboard (tile-level deletion lives under `dashboards:write`). |
### Saved queries (SQL composer)
| Permission | HTTP routes | What it allows |
| ---------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `queries:read` | `GET /queries`, `GET /queries/:id`, `GET /queries/schema` | List saved queries, load one, and inspect the read-only schema the composer targets. |
| `queries:write` | `POST /queries`, `PUT /queries/:id` | Create and edit saved queries. SQL is still routed through the same read-only role and guarded SQL checks as a `queries:run` call. |
| `queries:delete` | `DELETE /queries/:id` | Delete a saved query. |
| `queries:run` | `POST /queries/run` | Execute saved or ad-hoc SQL against the read-only role used by the composer. |
### AI assistant
| Permission | HTTP routes | What it allows |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent:use` | `GET /agent/conversations`, `POST /agent/conversations`, `GET /agent/conversations/:id`, `PATCH /agent/conversations/:id`, `DELETE /agent/conversations/:id`, `PUT /agent/conversations/:id/messages` | Talk to the AI assistant and manage your own (private) conversations. Required on the **user** to see the assistant dock; the assistant's own key is `dashboard-assistant` and is seeded separately (see below). |
### API keys
| Permission | HTTP routes | What it allows |
| ----------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `keys:create` | `POST /keys` | Create a new scoped API key. Does **not** grant editing an existing key's permissions (that is `keys:update`). |
| `keys:read` | `GET /keys` | List existing keys. Secrets are never returned by this endpoint. |
| `keys:update` | `PATCH /keys/:id` | Edit an existing key's permissions. A **human/dashboard-only** permission; it cannot be assigned to an API key (a bearer key may create keys but never edit them). |
| `keys:disable` | `POST /keys/:id/disable` | Revoke a key. Protected keys (`admin`, `dashboard-assistant`) can't be disabled; rotate them via env var + restart. |
| `keys:regenerate` | `POST /keys/:id/regenerate` | Rotate a key's secret. Protected keys can't be regenerated through this route. |
### Dashboard users
| Permission | HTTP routes | What it allows |
| -------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `users:create` | `POST /users`, `GET /users/defaults` | Invite a new dashboard user (issues an email + one-time passcode (OTP) login) and read the dashboard-configured default permission set used to seed the invite form. |
| `users:read` | `GET /users`, `GET /users/:id` | List users and load a single user record. |
| `users:update` | `PUT /users/:id` | Edit a user's permissions. Updates dispatch a permission-change email to the affected user and take effect on their next request; no relogin required. |
| `users:delete` | `DELETE /users/:id`, `POST /users/:id/enable` | Disable a user (revokes their sessions immediately) and re-enable a previously disabled user. |
These permissions back the dashboard's **Users** page, where each member's granted scopes are shown as chips:
### Operational settings
| Permission | HTTP routes | What it allows |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `settings:read` | `GET /settings`, `GET /settings/schema`, `GET /settings/model-context-windows`, `GET /settings/model-context-windows/resolve` | View dashboard-managed operational settings and their metadata; list per-model context-window overrides; and resolve the effective window for a model. |
| `settings:write` | `PUT /settings/:key`, `PUT /settings/model-context-windows`, `DELETE /settings/model-context-windows` | Edit operational settings and add, change, or remove per-model context-window overrides. Changes affect new events without restarting the server. |
### Alerts & incidents
| Permission | HTTP routes | What it allows |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- |
| `alerts:read` | `GET /alerts`, `GET /alerts/:id` | View configured alert definitions. |
| `alerts:write` | `POST /alerts`, `PUT /alerts/:id`, `DELETE /alerts/:id`, `POST /alerts/:id/test` | Create, edit, delete, and test-fire alert definitions. |
| `incidents:read` | `GET /alerts/incidents`, `GET /alerts/incidents/:iid`, `GET /alerts/incidents/:iid/comments`, `GET /alerts/incidents/:iid/subscribers` | View incidents and their triage trail. |
| `incidents:write` | `POST /alerts/:id/incidents` | Open an incident manually against an existing alert. |
| `incidents:ack` | `POST /alerts/incidents/:iid/ack`, `POST /alerts/incidents/:iid/assign`, `POST /alerts/incidents/:iid/resolve`, `POST /alerts/incidents/:iid/comments`, `POST /alerts/incidents/:iid/subscribe`, `POST /alerts/incidents/:iid/unsubscribe` | Acknowledge, assign, resolve, and comment on incidents. |
### Audits
| Permission | HTTP routes | What it allows |
| -------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `audits:read` | `GET /audits`, `GET /audits/:id`, `GET /audits/:id/runs`, `GET /audits/findings`, `GET /audits/findings/:fid` | View audit definitions, run history, and findings. |
| `audits:write` | `POST /audits`, `PUT /audits/:id`, `DELETE /audits/:id`, `POST /audits/:id/run`, `POST /audits/findings/:fid/status` | Create, edit, delete, and run audits; triage findings (acknowledge / mute / dismiss / resolve / reopen / assign). |
> **Note:** To give a key the audit surface, grant `audits:*` to it explicitly. See [Upgrade and backward-compatibility notes](#upgrade-and-backward-compatibility-notes) for how existing grantees were migrated when Audits shipped.
> The recipient-picker endpoint `GET /alerts/recipients` (which lists the member emails an alert editor can notify) is reachable by a holder of **either** `alerts:read` **or** `alerts:write`, so alert editors can populate the picker without being granted `users:read`.
> A dashboards viewer needs **both** `dashboards:read` (to load the saved views) and `evaluations:read` (the health metrics are computed from evaluation data). Grant `dashboards:write` to let a user create or edit dashboards, and `dashboards:delete` to remove them.
> `/health` and `/auth/*` (OTP request, OTP verify, session check, logout) are unauthenticated by design; they're the login flow and liveness probe. `GET /access-granters` requires a valid key but no specific permission, so any logged-in user can see which admins to contact about access changes.
***
## Permission Sets
Permission sets let you apply a named role instead of hand-picking individual tokens every time. Rather than selecting a dozen permissions one by one for each new dashboard user or API key, you choose a set, and everyone assigned to it carries a consistent, reviewable grant. Editing a custom set re-applies the new grant to every user already assigned to it, so a role change is one edit rather than a sweep through every member.
Every organization is seeded with three built-in sets:
| Set | Permissions | Intended for |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `read-only` | `events:read`, `keys:read`, `users:read`, `evaluations:read`, `dashboards:read`, `queries:read`, `settings:read`, `alerts:read`, `audits:read`, `incidents:read` | View-only access across every operational surface. |
| `standard` | everything in `read-only`, plus `evaluations:trigger`, `queries:run`, `incidents:ack`, `agent:use` | Read-only plus the everyday on-caller actions: run queries, re-evaluate sessions, acknowledge incidents, and use the AI assistant. |
| `admin` | every assignable permission | Full control of the org. |
The three built-in sets are **immutable**; their names always mean the same thing, so `read-only`, `standard`, and `admin` are safe to reference in policy and onboarding. An operator can create additional **custom sets** to model roles specific to your organization (for example, a "dashboard author" role or a "collector-only" role).
Sets are surfaced in the dashboard and managed over the API at `GET /permission-sets` (list, gated by `users:read`) and `POST /permission-sets` / `PUT /permission-sets/:name` / `DELETE /permission-sets/:name` (create, edit, delete a custom set, gated by `settings:write`). Deleting or editing a built-in set is refused.
Set membership is what backs two other features:
* **`DEFAULT_USER_PERMISSIONS`** (the grant preselected when an admin opens **+ new user**) defaults to the `standard` set.
* **The `--set` flag** on `agenteye-orgctl` (operator member management) starts a member from a named set, which you then fine-tune with `--add` / `--remove`.
> **Note:** When a set includes a permission that is not key-assignable (for example a custom set carrying `keys:update`), seeding a key from that set drops the non-assignable tokens; the server would otherwise reject the key with HTTP 422. Dashboard users are not subject to that restriction.
***
## Bootstrap Admin Key
The admin key is the single root credential that lets an operator bring up access from nothing: with it you can mint every other scoped key, invite the first dashboard users, and configure the instance before any other key exists. It is the one key you do not create through the keys API; it is provisioned from the environment so the server is reachable on first boot.
Set the `ADMIN_KEY` environment variable on the server. On every startup the server upserts this value as an admin key with all permissions.
To rotate: change `ADMIN_KEY` to a new secret and restart the server.
***
## Organization scoping
**Organizations themselves are created and managed out-of-band by an operator, not through this keys API.** Org and member lifecycle (create / rename / delete / purge an org; add / update / remove a member) is done with the **`agenteye-orgctl`** CLI; there is no HTTP API or dashboard button for it. What *is* unchanged: **per-org API keys are still minted in the dashboard (or via this keys API)** by org members.
In a multi-org deployment, every key an org member creates (through this keys API or the dashboard **Keys** page) belongs to **one organization** and can only ever read or write that org's data; the org is stamped on the key at creation and enforced on every request. The two bootstrap keys are the only exception: the `admin` key (seeded from `ADMIN_KEY`) and the `dashboard-assistant` key (seeded from `AGENT_API_KEY`) are **instance-scoped** (they carry no org). The dashboard authenticates with the `admin` key so it can proxy per-org requests on behalf of signed-in members. Single-tenant deployments need not think about this; all keys belong to the built-in `default` org.
***
## Creating Keys
Use the admin key (or any key with `keys:create` permission) to create additional scoped keys.
### Collector key (ingest only)
```bash theme={null}
curl -s -X POST http://your-server/keys \
-H "Authorization: Bearer $ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "prod-collector",
"key": "your-collector-secret",
"permissions": ["events:add"]
}'
```
### Dashboard key (read only)
```bash theme={null}
curl -s -X POST http://your-server/keys \
-H "Authorization: Bearer $ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "dashboard",
"key": "your-dashboard-secret",
"permissions": ["events:read", "keys:read"]
}'
```
When you create a key over the HTTP API, you provide the `key` value yourself; choose a strong secret and store it securely. (The dashboard works the other way: it generates a strong secret for you and shows it once at creation; see [Key Management in the Dashboard](#key-management-in-the-dashboard).) The response confirms the key was created:
```json theme={null}
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "prod-collector",
"permissions": ["events:add"],
"created_at": "2026-04-01T12:00:00Z"
}
```
***
## Listing Keys
```bash theme={null}
curl -s http://your-server/keys \
-H "Authorization: Bearer $ADMIN_KEY"
```
Key secrets are not returned in list responses, only IDs, names, and permissions.
***
## Disabling a Key
Disabling revokes access immediately without deleting the key record.
```bash theme={null}
curl -s -X POST http://your-server/keys//disable \
-H "Authorization: Bearer $ADMIN_KEY"
```
***
## Regenerating a Key
Generates a new secret for an existing key. The old secret is invalidated immediately.
```bash theme={null}
curl -s -X POST http://your-server/keys//regenerate \
-H "Authorization: Bearer $ADMIN_KEY"
```
The response includes the new plaintext secret, **shown only once**.
***
## Key Management in the Dashboard
The **Keys** page in the dashboard provides a UI for all of the above operations. You need a key with `keys:read` permission to view the list, and `keys:create` / `keys:update` / `keys:disable` / `keys:regenerate` for the create / edit / disable / regenerate actions respectively. Editing a key's permissions (`keys:update`) is separate from creating one (`keys:create`), so you can grant an operator the ability to mint keys without the ability to re-scope existing ones, or vice versa. The admin key covers all of these.
When you create a key from the dashboard you do not supply the secret; the dashboard generates a strong secret for you and displays it **once** at creation. Copy it immediately and store it securely; it is never shown again, exactly as with a regenerate. You can still pick the key's permissions directly, or seed them from a permission set (see below).
***
## Recommended Key Layout
| Key | Permissions | Used by |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `admin` (bootstrap via `ADMIN_KEY` env var) | all | Ops/setup, and the dashboard (authenticates with `ADMIN_KEY`, proxies user requests with permission checks) |
| Per-host collector key | `events:add` | Collector on each agent machine |
| `dashboard-assistant` (bootstrap via `AGENT_API_KEY` env var) | `events:read`, `evaluations:read`, `dashboards:read`, `dashboards:write`, `queries:read`, `queries:write`, `queries:run` | AI assistant, seeded automatically, **protected**; can't be edited through the API |
| Assistant telemetry key (optional) | `events:add` | AI assistant self-instrumentation, if enabled |
> **Note:** The assistant's key is **seeded automatically** by the server from the `AGENT_API_KEY` env var (the same secret the agent presents as `AGENTEYE_API_KEY`); there is no manual key-minting step and no admin key involved. Its permissions are fixed in source code so scope can't be widened by misconfiguration: read across events / evaluations / dashboards, plus dashboards-write and queries-read / write / run for the "Ask AI to write a query" authoring flow. All SQL still goes through the same read-only role and guarded SQL path as a user-written query, so this widens the *authoring surface*, not the data surface; destructive operations (`queries:delete`, `dashboards:delete`) deliberately stay off the assistant key. Like the `admin` key, it is **protected**: it can't be disabled or regenerated through the keys API, only rotated by changing `AGENT_API_KEY` and restarting. Dashboard *users* additionally need the `agent:use` permission to see and use the assistant. If you enable self-instrumentation, give the assistant a separate `events:add`-only key.
***
## Upgrade and backward-compatibility notes
You only need these if you are upgrading an existing instance; new deployments can skip them.
> When Audits shipped, existing grantees were widened along the same role shapes as alerts: every user and permission set holding `alerts:read` gained `audits:read`, and every holder of `alerts:write` gained `audits:write`. Existing API keys were **not** widened. Grant `audits:*` to a key explicitly if it needs the audit surface.
> Stored grants of the legacy `alerts:ack` token are parsed as `incidents:ack` so on-callers retain access without rekeying. The token is no longer assignable from the dashboard's user editor; the matrix offers `incidents:ack` instead.
***
## Next steps
* [Python SDK](/agenteye/python-sdk): how your agent code authenticates when sending events.
* [Security](/agenteye/security): how sign-in, access control, and per-organization data isolation work.
# AI Assistant
Source: https://docs.befailproof.ai/agenteye/assistant
Ask your agent data a question in plain English and get an answer that links straight to the evidence.
Ask your agent data a question in plain English and get an answer that links straight to the evidence. No SQL to write, no dashboards to dig through — the **Failproof AI Observability** assistant is the fastest way for anyone on your team to get answers about your agents.
*Ask in plain English and get an answer built from your own data. Here it breaks down which agents are busiest and which models they use, and shows the queries it ran so you can verify every number.*
There is nothing to learn. Open the chat, type what you want to know, and follow the links it hands back:
```
You: which sessions errored today?
AI: 5 sessions errored today, newest first. Each one is linked:
• checkout-agent 14:02 tool timeout
• billing-agent 11:47 unhandled error
• ...and 3 more
You: summarize this session (asked while viewing a run)
AI: This run took 12 steps across 3 tools and failed near the end when a
payment tool returned an error. It scored low on your "resolved" eval.
Links: the session, the failing event, and that evaluation.
```
## Just ask, and jump straight to the proof
You stop guessing and you stop writing queries. Ask "how is quality trending in prod this week?", "which sessions errored today?", or "summarize this session," and you get a straight answer in seconds instead of building a query and reading it yourself.
Every answer comes with its receipts. The assistant links the exact sessions, saved queries, and dashboards it used to reach the answer, so you can click through and confirm rather than take its word for it. It is also **page-aware**: ask about "this session" while you are viewing one and it already knows which run you mean. Reopen any earlier conversation later from the history switcher and pick up where you left off.
## Turn a good answer into a saved query or dashboard
When an answer is worth keeping, ask the assistant to save it. It drafts the SQL for a saved query, or assembles a dashboard from those queries, then shows you an **Approve / Reject** card. Nothing is written until you click Approve, so you get the speed of "just ask" with the last word always yours.
On the **Queries** page it goes a step further and becomes a SQL author: describe the query you want ("show error rate by agent for the last 7 days") and it streams SQL straight into the editor, opening a diff view so you can **Accept** or **Reject** the change before it lands.
*The Queries page: this editor is where the assistant streams a draft, read-only query for you to accept or reject.*
Authoring SQL by asking here uses the `queries:run` permission, the same one behind the editor's **Run** button. Chat everywhere else needs `agent:use`.
## Safe to hand to the whole team
You can open the assistant up to everyone without worrying about what it might touch:
* **It reads only what you can already see.** Answers are scoped to your own read permissions, so it never widens your data surface.
* **Every write waits for you.** Saved queries and dashboards are created only after your explicit Approve click, and there is no setting that turns that gate off.
* **It can never delete anything.** No delete tool is exposed and the assistant holds no delete permission. Deletions stay in your hands, in the dashboard.
* **It stays inside your org.** The assistant only ever sees the organization you are currently viewing.
* **Your questions stay yours.** Prompts and answers live in your own Observability database; product analytics records usage metadata only, never your prompt text.
## Where to find it
The assistant rides along on the right edge of every page under your org (`//...`). Click the rail, or press `⌘J` / `Ctrl+J`, to expand it into the full chat panel, and drag its edge to resize; your width is remembered across reloads. You need the **`agent:use`** permission to use it, otherwise the rail is greyed out. If it has not been switched on for your deployment yet (it needs an LLM connection), you will see a muted rail in place of a working chat.
## Related
* [CLI and agents](/agenteye/cli-and-agents)
* [Queries](/agenteye/queries)
* [Dashboards](/agenteye/dashboards)
* [Evaluation suite](/agenteye/evaluation-suite)
# Audits: your automatic reliability analyst
Source: https://docs.befailproof.ai/agenteye/audits
Failproof AI Observability goes looking for the failures you never wrote a rule for and hands you a ranked, evidence-backed to-do list of exactly what to fix.
Failproof AI Observability goes looking for the failures you never wrote a rule for and hands you a ranked, evidence-backed to-do list of exactly what to fix. It is like having an analyst comb your logs every night, then leaving the short list on your desk by morning.
*A two-minute tour: from a scheduled run to a fix you can act on.*
*Each audit is a recurring job that mines your sessions and writes up ranked, evidence-backed recommendations.*
## Stop guessing what to fix next
Alerts catch the problems you already know to watch for. Audits catch the ones you don't. On a schedule you set, an audit reads across all of your agent sessions and hunts for the patterns worth fixing, so you spend your time acting on findings instead of scrolling logs hoping to spot them yourself.
A single run goes after the failure modes that actually break agents in production:
* **Error clusters**: the same failure repeating under a shared root cause.
* **Drift versus a baseline**: behaviour quietly sliding away from a known-good window.
* **Goal failure in transcripts**: runs that technically finished but never did the job.
* **Tool misuse**: the wrong tool, bad arguments, or loops that burn calls.
* **Quality and cost trade-offs**: where you are overpaying for output you could get cheaper.
* **Coverage gaps**: behaviour that no eval or alert is watching.
You decide how hard it looks with a single **sensitivity** setting (low, medium, or high), so a noisy staging agent and a locked-down production one can each be tuned to the signal you want.
## Every recommendation comes with receipts
You never have to take a finding on faith. Each recommendation cites the exact sessions it came from and the SQL that surfaced it, so you can open the evidence and confirm the problem in a click instead of reverse-engineering a claim.
That is also what keeps audits honest. The server checks that every cited session actually exists and **discards any recommendation whose evidence does not hold up**, so the audit investigates but never invents. What lands on your list is real, reproducible, and ranked by how much it matters, with the biggest wins at the top.
## Turn a fix into a guardrail
Fixing an issue is only half the win. The other half is making sure it cannot quietly come back. Every finding carries a **one-click shortcut that drafts a recurrence alert**, prefilled with a sensible starting trigger you can tune. Close the finding, arm the alert, and the next time that pattern reappears you get paged instead of rediscovering it in a future audit.
## Where to find it
Audits live in the dashboard at **`//audits`** (sidebar to *analyze* to *audits*). Viewing runs and findings needs **`audits:read`**; creating, editing, and triaging audits needs **`audits:write`**. Set an audit's scope and cadence, then hit **Run now** whenever you want results immediately instead of waiting for the next scheduled pass.
## Related
* [Alerts](/agenteye/alerts): get paged the moment a threshold you already know about is crossed.
* [Evaluations](/agenteye/evaluations): score every run so quality regressions surface on their own.
* [Error tracking](/agenteye/error-tracking): group and follow the errors your agents throw.
* [Incidents](/agenteye/incidents): track an issue an audit turns up through to its fix.
# CLI
Source: https://docs.befailproof.ai/agenteye/cli
Drive all of Failproof AI Observability from the terminal or a script: no dashboard round-trips.
Drive all of Failproof AI Observability from the terminal or a script: no dashboard round-trips. The `agenteye` CLI queries your data (sessions, event logs, evaluations) and administers your org (API keys, users, settings, alerts, incidents, saved queries), so reach for it when you want to automate a check, wire Observability into CI, or let a coding agent inspect production. Every command supports a `--json` flag, so it works equally well for you at a prompt or for a coding agent (Claude Code, Cursor) shelling out and parsing the result.
With one binary you can:
* **Read your data**: `sessions`, `events`, `evals`, `errors` (filter by time, agent, env, score).
* **Manage your org**: `keys`, `users`, `settings`, `alerts`, `incidents`.
* **Run analytics**: saved SQL and an ad-hoc query runner (`query`).
* **Ask the AI assistant**: the same read-only analyst you chat with in the dashboard (`agent`).
> **Note:** This is the `agenteye` CLI, a different tool from the collector daemon (`agenteye-collector`). The CLI talks to your dashboard; the collector ships events to the server.
***
## Quickstart
From nothing to your first result in four lines. Point the CLI at your dashboard, sign in, confirm who you are, then pull the last day of runs:
```bash theme={null}
pipx install agenteye
agenteye --base-url https://agenteye.example.com login --email you@example.com # emailed 6-digit code
agenteye whoami # confirm user + active org
agenteye --json sessions --since 24h # one row per agent run, last 24h
```
That last command prints a JSON object of the most recent sessions (newest first, capped at 50 by default). Pipe it into `jq` to slice it, or drop `--json` for a boxed, colourised table. Each row carries the run's status and, if an evaluator scored it, its metric scores (abbreviated here):
```json theme={null}
{
"sessions": [
{
"session_id": "run-8f2a",
"agent_id": "checkout-bot",
"environment": "prod",
"status": "error",
"scores": { "helpfulness": 0.42, "tool_efficiency": 0.55 },
"event_count": 37,
"started_at": "2026-07-16T09:14:02Z",
"last_event_at": "2026-07-16T09:14:48Z"
}
],
"next_cursor": null
}
```
The rest of this page explains each piece: [installing](#installation) in isolation, [signing in](#authentication), [configuration](#configuration), the [global conventions](#global-options--conventions) every command shares, and the [full command reference](#command-reference).
***
## Installation
The CLI is a public PyPI package named **`agenteye`**. Install it in an isolated environment so it always has its own dependencies:
```bash theme={null}
pipx install agenteye
# or
uv tool install agenteye
```
It requires Python 3.10+. The installed command is **`agenteye`**:
```bash theme={null}
agenteye --version
agenteye --help
```
> **Note:** The Failproof AI Observability Python SDK also uses the `agenteye` distribution name. Installing the CLI with `pipx` or `uv tool` (rather than `pip install` into a shared virtualenv) keeps the two from colliding. A plain `pip install agenteye` is fine only if the SDK is not installed in the same environment.
***
## Authentication
The CLI authenticates to the **dashboard** with an emailed one-time code:
```bash theme={null}
agenteye login --email you@example.com
# A 6-digit code is emailed to you; paste it at the prompt.
```
The session token is stored in `~/.agenteye/cli.json` (readable only by you, mode `0600`) and is valid for 24 hours by default. When it expires, run `agenteye login` again.
```bash theme={null}
agenteye whoami # show the current user, active org, and permissions
agenteye logout # revoke the session and clear the stored token
```
`whoami` never errors on a missing or expired session; it reports `logged_in: false` instead, so a script or agent can probe auth state safely (it can still exit non-zero if no base URL is set or the dashboard is unreachable).
**Requirements:** your email must be permitted to sign in to the dashboard (ask your Failproof AI Observability administrator), and the dashboard must be reachable at its base URL (see [Configuration](#configuration)). If you request a code and none arrives, your email is likely not yet enabled for dashboard access.
***
## Choosing your org (multi-tenant)
If your account belongs to more than one org, choose the active one **at login**; it is saved and used for every later command:
```bash theme={null}
agenteye login --org acme # authenticate and set the active tenant in one step
agenteye orgs list # the orgs you can access (the active one is marked)
agenteye orgs switch globex # change the saved default
agenteye --org globex sessions # override for a single command
```
If you belong to exactly one org it is selected automatically and you can ignore `--org` entirely. If you belong to several and don't pick one, the CLI lists them and asks you to re-run with `--org `. The active org is sent to the dashboard on every request, and your permissions are resolved **per org**; `agenteye whoami` shows the active org, your permissions in it, and all your memberships.
***
## Configuration
| Setting | Flag | Environment variable | Default |
| ------------------------- | ------------------------- | ------------------------------------------------- | ------------------------------------------------ |
| Dashboard base URL | `--base-url` | `AGENTEYE_DASHBOARD_URL` | **required** (no default) |
| Active org/tenant | `--org` | `AGENTEYE_ORG` | chosen at login; saved in `~/.agenteye/cli.json` |
| Session token | `--token` | `AGENTEYE_CLI_TOKEN` | from `~/.agenteye/cli.json` |
| JSON output | `--json` | `AGENTEYE_CLI_JSON` | off |
| Skip TLS verification | `--insecure` / `--secure` | `AGENTEYE_INSECURE` | off (saved at login) |
| Request timeout (seconds) | `--timeout` | *(none)* | 30 |
| Disable usage telemetry | *(none)* | `AGENTEYE_ANALYTICS_DISABLED` (or `DO_NOT_TRACK`) | telemetry is currently disabled; nothing is sent |
Resolution order is **flag → environment variable → config file**. There is no default; you must point the CLI at your dashboard, either per-command (`--base-url https://agenteye.example.com`) or once via the environment (it's also saved after your first `login`):
```bash theme={null}
export AGENTEYE_DASHBOARD_URL=https://agenteye.example.com
```
The configuration directory honours `AGENTEYE_HOME` (the same convention used by the SDK and collector); if set, `cli.json` lives in `$AGENTEYE_HOME/cli.json`.
### Self-signed or internal TLS
If your dashboard is served over HTTPS with a self-signed or internal certificate (for example, a raw load-balancer hostname), TLS verification rejects it with a `CERTIFICATE_VERIFY_FAILED` error. Pass `--insecure` to skip certificate verification:
```bash theme={null}
agenteye --base-url https://agenteye.internal --insecure login
```
`--insecure` is **saved to `cli.json` when you log in**, so later commands skip verification automatically; you don't have to repeat the flag. Pass `--secure` for a one-off verified call, or to save verification back on at your next login. The CLI prints a warning to stderr before any command that contacts the dashboard while verification is disabled. Skipping verification removes protection against man-in-the-middle attacks; ensure you trust the network path to your dashboard (VPN, private subnet, etc.) before relying on it.
***
## Telemetry & privacy
> **Note:** The shipped CLI sends **no usage telemetry today.** A master kill switch is on, so nothing is transmitted regardless of your environment. The section below describes the opt-out capability for if and when telemetry is ever enabled.
Even when enabled, telemetry would be **anonymous usage analytics only**, never your agent, session, or event data:
* **No agent, session, or event data ever leaves your infrastructure.** Only CLI usage would be reported: the command and subcommand name (e.g. `keys create`), the **names** of the flags you used (never their values), success/exit status, and duration, plus a per-action event for mutations (e.g. `api_key_created`, `query_run`) carrying only static names/enums and coarse counts. Your dashboard URL, session token, email, org slug, resource ids, SQL, key secrets, and query filters would **never** be sent. Operators would be identified only by an opaque internal id, never by email.
* **Opt out ahead of time** by setting `AGENTEYE_ANALYTICS_DISABLED=1` in the CLI's environment (the CLI also honours the cross-tool `DO_NOT_TRACK=1` convention). This takes effect the moment telemetry is ever turned on, so a privacy-conscious environment can stay opted out permanently.
* If telemetry were enabled, the CLI would send directly to PostHog (`https://us.i.posthog.com`); a machine with that host blocked would silently send nothing and the CLI would be unaffected.
***
## Global options & conventions
Read this once; it applies to every command.
* **Global options go BEFORE the command.** `agenteye --json sessions` is correct; `agenteye sessions --json` is a usage error. The globals are `--json`, `--base-url`, `--org`, `--token`, `--insecure`/`--secure`, `--timeout`, `--quiet`, and `--no-color`.
* **`--json` prints pure JSON to stdout, and nothing else.** Human status lines, warnings, and errors go to **stderr**, so a `--json` stdout capture stays clean to pipe into `jq` even when a status line is shown. Without `--json` you get a boxed, colourised view for human eyes.
* **Discover with `--help`.** Every command and subcommand has `--help` (and the `-h` alias): `agenteye -h`, `agenteye sessions -h`, `agenteye keys create -h`. The top-level help also lists the exit codes and global options. There is no global machine-readable surface dump; use per-command `--help`, plus the domain-specific `agenteye query schema` and `agenteye settings schema` for those two registries.
* **Confirmations auto-skip for scripts and agents.** Create/update/delete commands prompt "are you sure?" in an interactive terminal, but **auto-skip that prompt under `--json` or whenever stdin is not a TTY** (a TTY is an interactive terminal session; a pipe or a CI runner is not), so scripts and agents never hang. Pass `--yes`/`-y` to skip it explicitly. Because the prompt won't fire for an agent, an agent should confirm destructive actions with the human first.
* **Pagination:** results are newest-first and cursor-paginated (each page returns a token you use to fetch the next). `--limit N` (alias `-n`) caps rows and **defaults to 50**; `--all` auto-paginates (in 200-row chunks) **up to `--limit`**, so a bare `--all` still stops at 50. For a full sweep pass a high explicit cap: `--all --limit 1000`. `--page-size N` controls the per-request chunk (max 200); `--cursor ` resumes from a prior page's `next_cursor`.
* **Time filters:** `--since` takes a relative window: `15m`, `1h`, `6h`, `24h`, `7d`, or `all` (the dashboard's presets). For a longer or custom range (say the last 30 days), use `--from`/`--to`: explicit ISO-8601 UTC timestamps **with `T` and a timezone** (e.g. `2026-06-01T00:00:00Z`) that override `--since`. A space-separated or timezone-less value is a usage error.
* **`--fields a,b,c`** (on `events`, `sessions`, `evals`, `errors`) restricts the output to those keys, for both the table and `--json`. Unknown names are rejected with the valid list, a cheap way to discover field names.
* **`--file payload.json`** (or `--file -` to read stdin) supplies a full JSON request body where a resource has a complex shape (on `alerts create/update`, `settings set`, and `users create/update`). Saved-query SQL uses `--sql @file.sql` instead.
* **Multi-value filters** are comma-separated → matched as a set (union within one filter, AND across filters): `--event-type tool_use,tool_result`. Click options are not variadic, so `--add a b` breaks. Use `--add a,b`, repeat the flag (`--add a --add b`), or quote (`--add "a b"`).
***
## Command reference
### You'll use these 5 commands most
Most day-to-day work runs through a handful of read commands. Start here, then reach for the full surface below when you need it:
| Command | What it does | Try it |
| ---------- | ---------------------------------------------------------------- | --------------------------------------------------------- |
| `sessions` | One row per agent run: time, env, agent, status, latest score. | `agenteye --json sessions --since 24h --status error` |
| `events` | The raw per-step trail inside a run (add `--full` for payloads). | `agenteye --json events --session-id run-001 --all` |
| `evals` | Evaluation results and scores; `--aggregate` rolls them up. | `agenteye --json evals --aggregate --since 7d --env prod` |
| `errors` | Just the errored events; `--aggregate` for counts by type. | `agenteye --json errors --since 24h --aggregate` |
| `list` | Discover the valid filter values (agents, envs, models, …). | `agenteye list agents` |
### Everything the CLI can do
The full surface follows. The CLI has **18 top-level commands**. All read commands accept `--json` and the global options above; run `agenteye -h` (or ` -h`) for the exhaustive flag list and JSON shape of any one.
### Identity: `login` · `logout` · `whoami` · `orgs` · `version` · `help`
```bash theme={null}
agenteye login --email you@example.com [--org acme] # emailed one-time code; saves the session
agenteye logout # clear the saved session on this machine
agenteye whoami # current user, active org, permissions
agenteye version # print the CLI version (same as --version)
agenteye help # top-level help (same as --help)
```
`orgs` inspects and switches the active tenant:
```bash theme={null}
agenteye orgs list # your orgs + your role in each (active one marked)
agenteye orgs switch acme # change the saved active org (omit the slug to pick from a list on a TTY)
agenteye orgs current # identity card for the active org
agenteye orgs perms # your permissions in the active org, grouped by resource
```
### Observe (read-only): `events` · `sessions` · `evals` · `errors` · `list`
None of these need a confirmation. Shared filters: `--session-id`, `--agent-id`, `--env` (**not** `--environment`), and the time range (`--since` / `--from` / `--to`).
```bash theme={null}
# events (alias: the raw per-step trail), newest first
agenteye --json events --session-id run-001 --event-type tool_use,tool_result --all --limit 1000
agenteye --json events --since 1h --search timeout --all | jq '.events[].payload'
# sessions: one row per agent run (time/env/agent/session/status; no score filtering)
agenteye --json sessions --since 24h --status error
agenteye --json sessions --agent-id checkout-bot --env prod --all --limit 1000
# evals: evaluation results + scores; --score filters by metric, --aggregate rolls up
agenteye --json evals --score helpfulness:0.5..0.8 --score tool_efficiency:..0.3
agenteye --json evals --aggregate --since 7d --env prod # status mix + per-key score stats
# errors: errored events; --aggregate for counts/sessions/agents/last-seen
agenteye --json errors --since 24h --aggregate
agenteye --json errors --since 24h --error-type timeout --all --limit 1000
# list: discover valid filter values before you filter
agenteye list envs # also: agents event_types score_filters models hooks tools error_types
```
`--score KEY:MIN..MAX` (on **`evals`**, not `sessions`) is repeatable and AND-combined; either bound is optional (`..0.5` means ≤ 0.5, `0.9..` means ≥ 0.9). Up to 20 score filters per request. `evals --scores-full` is a display flag for the **human table only**; it shows every score pair instead of the first few plus a `+N` count. It has no effect under `--json`, which always returns the complete score object. To read **one session end-to-end**, combine the event trail with its evaluation:
```bash theme={null}
agenteye --json events --session-id run-001 --all --limit 1000 | jq '.events[] | {ts, event_type}'
agenteye --json evals --session-id run-001 # its scores + status
```
### Manage (permission-gated): `keys` · `users` · `settings` · `alerts` · `incidents`
**`keys`**: API keys. The secret is generated locally, sent to the server (which stores only a hash), and **shown once** on create/regenerate; capture it then. With `--json` it appears only in the `key` field. Referenced by **name**.
```bash theme={null}
agenteye keys list # active keys first, then revoked
agenteye keys show ci-bot
agenteye keys create ci-bot --add events:read.add # scope to what you need; prints the secret ONCE
agenteye keys create ops --permission-set standard --remove queries:run # seed a preset, then trim
agenteye keys update ci-bot --add evaluations:read --yes
agenteye keys regenerate ci-bot --yes # rotate the secret (the old one stops working)
agenteye keys disable ci-bot --yes # revoke
```
Permissions work as `(permission-set ∪ --add) − --remove`. Tokens are `slug:action` (e.g. `events:read`) or `slug:action.action` to expand several on one resource (`events:read.add` → `events:read`, `events:add`). Presets: `read-only`, `standard`, `admin`. Human-only permissions (`keys:update`) can't be granted to a key.
**`users`**: org members, referenced by **email** (a UUID id is also accepted).
```bash theme={null}
agenteye users list [--active-only]
agenteye users show dev@corp.com
agenteye users create dev@corp.com --permission-set standard
agenteye users update dev@corp.com --add alerts:write --remove queries:delete # predicts + confirms
agenteye users disable dev@corp.com --yes # has protected/self guards
agenteye users enable dev@corp.com
```
**`settings`**: a fixed registry (you read and change existing keys; you cannot create new ones).
```bash theme={null}
agenteye settings list # key · value · type · updated (secrets masked)
agenteye settings schema # what each key accepts (type · range · description)
agenteye settings set session_ttl_secs --value 86400 --yes
```
**`alerts`**: alert definitions, referenced by **name**. `create` takes a positional NAME plus flags or a full JSON body via `--file`.
```bash theme={null}
agenteye alerts list
agenteye alerts show high-errors
agenteye alerts create high-errors --file alert.json # NAME is required (positional)
agenteye alerts update high-errors --severity critical --yes
agenteye alerts test high-errors --yes # fire a test notification
agenteye alerts delete high-errors --yes
```
**`incidents`**: alert incidents, referenced by id (short ids accepted). `show` prints the full activity log; read it before acting.
```bash theme={null}
agenteye incidents list --state firing # also: acknowledged, resolved
agenteye incidents count
agenteye incidents show
agenteye incidents ack
agenteye incidents assign you@corp.com # assignee must be an operator
agenteye incidents resolve --yes
agenteye incidents open --alert-id --severity critical # open one manually against an alert
agenteye incidents comment-add "root cause: upstream 5xx"
agenteye incidents comment-list ; agenteye incidents comment-delete
agenteye incidents subscribe ; agenteye incidents unsubscribe ; agenteye incidents subscribers
```
### Analytics & assistant: `query` · `agent`
**`query`**: saved SQL against your analytics store plus an ad-hoc runner. Saved queries are referenced by **name**; the SQL is validated server-side (SELECT/WITH only, statement timeout, row cap).
```bash theme={null}
agenteye query schema [TABLE] # column layout of the analytics views
agenteye query run --sql "select count(*) from analytics.events"
agenteye query run errs --arg prod --limit 100 # run a saved query + a positional $1
agenteye query list ; agenteye query show errs
agenteye query create errs --sql @errs.sql --description "errored events (24h)"
agenteye query update errs --sql @errs.sql --yes ; agenteye query delete errs --yes
```
**`agent`**: talks to the built-in **AI assistant** (the same read-only analyst you can chat with in the dashboard). Chats are referenced by a short chat-id (prefix-resolved).
```bash theme={null}
agenteye agent health # is the AI assistant configured/reachable
agenteye agent models # models you can pass to --model (default marked)
agenteye agent ask "which agents errored most in the last day?" # starts a chat; prints its short id
agenteye agent ask --chat "and which tools did they call?" # continue that chat
agenteye agent chats ; agenteye agent show
agenteye agent rename --title "error triage" ; agenteye agent delete
```
***
## Exit codes
| Code | Meaning |
| ---- | ------------------------------------------------------------------------------------ |
| 0 | Success |
| 1 | Unexpected error (e.g. the dashboard returned a 5xx) |
| 2 | Usage error (invalid arguments, unknown command/flag, name collision) |
| 3 | Cannot reach the dashboard |
| 4 | Not logged in or session expired; run `agenteye login` |
| 5 | Authenticated, but your account lacks the required permission (the message names it) |
| 6 | The requested resource was not found (e.g. unknown session or incident id) |
These make the CLI safe to script: a coding agent can branch on a `4` to prompt you to re-authenticate, or a `5` to surface the missing permission. See [CLI recipes for agents](/agenteye/cli-recipes) for exit-code-handling patterns and JSON output shapes.
***
## Next steps
* **[CLI recipes for agents](/agenteye/cli-recipes)**: copy-paste query patterns, `jq` one-liners, `--fields` projections, exit-code handling, and JSON output shapes, written for coding agents driving the CLI.
* **[CLI agent skill](/agenteye/cli-skill)**: package this CLI as an installable Claude Code / Codex *skill* so a coding agent drives Failproof AI Observability from plain-English requests.
* **[API keys](/agenteye/api-keys)**: the permission model behind `keys create --add …`.
* **[AI assistant](/agenteye/assistant)**: enabling the assistant that `agent ask` talks to.
# CLI
Source: https://docs.befailproof.ai/agenteye/cli-and-agents
Your entire Failproof AI Observability deployment, one command away.
Your entire Failproof AI Observability deployment, one command away. Check production, cut an API key, or ack an incident without leaving your terminal, then script any of it into CI, or let a coding agent do it for you in plain English.
```bash theme={null}
pipx install agenteye
agenteye login --email you@example.com # a 6-digit code lands in your inbox
agenteye --json sessions --since 24h # every agent run from the last day, newest first
```
*The `agenteye` CLI talks to your dashboard. It is a different tool from the collector, which ships events to the server.*
## Your whole deployment, one command away
Stop tab-hopping to answer a quick question. The `agenteye` CLI reads your data and administers your org from a single binary, so a check that used to mean clicking through the dashboard becomes one line you can rerun, alias, or paste into a runbook. You get four surfaces:
* **Read your data:** `sessions`, `events`, `evals`, and `errors`, filtered by time, agent, and environment.
* **Manage your org:** `keys`, `users`, `settings`, `alerts`, and `incidents`.
* **Run analytics:** saved SQL plus an ad-hoc `query` runner over your event data.
* **Ask the assistant:** `agent ask` reaches the same read-only analyst you chat with in the dashboard.
Install it once with `pipx`, sign in with an emailed 6-digit code, and you are ready. The session lasts about a day; rerun `agenteye login` when it expires. Reach for it to spot-check production, provision a key, or triage a firing incident, all without opening a browser:
```bash theme={null}
agenteye errors --since 24h --aggregate # what is breaking, grouped by error type
agenteye incidents list --state firing # what is on fire right now
agenteye keys create ci --add events:add # a key that can only push events, secret shown once
```
One habit to know: global options like `--json` go before the command. `agenteye --json sessions` is right; `agenteye sessions --json` is not.
## Script it, wire it into CI
Every command takes `--json`, and that changes everything. Clean JSON goes to stdout while human status and warnings go to stderr, so a `--json` capture pipes straight into `jq` with no stray line to strip. That is what makes the CLI equally good for you at a prompt and for a coding agent parsing output:
```bash theme={null}
agenteye --json sessions --since 24h --status error | jq -r '.sessions[].session_id'
```
It is built to run unattended. Confirmation prompts auto-skip when no terminal is attached, so nothing hangs in a pipeline, and every command returns a meaningful exit code: `0` success, `4` not logged in, `5` missing a permission (the message names it, for example `alerts:write`), `3` dashboard unreachable. A script can branch on a `4` to reauthenticate or a `5` to tell you exactly what to ask an admin for, instead of failing blind.
## Let a coding agent drive it in plain English
Better yet, you should not have to remember any of these flags at all. The **CLI skill** is a small Agent Skill folder named `agenteye-cli` that teaches a coding agent such as Claude Code or Codex to drive the CLI from plain-English requests. Ask "is anything broken today?" and the agent picks the command, runs it as you, and answers in prose.
For Claude Code, drop the `agenteye-cli` folder into `~/.claude/skills/` and it is auto-discovered. Failproof AI Observability provides the folder; there is nothing extra to install, because it only drives the CLI you already installed. Log in yourself first: the skill cannot complete the emailed-code login for you.
Because the agent runs the CLI as you, it can do everything your login permits, reads and writes alike: create keys, change settings, resolve incidents. The CLI's "are you sure?" prompt does not fire for an agent, so the skill is written to state the exact command and wait for your OK before any change. You are the confirmation step.
```text theme={null}
you Why did session run-001 fail?
agent Running: agenteye --json events --session-id run-001 --all
The checkout-agent hit a TimeoutError on its third tool call.
```
Reads stay instant, and every write pauses for you:
```text theme={null}
you Give CI a key that can only push events.
agent This creates an API key. I'll run:
agenteye keys create ci --add events:add
OK to proceed?
you yes
agent Done. Key "ci" created with events:add only. The secret is shown once, so store it now.
```
## Related
* [CLI reference](/agenteye/cli): every command, flag, and JSON shape.
* [CLI recipes for agents](/agenteye/cli-recipes): copy-paste `jq` patterns and exit-code handling.
* [CLI agent skill](/agenteye/cli-skill): install and run the `agenteye-cli` skill.
* [AI assistant](/agenteye/assistant): the in-dashboard analyst that `agent ask` talks to.
# CLI recipes for agents
Source: https://docs.befailproof.ai/agenteye/cli-recipes
Copy-paste query patterns and jq recipes that turn session, event, and evaluation data into something a script or coding agent can automate.
Pull session, event, and evaluation data (and trigger re-evaluations) straight from a script or coding agent, with clean JSON on stdout that pipes directly into `jq`. These recipes turn Failproof AI Observability's data into something a terminal user or an AI coding agent (Claude Code, Cursor) can query and automate, without clicking through the dashboard.
The patterns below are copy-paste ready for the Failproof AI Observability CLI (`agenteye`). For installation, authentication, and the full option list see [CLI](/agenteye/cli); run `agenteye -h` or `agenteye -h` for the built-in help.
## Golden rules
1. **Global options go *before* the command.** `agenteye --json sessions` is correct; `agenteye sessions --json` is not. The globals are `--json`, `--base-url`, `--org`, `--token`, `--insecure`/`--secure`, `--timeout`, `--quiet`, `--no-color`.
2. **Pass `--json` whenever you parse output.** Data goes to **stdout** as JSON; human status and errors go to **stderr**, so stdout stays clean to pipe into `jq`.
3. **Branch on the exit code**, not on stderr text: `0` ok · `1` unexpected error · `2` bad arguments · `3` cannot reach the dashboard · `4` not logged in or expired · `5` missing permission · `6` resource not found.
4. **Discover with `-h`.** Every command documents its filters, value formats, and JSON shape.
## One-time setup
```bash theme={null}
export AGENTEYE_DASHBOARD_URL=https://agenteye.example.com # so you don't repeat --base-url
agenteye login --email you@example.com # paste the emailed code; valid ~24h
```
## Confirm auth before doing work
`whoami` never errors on a missing or expired session; it reports `logged_in:false` instead, so an agent can probe auth state safely. (It can still exit non-zero if no base URL is set or the dashboard is unreachable.)
```bash theme={null}
if [ "$(agenteye --json whoami | jq -r .logged_in)" != "true" ]; then
echo "Not authenticated. Run: agenteye login" >&2; exit 1
fi
```
## Find failing or low-scoring sessions
```bash theme={null}
# sessions in the last 24h whose evaluation errored
agenteye --json sessions --since 24h --status error | jq -r '.sessions[].session_id'
# evaluations scoring <= 0.5 on helpfulness, for one agent
agenteye --json evals --agent-id checkout-bot --score helpfulness:..0.5 \
| jq '.evaluations[] | {session_id, scores}'
```
Score filtering lives on **`evals`**, not `sessions`. `--score KEY:MIN..MAX` is repeatable and AND-combined; either bound is optional (`..0.5` means ≤ 0.5, `0.9..` means ≥ 0.9). You can pass up to 20 score filters per request; more returns HTTP 400. `sessions` shares the `--env`, `--status`, `--agent-id`, `--session-id`, and time-range filters with `evals`, but has no `--score`.
## Read one session end-to-end
There is no single `session show` command. Combine the event trail with the session's evaluation:
```bash theme={null}
# the session's latest evaluation (status + scores)
agenteye --json evals --session-id run-001 | jq '.evaluations[0] | {status, scores}'
# every event in the run (raise --limit for a full sweep)
agenteye --json events --session-id run-001 --all --limit 1000 | jq '.events[] | {ts, event_type}'
# just the tool calls in a session (--full is required to get the raw payload)
agenteye --json events --full --session-id run-001 --event-type tool_use,tool_result --all \
| jq '.events[].payload'
```
> **Note:** By default, `events` reads a fast, payload-free feed. Each event carries a server-computed one-line `summary` plus flags like `is_error` and token counts, but `payload` comes back as `{}`. To pull the raw payload, add `--full` (or `--fields payload`). The full feed is slower at scale, so keep it bounded: pair `--full` with a single `--session-id`.
## Fetch everything (pagination)
Results are newest-first and cursor-paginated.
```bash theme={null}
# one shot: fetch up to 500 rows in 200-row pages
agenteye --json events --session-id run-001 --limit 500 --all > events.json
# manual paging: feed next_cursor back in
page=$(agenteye --json events --limit 100)
cursor=$(echo "$page" | jq -r '.next_cursor // empty')
[ -n "$cursor" ] && agenteye --json events --limit 100 --cursor "$cursor"
```
## Slim the output with --fields
Restrict the keys (in both the table and `--json`) to reduce what an agent must read.
```bash theme={null}
agenteye --json sessions --since 7d --fields session_id,status,scores | jq -c '.sessions[]'
agenteye --json events --session-id run-001 --fields ts,event_type --all
```
Unknown field names are rejected (exit `2`) with the valid list, a cheap way to discover field names.
## Discover valid filter values
```bash theme={null}
agenteye --json list envs | jq -r '.values[]' # values for --env
agenteye --json list tools | jq -r '.values[]' # tool names; also agents, models, event_types, …
agenteye --json list score_filters | jq -r '.values[]' # valid KEY for --score KEY:MIN..MAX
```
## Pick your org (multi-tenant)
If you belong to more than one org, choose the active tenant at login (it's saved):
```bash theme={null}
agenteye login --org acme --email you@corp.com # set the tenant in the same step as login
agenteye --json orgs list | jq -r '.orgs[].org_slug'
agenteye --org globex --json sessions --since 24h # override for one command
```
A multi-org login without `--org` exits non-zero and prints the orgs to choose from.
## Provision an API key for the SDK/collector
```bash theme={null}
# the secret is printed ONCE, with --json it's the .key field
key=$(agenteye --json keys create ci-bot --add events:read.add | jq -r '.key')
agenteye keys regenerate ci-bot --yes # rotate; agenteye keys disable ci-bot --yes to revoke
```
## Run a saved or ad-hoc query
```bash theme={null}
agenteye --json query run --sql "select count(*) from analytics.events" | jq '.rows'
agenteye --json query run errs --arg prod | jq '.rows' # a saved query + a positional $1
```
## Triage an incident non-interactively
```bash theme={null}
id=$(agenteye --json incidents list --state firing | jq -r '.incidents[0].id')
agenteye incidents ack "$id"
agenteye incidents assign "$id" --assignee you@corp.com
agenteye incidents resolve "$id" --yes
```
> **Note:** Mutations auto-skip their confirmation prompt under `--json` or when stdin isn't a TTY, so agents never hang; pass `--yes`/`-y` to skip it explicitly elsewhere.
## Exit-code handling in a script
```bash theme={null}
out=$(agenteye --json sessions --since 1h) || code=$?
case "${code:-0}" in
0) echo "$out" | jq '.sessions | length' ;;
4) echo "Session expired - run 'agenteye login'." >&2 ;;
5) echo "Missing permission (ask an admin for evaluations:read)." >&2 ;;
3) echo "Dashboard unreachable - check the URL." >&2 ;;
*) echo "Unexpected error (exit ${code})." >&2 ;;
esac
```
## JSON output shapes
| Command | stdout JSON (with `--json`) |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `whoami` | `{"logged_in": true, "id", "email", "is_instance_admin", "active_org", "permissions": [...], "memberships": [...]}` or `{"logged_in": false}` |
| `orgs list` | `{"active_org", "orgs": [{"org_slug","org_name","permission_set","permissions"}]}` |
| `events` | `{"events": [...], "next_cursor": }` |
| `evals` | `{"evaluations": [...], "next_cursor": }` |
| `sessions` | `{"sessions": [...], "next_cursor": }` |
| `errors` | `{"errors": [...], "next_cursor": }` |
| `list ` | `{"kind", "values": [...]}` |
| `keys list` / `keys create` | `{"keys": [...]}` / `{id, name, permissions, created_at, key}` (`key` shown once) |
| `query run` | `{columns: [{name,type}], rows: [[...]], truncated, elapsed_ms}` |
| `users list` / `settings list` | `{"users": [...]}` / `{"settings": [...]}` |
| `alerts list` / `incidents list` | `{"alerts": [...]}` / `{"incidents": [...]}` |
| create/update/delete (any) | the resource object, or `{"deleted": true, "id"}` for deletes |
| failure (any, with `--json`) | `{"error": "...", "exit_code": , "status"?: , "hint"?: "..."}` on stdout |
* Each **event** item (`events`): `id, session_id, agent_id, event_type, ts, payload, environment, summary, is_error, error_type, output_tokens, context_window, context_fill`. Note that `payload` is `{}` unless you request the full feed with `--full` (or `--fields payload`).
* Each **evaluation** item (`evals`): `id, session_id, agent_id, environment, status, scores, reasoning, summary, error, attempt_count, duration_ms, completed_at, created_at`.
* Each **session** item (`sessions`): `session_id, agent_id, environment, status, scores, event_count, started_at, last_event_at, first_event_id, last_event_id, latest_evaluation`.
Each command's `--fields` accepts exactly its own item's field names. The set differs between `sessions` and `evals`, so a name valid for one may be rejected by the other.
## Next steps
* [CLI](/agenteye/cli): installation, authentication, and the full option reference for every command.
* [CLI agent skill](/agenteye/cli-skill): package these recipes as a skill your coding agent can load.
* [API keys](/agenteye/api-keys): create and scope the keys the CLI, SDK, and collector authenticate with.
* [Python SDK](/agenteye/python-sdk): send events into Failproof AI Observability so there is data for these recipes to query.
# Failproof AI Observability CLI Agent Skill
Source: https://docs.befailproof.ai/agenteye/cli-skill
Ask your coding agent "is anything broken today?" and let it answer from your live Failproof AI Observability data, with no commands to memorize.
Ask your coding agent *"is anything broken today?"* and let it answer from your live Failproof AI Observability data, with no commands to memorize. The **Failproof AI Observability CLI skill** (`agenteye-cli`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to operate your Observability deployment through the [`agenteye` CLI](/agenteye/cli) from plain-English requests like *"give CI a key that can only push events"* or *"ack the firing incident and assign it to me."*
It is **not** a service or a separate binary; there is nothing to deploy. It rides on top of the CLI you have already installed: the agent shells out to `agenteye --json …`, parses the clean JSON, and answers you in prose. Everything it can do, you could do yourself by typing the same commands.
***
## How it relates to the other Failproof AI Observability interfaces
Failproof AI Observability gives you four ways to reach the same data and controls. They complement each other:
| Interface | What it is | Where it runs | Reach for it when |
| ---------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------- | ------------------------------------------------------------ |
| **[CLI](/agenteye/cli)** | The command/flag reference for `agenteye` | Your terminal | You want to run or script a specific command |
| **[CLI recipes](/agenteye/cli-recipes)** | Copy-paste `jq`/pipeline patterns | Your terminal / scripts | You're wiring the CLI into automation |
| **CLI skill** (this doc) | A natural-language front door on the CLI | Your coding agent, on your workstation | You want to *just ask* and let the agent pick the command |
| **[Evaluator skill](/agenteye/evaluator-skill)** | A sibling skill that designs and builds your scoring service | Your coding agent, on your workstation | You want to *produce* eval scores rather than read them |
| **[Python SDK skill](/agenteye/python-sdk-skill)** | A sibling skill that instruments your agent so it emits telemetry at all | Your coding agent, on your workstation | You want your agent to *produce* the events this skill reads |
| **[In-dashboard AI assistant](/agenteye/assistant)** | A chat embedded in the dashboard | Server-side (in the dashboard) | You want in-dashboard Q\&A over your data |
The skill itself has no privileges of its own; it just turns your words into CLI calls that run as you:
```mermaid theme={null}
flowchart TD
YOU["you: 'ack the firing incident'"] --> AGENT["coding agent (Claude Code / Codex) loads the agenteye-cli skill"]
AGENT --> CLI["agenteye --json incidents ack ..."]
CLI -->|your authenticated CLI session| API["Observability dashboard API"]
```
### vs. the in-dashboard AI assistant: an important distinction
These are two different tools with very different blast radii:
* The **in-dashboard AI assistant** ([AI assistant](/agenteye/assistant)) is a chat embedded in the dashboard, backed by the agent service. It is **read-only plus approval-gated authoring**: it can draft saved queries and dashboards, but every write pauses for your explicit click-approval, and it never deletes. It is gated by the `agent:use` permission and only ever sees data for the org you're viewing.
* The **CLI skill** runs on *your* workstation inside *your* coding agent and drives the `agenteye` CLI as **you**. It can perform the CLI's **full surface, including mutations** (create/rotate/disable API keys, change org settings, resolve incidents, delete saved queries), bounded only by the permissions of your CLI login. Treat it exactly as carefully as you would treat running those commands by hand.
***
## Prerequisites
1. The **`agenteye` CLI installed** and on `PATH` (see the [CLI](/agenteye/cli) reference: `pipx install agenteye`).
2. Your **dashboard URL** set (`AGENTEYE_DASHBOARD_URL`, or the agent passes `--base-url`).
3. A **logged-in session**: run `agenteye login` yourself first. The skill **cannot** complete the emailed one-time-code login for you; it will tell you to run `agenteye login` if the session is missing or expired (CLI exit code `4`).
***
## Where to get it
The skill is published in Failproof AI's public skills collection:
**[github.com/FailproofAI/skills](https://github.com/FailproofAI/skills)** → [`skills/agenteye-cli/`](https://github.com/FailproofAI/skills/tree/main/skills/agenteye-cli)
Nothing about it is gated — the repository is public and the skill needs no credential of its own, because it only drives the **public** `agenteye` CLI against *your* dashboard, using the session *you* logged in with. You do not need to ask anyone for it.
Note it ships as its own folder and is **not** inside the `pipx install agenteye` package, so don't look for it there.
## Installing the skill
The quickest path is the [`skills`](https://skills.sh) CLI, which fetches the folder and drops it where your agent looks:
```bash theme={null}
# Claude Code, this project only
npx skills add FailproofAI/skills --skill agenteye-cli -a claude-code
# every project (installs to ~/.claude/skills/)
npx skills add FailproofAI/skills --skill agenteye-cli -a claude-code -g --copy
# Codex instead
npx skills add FailproofAI/skills --skill agenteye-cli -a codex
```
Then manage it like any other skill:
```bash theme={null}
npx skills list -a claude-code # what's installed
npx skills update agenteye-cli # pull the latest version
npx skills remove agenteye-cli # remove it
```
Prefer to install by hand? An Agent Skill is just a folder containing a `SKILL.md` (plus optional references), so copying it works too:
* **Claude Code**: put the `agenteye-cli/` folder in `~/.claude/skills/` (every project) or `/.claude/skills/` (that repo only). Claude Code auto-discovers it — verify with the `/skills` list, or simply ask a question that matches its description.
* **Codex (OpenAI)**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so Codex auto-selects the skill when a task matches; otherwise invoke it explicitly as `$agenteye-cli`.
***
## Safety: mutations do NOT prompt when an agent runs the CLI
> **Warning:** Read this before letting an agent make changes.
The `agenteye` CLI normally asks *"are you sure?"* before a destructive action. It **auto-skips that confirmation whenever it is not attached to a terminal (which is exactly how a coding agent runs it), and `--json` skips it too.** So the safety prompt will **not** fire for the agent.
The skill is written to compensate: it is instructed to state the exact command it will run and get your explicit **OK before any state change**. Keep that discipline. When you drive Failproof AI Observability through an agent, *you* are the confirmation step. The state-changing commands to watch for:
* `keys create` / `update` / `disable` / `regenerate`
* `users create` / `update` / `disable` / `enable`
* `settings set`
* `alerts create` / `update` / `delete` / `test`
* the writing `incidents` subcommands: `ack` / `assign` / `resolve` / `open` / `comment-add` / `comment-delete` / `subscribe` / `unsubscribe`
* `query create` / `update` / `delete`
* `agent rename` / `delete`
* `orgs switch`
Everything under **Observe** (`events`, `sessions`, `evals`, `errors`, `list`, `whoami`, `orgs list/current/perms`) is read-only and changes nothing.
Because the agent acts as **you**, it can only do what your login is permitted to do; permissions are resolved **per org** (see [API keys](/agenteye/api-keys)). A command you lack permission for returns exit code `5` with the exact permission named, so the agent can tell you precisely what to ask an admin for rather than failing opaquely.
***
## What you can ask it
Here's what a real round-trip looks like, a read, then a change that waits for your OK:
```text theme={null}
you ▸ Is anything broken in the last day?
agent ▸ Running: agenteye --json errors --since 24h --aggregate
3 error types in the last 24h. The biggest is TimeoutError
(18 hits, all on the checkout-agent), then RateLimited (5).
Want the failing sessions for TimeoutError?
you ▸ Give CI a key that can only push events.
agent ▸ This creates an API key. I'll run:
agenteye keys create ci --add events:add
OK to proceed?
you ▸ yes
agent ▸ Done. Key "ci" created with events:add only.
The secret is shown only once, so store it now. I can't reprint it.
```
The skill maps each plain-English intent to the right `agenteye` command, discovering valid values first (`list `, `whoami`) so it doesn't guess, and stating the exact command before any change. More examples:
* *"Is anything broken / failing in the last 24 hours?"* → `errors --since 24h --aggregate`, then a breakdown.
* *"Why did session `run-001` fail?"* → `events --session-id run-001 --all` + `evals --session-id run-001`.
* *"How is quality trending this week?"* → `evals --aggregate --since 7d`, then drill into low-scoring runs.
* *"Give CI a key that can only push events."* → `keys create ci --add events:add` (it states the command, then creates it and captures the one-time secret).
* *"Who has access? Make Dana read-only."* → `users list` → `users update dana@… --permission-set read-only` (after confirming with you).
* *"Ack the firing incident and assign it to me."* → `incidents list --state firing` → `incidents ack ` / `incidents assign you@…`.
For the exact commands, flags, and JSON shapes behind these, see the [CLI](/agenteye/cli) reference and [CLI recipes for agents](/agenteye/cli-recipes).
***
## Next steps
* **[CLI](/agenteye/cli)**: full command and flag reference for `agenteye`.
* **[CLI recipes for agents](/agenteye/cli-recipes)**: copy-paste `jq` patterns and exit-code handling.
* **[Evaluator agent skill](/agenteye/evaluator-skill)**: the sibling skill, for building the evaluator whose scores `agenteye evals` reads.
* **[Python SDK agent skill](/agenteye/python-sdk-skill)**: the sibling skill, for instrumenting an agent so it emits the telemetry `agenteye` reads.
* **[AI assistant](/agenteye/assistant)**: the in-dashboard assistant (not to be confused with this terminal skill).
* **[API keys](/agenteye/api-keys)**: the per-org permission model that bounds what the skill can do.
# Codex session capture
Source: https://docs.befailproof.ai/agenteye/codex-capture
Tail your team's local OpenAI Codex sessions into AgentEye as ordinary sessions and events — with no change to how they run Codex.
Your engineers already run OpenAI Codex every day. Codex session capture brings those coding sessions into AgentEye as ordinary sessions and events, so you can search, replay, and evaluate them next to everything else you observe. It complements the [Python SDK](/agenteye/python-sdk): the SDK instruments agents you write, while this captures the Codex work your team already does — with no change to how they run it.
A small background collector reads Codex's local session transcripts as they are written and ships them to AgentEye. One collector per machine captures every local Codex surface at once — there is no per-surface setup.
The same collector captures other agents too — see [OpenClaw](/agenteye/openclaw-capture) and [Hermes](/agenteye/hermes-capture). Enable each one you run; a single collector can capture several at once.
***
## What it captures
Every Codex surface that runs **locally** produces the same on-disk session transcripts, and the collector picks up all of them:
* the Codex **CLI** and `codex exec`
* the **VS Code / IDE extension**
* the **desktop app**, when it runs a session locally
Each Codex session becomes an AgentEye [session](/agenteye/sessions); its user and assistant messages, reasoning, tool calls, tool results, and token usage become the matching [events](/agenteye/event-stream). The surface each session came from (CLI, IDE, or desktop) is recorded, so you can tell them apart.
> **Cloud sessions are not captured.** The desktop app increasingly runs sessions in the Codex cloud and keeps only their metadata on the machine — there is no local transcript to read. Only locally-executed sessions are captured.
***
## Turn it on
Capture is off until you enable it. Install the collector with an API key that has the `events:add` permission (see [API keys](/agenteye/api-keys)), and turn on Codex capture:
```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/FailproofAI/agenteye-collector/main/install.sh \
| sh -s -- --key --codex-enabled
```
That installs the collector, registers it as a background service, and starts capturing. Confirm it is running:
```bash theme={null}
agenteye-collector health
```
On first run, your existing Codex sessions are backfilled once and new activity then streams within seconds. Codex's own files are only ever read — never modified, moved, or deleted — and each session is shipped exactly once, even across restarts.
***
## Where it shows up
Captured sessions appear in **Sessions**, and their events in the **Events** stream, the same as any other agent you observe — so [session replay](/agenteye/sessions), [search](/agenteye/queries), [evaluations](/agenteye/evaluations), and [alerts](/agenteye/alerts) all work on them. Filter by the Codex agent to see them on their own.
***
## Privacy
Codex transcripts contain the full session — including command output, file contents, and anything Codex read or wrote — and can contain secrets. Captured sessions are shipped as-is, so enable capture only on machines and for teams where centralizing that content in AgentEye is appropriate, and give the collector a key scoped to `events:add` only. See [Security](/agenteye/security) for how your data is kept isolated.
# Concepts
Source: https://docs.befailproof.ai/agenteye/concepts
The vocabulary behind Failproof AI Observability — events, sessions, evaluations, audits, findings, and incidents — defined in one place.
This page defines the vocabulary Failproof AI Observability uses. If a term in another guide is unfamiliar, it's defined here. You don't need to read it end to end: skim it, or jump back when you hit a word you want pinned down.
***
## The data model
**Event**
The smallest unit of data. One event records a single step your agent took: a `tool_use`, a `model_request`, a `hook_completed`, an `error`, and so on. Your agent emits events through the [Python SDK](/agenteye/python-sdk); they show up live on the **Events** page.
**Session**
One agent run, identified by a `session_id`. A session is all the events that share that id, rolled up into a single row on the **Sessions** page and drawn as an execution graph on its detail page. A session usually starts with `agent_start` and ends with `agent_end`.
**Agent**
A named actor inside a run, identified by an `agent_id`. A run can involve several agents: a planner that spawns a summarizer sub-agent, for example. Sub-agents carry a `parent_id`, which is what lets Failproof AI Observability draw them on their own lanes in the execution graph.
**Environment**
A label for where the run happened: `production`, `staging`, `dev`. You set it once when you configure the SDK. Almost every dashboard page can filter by environment.
**Context-window fill**
The percentage of a model's context window a response consumed. Failproof AI Observability stamps it on `model_response` events for models it recognizes, so prompt growth and impending compaction are visible right in the event stream.
***
## Quality
**Evaluation**
A quality score for a finished session, produced by a scoring service you run. Evaluations are opt-in: until you connect an evaluator, sessions are recorded but not scored. Each evaluation can carry several named scores (for example `helpfulness`, `factuality`, `tool_efficiency`), each with a short reasoning note. See [Evaluation suite](/agenteye/evaluation-suite).
**Score key**
The name of one dimension an evaluator reports, such as `helpfulness`. Alerts and audits can watch a specific score key over time.
**Evaluator**
Your scoring service. Failproof AI Observability POSTs a finished run's transcript to it and stores the scores it returns. It does not ship a default evaluator; the scoring logic is yours.
***
## Finding and fixing failures
**Hook**
A guardrail or side-effect your agent framework runs around a step: a content-safety check, PII redaction, a budget guard. Hooks emit `hook_triggered` / `hook_completed` events with an `outcome` (allow, deny, modify), and get their own observe page.
**Alert rule**
A rule that fires when a metric crosses a threshold you set: error rate, p95 latency, token cost, or an evaluator score. When a rule fires, it opens an incident and notifies your chosen channels (email, Slack, webhook, in-dashboard). See [Alerts](/agenteye/alerts).
**Incident**
An open issue created when an alert rule fires. Incidents have a lifecycle (acknowledge, assign, resolve) and an activity timeline that records every action. You can also open one manually.
**Audit**
A recurring investigation (hourly to weekly) that mines your logs *across* sessions for failure patterns you haven't written a rule for: error clusters, low scores, latency outliers, tool-call loops, and runs that never finished. Where an alert watches a metric you already know about, an audit tells you what to look at next. See [Audits](/agenteye/audits).
**Finding**
One ranked, evidence-backed result from an audit run. A finding names a pattern, links to the exact sessions behind it, and carries a triage lifecycle (acknowledge, resolve, mute, dismiss). Failproof AI Observability deduplicates findings run-over-run so a known pattern updates instead of piling up.
**The AI assistant**
The in-dashboard chat that answers questions about your agents in plain English, over your own data. It is read-only by default; anything it creates (a saved query, a dashboard) is approval-gated, and it can never delete. See [AI assistant](/agenteye/assistant).
***
## Running it
**Organization (tenant)**
An isolated workspace. One Failproof AI Observability instance can host many organizations, each with its own users, keys, and data. Every dashboard URL is scoped under your org slug (`//…`).
**Collector**
`agenteye-collector`, the lightweight daemon that runs on each agent machine, batches the events the SDK writes to disk, and ships them to the server.
**API key**
A scoped token that authenticates a client against the server. Keys carry granular permissions (for example `events:add` for the collector, read-only scopes for a dashboard key). See [API keys](/agenteye/api-keys).
**Server**
The ingest and API service. It ingests events, stores operational state in your databases, and serves the dashboard and CLI.
**Dashboard**
The web UI. Every page is scoped to an organization and reads through the server's API.
***
## Next steps
* [Overview](/agenteye/overview): how these pieces fit together.
* [Observability](/agenteye/observability): the observe surfaces (Events, Sessions, Models, Tools, Hooks, Errors).
# Dashboards
Source: https://docs.befailproof.ai/agenteye/dashboards
Turn your live agent data into one shared picture your whole team watches.
Turn your live agent data into one shared picture your whole team watches. Pin the queries that matter as charts, and everyone opens the same numbers at a glance, without re-running a single query.
*One board, four saved queries: events per hour, errors by type, latency, and tokens by model.*
## Everyone sees the same truth
Stop pasting screenshots into chat and stop re-running the same query five times a day. A dashboard is a shared, org-wide board anyone on your team can open to the exact same view. When the underlying data moves, the charts move with it, so the board is always current and nobody is arguing over stale numbers.
The fleet dashboard above is a good starting shape for day-to-day operations:
* an **events-per-hour** line, so you can watch throughput and catch a sudden drop
* an **errors-by-type** bar, so your biggest failure categories jump out
* a **latency** area chart, so slow-downs show up before users complain
* a **tokens-by-model** breakdown, so cost stays in view
You'll find your boards at `//dashboards`.
## Pin the queries you already saved
Every tile starts as a saved query. Build and save the query you care about in the [Queries](/agenteye/queries) library (built-in presets plus your own, over your events and evaluations), then pin it to a dashboard as the chart that fits the data: a **line** for trends over time, a **bar** for comparing categories, an **area** for volume, or a **pie** for a share breakdown.
Because a tile is just your saved query rendered as a chart, there's nothing to keep in sync by hand. Update the query once and every dashboard that uses it updates too.
## Watch quality, not just volume
Volume tells you the agents are busy. Quality tells you they're actually doing the job. Point a dashboard at your [evaluation scores](/agenteye/evaluations) and you get a board that tracks how well runs are going over time, so a quality regression shows up as a dip on a chart instead of a surprise from a customer.
*A quality board keeps your evaluation scores front and center, right beside the operational numbers.*
Keep an operations board and a quality board side by side and your team has one place to answer both "is it working?" and "is it good?", without anyone re-running a query.
## Related
* [Queries](/agenteye/queries): build and save the queries that become your tiles.
* [Evaluations](/agenteye/evaluations): score your runs so you can chart quality over time.
* [Alerts](/agenteye/alerts): turn a threshold on any of these metrics into a page.
# Error Tracking
Source: https://docs.befailproof.ai/agenteye/error-tracking
See every failure your agents produce in one place, grouped so a noisy burst reads as a single problem.
See every failure your agents produce in one place, grouped so a noisy burst reads as a single problem. You get a one-click path from "something is red" to the exact run that broke, without scrolling a live feed to find it.
*The Errors page: a histogram of failures over time, with repeat failures collapsed into one row per incident.*
## Every failure, already collected for you
When an agent breaks, you should not have to scroll a live event stream hoping to catch the red rows before they scroll away. The **Errors** page does the collecting for you. It pulls together everything the dashboard would paint red into one triage surface, so the first thing you see is what is failing, not where to go looking for it.
And it catches more than the obvious ones. Alongside explicit `error` events, Failproof AI Observability surfaces the quiet failures too: any `tool_result`, `hook_completed`, or `agent_end` whose payload carries a failure shows up here. A tool that returned an error, or a hook that exited badly, no longer slips past you just because nothing threw a loud exception.
Across the top, a histogram plots errors over time. One look tells you whether this is a steady background trickle or a spike that started a few minutes ago, so you know right away whether to drop what you are doing.
Like every observe surface, the Errors page is scoped to your organization and filters by date range, environment, agent, and session. That means you can take a fleet-wide list and narrow it to the one agent or one environment you actually care about.
## One incident, not a hundred identical rows
A single broken dependency can fire the same error hundreds of times a minute. Left raw, that is a wall of near-identical lines that buries the one thing you actually need to see.
Failproof AI Observability collapses repeat failures that share the same session and error type into a single row. A burst reads as one incident. You end up counting problems, not log lines, and the signal that matters stays on top instead of being drowned out by its own volume.
## From "something is red" to the exact event
Click any row to land straight inside that run's session, positioned on the exact event that failed. No copying session IDs, no scrolling to hunt for the moment it went wrong: you arrive right on it, with the full execution graph one glance away so you can see what the agent did in the moments before it broke.
If you have `alerts:write`, every row also carries a **+ alert** button. Click it and Observability opens a new alert rule already filled in to catch that same failure again. The incident you just triaged becomes the one that pages you next time, instead of surprising you twice.
**Where to find it:** the **Errors** page lives in the observe section of the dashboard, at `//errors`.
## Related
* [Alerts](/agenteye/alerts): turn any failure into a paging rule.
* [Incidents](/agenteye/incidents): track a firing alert from open to resolved.
* [Sessions](/agenteye/sessions): open the full run behind any error.
* [Audits](/agenteye/audits): let Observability find failure patterns across your runs for you.
# Evaluation Suite
Source: https://docs.befailproof.ai/agenteye/evaluation-suite
Failproof AI Observability can automatically score every finished agent run for quality: you supply a small scoring service, and Observability handles the rest.
Failproof AI Observability can automatically score every finished agent run for quality: you supply a small scoring service, and Observability handles the rest. Use it to track the dimensions you care about (helpfulness, tool efficiency, factuality, safety; you choose), catch regressions early, and compare agents or environments at a glance. Scoring is opt-in: the pipeline does nothing until you set `EVALUATOR_ENDPOINT` on the server.
> **Note:** You define the score dimensions. Your evaluator can return any numeric keys it likes; Observability stores, trends, and displays whatever you send back.
## At a glance
1. **Write a scorer.** Stand up a small HTTP service that reads a session transcript and returns scores. Observability ships a working reference you can copy. See [Writing an evaluator with the SDK](#writing-an-evaluator-with-the-sdk).
2. **Point Observability at it.** Set `EVALUATOR_ENDPOINT` (and a shared `EVALUATOR_TOKEN`) on the server process.
3. **Watch the scores land.** Every completed session is scored automatically; results show up on the session detail page, the sessions grid, and saved dashboards.
*Once an evaluator is configured, each completed run is scored and the results appear in the session's right rail: the summary on top, then per-dimension score bars with reasoning.*
***
## How it works
```mermaid theme={null}
flowchart LR
ING["ingest /events agent_end"] --> SRV["Observability server"]
SRV -->|"POST /evaluate"| EV["Evaluator service"]
EV -->|"done or pending"| SRV
SRV -->|"poll GET /evaluate/{job_id}"| EV
EV -->|"done"| SRV
SRV --> RES["evaluations terminal results"]
```
When the Observability SDK emits an `agent_end` event for a session, the server
schedules an evaluation. It then POSTs the full event transcript to your
evaluator service, which can either:
* **Return the result inline** with `{"status":"done", "scores":{...}, "reasoning":{...}, "summary":"..."}`. The
result is appended to the session's evaluation timeline. `reasoning` and
`summary` are optional.
* **Defer** with `{"status":"pending", "job_id":"abc-123"}`. Observability then
calls `GET {EVALUATOR_ENDPOINT}/evaluate/abc-123` until your evaluator
returns `{"status":"done", ...}` or `{"status":"error", "error":"..."}`.
The polling cadence is per-job: a `pending` response may include
`next_poll_secs` to override; otherwise Observability uses the
`default_poll_interval_secs` value from `GET /config`; otherwise the server
falls back to `EVALUATOR_POLLING_INTERVAL_SECS` (default 10s). All values
are clamped to \[1s, 1h].
Sessions that never emit `agent_end` (for example, a crashed agent process)
can also be picked up: the evaluator's `GET /config` may return
`{"inactivity_timeout_secs": 1800}`, and Observability will evaluate any session
that has gone idle for that long. Set the field to `null` or omit it to
disable this fallback.
The pipeline is fully no-op when `EVALUATOR_ENDPOINT` is unset.
A session can accumulate **multiple terminal evaluations over time**: each
`agent_end` event (and each manual re-eval from the dashboard) appends a
fresh evaluation row. This is the supported way to evaluate a resumed
conversation: a user ends an agent, comes back later, sends more events,
ends the agent again, and a second evaluation runs against the full updated
transcript. The dashboard renders the most-recent evaluation as the
headline and the prior evaluations as a collapsible timeline. While one
evaluation is running for a session, additional `agent_end` events for that
session are ignored; the next one after the running evaluation completes
will enqueue a fresh evaluation as usual.
The inactivity fallback re-engages on resumed sessions too: if new events
arrive after a previous terminal evaluation and the session then goes idle
past `inactivity_timeout_secs`, a fresh evaluation is enqueued.
Transient failures (5xx, 429, timeouts, network errors) are retried with
exponential backoff up to `EVALUATOR_MAX_ATTEMPTS`; 4xx responses are
terminal. Observability is safe to run with multiple horizontally-scaled server
instances; work is partitioned so the same session is never dispatched
twice concurrently.
***
## HTTP contract
Every authenticated route uses **bearer token auth**. The same value must be
configured on both sides:
* Observability server: env var `EVALUATOR_TOKEN`
* Evaluator service: configured the same way (the `agenteye-evaluator` SDK
reads `EVALUATOR_TOKEN` by convention)
If `EVALUATOR_TOKEN` is unset, the server sends no `Authorization` header; the
evaluator may then accept anonymous requests, which is fine for an
internal-only network but discouraged on the public internet.
### Routes the evaluator must serve
| Route | Body / params | Response |
| -------------------- | ------------------ | -------------------------------------------------------------------------------------------- |
| `GET /health` | none | `{"status":"ok"}` (open, no auth) |
| `GET /config` | none | `{"inactivity_timeout_secs": \| null, "default_poll_interval_secs": \| omitted}` |
| `POST /evaluate` | `EvalRequest` JSON | `{"status":"done", ...}` or `{"status":"pending", "job_id":"..."}` |
| `GET /evaluate/{id}` | none | same response shape as `/evaluate` |
### `EvalRequest` body sent by the server
```json theme={null}
{
"schema_version": "1",
"session_id": "session-abc123",
"agent_id": "planner",
"environment": "production",
"started_at": "2026-05-10T12:00:00Z",
"ended_at": "2026-05-10T12:05:00Z",
"events": [
{ "id": 1234, "ts": "...", "event_type": "agent_start", "payload": { ... } },
...
]
}
```
### Response shapes
**Sync (done):**
```json theme={null}
{
"status": "done",
"scores": { "helpfulness": 0.85, "tool_efficiency": 0.6 },
"reasoning": {
"helpfulness": "answered the question directly with citations",
"tool_efficiency": "called list_files three times when one would have done"
},
"summary": "strong answer quality, weak tool selection"
}
```
`reasoning` (a per-score justification map) and `summary` (an overall
one-paragraph narrative) are both optional. Keys in `reasoning` should
mirror keys in `scores`; the dashboard renders each entry inline under
its score bar. Older evaluators that return only `scores` continue to
work unchanged; `reasoning` and `summary` simply read as null and
the corresponding UI affordances are omitted.
**Async (deferred):**
```json theme={null}
{ "status": "pending", "job_id": "abc-123", "next_poll_secs": 30 }
```
`next_poll_secs` is optional; if omitted the server falls back to the
evaluator's `default_poll_interval_secs` from `/config`, then to its own
`EVALUATOR_POLLING_INTERVAL_SECS` env var.
**Terminal evaluator-side error:**
```json theme={null}
{ "status": "error", "error": "model service unavailable" }
```
The server treats any other 2xx body as a protocol error and records a
terminal `error` for the session.
***
## Writing an evaluator with the SDK
You don't have to implement the HTTP contract by hand. The `agenteye-evaluator`
Python package gives you a typed FastAPI wrapper that handles auth, routing, and
the request/response shapes for you.
Failproof AI Observability also ships a **working reference evaluator** that
scores `helpfulness`, `tool_efficiency`, and `factuality` from the shape of the
transcript. Copy it as a starting point and swap in your own logic: an LLM
judge, a rule engine, whatever fits your quality bar.
Minimum viable evaluator:
```python theme={null}
import os
from agenteye_evaluator import Evaluator, EvalRequest, EvalResponse
app = Evaluator(token=os.environ["EVALUATOR_TOKEN"])
@app.evaluator
def run(req: EvalRequest) -> EvalResponse:
# Inspect req.events (the full session transcript) and return scores.
tool_calls = sum(1 for e in req.events if e.event_type == "tool_use")
return EvalResponse(
scores={"tool_calls": float(tool_calls)},
reasoning={"tool_calls": f"{tool_calls} tool invocations in the transcript"},
summary="tight tool loop" if tool_calls < 5 else "agent looped on tools",
)
```
The `app` instance runs under any ASGI server, so `uvicorn module:app` starts it.
For evaluators that need to defer expensive work, return `JobPending`
instead and register a `@app.job_lookup` handler; the Observability server
polls `GET /evaluate/{job_id}` until you return a terminal status or the
`EVALUATOR_MAX_POLL_DURATION_SECS` cap (default 1 h) elapses.
The full API reference, async pattern, and event schema are documented in the
`agenteye-evaluator` SDK's README.
***
## Running your evaluator
The evaluator is **your service** — Failproof AI Observability does not ship a
default evaluator, so you build and run it wherever you run your own services.
It runs under any ASGI server (for example `uvicorn my_evaluator:app`); serve
the `/health`, `/config`, and `/evaluate` routes from the
[HTTP contract](#http-contract), then point the server at it (see
[Configuring the server](#configuring-the-server)).
Once the evaluator is reachable, `GET /health` returns `{"status":"ok"}`. After
an agent runs end-to-end, `GET /evaluations` on the server returns a row with
`status: "done"` and the scores your evaluator produced.
***
## Configuring the server
Set on the server process:
| Env var | Meaning |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `EVALUATOR_ENDPOINT` | Base URL of your evaluator (`http://evaluator:9000`). Unset = pipeline disabled. |
| `EVALUATOR_TOKEN` | Bearer token. Must equal the value the evaluator service is configured with. |
| `EVALUATOR_WORKERS` | Worker tasks per server instance (default 2). |
| `EVALUATOR_CLAIM_BATCH` | Rows claimed per worker tick (default 4). Batches are processed **concurrently**; effective concurrency on your evaluator endpoint is `EVALUATOR_WORKERS × EVALUATOR_CLAIM_BATCH`. |
| `EVALUATOR_POLL_IDLE_SECS` | How long a worker sleeps between dispatch attempts when no evaluation is due (default 2s). |
| `EVALUATOR_POLLING_INTERVAL_SECS` | Final fallback for `GET /evaluate/{id}` cadence when neither the per-response `next_poll_secs` nor the evaluator's `default_poll_interval_secs` is set (default 10s). |
| `EVALUATOR_REQUEST_TIMEOUT_MS` | Per-request timeout (default 30000). |
| `EVALUATOR_MAX_ATTEMPTS` | After this many transient failures the result is recorded as terminal `error` (default 5). |
| `EVALUATOR_CONFIG_REFRESH_SECS` | `GET /config` cadence (default 300). |
| `EVALUATOR_MAX_POLL_DURATION_SECS` | Maximum wallclock time a session may remain in the polling queue before it's terminated as `timeout` (default 3600s). Guards against an evaluator that keeps returning `pending` forever. |
To turn on automatic scoring, set both `EVALUATOR_ENDPOINT` and
`EVALUATOR_TOKEN` on the server, then restart it to pick up the change. With
`EVALUATOR_ENDPOINT` unset the pipeline stays a no-op.
The tuning knobs above are optional; set the corresponding environment
variables on the server only if you need to override the defaults.
***
## API reference
| Method | Path | Required permission | Purpose |
| ------ | ----------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET` | `/evaluations` | `evaluations:read` | Query terminal results. Supports `session_id`, `agent_id`, `environment`, `status` (`done`/`error`/`timeout`), `ts_from`, `ts_to`, `cursor`, `limit`, `score_filters`, `latest_per_session`. `limit` defaults to 50 and is capped at 200 (note this differs from `/events`, which caps at 1000). `environment` accepts a comma-separated list (e.g. `environment=prod,staging`); single values still work. With `latest_per_session=true` the response contains at most one row per `session_id` (the most recent by `completed_at`) used by the sessions-list page to collapse a session's evaluation timeline to its current headline. Defaults to false (returns the full history). |
| `GET` | `/evaluations/aggregate` | `evaluations:read` | Rolled-up eval health for a filtered slice: total count, a done/error/timeout breakdown, per-score-key stats (count/avg/min/max/p50 over the arbitrary `scores` keys), and a time-bucketed timeline. Accepts the **same filter params as `/evaluations`** plus `featured_keys` (CSV of score keys to trend) and `latest_per_session`. Powers the Dashboards feature; metrics are exact over the whole matching set, not sampled. |
| `GET` | `/evaluations/environments` | `evaluations:read` | Distinct environment values from the `evaluations` table. Used to populate filter dropdowns scoped to evaluation-readable data. |
| `GET` | `/evaluation-jobs` | `evaluations:read` | Visibility into in-flight evaluations. Filter by `status` (`pending`/`polling`). |
| `GET` | `/events` | `events:read` | Stream a session's raw events. Supports `session_id`, `agent_id`, `event_type` (CSV), `environment` (CSV), `ts_from`, `ts_to`, `cursor`, `limit`, and `order`. `order` is `desc` (newest-first, the default) or `asc` (oldest-first); an unrecognized value falls back to `desc`. Cursor-paginate via the response's `next_cursor` (an event id): pass it back as `cursor` to get the next page; with `asc` the next page is the events after that id, with `desc` the events before it. `limit` defaults to 50 and is capped at 1000. |
| `GET` | `/sessions/:session_id/export` | `events:read` | Returns the exact JSON body the evaluator would receive for this session, served as a downloadable attachment named `session-.json`. Useful for replaying production sessions through `agenteye-evaluator` for offline testing. The bytes are byte-identical to what the evaluator pipeline sends. |
| `POST` | `/sessions/:session_id/re-evaluate` | `evaluations:trigger` | Enqueue a fresh evaluation for a session; runs whether or not a prior evaluation exists. The new result is **appended** to the session's evaluation timeline rather than overwriting the previous one, so prior scores remain visible as history. Returns `202` on enqueue, `404` for an unknown session, `409` if an evaluation is already in flight. Use this after deploying a new evaluator, or for sessions that never emitted `agent_end`. |
### Filtering by score range: `score_filters`
`GET /evaluations` accepts an optional `score_filters` parameter that
narrows results by numeric values inside the `scores` object. The
parameter is a comma-separated list of `key:min..max` entries; either
bound may be omitted. Multiple entries combine with logical AND. Rows
where the named key is absent or non-numeric are excluded. A request may
carry at most 20 filter entries; exceeding that returns HTTP 400.
Examples:
```text theme={null}
# helpfulness in [0.5, 0.8]
GET /evaluations?score_filters=helpfulness:0.5..0.8
# tool_efficiency at most 0.3 (no lower bound)
GET /evaluations?score_filters=tool_efficiency:..0.3
# helpfulness >= 0.5 AND factuality >= 0.9
GET /evaluations?score_filters=helpfulness:0.5..,factuality:0.9..
```
Each `/evaluations` response object has these fields:
| Field | Type | Notes |
| --------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluation_id` | string (UUID) | The canonical identifier for this terminal evaluation. Each terminal evaluation gets a new UUID; a single session can hold multiple. |
| `id` | string (UUID) | Backwards-compatibility alias carrying the same value as `evaluation_id`. |
| `session_id` | string | The session this evaluation ran against. A session can have multiple evaluations in the timeline. |
| `agent_id` | string | Identifies the agent that produced the session. |
| `environment` | string | Environment label copied from the session. |
| `status` | enum | One of `"done"`, `"error"`, `"timeout"`. |
| `scores` | object \| null | Scores returned by your evaluator. |
| `reasoning` | object \| null | Optional per-score justification map returned by your evaluator. Keys typically mirror those in `scores`. The dashboard renders each entry under its score bar. |
| `summary` | string \| null | Optional one-paragraph overall narrative returned by your evaluator. The dashboard renders this above the per-score breakdown as the evaluation's headline. |
| `error` | string \| null | Populated on `"error"` / `"timeout"` only. |
| `attempt_count` | integer | Number of dispatch attempts (≥ 1). |
| `duration_ms` | integer \| null | Duration of the final attempt. |
| `completed_at` | string (ISO 8601 UTC) | When the terminal result was recorded. Results are ordered by `completed_at` (newest first). |
| `created_at` | string (ISO 8601 UTC) | Carries the same timestamp as `completed_at` (write-once semantics). |
***
## Permissions
| Permission | Grants |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `evaluations:read` | List evaluation results, view scores in the dashboard, and load dashboard health metrics. |
| `evaluations:trigger` | Manually enqueue an evaluation for a session via `POST /sessions/:session_id/re-evaluate` or the dashboard's re-evaluate button. |
| `dashboards:read` | View saved dashboards (also needs `evaluations:read` to load their metrics). |
| `dashboards:write` | Create and edit dashboards. |
| `dashboards:delete` | Delete dashboards. |
The bootstrap admin (`ADMIN_KEY`, `ADMIN_EMAIL`) automatically receives these.
***
## Viewing results
* **`/sessions/`**: events timeline + a right rail showing the session's
scores and any error from the dispatch attempt. If your key has
`evaluations:trigger`, a **re-evaluate** button appears next to the export
button, useful for sessions that never emitted `agent_end`, or for
refreshing scores after deploying a new evaluator. The dashboard polls for
the new result and updates the right rail when it lands.
* **`/sessions`**: filterable session grid; the score column shows each
session's evaluation status and scores at a glance.
* **`/dashboards`**: saved eval-health views (see [Dashboards](#dashboards) below).
*The sessions grid shows each run's evaluation status and scores at a glance; red/amber/green badges make low scores jump out.*
***
## Dashboards
The **Dashboards** page (`/dashboards`) lets you save a combination of evaluation
filters as a named, reusable view and watch how that slice of evaluations is
doing at a glance. Dashboards are **shared across your whole organization**;
everyone with `dashboards:read` sees the same set.
Each dashboard pins:
* **Filters**: the same controls as the sessions page: environment, status,
agent, a rolling time window, and score-range filters (`key:min..max`).
* **A display configuration**: which score keys to feature, the green/amber/red
health thresholds, which panels to show, and whether to collapse to the latest
evaluation per session.
Each card shows the number of matching sessions, a done/error/timeout breakdown,
the average of each featured score, and a small trend sparkline. Opening a
dashboard shows the full-size panels; **"open in sessions"** drops you into the
sessions page pre-filtered to exactly that slice. Metrics are computed
server-side over the whole matching set (via `GET /evaluations/aggregate`), so
the numbers are exact rather than sampled.
**Permissions:** viewing needs both `dashboards:read` and `evaluations:read`;
creating and editing needs `dashboards:write`; deleting needs `dashboards:delete`.
The bootstrap admin receives all of these automatically.
***
## Troubleshooting
**Sessions exist but no evaluations are created.** Confirm `EVALUATOR_ENDPOINT`
is set on the server process, that the server and evaluator share the same
`EVALUATOR_TOKEN` value, and that the evaluator's `/health` endpoint is
reachable from the server. With `EVALUATOR_ENDPOINT` unset the pipeline is a
no-op.
**In-flight evaluations pile up.** Query `GET /evaluation-jobs` to see the
in-flight queue. Inspect `attempt_count`, `next_attempt_at`, and `last_error`
on each row. Common causes: evaluator service unreachable or returning 5xx
(retried with backoff), wrong `EVALUATOR_TOKEN` (401 is terminal), or an
async evaluator that returns `pending` indefinitely (see below).
**Sessions completed but no terminal evaluation.** Query
`GET /evaluation-jobs?status=polling`; the result may still be in flight.
If a job is stuck in `pending`, the server is having trouble reaching the
evaluator; check that the evaluator is up and that `EVALUATOR_TOKEN` matches.
**`HTTP 401 from evaluator: invalid bearer token`.** The `EVALUATOR_TOKEN`
on the server does not match the value the evaluator service is configured
with. They must be identical.
**Async evaluator returns `pending` forever.** The server polls
`GET /evaluate/{job_id}` until the evaluator returns `done` or `error`, or
until `EVALUATOR_MAX_POLL_DURATION_SECS` (default 1 h) elapses. After the cap
the evaluation is recorded as `timeout` and removed from the in-flight queue.
Raise `EVALUATOR_MAX_POLL_DURATION_SECS` if your evaluator legitimately needs
longer than the default.
***
## Next steps
* [Evaluator agent skill](/agenteye/evaluator-skill): have a coding agent design your dimensions against real sessions and build this service for you.
* [Python SDK](/agenteye/python-sdk): emit the `agent_end` events that trigger scoring.
* [API keys](/agenteye/api-keys): the `evaluations:read` and `evaluations:trigger` permissions.
* [Audits](/agenteye/audits): Observability's other automated quality feature, for policy-based review.
# Evaluations
Source: https://docs.befailproof.ai/agenteye/evaluations
Quality problems find you now, instead of you hearing about them in a user complaint.
Quality problems find you now, instead of you hearing about them in a user complaint. Connect your own scoring service once and Failproof AI Observability grades every finished run automatically, so a drop in helpfulness or a spike in hallucinations shows up on its own, before a customer feels it.
*Every run on the sessions grid carries its scores; red, amber, and green badges make the weak runs jump out without you opening a single transcript.*
## Stop sampling runs by hand
You used to spot-check a handful of runs and hope the rest were fine. Now every completed session is scored the moment it finishes, on the dimensions you care about: helpfulness, tool efficiency, factuality, safety, whatever your quality bar is. You define the score keys; Failproof AI Observability stores, trends, and displays whatever your evaluator sends back. No run slips through unscored, and you stop learning about a regression from a support ticket.
The scores ride along on the sessions grid at **`//sessions`** (sidebar → *observe* → *sessions*), one badge cluster per row. Want just the runs that fell short? Filter the grid by score range, say helpfulness below 0.5, and pull up exactly the runs worth reading. Viewing scores needs the `evaluations:read` permission.
## See why a run scored low
A number tells you a run was weak; the session page tells you why. Open any run and the right rail leads with the headline summary, then shows a bar per dimension with your evaluator's own reasoning under each one, so you go from "this scored 0.4 on factuality" to the exact claim it got wrong in seconds.
*The session detail view: summary, per-dimension score bars, and the reasoning behind each score, right next to the run's event timeline.*
Shipped a sharper evaluator, or looking at a run that crashed before it could be scored? A **re-evaluate** button (gated by `evaluations:trigger`) re-scores the session in place and appends the fresh result to its timeline, so earlier scores stay visible as history. You will find it at **`//sessions/`**.
## Watch quality trend across the fleet
One run scoring low is noise; a whole cohort sliding is a signal. Saved dashboards turn your scores into a trend you can watch at a glance: average helpfulness this week against last, per agent, per environment.
*A saved quality dashboard trends the score keys you feature, so a slow drift is obvious long before it becomes an incident.*
Dashboards live at **`//dashboards`** (sidebar → *analyze* → *dashboards*), are shared across your whole organization, and each card rolls up the matching sessions: how many, the average of each featured score, and a trend sparkline. "Open in sessions" drops you straight into the pre-filtered runs behind any number. Viewing needs `dashboards:read` plus `evaluations:read`.
## Connect an evaluator once
Scoring is opt-in and stays completely off until you point Failproof AI Observability at a scorer. You stand up one small HTTP service (Observability ships a working reference you can copy), set two values on your server, and every run from then on is scored for you. The full walkthrough, the scoring contract, and the SDK live in the deep guide.
Not sure which dimensions are worth scoring in the first place? The [evaluator agent skill](/agenteye/evaluator-skill) has your coding agent work that out against your own sessions, then build and deploy the service.
## Related
* [Evaluation suite](/agenteye/evaluation-suite): connect your evaluator, the scoring contract, and the SDK.
* [Evaluator agent skill](/agenteye/evaluator-skill): let a coding agent pick your score dimensions and build the evaluator.
* [Sessions](/agenteye/sessions): the run-by-run grid where scores appear.
* [Dashboards](/agenteye/dashboards): save and share quality trends across your org.
* [Audits](/agenteye/audits): Observability's other automatic quality feature, for cross-session investigations.
# Failproof AI Observability Evaluator Agent Skill
Source: https://docs.befailproof.ai/agenteye/evaluator-skill
Go from "I think our agent is sometimes bad" to a deployed scoring service, with your coding agent doing both the deciding and the building.
Go from *"I think our agent is sometimes bad"* to a deployed scoring service, with your coding agent doing both the deciding and the building. The **Failproof AI Observability evaluator skill** (`agenteye-evaluator`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to work out which quality dimensions are worth tracking for *your* agent, then write, test, and deploy the [evaluator service](/agenteye/evaluation-suite) that scores them.
It is **not** a hosted scorer, a registry you upload to, or a plugin system. Your evaluator stays your own HTTP service on your own infrastructure, exactly as described in the [Evaluation suite](/agenteye/evaluation-suite) guide. The skill only teaches your agent to build it well, so everything it does, you could do yourself by writing the same code.
***
## The hard part is deciding what to score
The SDK surface is small — a decorator and two models — and an agent can write that from the [contract](/agenteye/evaluation-suite#http-contract) alone. That's not where evaluators fail. They fail because they score the wrong thing, and an evaluator that scores the wrong thing is worse than none: it produces a dashboard everyone learns to ignore.
So most of the skill is the part before any code exists. It has the agent interview you (*"describe a run that went well; now one that went badly"*), then pull your real sessions through the [`agenteye` CLI](/agenteye/cli) and read them end to end. Those two halves usually disagree, and the gap is the point: what you intend to measure versus what your transcripts can actually support. A dimension only survives if it is **computable** from the events and **discriminating** — if it scores 0.9 on both your good run and your bad one, it teaches nothing and gets cut.
What comes back is a proposal of 2-4 dimensions with the reasoning attached, for you to sign off on before a line is written.
```mermaid theme={null}
flowchart TD
YOU["you: 'I want evals for my support bot'"] --> AGENT["coding agent (Claude Code / Codex) loads the agenteye-evaluator skill"]
AGENT -->|"interview: what does good vs bad look like?"| YOU
AGENT -->|"agenteye --json sessions / events"| DATA["your real sessions what actually happens"]
DATA --> DIMS["2-4 dimensions, you sign off"]
DIMS --> SVC["your evaluator service agenteye-evaluator SDK"]
SVC --> SCORES["scores land in the dashboard and agenteye evals"]
```
***
## How it relates to the other evaluation pieces
Four docs cover scoring, and they hand off to each other in order:
| Page | What it is | Reach for it when |
| -------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------ |
| **[Evaluations](/agenteye/evaluations)** | The feature: scores on the sessions grid, dashboards, re-evaluate | You want to know what automatic scoring gets you |
| **[Evaluation suite](/agenteye/evaluation-suite)** | The HTTP contract, the SDK, the server env vars | You're implementing or debugging the evaluator yourself |
| **Evaluator skill** (this doc) | A natural-language front door on designing *and* building the scorer | You want to go from "I want evals" to a running service |
| **[CLI skill](/agenteye/cli-skill)** | A natural-language front door on the `agenteye` CLI | You want to *read* the scores you already have |
| **[Python SDK skill](/agenteye/python-sdk-skill)** | A natural-language front door on instrumenting your agent | Your agent isn't emitting sessions yet — there is nothing to score |
### vs. the CLI skill: build versus read
The two skills are deliberately non-overlapping, and installing both is the normal setup — the agent picks between them based on what you ask:
* **`agenteye-evaluator`** (this doc) builds the thing that *produces* scores. Its job ends when scores land for the first time.
* **[`agenteye-cli`](/agenteye/cli-skill)** reads scores that already exist (`agenteye evals`). *"Did quality drop this week?"* is its question, not this skill's.
***
## Prerequisites
1. The **`agenteye` CLI installed and logged in** (`pipx install agenteye`, then `agenteye login`). The skill leans on it twice: to pull the real sessions it designs against, and to confirm your scores landed at the end. Your login needs `events:read`, plus `evaluations:read` for that final check. As with the CLI skill, it **cannot** complete the emailed one-time-code login for you.
2. **Somewhere for the evaluator to live.** It gets built into an image and run as a long-running service, so it needs a real repo, not a scratch file. Evaluators often live in their own repo, separate from the agent being scored — the skill looks for an existing one and asks before scaffolding a new one.
3. **The `agenteye-evaluator` SDK wheel** — read the next section before your agent starts typing `pip` commands.
***
## Where to get it
The skill is published in Failproof AI's public skills collection:
**[github.com/FailproofAI/skills](https://github.com/FailproofAI/skills)** → [`skills/agenteye-evaluator/`](https://github.com/FailproofAI/skills/tree/main/skills/agenteye-evaluator)
The repository is public and the skill needs no credential of its own — it only drives the `agenteye` CLI with the session *you* logged in with, and writes code in *your* repo. Note it ships as its own folder and is **not** inside the `pipx install agenteye` package, so don't look for it there.
## Installing the skill
The quickest path is the [`skills`](https://skills.sh) CLI, which fetches the folder and drops it where your agent looks:
```bash theme={null}
# Claude Code, this project only
npx skills add FailproofAI/skills --skill agenteye-evaluator -a claude-code
# every project (installs to ~/.claude/skills/)
npx skills add FailproofAI/skills --skill agenteye-evaluator -a claude-code -g --copy
# Codex instead
npx skills add FailproofAI/skills --skill agenteye-evaluator -a codex
```
Then manage it like any other skill:
```bash theme={null}
npx skills list -a claude-code # what's installed
npx skills update agenteye-evaluator # pull the latest version
npx skills remove agenteye-evaluator # remove it
```
Prefer to install by hand? An Agent Skill is just a folder containing a `SKILL.md` (plus optional references), so copying it works too:
* **Claude Code**: put the `agenteye-evaluator/` folder in `~/.claude/skills/` (every project) or `/.claude/skills/` (that repo only). Claude Code auto-discovers it — verify with the `/skills` list, or just ask for evals.
* **Codex (OpenAI)**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so Codex auto-selects the skill when a task matches; otherwise invoke it explicitly as `$agenteye-evaluator`.
***
## The SDK is not on public PyPI
> **Warning:** Read this before letting an agent install the SDK.
The skill is public; the SDK it drives is not. `agenteye-evaluator` ships only as a private release artifact, and unlike `agenteye`, the name is **unclaimed on public PyPI** — so a bare `pip install agenteye-evaluator` could pull a stranger's package into the service that reads your production transcripts. That's a supply-chain problem, not a typo.
The skill knows this and works down an install ladder instead, stopping at the first rung that applies: the monorepo source if you're inside the AgentEye repo, otherwise the private release wheel from GitHub Releases (needs access), and if neither is reachable it **stops and tells you to ask your Failproof AI contact for the wheel** rather than improvising.
So if your agent proposes a bare `pip install agenteye-evaluator` from public PyPI, that is the tell that the skill never loaded. Stop there and check it's installed.
***
## What you can ask it
A real round-trip starts with a vague ask and ends with a signed-off design, not with code:
```text theme={null}
you ▸ I want evals for our support bot. I think it's sometimes bad.
agent ▸ Before I write anything: describe a run that went well,
and one that went badly.
you ▸ Good: it answers from the docs. Bad: it makes stuff up,
or gives up and says "contact support".
agent ▸ Running: agenteye --json sessions --status error --since 7d
Reading 5 sessions, including two you'd call bad.
Your words say factuality and deflection. The transcripts
agree on deflection (2 of 5 end with a support hand-off,
and it's plain in model_response.content), but factuality
needs a ground truth your events don't carry.
So I'd score three things:
- deflection rules, cheap, separates your two runs
- tool_efficiency rules, catches the retry loop in run-004
- groundedness LLM judge: is every claim in the docs it retrieved?
"factuality" would need ground truth we don't have; groundedness
is the closest thing your transcripts can actually support.
OK with those three names? Renaming later splits the history.
you ▸ yes
agent ▸ No evaluator in this repo. Should I scaffold one here, or do
you have one elsewhere?
```
From there it writes the rules-based dimensions first (free, instant, deterministic), tests them against a real captured session including the empty and never-finished ones that crash naive evaluators, and only reaches for an LLM judge on the subjective dimension. It knows the [dispatcher's limits](/agenteye/evaluation-suite#configuring-the-server) — a 30s request timeout and 8 concurrent calls deployment-wide — so if the judge won't reliably fit, it goes async with `JobPending` rather than letting your judge get cancelled and retried five times at five times the cost.
Then it deploys, sets the two server env vars, and confirms with `agenteye --json evals --session-id ` that scores actually landed. Scores landing is the only proof.
***
## What to watch for
* **Dimension names are close to permanent.** Score keys are arbitrary strings and the platform trends whatever you send, which means nothing downstream corrects a bad choice. Rename later and the history splits: old sessions keep the old key and the trend breaks. This is why the skill gets explicit sign-off before writing code — take that prompt seriously.
* **Fixtures are real production transcripts.** Designing against real sessions means pulling them to disk, and they can contain customer data. The skill asks before committing them to git; if in doubt, keep `fixtures/` out of the repo and have each developer pull their own.
* **The agent writes and deploys a service that reads every transcript.** It acts as you, bounded by your CLI login's permissions, but review the evaluator like any other code that touches production data.
***
## Next steps
* **[Evaluation suite](/agenteye/evaluation-suite)**: the HTTP contract, the SDK, and the server env vars the skill configures.
* **[Evaluations](/agenteye/evaluations)**: where the scores show up once they land.
* **[CLI skill](/agenteye/cli-skill)**: the sibling skill, for reading results rather than building the scorer.
* **[CLI](/agenteye/cli)**: the command reference behind the session data the skill designs against.
# Event Stream
Source: https://docs.befailproof.ai/agenteye/event-stream
The moment your agent does something, you see it.
The moment your agent does something, you see it. The Event Stream is your live pulse on every agent in production: no waiting, no grepping logs, no guessing what just happened.
*Every event from every agent in your org, newest first, updating as it happens.*
## Your live pulse on every agent
When an agent starts a run, calls a model, fires a tool, runs a hook, or hits an error, the row appears at the top of the stream the moment it happens. It tails every event across every agent in your organization, newest first, so you always have a current picture instead of a stale one.
That means no tailing log files on a box somewhere, no grepping across machines, no stitching timestamps together by hand. You open one page and you are already watching production.
Rows are colour-coded by type, so you can read the stream at a glance instead of parsing every line. At a glance, each row shows you:
* **Its type**, colour-coded: `agent_start`, `model_response`, `tool_use`, `hook_completed`, `error`, and more.
* **A one-line summary** of what happened, so you rarely need to open anything just to get the gist.
* **Token counts** for the step.
* **A context-window fill badge** where it applies, so prompt growth and an approaching compaction are visible before they bite.
Watching it live means you catch a bad deploy, a runaway loop, or a burst of errors as it happens, not in tomorrow's log review.
## Find the one run that matters
When something looks off, you don't want the firehose. You want the single run that broke. The stream filters down fast: by environment, by agent, by session, by event type, or by free text.
Filter by session id or agent id to follow one run from its first event to its last. Filter by event type to isolate a single kind of activity, for example every `error` across the org in one view. Stack filters to narrow from "everything, everywhere" to "this agent, in prod, erroring" in a couple of clicks, then act on what you find.
Free-text search cuts straight to a message, a tool name, or an id you already have in hand, so a customer report turns into the exact run in seconds.
## Where to find it
The Event Stream is your org home. Sign in and it is the first surface you land on, at `//`, so triage starts the second you arrive.
Behind it, your agents emit events through the SDK, the collector ships them to your Failproof AI Observability server, and the stream tails them as they arrive in infrastructure you control. When you want the rolled-up view instead of the raw trail, each run's events collapse into a single row on Sessions, one click away.
This is the raw source of truth that every other observe surface builds on, so when a number looks wrong elsewhere, the stream is where you confirm what actually happened.
## Related
* [Sessions](/agenteye/sessions): the same events rolled up into one row per run, with a git-style execution graph.
* [Telemetry](/agenteye/telemetry): what your agents send and how events reach the stream.
* [Error tracking](/agenteye/error-tracking): one triage surface for everything that went wrong.
* [Alerts](/agenteye/alerts): turn any threshold into a paging rule.
* [CLI and agents](/agenteye/cli-and-agents): the same live trail from your terminal.
# Hermes session capture
Source: https://docs.befailproof.ai/agenteye/hermes-capture
Bring your team's Hermes gateway sessions — Slack, Telegram, CLI, and scheduled runs — into AgentEye as ordinary sessions and events.
[Hermes](https://hermes-agent.nousresearch.com) answers your team from wherever they already work — Slack, Telegram, the CLI, scheduled runs. Hermes session capture brings all of it into AgentEye as ordinary sessions and events, so the assistant your team talks to every day is as observable as the agents you write yourself.
A small background collector reads Hermes's local session store as it is written and ships sessions to AgentEye. It works the same way as [Codex](/agenteye/codex-capture) and [OpenClaw](/agenteye/openclaw-capture) capture, and one collector can capture several at once.
***
## What it captures
Every Hermes session on the machine is captured, whichever channel it came from. Each one becomes an AgentEye [session](/agenteye/sessions); its user and assistant messages, tool calls, and tool results become the matching [events](/agenteye/event-stream).
The channel a session started from — Slack, Telegram, CLI, or a scheduled run — is recorded on the session, so you can tell them apart and filter to one at a time. Alongside it come the model the session ran on, the chat and person it was started from, and, when a session spawned another, the link back to its parent.
Sessions appear as soon as Hermes starts them, whether or not anything has been said yet, and a turn's reply and its tool calls stay in the order they actually happened. When a session ends you also get why it ended, what it cost, and how many tokens it used.
***
## Turn it on
Capture is off until you enable it. Install the collector with an API key that has the `events:add` permission (see [API keys](/agenteye/api-keys)), and turn on Hermes capture:
```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/FailproofAI/agenteye-collector/main/install.sh \
| sh -s -- --key --hermes-enabled
```
That installs the collector, registers it as a background service, and starts capturing. Confirm it is running:
```bash theme={null}
agenteye-collector health
```
Capturing more than one agent on the same machine? Add each one's flag to the same command — for example `--hermes-enabled --codex-enabled`.
On first run, your existing Hermes sessions are backfilled once and new activity then streams within seconds. Hermes's own data is only ever read — never modified or deleted — and each message is shipped once, even across restarts.
`health` also tells you whether everything the collector captured actually reached AgentEye. If a batch could not be delivered it is kept and retried rather than discarded, and the check reports unhealthy while anything is still outstanding — so "healthy" means your data arrived, not merely that the process is alive.
***
## Where it shows up
Captured sessions appear in **Sessions**, and their events in the **Events** stream, the same as any other agent you observe — so [session replay](/agenteye/sessions), [search](/agenteye/queries), [evaluations](/agenteye/evaluations), and [alerts](/agenteye/alerts) all work on them. Filter by the Hermes agent to see them on their own.
***
## Privacy
Hermes sessions contain the full transcript — including command output, file contents, and anything the agent read or wrote — and can contain secrets. Captured sessions are shipped as-is, so enable capture only where centralizing that content in AgentEye is appropriate, and give the collector a key scoped to `events:add` only. See [Security](/agenteye/security) for how your data is kept isolated.
# Incidents
Source: https://docs.befailproof.ai/agenteye/incidents
When an alert fires, everyone can see the incident is open, who owns it, and what has happened so far — on one attributed timeline.
When an alert fires, the first question is always "who's on it?" Incidents answer it: the moment something breaches, everyone can see the incident is open, who owns it, and exactly what has happened so far, with a clean, attributed record you can hand straight to a post-mortem.
*The inbox groups open incidents by state and filters by severity and assignee, so you see what needs a human now.*
## Know who has it, at a glance
No more "is anyone looking at this?" in a chat thread. A breach opens an incident automatically and drops it into a shared inbox, grouped by state. Acknowledge it and your name is on it, so the rest of the team knows it is handled. Acknowledgement is shared: several operators can ack the same incident and each is recorded on its own, so a full war room shows up by name instead of stepping on each other. Assign one owner for triage, and filter the inbox by severity or assignee to cut it down to what is yours.
## The whole story, in one timeline
When the incident is over, you already have the write-up. Open any incident and you get the breach evidence, its assignees and subscribers, a comment thread for coordinating in place, and an append-only activity timeline.
*Everything that happened, in order, each line signed by whoever did it.*
Every action (opened, acknowledged, resolved, and so on) is written to that timeline and never edited away. Each entry is attributed: to the operator who took it, by email, or to **automated** for anything Failproof AI Observability did on its own, like opening the incident on the breach. Nothing is anonymous and nothing is lost, so the post-mortem more or less writes itself.
## How an incident moves
```mermaid theme={null}
stateDiagram-v2
[*] --> firing
firing --> acknowledged: an operator acks
firing --> resolved: an operator resolves
acknowledged --> resolved: an operator resolves
resolved --> [*]
```
* **Open (firing):** the breach opens the incident and pages your channels once. Repeated breaches fold into the same incident and refresh its evidence instead of paging you again and again.
* **Acknowledged:** an operator picks it up. It stays open, and later breaches update the evidence quietly.
* **Resolved:** an operator closes it out. Automatic resolution when the condition clears is planned but not yet enabled, so an incident stays open until a human resolves it, which keeps everyone honest about what has actually cleared. A fresh incident can open on the same alert later.
One alert holds at most one open incident at a time, so a flapping rule cannot bury you in duplicates. You can also open an incident by hand: a standalone one for something no alert caught, or one attached to an existing alert, if you have `incidents:write`.
## Where to find it
Incidents live at `//incidents`. Viewing needs **`incidents:read`**; opening a manual incident needs **`incidents:write`**; acknowledging, assigning, commenting, and resolving need **`incidents:ack`**. Older keys granted the retired `alerts:ack` keep working, since it is honored as `incidents:ack`, so your on-call rotation does not need re-issuing.
## Related
* [Alerts](/agenteye/alerts): the rules that open these incidents when a threshold breaches.
* [Error tracking](/agenteye/error-tracking): see every failure in one place and promote one to an alert.
* [Audits](/agenteye/audits): the scheduled analyst that finds the failures no rule was watching.
# Observe
Source: https://docs.befailproof.ai/agenteye/observability
The observe surfaces are where you watch what your agents are doing right now and drill into any single run.
The observe surfaces are where you watch what your agents are doing right now and drill into any single run. Everything here is live, scoped to your organization, and filterable by date range, environment, agent, and session, so you go from "something feels off" to the exact run in seconds.
Four surfaces, each with its own page:
* **[Event stream](/agenteye/event-stream)**: the live, per-step trail of every run across every agent, newest first. Your org home and first stop for triage.
* **[Sessions and execution graph](/agenteye/sessions)**: those events rolled up into one row per run, plus a git-style picture of how each run unfolded.
* **[Performance metrics](/agenteye/telemetry)**: latency heat-maps and p50/p95/p99 vitals for your models, tools, and hooks, so a tail spike stands out from the median.
* **[Error tracking](/agenteye/error-tracking)**: one triage surface for everything that went wrong, one click from a firing alert to the run that broke.
## Related
* [Evaluations](/agenteye/evaluations): score every run for quality.
* [Alerts](/agenteye/alerts): turn any threshold into a paging rule.
* [Audits](/agenteye/audits): let Failproof AI Observability find failure patterns across sessions for you.
* [CLI and agents](/agenteye/cli-and-agents): the same observability from your terminal.
# OpenClaw session capture
Source: https://docs.befailproof.ai/agenteye/openclaw-capture
Tail your team's local OpenClaw sessions into AgentEye as ordinary sessions and events — with no change to how OpenClaw runs.
If your team runs [OpenClaw](https://docs.openclaw.ai), OpenClaw session capture brings those sessions into AgentEye as ordinary sessions and events, so you can search, replay, and evaluate them next to everything else you observe. It complements the [Python SDK](/agenteye/python-sdk): the SDK instruments agents you write, while this captures the OpenClaw work your team already does — with no change to how they run it.
A small background collector reads OpenClaw's local session transcripts as they are written and ships them to AgentEye. It works the same way as [Codex capture](/agenteye/codex-capture), and one collector can capture both at once.
***
## What it captures
Every agent configured in a machine's OpenClaw setup is captured by that machine's collector — there is no per-agent setup.
Each OpenClaw session becomes an AgentEye [session](/agenteye/sessions); its user and assistant messages, tool calls, and tool results become the matching [events](/agenteye/event-stream).
***
## Turn it on
Capture is off until you enable it. Install the collector with an API key that has the `events:add` permission (see [API keys](/agenteye/api-keys)), and turn on OpenClaw capture:
```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/FailproofAI/agenteye-collector/main/install.sh \
| sh -s -- --key --openclaw-enabled
```
That installs the collector, registers it as a background service, and starts capturing. Confirm it is running:
```bash theme={null}
agenteye-collector health
```
Capturing more than one agent on the same machine? Add each one's flag to the same command — for example `--openclaw-enabled --codex-enabled`.
On first run, your existing OpenClaw sessions are backfilled once and new activity then streams within seconds. OpenClaw's own files are only ever read — never modified, moved, or deleted — and each session is shipped exactly once, even across restarts.
***
## Where it shows up
Captured sessions appear in **Sessions**, and their events in the **Events** stream, the same as any other agent you observe — so [session replay](/agenteye/sessions), [search](/agenteye/queries), [evaluations](/agenteye/evaluations), and [alerts](/agenteye/alerts) all work on them. Filter by the OpenClaw agent to see them on their own.
***
## Privacy
OpenClaw transcripts contain the full session — including command output, file contents, and anything the agent read or wrote — and can contain secrets. Captured sessions are shipped as-is, so enable capture only on machines and for teams where centralizing that content in AgentEye is appropriate, and give the collector a key scoped to `events:add` only. See [Security](/agenteye/security) for how your data is kept isolated.
# Failproof AI: Observe Agents for Failures
Source: https://docs.befailproof.ai/agenteye/overview
Failproof AI Observability is a self-hosted platform for observing, evaluating, and improving your AI agents in production.
Failproof AI Observability is a self-hosted platform for observing, evaluating, and improving your AI agents in production. It records everything your agents do (every tool call, model request, hook, and error), scores the quality of each run, and surfaces the failures you didn't know to look for, all in a dashboard you run inside your own infrastructure.
If you ship AI agents and you're tired of guessing why a run went wrong, this is the page to start on. It explains what Failproof AI Observability gives you and how the pieces fit together, before you install anything.
> **Failproof AI Observability is an enterprise product from Failproof AI.** Want to see it in action? Request a demo: email [nikita@befailproof.ai](mailto:nikita@befailproof.ai).
*Every agent run is drawn as a git-style execution graph (left) beside its event timeline. Parallel sub-agents each get their own lane; the right rail breaks down the tools, models, hooks, and token spend for the run.*
***
## See it in action
Two short videos show the two things teams reach for first: tracing a run, and finding failures automatically.
*Agent tracing: follow a single run step by step, from goal to tools to final answer.*
*Failproof Audit: let Failproof AI Observability mine your logs across sessions and tell you what to fix.*
***
## Why teams use it
* **See what your agent actually did.** Every run becomes a readable, git-style execution graph: which tools ran in parallel, which sub-agents branched off, where it stalled, and what it spent.
* **Catch quality regressions automatically.** Connect a small scoring service and Failproof AI Observability scores every finished run, so a drop in helpfulness or a spike in hallucinations shows up on its own.
* **Find failures you didn't write a rule for.** Recurring audits mine your logs across sessions for error clusters, latency outliers, low scores, and stuck runs, then hand you ranked, evidence-backed findings.
* **Get paged when it matters.** Threshold rules fire on error rate, latency, cost, or evaluator scores and open incidents you can acknowledge, assign, and resolve.
* **Ask questions in plain English.** An in-dashboard AI assistant answers "how is quality trending in prod this week?" over your own data. Any change it makes is approval-gated.
* **Keep your data.** Failproof AI Observability is self-hosted: events, prompts, and analytics stay in infrastructure you control.
***
## What you get
Failproof AI Observability is organized around three ideas (**observe**, **analyze**, and **admin**), mirrored in the dashboard's left sidebar.
**Observe** (the raw truth of what happened):
* **[Event stream](/agenteye/event-stream)**: the live, per-step trail of every run (tool calls, model calls, hooks, errors).
* **[Sessions](/agenteye/sessions)**: those events rolled up into one row per run, each ready to be scored, with a git-style execution graph.
* **[Performance metrics](/agenteye/telemetry)**: per-surface latency heat-maps and p50/p95/p99 vitals for models, tools, and hooks, so a tail spike stands out from the median.
* **[Error tracking](/agenteye/error-tracking)**: one triage surface for everything that went wrong, one click from a firing alert.
*Each observe surface pairs a sparkline and p50/p95/p99 vitals with a latency heat-map and a percentile band. Shown here: Tools.*
**Analyze** (turn activity into answers):
* **[Queries](/agenteye/queries)** and **[dashboards](/agenteye/dashboards)**: saved SQL over your events and evaluations, charted into shared, org-scoped dashboards.
* **[Evaluations](/agenteye/evaluations)**: quality scores produced by your own evaluator service, with per-score reasoning.
* **[Audits](/agenteye/audits)**: recurring investigations that surface failure patterns across sessions.
* **[Alerts](/agenteye/alerts)** and **[incidents](/agenteye/incidents)**: threshold rules that page you, plus an incident workflow to triage them.
**Interfaces** (reach your data your way):
* **[CLI](/agenteye/cli-and-agents)**: drive your whole deployment from the terminal or a script, and let a coding agent do it for you in plain English.
* **[AI assistant](/agenteye/assistant)**: ask questions about your agents in plain English, right inside the dashboard.
* **REST API**: everything the dashboard and CLI do is backed by a REST API you can call directly with a scoped [API key](/agenteye/api-keys) — ingest events, query sessions and evaluations, and manage dashboards, alerts, audits, users, and keys, so you can wire Failproof AI Observability into your own tooling.
**Admin** (run it for your team):
* **[API keys](/agenteye/api-keys)**: scoped tokens for the collector, the dashboard, and the assistant.
* **Users**: passwordless, email-based sign-in with an allowlist.
* **Settings**: per-org configuration, including model context-window overrides.
***
## How the pieces fit
Data flows in one direction, from your agent code to the dashboard: your agent (via the Python SDK) emits events to the agenteye-collector, which ships them to the server, which serves the dashboard. Two optional services round it out — a scoring service (evaluations) and an AI assistant service (the in-dashboard chat).
* **Python SDK**: you add a few `agenteye.event.*` calls to your agent; events are buffered locally.
* **agenteye-collector**: a lightweight daemon on each agent machine that batches events and ships them to the server.
* **Server**: ingests your events, keeps operational state in your own databases, and serves the REST API that the dashboard, CLI, and your own integrations all use.
* **Dashboard**: where you explore everything.
* **Optional services**: a scoring service (evaluations), and an AI assistant service (the in-dashboard chat).
For the vocabulary used throughout the docs (*event, session, evaluation, audit, finding, incident*), see [Concepts](/agenteye/concepts).
***
## Getting Failproof AI Observability
Failproof AI Observability is an enterprise product from Failproof AI, and it works alongside Failproof AI Enforcement — the policy and guardrail product — under the Failproof AI brand. It runs entirely in your own environment. If you don't have access to the packages yet, request a demo and we'll get you set up: email [nikita@befailproof.ai](mailto:nikita@befailproof.ai).
***
## Next steps
* [Concepts](/agenteye/concepts): the Failproof AI Observability vocabulary in one place.
* [Observability](/agenteye/observability): follow what your agents do, run by run.
* [Security](/agenteye/security): how Failproof AI Observability keeps your data isolated and in your control.
# Python SDK
Source: https://docs.befailproof.ai/agenteye/python-sdk
See exactly what your AI agents did in production: every agent run, tool call, model request, hook, and human intervention.
See exactly what your AI agents did in production: every agent run, tool call, model request, hook, and human intervention. The Failproof AI Observability Python SDK records that trail from inside your agent code so you can debug, audit, and evaluate what happened. Use it whenever you want Failproof AI Observability to observe your agents.
Under the hood, the SDK writes structured events to local JSONL files, and the collector daemon picks them up and ships them to the platform automatically. You do not manage those files yourself.
> **Tip:** New to Failproof AI Observability? This page is the complete SDK event reference.
***
## Installation
The SDK is distributed to customers as a private wheel rather than from a public package index. Your onboarding covers how to obtain it, install it, and pin it — talk to your Failproof AI contact if you need access.
Once it is installed, confirm you have it:
```bash theme={null}
python -c "import agenteye; print(agenteye.__version__)"
```
Prefer to let a coding agent do the whole integration? The [Python SDK Agent Skill](/agenteye/python-sdk-skill) knows the install path, plans the instrumentation points, writes them, and verifies the events land.
***
## Quick Start
```python theme={null}
import agenteye
agenteye.configure(environment="production")
agenteye.event.agent_start(session_id="run-001", agent_id="planner", goal="answer user query")
agenteye.event.tool_use(
session_id="run-001",
agent_id="planner",
tool_name="web_search",
tool_call_id="toolu_01",
input={"query": "latest AI research"},
)
agenteye.event.tool_result(
session_id="run-001",
agent_id="planner",
tool_name="web_search",
tool_call_id="toolu_01",
output={"results": ["..."]},
)
agenteye.event.agent_end(session_id="run-001", agent_id="planner", outcome="success")
```
### Instrumenting a real call
In practice you wrap your existing agent code. Bracket a model call with `model_request` before and `model_response` after, so the two events span the real request and Failproof AI Observability can pair them:
```python theme={null}
import anthropic
import agenteye
agenteye.configure(environment="production")
client = anthropic.Anthropic()
messages = [{"role": "user", "content": "Summarise today's incidents."}]
agenteye.event.model_request(
session_id="run-001",
agent_id="planner",
model="claude-sonnet-4-6",
messages=messages,
)
reply = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=512,
messages=messages,
)
agenteye.event.model_response(
session_id="run-001",
agent_id="planner",
model=reply.model,
stop_reason=reply.stop_reason,
input_tokens=reply.usage.input_tokens,
output_tokens=reply.usage.output_tokens,
content=[block.model_dump() for block in reply.content],
)
```
Wrap tool calls the same way with `tool_use` and `tool_result`, reusing one `tool_call_id` across the pair.
Here is what those events look like once they reach the dashboard, colour-coded by type and filterable by environment, agent, and session:
***
## configure()
```python theme={null}
agenteye.configure(
base_dir=None, # Path | str | None. Default: $AGENTEYE_HOME or ~/.agenteye
flush_interval=0.5, # float, seconds between flush cycles
environment=None, # str | None. Deployment environment label
)
```
Call once before any `event.*` call. Safe to omit; defaults work out of the box. All arguments are keyword-only; pass them by name as shown above.
When `base_dir` is `None` (the default), the SDK reads `$AGENTEYE_HOME` if set,
otherwise falls back to `~/.agenteye`. This matches the collector's own resolution,
so a single `AGENTEYE_HOME` env var configures the shared event spool for both
the SDK and the collector.
***
## Environment
Label every event with a deployment environment (`production`, `staging`, `qa`, `canary`, etc.). Set it once; the SDK attaches it to every event automatically.
**Option 1: via `configure()`:**
```python theme={null}
agenteye.configure(environment="production")
```
**Option 2: via environment variable:**
```bash theme={null}
export AGENTEYE_ENVIRONMENT=production
```
**Priority:** `configure(environment=...)` wins over the environment variable. If neither is set, defaults to `"dev"`.
The environment value appears as a first-class filter in the dashboard and is stored on the server for fast queries.
> **Warning:** Environment values must not contain a literal `,` comma. The dashboard filters use comma-separated multi-select on the wire (`?environment=prod,staging`), so an environment named `prod,blue` would be split into two values. Events with comma-containing environments are rejected at ingest time.
***
## Data and privacy
The SDK records only the fields you explicitly pass. Prompts, messages, tool inputs and outputs, and model content are captured solely because you hand them to an `event.*` call. Nothing is read from your process or captured implicitly. Any field you leave unset is omitted from the event entirely; it is not written to disk.
That makes redaction your choice and your responsibility. If a prompt or tool payload contains PII or secrets you would rather not store, strip or mask it before you pass it to the event method.
***
## Event Reference
Most events come in start/end pairs that share a correlation ID: `tool_use` and `tool_result` share a `tool_call_id`, `hook_triggered` and `hook_completed` share a `hook_id`, and `human_wait` and `human_input` share an `input_id`. Emit the start event, do the work, then emit the end event with the same ID. Failproof AI Observability matches the pair and computes `duration_ms` for you, so you never pass `duration_ms` yourself.
All event methods require these two fields:
| Field | Type | Description |
| ------------ | ----- | ----------------------------------------------------------- |
| `session_id` | `str` | Identifies the top-level agent run |
| `agent_id` | `str` | Identifies which agent within the session emitted the event |
All methods also accept arbitrary `**kwargs` for custom metadata (see [Custom Fields](#custom-fields)).
***
### `event.agent_start()`
Emitted when an agent begins work.
```python theme={null}
agenteye.event.agent_start(
session_id="run-001",
agent_id="planner",
goal="answer user query", # str | None
parent_id=None, # str | None - parent agent_id for nested agents
)
```
***
### `event.agent_end()`
Emitted when an agent finishes work.
```python theme={null}
agenteye.event.agent_end(
session_id="run-001",
agent_id="planner",
outcome="success", # str | None
summary="Answered query", # str | None
)
```
***
### `event.tool_use()`
Emitted when an agent invokes a tool. Pair with `tool_result`; the SDK auto-computes `duration_ms`.
```python theme={null}
agenteye.event.tool_use(
session_id="run-001",
agent_id="planner",
tool_name="web_search", # str, required
tool_call_id="toolu_01", # str, required - correlation key for the matching tool_result
input={"query": "..."}, # dict | None
)
```
***
### `event.tool_result()`
Emitted when a tool returns. Correlates with `tool_use` via `tool_call_id`.
```python theme={null}
agenteye.event.tool_result(
session_id="run-001",
agent_id="planner",
tool_name="web_search",
tool_call_id="toolu_01", # must match the prior tool_use
output={"results": ["..."]}, # Any | None
error=None, # str | None - set if the tool raised
# duration_ms is computed automatically - do not pass it
)
```
***
### `event.model_request()`
Emitted just before sending a prompt to an LLM.
```python theme={null}
agenteye.event.model_request(
session_id="run-001",
agent_id="planner",
model="claude-sonnet-4-6", # str | None - any provider/model string; not validated
messages=[ # list[dict] | None - conversation turns
{"role": "user", "content": "..."},
],
system="You are helpful.", # Any | None - str or list of content blocks
tools=[ # list[dict] | None - tool schemas offered to the model
{"name": "search", "input_schema": {"type": "object"}},
],
)
```
`messages` entries accept either a plain string `content` or Anthropic-style list-of-blocks `content`. Sampling params (`temperature`, `max_tokens`, etc.) can be passed as extra kwargs.
***
### `event.model_response()`
Emitted when the LLM returns a response.
```python theme={null}
agenteye.event.model_response(
session_id="run-001",
agent_id="planner",
model="claude-sonnet-4-6", # str | None - any provider/model string; not validated
stop_reason="end_turn", # str | None
input_tokens=1024, # int | None
output_tokens=256, # int | None
content=[ # Any | None - str, or list of content blocks
{"type": "text", "text": "..."},
],
role="assistant", # str | None
)
```
`content` accepts either a plain string (generic providers) or a list of Anthropic-style content blocks. Tool calls live inside `content` as `{"type": "tool_use", ...}` blocks, with no separate `tool_calls` field.
***
### `event.hook_triggered()`
Emitted when a hook fires. Pair with `hook_completed`; the SDK auto-computes `duration_ms`.
```python theme={null}
agenteye.event.hook_triggered(
session_id="run-001",
agent_id="planner",
hook_name="pre_tool_use", # str, required
hook_id="hook-abc", # str, required - correlation key
trigger_event="tool_use", # str | None
input={"tool": "search"}, # Any | None
)
```
***
### `event.hook_completed()`
Emitted when a hook finishes. Correlates with `hook_triggered` via `hook_id`.
```python theme={null}
agenteye.event.hook_completed(
session_id="run-001",
agent_id="planner",
hook_name="pre_tool_use",
hook_id="hook-abc", # must match the prior hook_triggered
outcome="allow", # str | None
output=None, # Any | None
error=None, # str | None
# duration_ms is computed automatically - do not pass it
)
```
***
### `event.error()`
Emitted when an unhandled error occurs.
```python theme={null}
agenteye.event.error(
session_id="run-001",
agent_id="planner",
error_type="TimeoutError", # str, required
message="timed out", # str, required
traceback="Traceback...", # str | None
)
```
***
## Human-in-the-Loop Events
Human-in-the-loop events give you oversight over the moments where a person steps into the agent's execution (waiting for approval, providing input, pausing, or stopping the agent). They let you measure how long humans take to respond (the SDK auto-computes `duration_ms` on the paired events), audit who paused or interrupted an agent, and build approval and oversight workflows that surface in the dashboard.
### `event.human_wait()`
Emitted when the agent pauses execution to wait for a human to provide input. Pair with `human_input`; the SDK auto-computes `duration_ms` (how long the human took to respond).
```python theme={null}
agenteye.event.human_wait(
session_id="run-001",
agent_id="planner",
input_id="inp-abc", # str, required - correlation key for the matching human_input
prompt="Do you approve this action?", # str | None - the question shown to the human
options=["approve", "reject", "defer"], # list[str] | None - choices presented to the human
reason="approval_required", # str | None - why the agent is waiting
)
```
### `event.human_input()`
Emitted when a human provides input and the agent resumes. Correlates with `human_wait` via `input_id`. `duration_ms` is auto-computed and must not be passed by the caller.
```python theme={null}
agenteye.event.human_input(
session_id="run-001",
agent_id="planner",
input_id="inp-abc", # str, required - must match the prior human_wait
response="approve", # str | None - the human's answer (free text or selected option)
# duration_ms is computed automatically - do not pass it
)
```
### `event.human_pause()`
Emitted when a human actively pauses the agent (e.g. via a dashboard control). The agent is suspended but not terminated.
```python theme={null}
agenteye.event.human_pause(
session_id="run-001",
agent_id="planner",
reason="user_requested", # str | None
user_id="usr_42", # str | None - who paused the agent
)
```
### `event.human_interrupt()`
Emitted when a human actively stops the agent mid-execution. Unlike `human_pause`, the agent's work is terminated rather than suspended.
```python theme={null}
agenteye.event.human_interrupt(
session_id="run-001",
agent_id="planner",
reason="output_incorrect", # str | None
user_id="usr_42", # str | None - who interrupted the agent
at_step="tool_use:web_search", # str | None - what the agent was doing when stopped
)
```
***
## Custom Fields
Any extra keyword arguments are appended to the event after the standard fields:
```python theme={null}
agenteye.event.tool_use(
session_id="run-001",
agent_id="planner",
tool_name="db_query",
tool_call_id="toolu_02",
tenant_id="acme", # custom field
region="us-east-1", # custom field
)
```
`timestamp`, `type`, and `environment` are reserved and raise `ValueError` (`Reserved field names cannot be used as custom fields: [...]`) if passed as custom fields. `session_id` and `agent_id` are required parameters on every event method and cannot be supplied a second time; Python raises `TypeError` if you do. Set the environment with `configure(environment=...)` (or the `AGENTEYE_ENVIRONMENT` variable) instead.
Keep payloads as structured JSON when you want to query their fields. Values JSON does not natively support—such as datetimes, UUIDs, decimals, sets, bytes, or model objects—are converted to strings so recording continues safely.
***
## How Events Are Written
Events are buffered in-process and flushed to disk every `flush_interval` seconds (default 500 ms). Each flush writes one JSONL file:
```text theme={null}
~/.agenteye/events/event-2026-04-01T12-00-00-000Z.jsonl
```
The collector watches this directory and uploads files automatically. You do not need to manage these files directly.
Each file is written atomically: the SDK writes to a temporary file and then renames it into place, so the collector never sees a half-written file. A final flush also runs when your process exits, so events buffered in the last interval are not lost. If the collector is offline, events simply accumulate as files on disk and ship once it comes back.
***
## Next steps
* [Event stream](/agenteye/event-stream): watch these events arrive live, colour-coded and filterable by environment, agent, and session.
* [Sessions](/agenteye/sessions): see how the paired events reconstruct each agent run as an execution graph and timeline.
# Failproof AI Observability Python SDK Agent Skill
Source: https://docs.befailproof.ai/agenteye/python-sdk-skill
Go from an uninstrumented agent to events you can see, with your coding agent finding the instrumentation points, writing them, and proving they landed.
Tell your coding agent *"add Failproof AI Observability to this agent"* and let it read your loop, work out where the instrumentation belongs, write it, and verify the events before it calls the job done.
The **Python SDK skill** (`agenteye-python-sdk`) is an *Agent Skill*: a folder of instructions that a coding agent such as Claude Code or Codex loads on demand when a task matches it. It teaches the agent to use the [Python SDK](/agenteye/python-sdk) — it is not a library, and it changes nothing about how the SDK works.
## Instrumentation is easy to write and easy to get quietly wrong
The SDK is small: thirteen event methods, all keyword-only. A coding agent can read the [Python SDK](/agenteye/python-sdk) reference and produce plausible instrumentation in a minute.
The catch is that this SDK does not raise when you get it wrong, and wrong instrumentation looks exactly like right instrumentation until someone opens a dashboard and finds it empty. The mistakes that cost real time are all silences:
| The mistake | What you see |
| --------------------------------- | -------------------------------------------------------------------------- |
| No `agent_start` | Every event lands. Zero sessions. |
| Environment never set | Everything works, filed under `dev`. |
| `outcome="failure"` | The run shows green — only `failed`, `error`, `timeout`, `rejected` count. |
| A typo'd field name | Accepted and stored as a new field. |
| Events emitted from a thread pool | Silently dropped. |
None of these raise. None show up in tests. Every one is in the skill, stated as a contract with the check that catches it.
## What it does, in order
The skill runs the same three steps a careful engineer would:
1. **Plan.** It reads your agent loop and asks the two questions only you can answer: what counts as one run (your `session_id`), and who the distinguishable actors are (your `agent_id`). It gets those agreed before writing code, because changing them later splits your history and breaks the trends.
2. **Write.** It binds identity once per run rather than threading it through every call site, and it picks a concurrency-safe shape — a detail that matters, because the obvious shortcut silently mixes two overlapping runs into one session.
3. **Verify.** It runs your agent and reads the resulting event files, checking that `agent_start` is present, the environment is right, and one run produced one session.
That third step is the one people skip. The SDK writes events to local files, so a complete integration can be proven on a laptop with no server, no API key, and no network — which is exactly why the skill insists on doing it.
## How it relates to the other skills
Three skills, one clean split:
| Skill | Reach for it when | What it touches |
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **Python SDK skill** (this page) | You want your agent to *emit* telemetry — "add observability", "why isn't my agent showing up?" | Writes code in your agent's repo. Reads nothing. |
| **[Evaluator skill](/agenteye/evaluator-skill)** | You want to *score* runs — "what should we even measure?" | Writes code in your repo; reads telemetry |
| **[CLI skill](/agenteye/cli-skill)** | You want to *read* what happened, or operate your deployment | Drives the CLI as you, including changes |
They hand off in that order: this skill gets events flowing, the evaluator scores them, the CLI reads them back. There is nothing to evaluate and nothing to read until your agent emits sessions, so if you are starting from scratch, start here.
## Prerequisites
1. **Python 3.10+** and the agent codebase you want to instrument.
2. **The SDK.** It is distributed to customers as a private wheel rather than from a public index — your onboarding covers how to get it and install it. The skill knows the install path and will ask you rather than guess if it cannot find it.
3. **Nothing else.** No dashboard login, no API key, no network. The skill verifies against the event files the SDK writes, so it can finish and prove its work offline.
## Where to get it
The skill lives in the public [`FailproofAI/skills`](https://github.com/FailproofAI/skills) collection:
```bash theme={null}
npx skills add FailproofAI/skills --skill agenteye-python-sdk -a claude-code
```
Add `-g` to install it for every project instead of just the current one, and `--copy` if your environment does not follow symlinks. For Codex, pass `-a codex`.
## Installing it by hand
Agent Skills are folders containing a `SKILL.md` plus references. If you would rather not use the installer:
* **Claude Code**: copy the `agenteye-python-sdk/` folder into `~/.claude/skills/` (every project) or `/.claude/skills/` (that repo only). Claude Code discovers it automatically — check the `/skills` list, or just ask something that matches it.
* **Codex**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so it is auto-selected when a task matches; otherwise invoke it as `$agenteye-python-sdk`.
Run your agent **in the repository holding the code you want instrumented** — the skill reads your agent loop before it proposes anything.
## What a session looks like
```text theme={null}
you ▸ Add Failproof AI Observability to this agent.
agent ▸ Read your loop — one tool dispatcher in tools.py, one Anthropic
wrapper in llm.py. Two questions before I write anything.
What counts as one run? You handle requests concurrently, so I'd
use the request id you already have in ctx.request_id — then a
session here and a request in your logs are the same string.
And who are the actors? I see a planner path and a researcher
path. Label them separately, or is that one agent to you?
you ▸ Request id is right. They're one agent, call it "assistant".
agent ▸ Done. Wrapped the dispatcher and the LLM client; agent_start and
agent_end bracket handle_request, including the error path. Used
a context-local for identity — your dispatcher runs tools in a
thread pool, and the simpler approach would have mixed concurrent
runs into one session without complaining.
Verified against a real run: 11 events across 4 files —
agent_start, model_request/response, tool_use/tool_result x3,
agent_end (outcome=success). environment=production, one session
per request, durations paired.
One thing I did NOT do: your worker pool in batch.py submits
straight to the executor, so events from there would be dropped.
Want me to fix those too?
```
The pattern to notice: it read the code before proposing, asked only the questions you can answer, reused an id you already had, chose the concurrency-safe shape *because* it saw a thread pool, and **verified by reading the actual events** rather than declaring success — then flagged the one place it knew would fail quietly.
## What you can ask it
* *"Why isn't my agent showing up on the dashboard?"* → walks the ladder: are events being written, is `agent_start` there, is the environment right, is the collector reading the same place.
* *"Everything's landing under dev."* → the environment was never set, or was reset by a later call.
* *"Add token tracking."* → finds your LLM wrapper and records model, stop reason, and usage.
* *"Instrument the sub-agents too."* → one session, distinct agent labels, nested under their parent.
* *"Write tests for the instrumentation."* → points the SDK at a temporary directory and asserts on the events it wrote.
## What to watch for
**Let it verify.** The step that makes this skill worth using is the last one — running your agent and reading the events back. An agent that writes instrumentation and stops has done the easy half, and the half that fails silently is the other one.
**Agree the names before the code.** `session_id` and `agent_id` are the axes every surface groups by. Renaming them later splits the history: old runs keep the old labels and your trends break. The skill will ask; the answer is worth a minute's thought.
**If your agent proposes installing the SDK from a public index, the skill did not load.** The SDK is distributed privately. That proposal is a reliable tell that your coding agent is guessing rather than following the skill — stop it there and check the skill is installed.
Beyond that its blast radius is small: it writes code in your working directory and event files where you tell it. It reads nothing from your deployment and changes nothing about it.
## Next steps
* **[Python SDK](/agenteye/python-sdk)**: the complete event reference — every event type and field — behind what this skill automates.
* **[Sessions](/agenteye/sessions)**: what your instrumentation produces once events land.
* **[Evaluator Agent Skill](/agenteye/evaluator-skill)**: the next step once runs are landing — scoring them.
* **[CLI Agent Skill](/agenteye/cli-skill)**: reading your telemetry back.
# Queries
Source: https://docs.befailproof.ai/agenteye/queries
Ask any question of your agent data and get an answer in seconds.
Ask any question of your agent data and get an answer in seconds. Failproof AI Observability gives you a library of saved, ready-to-run queries over your events and evaluations, so you start from a working example instead of a blank SQL editor.
*Your saved-queries library at `//queries`: built-in presets sitting alongside the queries your team has saved.*
## Start from a preset, not a blank page
You do not have to remember table names or write SQL from scratch. The library opens with built-in presets for the questions teams ask most, sitting right next to the queries your own team has saved and named. Pick one that is close to what you want and you are most of the way to an answer.
Every saved query is org-scoped and shared, so the useful ones your teammates write become yours too. Name a query and give it a description once, and anyone in your org can find it, run it, or pin its results onto a dashboard later.
Find it at `//queries`.
## Tweak it and run it in the SQL composer
Open any query and it lands in the SQL composer, where you can adjust it and see the answer immediately: no export, no round-trip, no waiting on someone else.
*The SQL composer: your query on the left, a schema sidebar so you never guess a column name, and a live result grid below.*
* **A schema sidebar** lays out the analytics tables and their columns, so you can shape a query without hunting for field names.
* **A live result grid** returns rows the moment you run, so you iterate in seconds rather than guessing and re-guessing.
* **Read-only by design.** Queries run against your event store and are validated on the server: only `SELECT` and `WITH` statements are allowed, with a statement timeout and a row cap. An exploratory query can never modify your data, and a runaway one gets stopped for you.
Happy with the result? Save it back to the library so the whole team inherits it, or pin its output onto a dashboard as a line, bar, area, or pie tile.
## Run them from the terminal, or let the assistant write them
The same saved queries follow you wherever you work:
* **From the terminal.** The `agenteye` CLI lists, runs, and saves the very same queries, so you can drop a result into a script, wire it into CI, or hand it to a coding agent.
```bash theme={null}
agenteye query list # the same saved queries, from your terminal
agenteye query run errs --arg prod # run one and print the rows (add --json to pipe it)
```
See [CLI and agents](/agenteye/cli-and-agents) for the full command set.
* **From the AI assistant.** Not sure how to phrase the SQL? Ask the in-dashboard [AI assistant](/agenteye/assistant) in plain English and it will draft the query and save it to your library for you.
Running a saved query is gated by the `queries:run` permission, kept separate from the permissions to create or delete queries, so you can grant read access without letting everyone rewrite the library.
## Related
* [Dashboards](/agenteye/dashboards): pin query results into shared, org-wide charts.
* [AI assistant](/agenteye/assistant): ask questions in plain English and get a query back.
* [CLI and agents](/agenteye/cli-and-agents): run and save the same queries from your terminal.
# Security
Source: https://docs.befailproof.ai/agenteye/security
Failproof AI Observability is built to sit close to your production agents, which means it sees your prompts, tool inputs, and outputs.
Failproof AI Observability is built to sit close to your production agents, which means it sees your prompts, tool inputs, and outputs. This page explains how it keeps that data isolated, controlled, and in your hands. If you're evaluating Failproof AI Observability for a security review, start here.
***
## Your data stays in your environment
Failproof AI Observability is self-hosted. Events, prompts, model responses, and analytics are stored in your own databases, in your own environment. Nothing is sent to a third-party SaaS for storage, and your data stays in your own cloud account.
***
## Tenant isolation
One Failproof AI Observability instance can host many organizations, and each is isolated at the storage layer — enforced by the database, not just the UI:
* An organization's operational data (users, keys, dashboards, saved queries) is scoped to that org, and cross-org reads are blocked by the database itself.
* Every ingested event is stamped with its owning org, so one organization's events can never be read by another.
Every dashboard route is scoped under an org slug (`//…`).
***
## Sign-in
Failproof AI Observability uses passwordless, email-based sign-in. There is no password to phish or leak. A user requests a one-time code (or a one-click magic link), which is emailed to them and expires quickly. Sign-in is gated by an **allowlist**: only email addresses (or domains) you permit can authenticate.
***
## Scoped access with API keys
Every client authenticates with an API key that carries granular, least-privilege permissions. A collector needs only `events:add`; a dashboard or assistant key can be read-only; destructive actions (delete, regenerate) are separate grants you choose to include.
Keep the admin bootstrap key for setup, and issue narrow keys for everything else. See [API keys](/agenteye/api-keys).
***
## A read-only, approval-gated assistant
The in-dashboard [AI assistant](/agenteye/assistant) answers questions over your data, but it is constrained by design:
* It is **read-only by default**: its SQL runs through a guard that permits only `SELECT`/`WITH` queries, single-statement, with a row cap.
* Anything it creates (a saved query, a dashboard) is **approval-gated**: you review and approve every write before it happens.
* It **can never delete**.
So a teammate can ask "which agents errored most this week?" and act on the answer, without the assistant being able to change or remove your data on its own.
***
## In transit
All traffic runs over HTTPS. You terminate TLS with your own certificates, so collector-to-server and browser-to-server traffic is encrypted in transit.
***
## Next steps
* [Overview](/agenteye/overview): how Failproof AI Observability fits together.
* [API keys](/agenteye/api-keys): scope access for the collector, dashboard, and assistant.
* [Observability](/agenteye/observability): what Failproof AI Observability captures from your agents.
# Sessions & Execution Graph
Source: https://docs.befailproof.ai/agenteye/sessions
Every event from a run, rolled into one readable row and drawn as a git-style execution graph you can read in seconds.
Stop guessing why a run failed. Failproof AI Observability rolls every event from a run into one readable row, then draws the whole run as a git-style picture you can read in seconds, so you see exactly what your agent did, step by step.
*One row per run: the status pill tells you how the run ended at a glance, and a score badge rides along once an evaluator is connected.*
*Agent tracing: follow a single run step by step, from goal to tools to final answer.*
***
## See every run at a glance
The raw event trail is the truth of every step, but when you have thousands of steps across dozens of runs, you need the run, not the step. The Sessions page rolls all of a run's events up into one row, so a day of activity becomes a scannable list instead of a firehose.
Each row carries a status pill, so a failed run stands out from a healthy one before you click anything. Filter by date range, environment, agent, or session to go from "everything" to "the run I care about" in a couple of clicks.
Once you connect an evaluator, every completed run is scored automatically and its latest score shows up on the row as a badge. You can filter by any score range, so "show me every low-scoring prod run this week" is a filter, not a manual review. Until you set one up, sessions still capture the full run; they just don't carry a score yet.
***
## Read the whole run as a picture
*The execution graph (left) sits beside the event timeline; the right rail breaks down the tools, models, hooks, and token spend for the run.*
Click any session to open its execution graph: a git-style view of how agents, tools, hooks, and model calls unfolded over time. Parallel sub-agents each branch onto their own lane, so you can see which work ran side by side, which sub-agent stalled, and where the run went off course, without replaying it in your head from a wall of logs.
The right rail gives you the per-run breakdown: which tools and models ran, which hooks fired, and what the run spent in tokens. That is the answer to "why did this run cost so much?" or "which tool is the slow one?" sitting right next to the graph that caused it.
***
## Where to find it
Every dashboard page is scoped to your org (`//…`). Sessions lives under **Observe** in the left sidebar, next to Events, with the date range, environment, agent, and session filters across the top of the list. Every row is one click from its full execution graph.
To turn on the score badges and score-range filtering, connect an evaluator: see [Evaluations](/agenteye/evaluations).
***
## Related
* [Event stream](/agenteye/event-stream): the raw, per-step trail every session is rolled up from.
* [Evaluations](/agenteye/evaluations): connect an evaluator so each run gets a score badge you can filter by.
* [Telemetry](/agenteye/telemetry): how runs get from your agent into these sessions.
# Performance Metrics
Source: https://docs.befailproof.ai/agenteye/telemetry
See the instant your models, tools, or hooks slow down or run up a bill, and catch a tail-latency spike before your users ever feel it.
See the instant your models, tools, or hooks slow down or run up a bill, and catch a tail-latency spike before your users ever feel it. Three dedicated pages turn raw timings into p50, p95, and p99 you can read at a glance.
*The Models page: a latency heat-map, a percentile band, and per-model tokens, estimated cost, and context-window fill.*
## Stop letting averages hide your worst runs
An average latency number is comforting and useless: it smooths over the one call in fifty that stalls and pages your on-call at 2am. The Models, Tools, and Hooks pages refuse to do that. Each shares the same shape, so you learn it once:
* A **24-bin sparkline** for the trend at a glance: is this getting worse?
* A **vitals strip** with p50, p95, and p99 latency, so the typical run and the tail sit side by side.
* A **latency heat-map**, 24 time bins by latency buckets, that shows *when* the slow calls clustered.
* A **percentile band**: a p50 line with p25 to p75 and p10 to p90 shaded ribbons and p99 dots, so the spread stays visible instead of averaged away.
A shared hover crosshair links the heat-map and the band, so a tail spike lines up in time across both instead of hiding behind a single mean line. Find all three pages in the **observe** section of your dashboard, each scoped to your organization and filterable by date range, environment, agent, and session.
## Models: see exactly what each model costs you
The Models page (shown up top) answers the two questions a bill always raises: which model, and how much. On top of the shared latency view, it adds **per-model token consumption**, **estimated cost**, and **context-window fill**, so runaway prompt growth and an impending compaction are visible before they surprise you.
Failproof AI Observability recognizes common model IDs automatically. If a window looks wrong, or you run a private model of your own, correct it or add one under **Settings**, in **model context windows**, and the fill readouts follow.
## Tools: tell the slow apart from the broken
A tool call can be slow, or it can be quietly failing, and you want to know which one in seconds, not after digging through logs.
*The Tools page: the same heat-map and percentile band, plus a success and failure breakdown and a tool-distribution bar.*
Alongside the shared latency view, the Tools page adds a **success and failure breakdown** and a **tool-distribution bar**, so you see at a glance which tools you lean on most and which are eating your error budget.
## Hooks: pinpoint the exact hook and trigger
When a lifecycle hook drags a run, "hooks are slow" is not something you can act on. The Hooks page gets you to the one that matters.
*The Hooks page: latency broken down by hook name and trigger event.*
Over the same latency heat-map and percentile band, the Hooks page breaks activity down by **hook name** and **trigger event**, so you land on the single hook and the single event that need attention.
## Related
* [Event stream](/agenteye/event-stream): the live, colour-coded trail of every event.
* [Sessions](/agenteye/sessions): roll events up into one row per run and open its execution graph.
* [Error tracking](/agenteye/error-tracking): one triage surface for everything the dashboard paints red.
* [Dashboards](/agenteye/dashboards): roll-up views across your fleet.
# Architecture
Source: https://docs.befailproof.ai/architecture
How the hook handler, config loading, and policy evaluation work internally
This document explains how failproofai works internally: how the hook system intercepts agent tool calls, how configuration is loaded and merged, how policies are evaluated, and how the dashboard monitors agent activity.
***
## Overview
failproofai has two independent subsystems:
1. **Hook handler** - A fast CLI subprocess that Claude Code invokes on every agent tool call. Evaluates policies and returns a decision.
2. **Agent Monitor (Dashboard)** - A Next.js web application for monitoring agent sessions and managing policies.
Both subsystems share configuration files in `~/.failproofai/` and the project's `.failproofai/` directory, but they run as separate processes and communicate only through the filesystem.
***
## Hook handler
### Integration with Claude Code
When you run `failproofai policies --install`, it writes entries like this into `~/.claude/settings.json`:
```json theme={null}
{
"hooks": {
"PreToolUse": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "failproofai --hook PreToolUse"
}
]
}
],
"PostToolUse": [ ... ]
}
}
```
Claude Code then invokes `failproofai --hook PreToolUse` as a subprocess before each tool call, passing a JSON payload on stdin.
### Payload format
```json theme={null}
{
"session_id": "abc123",
"transcript_path": "/home/user/.claude/projects/myproject/sessions/abc123.jsonl",
"cwd": "/home/user/myproject",
"permission_mode": "default",
"hook_event_name": "PreToolUse",
"tool_name": "Bash",
"tool_input": { "command": "sudo apt install nodejs" }
}
```
For `PostToolUse` events, the payload also contains `tool_result` with the tool's output.
The handler enforces a 1 MB stdin limit. Payloads exceeding this are discarded and all policies implicitly allow.
### Response format
**Deny (PreToolUse):**
```json theme={null}
{
"hookSpecificOutput": {
"permissionDecision": "deny",
"permissionDecisionReason": "Blocked by failproofai: sudo command blocked"
}
}
```
**Deny (PostToolUse):**
```json theme={null}
{
"hookSpecificOutput": {
"additionalContext": "Blocked by failproofai because: API key detected in output"
}
}
```
**Instruct (any event except Stop):**
```json theme={null}
{
"hookSpecificOutput": {
"additionalContext": "Instruction from failproofai: Verify tests pass before committing."
}
}
```
**Stop event instruct:**
* Exit code: `2`
* Reason written to stderr (not stdout)
**Allow:**
* Exit code: `0`
* Empty stdout
**Allow with message:**
`allow(message)` lets a policy send informational context back to Claude even when the operation is permitted. The hook handler writes the following JSON to **stdout** (not a config file — this is the handler's response to Claude Code, just like deny and instruct responses above):
```json theme={null}
// Written to stdout by the hook handler process
{
"hookSpecificOutput": {
"additionalContext": "All CI checks passed on branch 'feat/my-feature'."
}
}
```
* Exit code: `0` (operation is allowed)
* When multiple policies return `allow` with a message, their messages are joined with newlines into a single `additionalContext` string
* If no policy provides a message, stdout is empty (same as before)
### Processing pipeline
`src/hooks/handler.ts` implements the full pipeline:
```text theme={null}
stdin JSON
→ parse payload (max 1 MB)
→ extract session metadata (session_id, cwd, tool_name, tool_input, etc.)
→ readMergedHooksConfig(cwd) ← merges project + local + global config
→ register enabled builtin policies with resolved params
→ load custom policies from customPoliciesPath (if set)
→ register custom policies into policy registry
→ evaluate all policies (builtins first, then custom)
→ first deny short-circuits
→ instruct decisions accumulate
→ allow messages accumulate
→ write JSON decision to stdout
→ persist event to ~/.failproofai/hook-activity/current.jsonl
→ exit
```
The entire process runs in under 100ms for typical payloads with no LLM calls.
***
## Configuration loading
`src/hooks/hooks-config.ts` implements three-scope config loading.
```text theme={null}
[1] {cwd}/.failproofai/policies-config.json ← project (highest priority)
[2] {cwd}/.failproofai/policies-config.local.json ← local
[3] ~/.failproofai/policies-config.json ← global (lowest priority)
```
Merge logic:
* `enabledPolicies` - deduplicated union across all three files
* `policyParams` - per-policy key, first file that defines it wins entirely
* `customPoliciesPath` - first file that defines it wins
* `llm` - first file that defines it wins
The web dashboard uses `readHooksConfig()` (global only) for reading and writing, since it is not invoked with a project cwd.
***
## Policy evaluation
`src/hooks/policy-evaluator.ts` runs policies in order.
For each policy:
1. Look up the policy's `params` schema (if it has one).
2. Read `policyParams[policy.name]` from the merged config.
3. Merge user-provided values over schema defaults to produce `ctx.params`.
4. Call `policy.fn(ctx)` with the resolved context.
5. If the result is `deny`, stop immediately and return that decision.
6. If the result is `instruct`, accumulate the message and continue.
7. If the result is `allow`, continue to the next policy.
After all policies run:
* If any `deny` was returned, emit the deny response.
* If any `instruct` returns were collected, emit a single instruct response with all messages joined.
* Otherwise, emit an allow response (empty stdout, exit 0).
***
## Builtin policies
`src/hooks/builtin-policies.ts` defines all 39 built-in policies as `BuiltinPolicyDefinition` objects:
```typescript theme={null}
interface BuiltinPolicyDefinition {
name: string;
description: string;
fn: (ctx: PolicyContext) => PolicyResult;
match: {
events: HookEventType[];
tools?: string[];
};
defaultEnabled: boolean;
category: string;
beta?: boolean;
params?: PolicyParamsSchema;
}
```
Policies that accept `params` declare a `PolicyParamsSchema` with types and defaults for each parameter. The policy evaluator injects resolved values into `ctx.params` before calling `fn`. Policy functions read `ctx.params` without null-guarding because defaults are always applied first.
Pattern matching inside policies uses parsed command tokens (argv), not raw string matching. This prevents bypass via shell operator injection (e.g. a pattern for `sudo systemctl status *` cannot be bypassed by appending `; rm -rf /` to the command).
***
## Custom policies
`src/hooks/custom-hooks-registry.ts` implements a `globalThis`-backed registry:
```typescript theme={null}
const REGISTRY_KEY = "__failproofai_custom_hooks__";
export const customPolicies = {
add(hook: CustomHook): void { ... }
};
export function getCustomHooks(): CustomHook[] { ... }
export function clearCustomHooks(): void { ... } // used in tests
```
`src/hooks/custom-hooks-loader.ts` loads the user's policy file:
1. Read `customPoliciesPath` from config; skip if absent.
2. Resolve to absolute path; check file exists.
3. Rewrite all `from "failproofai"` imports to the actual dist path so `customPolicies` resolves to the same `globalThis` registry.
4. Recursively rewrite transitive local imports to ensure ESM compatibility.
5. Write temporary `.mjs` files and `import()` the entry file.
6. Call `getCustomHooks()` to retrieve registered hooks.
7. Clean up all temp files in a `finally` block.
On any error (file not found, syntax error, import failure), the error is logged to `~/.failproofai/hook.log` and the loader returns an empty array. Built-in policies are unaffected.
Custom policies are evaluated after all built-in policies. A custom policy `deny` still short-circuits further custom policies (but all built-ins have already run by that point).
***
## Activity logging
After each hook event, the handler appends a JSONL line to `~/.failproofai/hook-activity/current.jsonl`, which rotates into `page--.jsonl` once it reaches a page:
```json theme={null}
{
"timestamp": "2026-04-06T12:34:56.789Z",
"sessionId": "abc123",
"eventType": "PreToolUse",
"toolName": "Bash",
"policyName": "block-sudo",
"decision": "deny",
"reason": "sudo command blocked by failproofai",
"durationMs": 12
}
```
One line per policy that made a non-allow decision. Allow decisions are not logged (to keep the file small).
***
## Dashboard architecture
The dashboard is a **Next.js 16** application using the App Router with React Server Components and Server Actions.
```text theme={null}
app/
layout.tsx ← Root layout (theme, telemetry, nav)
projects/page.tsx ← Server component: list all Claude projects
project/[name]/page.tsx ← Server component: list sessions in a project
project/[name]/session/
[sessionId]/page.tsx ← Server component: render session viewer
policies/page.tsx ← Client component: policy management + activity log
actions/
get-hooks-config.ts ← Read config + policy list
update-hooks-config.ts ← Toggle policy on/off
update-policy-params.ts ← Update policy parameters
get-hook-activity.ts ← Paginate/search activity log
install-hooks-web.ts ← Install/remove hooks from the browser
api/
download/[project]/[session]/route.ts ← Per-CLI session export (JSONL or JSON)
```
**Data flow:**
* Page components call `lib/projects.ts` and `lib/log-entries.ts` to read project/session data directly from the filesystem (no API layer for reads).
* The Policies page uses Server Actions for all mutations (toggle, params update, install/remove).
* The session viewer parses Claude's JSONL transcript format and renders a timeline of messages and tool calls.
**Key design decisions:**
* No database - all persistent state is in plain files (`~/.failproofai/`, `~/.claude/projects/`).
* Server Actions for mutations - no REST API needed for CRUD operations.
* React Server Components for read pages - faster initial load, no client bundle for data fetching.
* Client components only where interactivity is needed (policy toggles, activity search, log viewer).
***
## File layout
```text theme={null}
failproofai/
├── bin/
│ └── failproofai.mjs # CLI router (hook / dashboard / install / etc.)
├── src/hooks/
│ ├── handler.ts # Hook event pipeline
│ ├── builtin-policies.ts # 39 policy definitions
│ ├── policy-evaluator.ts # Policy execution engine
│ ├── policy-registry.ts # Policy registration and lookup
│ ├── policy-types.ts # TypeScript interfaces
│ ├── hooks-config.ts # Multi-scope config loading
│ ├── custom-hooks-registry.ts # globalThis-backed hook registry
│ ├── custom-hooks-loader.ts # ESM loader for user JS hooks
│ ├── manager.ts # install / remove / list operations
│ ├── install-prompt.ts # Interactive policy selection prompt
│ ├── hook-logger.ts # Logging to hook.log
│ ├── hook-activity-store.ts # Persist activity to hook-activity/
│ └── llm-client.ts # LLM API client (for AI-powered policies)
├── app/ # Next.js dashboard (pages + server actions)
├── lib/ # Shared utilities
│ ├── projects.ts # Enumerate Claude projects from filesystem
│ ├── log-entries.ts # Parse Claude transcript JSONL format
│ ├── paths.ts # Resolve system paths
│ └── ...
├── components/ # Shared React UI components
├── contexts/ # React context providers (theme, auto-refresh, telemetry)
├── examples/ # Example custom hook files
└── __tests__/ # Unit and E2E tests
```
# Built-in Policies
Source: https://docs.befailproof.ai/built-in-policies
All 39 built-in policies that catch common agent failure modes
failproofai ships with 39 built-in policies that catch common agent failure modes. Each policy fires on a specific hook event type and tool name. Nineteen policies accept parameters that let you tune their behavior without writing code. Five workflow policies enforce a commit → push → PR → CI pipeline before Claude stops.
***
## Overview
Policies are grouped into categories:
| Category | Policies | Hook type |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [Dangerous commands](#dangerous-commands) | block-sudo, block-rm-rf, block-curl-pipe-sh, block-failproofai-commands, block-self-pause | PreToolUse |
| [Infra commands](#infra-commands) | block-kubectl, block-terraform, block-aws-cli, block-gcloud, block-az-cli, block-helm, block-gh-pipeline | PreToolUse |
| [Secrets (sanitizers)](#secrets-sanitizers) | sanitize-jwt, sanitize-api-keys, sanitize-connection-strings, sanitize-private-key-content, sanitize-bearer-tokens | PostToolUse |
| [Environment](#environment) | block-env-files, protect-env-vars | PreToolUse |
| [File access](#file-access) | block-read-outside-cwd, block-secrets-write | PreToolUse |
| [Git](#git) | block-push-master, block-work-on-main, block-force-push, warn-git-amend, warn-git-stash-drop, warn-all-files-staged | PreToolUse |
| [Database](#database) | warn-destructive-sql, warn-schema-alteration | PreToolUse |
| [Warnings](#warnings) | warn-large-file-write, warn-package-publish, warn-background-process, warn-global-package-install | PreToolUse |
| [Package managers](#package-managers) | prefer-package-manager | PreToolUse |
| [Workflow](#workflow) | require-commit-before-stop, require-push-before-stop, require-pr-before-stop, require-no-conflicts-before-stop, require-ci-green-before-stop | Stop |
* **`block-`** — stop the agent from proceeding.
* **`warn-`** — give the agent additional context so it can self-correct.
* **`sanitize-`** — scrub sensitive data from tool output before the agent sees it.
### Namespaces
Every policy lives in a `/` slot. Built-in policies belong to the
**`failproofai/`** namespace — for example, `failproofai/sanitize-jwt`. The
namespace prevents collisions when you also load custom or third-party policies
with similar short names.
In your config you can refer to a built-in by either its short name or its
qualified name; both forms resolve to the same policy:
```json theme={null}
{
"enabledPolicies": [
"sanitize-jwt",
"failproofai/block-rm-rf"
]
}
```
If a name has no `/`, failproofai treats it as belonging to the default
namespace `failproofai`. Names that already contain a `/` (e.g. `myorg/foo`,
`custom/my-hook`) are kept as-is.
* **`require-`** — block the Stop event until conditions are met.
***
Every policy supports an optional `hint` field in `policyParams`. The hint is appended to the deny or instruct message Claude sees, giving actionable guidance without modifying policy code. Works with built-in, custom, and convention policies. See [Configuration → hint](/configuration#hint-cross-cutting) for details.
***
## Dangerous commands
Prevent agents from running operations that are hard to undo or that could damage the host system.
### `block-sudo`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `sudo` or `doas` command.
Blocks a command that runs an elevation binary **in command position**. Matching is structural rather than textual: the command is split into segments the way a shell would, prefix assignments (`FOO=bar`), redirections, and runners with their flags (`env`, `nohup`, `timeout`, `xargs`, `sh -c`, …) are walked off, and the resulting binary is compared by **basename**. So `/usr/bin/sudo`, `env sudo`, `timeout 5 sudo`, `"sudo"`, `\sudo` and `bash -c "sudo …"` are all denied, and `doas` is treated as the same capability under a different name.
Because it anchors on command position rather than on the word appearing anywhere, it does **not** fire on commands that merely mention it — `grep -r sudo /etc`, `cat /etc/sudoers`, `git commit -m "fix sudo handling"`, or a `grep` alternation containing the word all run normally.
This stops the obvious attempt; it does not close the class. An agent that can run arbitrary shell can still reach elevation indirectly — through a variable (`S=sudo; $S …`), a base64-decoded pipe, or a wrapper script on disk — because no inspection of a single command string can follow those. Treat this as a guardrail against mistakes and casual escalation, not as a security boundary against a determined agent. A real boundary has to be enforced below the shell.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | ------------------------------------------------------------------------------------------------ |
| `allowPatterns` | `string[]` | `[]` | Exact command prefixes that are permitted. Each entry is matched against the parsed argv tokens. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-sudo": {
"allowPatterns": ["sudo systemctl status", "sudo journalctl"]
}
}
}
```
With this config, `sudo systemctl status nginx` is allowed, but `sudo rm /etc/hosts` is denied.
Patterns are matched against parsed tokens, not the raw command string. This prevents bypass via appended shell operators (e.g. `sudo systemctl status x; rm -rf /` does not match `sudo systemctl status *`).
***
### `block-rm-rf`
**Event:** PreToolUse (Bash)\
**Default:** Denies `rm -rf`, `rm -fr`, and similar recursive deletion forms.
**Parameters:**
| Param | Type | Default | Description |
| ------------ | ---------- | ------- | -------------------------------------------------------- |
| `allowPaths` | `string[]` | `[]` | Paths that are safe to recursively delete (e.g. `/tmp`). |
**Example:**
```json theme={null}
{
"policyParams": {
"block-rm-rf": {
"allowPaths": ["/tmp", "/var/cache"]
}
}
}
```
***
### `block-curl-pipe-sh`
**Event:** PreToolUse (Bash)\
**Default:** Denies `curl | bash`, `curl | sh`, `wget | bash`, and similar patterns.
No parameters.
***
### `block-failproofai-commands`
**Event:** PreToolUse (Bash)\
**Default:** Denies commands that would uninstall or disable failproofai itself (e.g. `npm uninstall failproofai`, `failproofai policies --uninstall`).
No parameters.
***
### `block-self-pause`
**Event:** PreToolUse (Bash)\
**Default:** Denies `failproofai config --pause`, which suspends enforcement for a session. Pausing is a human decision — an agent able to run it could switch off every other policy with a single command.
Narrower than [`block-failproofai-commands`](#block-failproofai-commands) on purpose, and not covered by it: that policy anchors on a command boundary, so `npx -y failproofai config --pause` does not match it, and being broad it is often switched off so agents can run `failproofai audit`. `--resume` and `--status` are allowed — neither removes enforcement.
This stops the direct attempt, not the whole class: an agent can still reach the same state through an alias or a wrapper script. Closing it fully requires the pause to be unreachable from a tool call at all.
No parameters.
***
## Infra commands
Stop coding agents from running infrastructure CLIs or triggering CI/CD pipelines. All policies in this category are **opt-in** (`defaultEnabled: false`) — agents that legitimately need to call `kubectl`, `terraform`, etc. will not be disrupted unless you enable the policy. When enabled, every invocation of the matched CLI is denied unless the command matches an entry in `allowPatterns`.
The pattern grammar is the same as [`block-sudo`](#block-sudo): tokens are matched against parsed argv, `*` is a wildcard for one token, and any command containing a standalone shell operator (`&&`, `||`, `|`, `;`) or a token with embedded shell metacharacters is rejected before allowlist matching to prevent injection bypasses.
### `block-kubectl`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `kubectl` invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | -------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | kubectl command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-kubectl": {
"allowPatterns": ["kubectl get *", "kubectl describe *", "kubectl logs *"]
}
}
}
```
With this config, `kubectl get pods` is allowed but `kubectl apply -f deploy.yaml` is denied.
***
### `block-terraform`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `terraform` or `tofu` (OpenTofu) invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | --------------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | terraform/tofu command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-terraform": {
"allowPatterns": ["terraform plan", "terraform validate", "terraform show *"]
}
}
}
```
***
### `block-aws-cli`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `aws` CLI invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | -------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | aws CLI command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-aws-cli": {
"allowPatterns": ["aws s3 ls *", "aws sts get-caller-identity"]
}
}
}
```
***
### `block-gcloud`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `gcloud` (Google Cloud) CLI invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | ------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | gcloud command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-gcloud": {
"allowPatterns": ["gcloud auth list", "gcloud config list"]
}
}
}
```
***
### `block-az-cli`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `az` (Azure) CLI invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | ------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | az CLI command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-az-cli": {
"allowPatterns": ["az account show", "az group list"]
}
}
}
```
***
### `block-helm`
**Event:** PreToolUse (Bash)\
**Default:** Denies any `helm` invocation.
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | ----------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | helm command prefixes that are permitted. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-helm": {
"allowPatterns": ["helm list", "helm status *"]
}
}
}
```
***
### `block-gh-pipeline`
**Event:** PreToolUse (Bash)\
**Default:** Denies the following `gh` CLI subcommands that mutate state or trigger pipelines:
* `gh workflow run`, `gh workflow enable`, `gh workflow disable`
* `gh run rerun`, `gh run cancel`
* `gh pr merge`
* `gh release create`, `gh release delete`
* `gh cache delete`
* `gh secret set`, `gh secret delete`
Read-only `gh` subcommands such as `gh pr view`, `gh pr list`, `gh run list`, `gh release view`, and `gh api repos/.../...` are **not** matched by this policy — they are routinely needed for workflow checks (including failproofai's own `require-ci-green-before-stop`).
**Parameters:**
| Param | Type | Default | Description |
| --------------- | ---------- | ------- | ---------------------------------------------------------------------------------- |
| `allowPatterns` | `string[]` | `[]` | Specific scripted invocations to allow even though they would otherwise be denied. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-gh-pipeline": {
"allowPatterns": ["gh run rerun *"]
}
}
}
```
***
## Secrets (sanitizers)
Stop agents from leaking credentials into their context or output. Sanitizer policies fire on **PostToolUse** events. When Claude runs a Bash command, reads a file, or calls any tool, these policies inspect the output before it is returned to Claude. If a secret pattern is detected, the policy returns a deny decision that prevents the output from being passed back.
### `sanitize-jwt`
**Event:** PostToolUse (all tools)\
**Default:** Redacts JWT tokens (three base64url segments separated by `.`).
No parameters.
***
### `sanitize-api-keys`
**Event:** PostToolUse (all tools)\
**Default:** Redacts common API key formats: Anthropic (`sk-ant-`), OpenAI (`sk-`), GitHub PATs (`ghp_`), AWS access keys (`AKIA`), Stripe keys (`sk_live_`, `sk_test_`), and Google API keys (`AIza`).
**Parameters:**
| Param | Type | Default | Description |
| -------------------- | ------------------------------------ | ------- | ---------------------------------------------- |
| `additionalPatterns` | `{ regex: string; label: string }[]` | `[]` | Additional regex patterns to treat as secrets. |
**Example:**
```json theme={null}
{
"policyParams": {
"sanitize-api-keys": {
"additionalPatterns": [
{ "regex": "myco_[A-Za-z0-9]{32}", "label": "MyCo internal API key" },
{ "regex": "pat_[0-9a-f]{40}", "label": "Internal PAT" }
]
}
}
}
```
***
### `sanitize-connection-strings`
**Event:** PostToolUse (all tools)\
**Default:** Redacts database connection strings that contain embedded credentials (e.g. `postgresql://user:password@host/db`).
No parameters.
***
### `sanitize-private-key-content`
**Event:** PostToolUse (all tools)\
**Default:** Redacts PEM blocks (`-----BEGIN PRIVATE KEY-----`, `-----BEGIN RSA PRIVATE KEY-----`, etc.).
No parameters.
***
### `sanitize-bearer-tokens`
**Event:** PostToolUse (all tools)\
**Default:** Redacts `Authorization: Bearer ` headers where the token is 20 or more characters.
No parameters.
***
## Environment
Protect sensitive environment configuration from being read or exposed by agents.
### `block-env-files`
**Event:** PreToolUse (Bash, Read)\
**Default:** Denies reading `.env` files via `cat .env`, `Read` tool calls with `.env` as the file path, etc.
Does not block `.envrc` or other environment-adjacent files - only files named exactly `.env`.
No parameters.
***
### `protect-env-vars`
**Event:** PreToolUse (Bash)\
**Default:** Denies commands that print environment variables: `printenv`, `env`, `echo $VAR`.
No parameters.
***
## File access
Keep agents working inside project boundaries and away from sensitive files.
### `block-read-outside-cwd`
**Event:** PreToolUse (Read, Bash)\
**Default:** Denies reading files outside the project root. The boundary is `CLAUDE_PROJECT_DIR` (set once per session by Claude Code), with a fallback to the session's current working directory when that variable is unset. Using the project root rather than the live `cwd` means the boundary stays stable even after Claude `cd`s into a subdirectory.
**Parameters:**
| Param | Type | Default | Description |
| ------------ | ---------- | ------- | --------------------------------------------------------------------------- |
| `allowPaths` | `string[]` | `[]` | Absolute path prefixes that are permitted even if outside the project root. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-read-outside-cwd": {
"allowPaths": ["/shared/data", "/opt/company/config"]
}
}
}
```
***
### `block-secrets-write`
**Event:** PreToolUse (Write, Edit)\
**Default:** Denies writes to files commonly used for private keys and certificates: `id_rsa`, `id_ed25519`, `*.key`, `*.pem`, `*.p12`, `*.pfx`.
**Parameters:**
| Param | Type | Default | Description |
| -------------------- | ---------- | ------- | --------------------------------------------------- |
| `additionalPatterns` | `string[]` | `[]` | Additional filename patterns (glob-style) to block. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-secrets-write": {
"additionalPatterns": [".token", ".secret"]
}
}
}
```
***
## Git
Prevent accidental pushes, force-pushes, and branch mistakes that are hard to undo.
### `block-push-master`
**Event:** PreToolUse (Bash)\
**Default:** Denies `git push origin main` and `git push origin master`.
**Parameters:**
| Param | Type | Default | Description |
| ------------------- | ---------- | -------------------- | ----------------------------------------------- |
| `protectedBranches` | `string[]` | `["main", "master"]` | Branch names that cannot be pushed to directly. |
**Example:**
```json theme={null}
{
"policyParams": {
"block-push-master": {
"protectedBranches": ["main", "master", "release", "prod"]
}
}
}
```
To allow pushing to all branches (effectively disabling this policy without removing it from `enabledPolicies`), set `protectedBranches: []`.
***
### `block-work-on-main`
**Event:** PreToolUse (Bash)\
**Default:** Denies `git commit`, `git merge`, `git rebase`, and `git cherry-pick` while the working tree is on `main` or `master`. Branch creation and switching (`git checkout`, `git checkout -b`, `git switch`, `git switch -c`) are not affected.
**Parameters:**
| Param | Type | Default | Description |
| ------------------- | ---------- | -------------------- | ---------------------------------------------------------------- |
| `protectedBranches` | `string[]` | `["main", "master"]` | Branch names on which commit/merge/rebase/cherry-pick is denied. |
***
### `block-force-push`
**Event:** PreToolUse (Bash)\
**Default:** Denies `git push --force` and `git push -f`.
No policy-specific parameters. Use the cross-cutting [`hint`](/configuration#hint-cross-cutting) to suggest alternatives:
```json theme={null}
{
"policyParams": {
"block-force-push": {
"hint": "Create a new branch from your current HEAD (e.g. `git checkout -b `) and push that instead."
}
}
}
```
***
### `warn-git-amend`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to proceed carefully when running `git commit --amend`. Does not block the command.
No parameters.
***
### `warn-git-stash-drop`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to confirm before running `git stash drop`. Does not block the command.
No parameters.
***
### `warn-all-files-staged`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to review what it is staging when it runs `git add -A` or `git add .`. Does not block the command.
No parameters.
***
## Database
Catch destructive SQL operations before they execute against your database.
### `warn-destructive-sql`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to confirm before running SQL containing `DROP TABLE`, `DROP DATABASE`, or `DELETE` without a `WHERE` clause.
No parameters.
***
### `warn-schema-alteration`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to confirm before running `ALTER TABLE` statements.
No parameters.
***
## Warnings
Give agents extra context before potentially risky but non-destructive operations.
### `warn-large-file-write`
**Event:** PreToolUse (Write)\
**Default:** Instructs Claude to confirm before writing files larger than 1024 KB.
**Parameters:**
| Param | Type | Default | Description |
| ------------- | -------- | ------- | ----------------------------------------------------------------- |
| `thresholdKb` | `number` | `1024` | File size threshold in kilobytes above which a warning is issued. |
**Example:**
```json theme={null}
{
"policyParams": {
"warn-large-file-write": {
"thresholdKb": 256
}
}
}
```
The hook handler enforces a 1 MB stdin limit on payloads. To test this policy with small content, set `thresholdKb` to a value well below 1024.
***
### `warn-package-publish`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to confirm before running `npm publish`.
No parameters.
***
### `warn-background-process`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to be careful when launching background processes via `nohup`, `&`, `disown`, or `screen`.
No parameters.
***
### `warn-global-package-install`
**Event:** PreToolUse (Bash)\
**Default:** Instructs Claude to confirm before running `npm install -g`, `yarn global add`, or `pip install` without a virtual environment.
No parameters.
***
## Package managers
Enforce which package managers the agent is allowed to use.
### `prefer-package-manager`
**Event:** PreToolUse (Bash)\
**Default:** Disabled. When enabled, blocks any package manager command not in the `allowed` list and tells Claude to rewrite the command using an allowed manager.
Detects: pip, pip3, python -m pip, npm, npx, yarn, pnpm, pnpx, bun, bunx, uv, poetry, pipenv, conda, cargo.
| Parameter | Type | Default | Description |
| --------- | --------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| `allowed` | string\[] | `[]` | Allowed package manager names. Any detected manager not in this list is blocked. When empty, the policy is a no-op. |
| `blocked` | string\[] | `[]` | Additional manager names to block beyond the built-in list (e.g. `['pdm', 'pipx']`). |
The built-in block list covers: pip, pip3, npm, npx, yarn, pnpm, pnpx, bun, bunx, uv, poetry, pipenv, conda, cargo. Use `blocked` to append managers not in this list.
**Example configuration:**
```json theme={null}
{
"enabledPolicies": ["prefer-package-manager"],
"policyParams": {
"prefer-package-manager": {
"allowed": ["uv", "bun"],
"blocked": ["pdm", "pipx"]
}
}
}
```
With this config, `pip install flask` and `pdm install flask` are both denied with a message telling Claude to use `uv` or `bun` instead. Commands like `uv pip install flask` are allowed because `uv` is in the allowlist and is checked first.
***
## AI behavior
Detect when agents get stuck or behave unexpectedly.
### `warn-repeated-tool-calls`
**Event:** PreToolUse (all tools)\
**Default:** Instructs Claude to reconsider when the same tool is called 3+ times with identical parameters - a common sign the agent is stuck in a loop.
No parameters.
***
## Workflow
Enforce a disciplined end-of-session workflow. These policies fire on the **Stop** event and deny the agent from stopping until each condition is met. They follow a natural dependency chain: commit → push → PR → CI. If a policy denies, later policies in the chain are skipped (deny short-circuits).
All workflow policies are **fail-open**: if the required tool is not available (e.g. `gh` not installed, no git remote), the policy allows with an informational message explaining why the check was skipped.
### Per-CLI Stop semantics
Stop enforcement looks slightly different across the six supported CLIs because each one exposes a different "agent finished" hook contract. The **outcome** is the same — the agent doesn't get away with stopping while a workflow gate is failing — but the **mechanics** differ. The table below summarizes; only Pi has a user-visible quirk worth understanding before you enable a `require-*-before-stop` policy.
| CLI | When the gate fires | What you see |
| ------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Claude Code | Same agent loop, immediately | Claude continues working — fixes the issue, then attempts to finish again. No interruption visible to you. |
| Codex | Same agent loop, immediately | Same as Claude. |
| GitHub Copilot CLI | Same agent loop, immediately | Same as Claude (uses Copilot's `{decision:"block", reason}` retry channel — verified empirically against Copilot CLI 1.0.41). |
| Cursor Agent | Same agent loop, immediately | Same as Claude (uses Cursor's `{followup_message}` channel — capped at `loop_limit`, default 5 retries). |
| OpenCode | Same agent loop, immediately | Same as Claude (uses OpenCode's `client.session.prompt(...)` SDK call routed through `hookSpecificOutput.additionalContext`). |
| **Pi (pi-coding-agent)** | **Next user turn** | **Pi visibly stops** when the gate fires — its agent loop exits and you're returned to the prompt. The gate then fires the next time you submit a prompt: failproofai prepends a `MANDATORY ACTION REQUIRED` directive to that turn's system prompt, instructing the LLM to complete the workflow step (commit, push, etc.) before doing whatever you asked. |
**Pi limitation.** Pi's `AgentEndEvent` (the upstream equivalent of Claude's `Stop` hook) has no Result type — by the time it fires, Pi's agent loop has already exited. Pi cannot be forced to retry the same loop the way Claude / Copilot / Cursor / OpenCode can. failproofai shifts the gate to Pi's `before_agent_start` event (which fires after the next user prompt) so the workflow check still enforces, just on the next turn rather than the current one.
**What this means in practice:**
* After Pi stops, the deny reason is captured in-memory keyed by Pi session id. The very next prompt you submit in the same Pi process drains it: the LLM sees the `MANDATORY ACTION REQUIRED` directive at the top of its system prompt, commits (or pushes / opens the PR / waits for CI), and only then continues with your request. The captured deny reason is one-shot — once drained, the gate is clear.
* The gate is bounded by Pi's process lifetime. If you `Ctrl+C` Pi or quit between turns, the in-memory entry is dropped along with the process and the gate is missed. Claude, Copilot, Cursor, and OpenCode have the same bound (kill the agent and the gate is missed) — Pi just makes it more visible because the agent visibly exits before the gate fires.
* A pending deny is also cleared on `session_shutdown` for any reason (`new` / `resume` / `fork` / `quit`), so a stale gate from a prior session cannot leak into a fresh session started in the same Pi process.
If you need Claude-style same-loop retry, run your `Stop` policies under any of the other five supported CLIs. We are tracking Pi upstream for a future Result type on `AgentEndEvent` that would let us close this gap.
### `require-commit-before-stop`
**Event:** Stop\
**Default:** Denies stopping when there are uncommitted changes (modified, staged, or untracked files). Returns an informational message when the working directory is clean.
No parameters.
***
### `require-push-before-stop`
**Event:** Stop\
**Default:** Denies stopping when there are unpushed commits or when the current branch has no remote tracking branch. Suggests `git push -u` to create a tracking branch if needed. Fails open if no remote is configured.
**Parameters:**
| Param | Type | Default | Description |
| -------- | -------- | ---------- | ----------------------- |
| `remote` | `string` | `"origin"` | Remote name to push to. |
**Example:**
```json theme={null}
{
"policyParams": {
"require-push-before-stop": {
"remote": "upstream"
}
}
}
```
***
### `require-pr-before-stop`
**Event:** Stop\
**Default:** Denies stopping when no pull request exists for the current branch, or when the existing PR is closed without merging. Instructs Claude to create a PR with `gh pr create`. When the PR is **merged**, the policy allows (the work has shipped) and the message hints to switch off the branch (`git checkout main && git pull`).
No parameters.
This policy requires [GitHub CLI](https://cli.github.com/) (`gh`) to be installed and authenticated.
Run `gh auth login` with a personal access token that has `repo` scope for read access to
pull requests. If `gh` is not installed or not authenticated, the policy fails open and reports the reason to Claude.
***
### `require-no-conflicts-before-stop`
**Event:** Stop\
**Default:** Denies stopping when the current branch cannot cleanly merge into the base branch. The policy first confirms there is an `OPEN` PR on GitHub for the branch — without one, there is no merge target to enforce, so the entire policy short-circuits to allow. Once an `OPEN` PR is confirmed, two independent probes run:
1. **Local** — `git merge-tree --write-tree --name-only origin/ HEAD`. On conflict, the deny message names the conflicted files so Claude knows exactly what to resolve.
2. **GitHub** — reuses the `gh pr view --json mergeable,state` result already fetched in the precheck. Catches conflicts that a stale local `origin/` would miss (e.g. someone landed a conflicting PR on `main` since the last fetch). A `CONFLICTING` result denies. An `UNKNOWN` result also denies and instructs Claude to wait \~10 seconds and re-check before attempting to stop again — this prevents false negatives while GitHub recomputes.
Skips entirely (allows) when: `gh` is not installed, no PR exists for the branch, the PR's state is not `OPEN` (e.g. `MERGED`, `CLOSED`), or `gh pr view` returns unparseable output. Also fails open when `origin/` is missing locally or when no commits are ahead of base — those Layer 1 fall-throughs still consult the cached PR mergeability before allowing.
**Parameters:**
| Param | Type | Default | Description |
| ------------ | -------- | -------- | ------------------------------------------- |
| `baseBranch` | `string` | `"main"` | Base branch to check for conflicts against. |
GitHub CLI (`gh`) is required for this policy. The policy uses `gh pr view` to confirm
an `OPEN` PR exists before running any conflict probe — without `gh`, the policy
short-circuits to allow. Run `gh auth login` with a personal access token that has
`repo` scope for read access to pull requests.
***
### `require-ci-green-before-stop`
**Event:** Stop\
**Default:** Denies stopping when CI checks are failing or still running on the current branch. Checks both GitHub Actions workflow runs and third-party bot checks (e.g. CodeRabbit, SonarCloud, Codecov). Treats `skipped`, `cancelled`, and `neutral` conclusions as non-failing (the latter covers e.g. Socket Security alerts on outside contributor PRs, where the app intentionally reports neutral rather than success/failure). Returns an informational message when all checks pass.
No parameters.
This policy requires [GitHub CLI](https://cli.github.com/) (`gh`) to be installed and authenticated.
Run `gh auth login` with a personal access token that has `repo` scope for read access to
Actions workflow runs and the Checks API. If `gh` is not installed or not authenticated, the policy fails open and reports the reason to Claude.
***
***
## Disabling individual policies
Remove a specific policy from `enabledPolicies` in your config, or toggle it off in the dashboard's Policies tab.
```json theme={null}
{
"enabledPolicies": [
"block-rm-rf",
"sanitize-api-keys"
]
}
```
Policies not listed in `enabledPolicies` do not run, even if `policyParams` entries exist for them.
# Audit past sessions (beta)
Source: https://docs.befailproof.ai/cli/audit
Count how often the agent did wasteful or risky things across past transcripts
**Beta feature.** The audit ships as beta while we collect early feedback.
The detector catalog and report format may change before the next stable
cut. Please open an issue if anything looks off.
The audit replays your past agent-CLI transcripts through failproofai's policy
engine and renders a shareable, visual report on the **`/audit` dashboard
page** — your agent's archetype, a 0–100 score, and exactly which policies
would have caught what.
## Run it
Three ways in — all land on the same `/audit` report.
```bash npx (no install) theme={null}
npx -y failproofai audit
```
```bash failproofai audit theme={null}
failproofai audit
```
```bash failproofai (dashboard) theme={null}
failproofai
```
`npx -y failproofai audit` fetches failproofai, runs the scan, and opens the
dashboard for you — nothing to install first.
`failproofai audit` runs the scan in your terminal, then opens
`localhost:8020/audit` automatically when it finishes.
Run `failproofai` and click **Audit** in the navbar (between Policies and
Projects), or open `/audit` directly.
Run `failproofai audit -h` (or `--help`) to see usage. The audit runs **fully
offline** — no account or network required — and the dashboard keeps serving
until you stop it with `Ctrl+C`.
The dashboard scans past agent CLI transcripts on this machine (Claude Code, Codex, Copilot, Cursor, OpenCode, Pi) and reports how often the agent did things failproofai is built to stop — env-var checks, force pushes, redundant `cd ` prefixes, sleep-polling loops, re-reading files just edited, and more.
For each transcript, every tool-use event is replayed through the 39 builtin policies **and** through 8 audit-only detectors that catch patterns not yet covered by runtime policies. Counts are aggregated per policy / detector across all sessions.
## What you get
The `/audit` page is a single-screen, shareable **poster** followed by four below-the-fold sections:
1. **Poster** — your agent's identity at a glance: its **archetype** (one of 8 — `optimist`, `cowboy`, `explorer`, `goldfish`, `paranoid architect`, `precision builder`, `hammer`, `ghost`), its persona keywords, how rare that archetype is, and a **0–100 score** with a tier band (`S` down to `bottom tier`). Built to share — post to X or LinkedIn, or download it as a PNG.
2. **`// strengths`** — what your agent already does well, as real numbers from the scan (e.g. clean-tool-call %, `0` push-to-main attempts), shown only where the relevant policy has a clean record.
3. **`// quirks`** — what slipped through: a ranked table of behaviors failproofai would have caught — *when* it last happened, *what slipped* (and the builtin that would have blocked it), its *severity*, and how often it was *seen* (`new` / `recurring` / `N× seen`).
4. **`// how to improve`** — the prescribed fix list: one row per policy with a copy-paste `failproofai policy add `, plus an **install all** button that enables every recommendation at once and shows your **projected score** if you did.
5. **`// come back better`** — build the habit: set a re-audit email **reminder** (`3d` / `7d` / `14d` / `30d`) or re-audit now, and **invite a friend** to run their own audit (sent from failproof.ai, Cc'd to you). Reminders and invites require sign-in.
## Scheduled audits
If you run the **failproofaid daemon** (see [`failproofai config`](/cli/install-policies)),
it can re-run the audit for you on a schedule and refresh the `/audit` report in
the background. It is **off by default**, because the scan reads the *contents*
of every agent session transcript on this machine — nothing scans on a timer
until you ask for it.
Turn it on in `~/.failproofai/config.toml`:
```toml theme={null}
[audit]
auto = true
interval_days = 7
```
| Key | Meaning |
| --------------- | --------------------------------------------------------------------------------------- |
| `auto` | `true` enables the scheduled scan. Anything else — absent, `false`, `"yes"` — is off. |
| `interval_days` | Days between scans. Clamped to 1–90; `0`, a negative or a non-number falls back to `7`. |
* The schedule is **wall-clock**, so it survives suspend and reboots: a laptop
that was asleep past its due time runs **once** on wake, never a backlog.
* Each run is a separate, low-priority (`nice 19`) process — never the daemon's
hook path, which stays free to answer tool calls.
* A scan is skipped if `failproofai audit` or the dashboard's re-run is already
in flight; it is retried shortly afterwards rather than treated as a failure.
* Progress is written to `~/.failproofai/state/audit-schedule.json` (last run,
next due). The daemon owns that file — change the cadence in `config.toml`.
If you enabled this on a machine set up by an older failproofai, run
`failproofai config` once. The daemon's service definition needs one extra
entry before it can launch the CLI, and the refresh is part of that command.
## Audit-only detectors
These detect "stupid behavior" patterns not (yet) enforced in real time. They run only during the audit and never block a live tool call.
| Detector | What it counts |
| --------------------------- | -------------------------------------------------------------------------------------- |
| `redundant-cd-cwd` | Bash commands starting with `cd && …` even though commands already run in `cwd`. |
| `prefer-edit-over-read-cat` | `cat`/`head`/`tail`/`less`/`more` on a single source file — use the `Read` tool. |
| `prefer-edit-over-sed-awk` | `sed -i` / `awk … > file` in-place edits — use the `Edit` tool. |
| `prefer-write-over-heredoc` | Heredoc / multi-line `echo > file` writing files — use the `Write` tool. |
| `sleep-polling-loop` | Long `sleep N` (≥ 30s) or `while …; sleep …; done` polling loops. |
| `find-from-root` | `find /`, `find /home`, `find /usr`, etc. — scope to `cwd` instead. |
| `git-commit-no-verify` | `git commit … --no-verify` / `-n`, skipping hooks. |
| `reread-after-edit` | `Read` of a file that was just `Edit`/`Write` in the same session. |
## Caches
* **Per-transcript cache** at `~/.failproofai/cache/audit/.json` keyed by `(mtime, size, engineVersion, detectorVersion)` — invalidates automatically when the transcript or the policy/detector code changes. Each entry also stores a `cachedAt` timestamp as **TTL metadata** (not part of the cache key); entries older than **7 days** are rejected on read so long-lived results don't outlive evolving detector intent.
* **Whole-result cache** at `~/.failproofai/audit-dashboard.json` (mode 0600). Lets the dashboard render instantly on navigation without re-running. Also rejected on read past the **7-day TTL** — `/audit` then falls through to its empty state and prompts a fresh run. Click `[ re-audit now ]` near the bottom of the report to refresh — re-audit sends `noCache: true`, so it bypasses the per-transcript cache and re-scans every transcript instead of returning the cached result; the run streams progress via a sticky top strip and swaps the result in place on success (no page reload; a failed re-audit keeps the previous report).
## Notes
* **No mutation.** The audit replays in read-only mode. `warn-repeated-tool-calls` is skipped because its per-session sidecar would otherwise be modified.
* **Workflow policies skipped.** `require-*-before-stop` policies fire only on `Stop` events and `execSync` against the live git state — they have no meaningful "what would have happened in 2025" interpretation, so they don't appear in audit counts.
* **Custom policies skipped.** User-supplied custom hooks are not replayed (they may have changed since the original session).
# View sessions
Source: https://docs.befailproof.ai/cli/dashboard
Launch the dashboard to browse agent sessions and manage policies
```bash theme={null}
failproofai
```
Starts the web dashboard at `http://localhost:8020`.
## Options
| Flag | Description |
| ----------------------------- | --------------------------------------------------------- |
| `--port ` | Port to listen on (default: `8020`) |
| `--allowed-origins ` | Comma-separated hosts/IPs allowed to access dev resources |
To point the dashboard at a non-default Claude project folder, set the `CLAUDE_PROJECTS_PATH` environment variable when launching.
## Examples
```bash theme={null}
# Launch on a different port
failproofai --port 9000
# Use a custom Claude projects path via environment variable
CLAUDE_PROJECTS_PATH=~/my-claude-projects failproofai
```
# Environment variables
Source: https://docs.befailproof.ai/cli/environment-variables
Configure failproofai behavior with environment variables
## Dashboard
| Variable | Description |
| --------------------------------------------- | ----------------------------------------------------------------------- |
| `PORT` | Dashboard port (default: `8020`) |
| `CLAUDE_PROJECTS_PATH` | Override where Claude Code project folders are found |
| `FAILPROOFAI_DISABLE_PAGES=policies,projects` | Comma-separated dashboard pages to hide |
| `FAILPROOFAI_ALLOWED_DEV_ORIGINS` | Hosts/IPs allowed to access dev resources. Same as `--allowed-origins`. |
## Logging
| Variable | Description |
| ----------------------------------------- | ---------------------------------------------------------------------------------- |
| `FAILPROOFAI_LOG_LEVEL=info\|warn\|error` | Server log level (default: `warn`) |
| `FAILPROOFAI_HOOK_LOG_FILE` | Custom hook log file path, or `true` for default (`~/.failproofai/logs/hooks.log`) |
## Telemetry
failproofai reports anonymous usage telemetry by default. There are two ways to
turn it off, and they resolve to whichever is more restrictive — an environment
variable can never re-enable something the config file switched off.
| Variable | Description |
| ---------------------------------- | -------------------------------------------------- |
| `FAILPROOFAI_TELEMETRY_DISABLED=1` | Disable anonymous usage telemetry for this process |
To disable it permanently for the machine, add this to `~/.failproofai/config.toml`:
```toml theme={null}
[telemetry]
enabled = false
```
The config file is the option to use if you run the **failproofaid daemon**.
The daemon is a system-scope service, and its environment does not include
variables exported from your shell — so `FAILPROOFAI_TELEMETRY_DISABLED` cannot
reach it. `[telemetry] enabled = false` is read by both the CLI and the daemon.
The daemon reports its own **lifecycle** only: that it started (and whether the
previous run exited cleanly), that it stopped, when its evaluation worker was
spawned or restarted, when a collector task failed, and the outcome of a
cloud-policy pull. These carry low-cardinality values and counts — never a file
path, a command, a policy, a prompt, or anything read out of a transcript. There
is no per-tool-call event.
## Authentication
| Variable | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FAILPROOF_API_URL` | Override the api-server base URL used by the dashboard auth dialog. Defaults to `https://api.befailproof.ai`; set to `http://localhost:8080` (or wherever) when running a local api-server. |
| `FAILPROOFAI_AUTH_DIR` | Override where `auth.json` is stored (default: `~/.failproofai`). Mostly useful for isolated tests. |
## First-run prompt
| Variable | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------ |
| `FAILPROOFAI_NO_FIRST_RUN=1` | Skip the prompt that offers to install policies on the first bare `failproofai` invocation |
## LLM (for policy evaluation)
| Variable | Description |
| -------------------------- | ------------------------------------------------------- |
| `FAILPROOFAI_LLM_BASE_URL` | LLM API endpoint (default: `https://api.openai.com/v1`) |
| `FAILPROOFAI_LLM_API_KEY` | API key for LLM-powered policies |
| `FAILPROOFAI_LLM_MODEL` | Model name (default: `gpt-4o-mini`) |
# Hook handler (internal)
Source: https://docs.befailproof.ai/cli/hook
The subprocess Claude Code calls on each tool event
```bash theme={null}
failproofai --hook
```
This is the command registered in Claude Code's `settings.json` by `failproofai policies --install`. You don't normally call this directly.
Reads a JSON payload from stdin, evaluates all enabled policies, and exits with a code indicating the decision:
| Exit code | Decision | Effect |
| --------- | ---------- | ------------------------------------------------ |
| `0` | `allow` | Permit the action |
| `1` | `deny` | Block the action - Claude sees the denial reason |
| `2` | `instruct` | Inject guidance into Claude's context |
### Supported event types
| Category | Events |
| --------------------- | ------------------------------------------------------------------------------------------ |
| **Tool execution** | `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `PermissionDenied` |
| **Session lifecycle** | `SessionStart`, `SessionEnd`, `Stop`, `StopFailure` |
| **User interaction** | `UserPromptSubmit`, `Notification`, `Elicitation`, `ElicitationResult` |
| **Subagents & tasks** | `SubagentStart`, `SubagentStop`, `TaskCreated`, `TaskCompleted`, `TeammateIdle` |
| **Configuration** | `InstructionsLoaded`, `ConfigChange`, `CwdChanged` |
| **File system** | `FileChanged`, `WorktreeCreate`, `WorktreeRemove` |
| **Context** | `PreCompact`, `PostCompact` |
# Install policies
Source: https://docs.befailproof.ai/cli/install-policies
Enable policies so they run on every agent tool call
```bash theme={null}
failproofai policies --install [policy-names...] [options]
```
Writes hook entries into your installed agent CLI's settings file (Claude Code, OpenAI Codex, or GitHub Copilot CLI *(beta)*) so failproofai intercepts tool calls.
Aliases: `failproofai p -i`
## Options
| Flag | Description |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--cli claude\|codex\|copilot` | Agent CLI(s) to install for; space-separated (e.g. `--cli claude codex copilot`) or repeated. Omit to detect installed CLIs and prompt. |
| `--scope user` | Install into the user-scope settings file (Claude: `~/.claude/settings.json`; Codex: `~/.codex/hooks.json`; Copilot: `~/.copilot/hooks/failproofai.json`). Default. |
| `--scope project` | Install into the project-scope settings file (Claude: `/.claude/settings.json`; Codex: `/.codex/hooks.json`; Copilot: `/.github/hooks/failproofai.json`). |
| `--scope local` | Claude only — installs into `/.claude/settings.local.json`. Codex and Copilot do not have a `local` scope. |
| `--custom ` / `-c` | Path to a JS file containing custom hook policies |
## Behavior
* **No policy names** - opens an interactive prompt to select policies
* **Specific names** - enables those policies (added to any already enabled)
* **`all`** - enables every available policy
Installation is additive: running `--install` again adds new policies without removing existing ones.
## Examples
```bash theme={null}
# Install all default policies globally (interactive)
failproofai policies --install
# Install specific policies for the current project
failproofai policies --install block-sudo sanitize-api-keys --scope project
# Enable all policies at once
failproofai policies --install all
# Install with a custom policies file
failproofai policies --install --custom ./my-policies.js
# Install for OpenAI Codex (project scope)
failproofai policies --install --cli codex --scope project
# Install for GitHub Copilot CLI (beta) for the current project
failproofai policies --install --cli copilot --scope project
# Install for all three CLIs at once
failproofai policies --install --cli claude codex copilot
```
When `--custom ` is provided, the file is validated immediately - it must call `customPolicies.add()` at least once. The resolved path is saved to `policies-config.json` as `customPoliciesPath`.
# List policies
Source: https://docs.befailproof.ai/cli/list-policies
See which policies are enabled, their parameters, and custom policies
```bash theme={null}
failproofai policies
```
Shows all policies with their status, configured parameters, and custom policies.
## Sample output
```text theme={null}
Failproof AI Hook Policies (user)
Status Name Description
────── ──────────────────────────────────────────────────────────────
✓ block-sudo Block sudo commands
allowPatterns: ["sudo systemctl status"]
✓ block-rm-rf Block recursive deletions
✗ block-curl-pipe-sh Block curl|bash patterns
✓ sanitize-api-keys Redact API keys from output
additionalPatterns: [{ regex: "MY_TOKEN_...", label: "..." }]
── Custom Policies (/home/alice/myproject/my-policies.js) ──────────────
✓ require-jira-ticket Block commits without ticket
✓ approval-gate Approval gate for destructive ops
```
Unknown keys in `policyParams` are flagged here so you can catch typos early.
# Uninstall policies
Source: https://docs.befailproof.ai/cli/remove-policies
Remove hook entries from Claude Code's settings
```bash theme={null}
failproofai policies --uninstall [policy-names...] [options]
```
Removes failproofai hook entries from Claude Code's `settings.json`.
Aliases: `failproofai p -u`
## Options
| Flag | Description |
| ----------------- | ------------------------------------------ |
| `--scope user` | Remove from global settings (default) |
| `--scope project` | Remove from project settings |
| `--scope local` | Remove from local settings |
| `--scope all` | Remove from all scopes at once |
| `--custom` / `-c` | Clear the `customPoliciesPath` from config |
## Behavior
* **No policy names** - removes all failproofai hook entries from the settings file
* **Specific names** - disables those policies but keeps hooks installed
## Examples
```bash theme={null}
# Remove all hooks globally
failproofai policies --uninstall
# Disable a specific policy (keeps hooks installed)
failproofai policies --uninstall block-sudo
# Remove hooks from every scope
failproofai policies --uninstall --scope all
# Clear custom policies path
failproofai policies --uninstall --custom
```
# Check version
Source: https://docs.befailproof.ai/cli/version
Print the installed failproofai version
```bash theme={null}
failproofai --version
# or
failproofai -v
```
Prints the installed version number.
# Configuration
Source: https://docs.befailproof.ai/configuration
Config file format, three-scope system, and merge rules
failproofai uses JSON configuration files to control which policies are active, how they behave, and where custom policies are loaded from. Configuration is designed to be easy to share with your team - commit it to your repo and every developer gets the same agent safety net.
***
## Configuration scopes
There are three configuration scopes, evaluated in priority order:
| Scope | File path | Purpose |
| ----------- | ----------------------------------------- | ----------------------------------------------- |
| **project** | `.failproofai/policies-config.json` | Per-repo settings, committed to version control |
| **local** | `.failproofai/policies-config.local.json` | Personal per-repo overrides, gitignored |
| **global** | `~/.failproofai/policies-config.json` | User-level defaults across all projects |
When failproofai receives a hook event, it loads and merges all three files that exist for the current working directory.
### Merge rules
**`enabledPolicies`** - the union of all three scopes. A policy enabled at any level is active.
```text theme={null}
project: ["block-sudo"]
local: ["block-rm-rf"]
global: ["block-sudo", "sanitize-api-keys"]
resolved: ["block-sudo", "block-rm-rf", "sanitize-api-keys"] ← deduplicated union
```
**`policyParams`** - first scope that defines params for a given policy wins entirely. There is no deep merging of values within a policy's params.
```text theme={null}
project: block-sudo → { allowPatterns: ["sudo apt-get update"] }
global: block-sudo → { allowPatterns: ["sudo systemctl status"] }
resolved: { allowPatterns: ["sudo apt-get update"] } ← project wins, global ignored
```
```text theme={null}
project: (no block-sudo entry)
local: (no block-sudo entry)
global: block-sudo → { allowPatterns: ["sudo systemctl status"] }
resolved: { allowPatterns: ["sudo systemctl status"] } ← falls through to global
```
**`customPoliciesPaths` / `customPoliciesPath`** - first scope that defines either form wins.
**`disabledCustomPolicies`** - union across all scopes. The dashboard writes a
source-qualified ID here when you switch off an individual policy from an
explicit or convention policy file. Policies not listed remain enabled by
default; IDs include the source file so same-named policies in multiple files
can be controlled independently.
**`llm`** - first scope that defines it wins.
***
## Config file format
```json theme={null}
{
"enabledPolicies": [
"block-sudo",
"block-rm-rf",
"block-push-master",
"sanitize-api-keys",
"sanitize-jwt",
"block-env-files",
"block-read-outside-cwd"
],
"policyParams": {
"block-sudo": {
"allowPatterns": ["sudo systemctl status", "sudo journalctl"]
},
"block-push-master": {
"protectedBranches": ["main", "release", "prod"]
},
"block-rm-rf": {
"allowPaths": ["/tmp"]
},
"block-read-outside-cwd": {
"allowPaths": ["/shared/data", "/opt/company"]
},
"sanitize-api-keys": {
"additionalPatterns": [
{ "regex": "myco_[A-Za-z0-9]{32}", "label": "MyCo API key" }
]
},
"warn-large-file-write": {
"thresholdKb": 512
}
},
"customPoliciesPath": "/home/alice/myproject/my-policies.js"
}
```
***
## Field reference
### `enabledPolicies`
Type: `string[]`
List of policy names to enable. Names must match exactly the policy identifiers shown by `failproofai policies`. See [Built-in Policies](/built-in-policies) for the full list.
Policies not in `enabledPolicies` are inactive, even if they have entries in `policyParams`.
### `policyParams`
Type: `Record>`
Per-policy parameter overrides. The outer key is the policy name; the inner keys are policy-specific. Each policy documents its available parameters in [Built-in Policies](/built-in-policies).
If a policy has parameters but you don't specify them, the policy's built-in defaults are used. Users who do not configure `policyParams` at all get identical behavior to previous versions.
Unknown keys inside a policy's params block are silently ignored at hook-fire time but flagged as warnings when you run `failproofai policies`.
#### `hint` (cross-cutting)
Type: `string` (optional)
A message appended to the reason when a policy returns `deny` or `instruct`. Use it to give Claude actionable guidance without modifying the policy itself.
Works with any policy type — built-in, custom (`custom/`), project convention (`.failproofai-project/`), or user convention (`.failproofai-user/`).
```json theme={null}
{
"policyParams": {
"block-force-push": {
"hint": "Try creating a fresh branch instead."
},
"block-sudo": {
"allowPatterns": ["sudo apt-get"],
"hint": "Use apt-get directly without sudo."
},
"custom/my-policy": {
"hint": "Ask the user for approval first."
}
}
}
```
When `block-force-push` denies, Claude sees: *"Force-pushing is blocked. Try creating a fresh branch instead."*
Non-string values and empty strings are silently ignored. If `hint` is not set, behavior is unchanged (backward-compatible).
### `customPoliciesPath`
Type: `string` (absolute path)
Path to a JavaScript file containing custom hook policies. This is set automatically by `failproofai policies --install --custom ` (the path is resolved to absolute before being stored).
The file is loaded fresh on every hook event - there is no caching. See [Custom Policies](/custom-policies) for authoring details.
### Convention-based policies
In addition to the explicit `customPoliciesPath`, failproofai automatically discovers and loads policy files from `.failproofai/policies/` directories:
| Level | Directory | Scope |
| ------- | ------------------------------------------ | ------------------------------------ |
| Project | `.failproofai/policies/` | Shared with team via version control |
| User | `~/.failproofai/policies/custom-policies/` | Personal, applies to all projects |
The user-level directory moved down a level in the home-directory
reorganisation. Files left at the old `~/.failproofai/policies/` are moved
into `custom-policies/` automatically the first time you run any `failproofai`
command after upgrading, and the command tells you which files it moved.
**File matching:** Only files matching `*policies.{js,mjs,ts}` are loaded (e.g. `security-policies.mjs`, `workflow-policies.js`). Other files in the directory are ignored.
**No config needed:** Convention policies require no entries in `policies-config.json`. Just drop files into the directory and they're picked up on the next hook event.
**Union loading:** Both project and user convention directories are scanned. All matching files from both levels are loaded (unlike `customPoliciesPath` which uses first-scope-wins).
See [Custom Policies](/custom-policies) for more details and examples.
### `llm`
Type: `object` (optional)
LLM client configuration for policies that make AI calls. Not required for most setups.
```json theme={null}
{
"llm": {
"model": "claude-sonnet-4-6",
"apiKey": "sk-ant-..."
}
}
```
***
## Managing configuration from the CLI
The `policies --install` and `policies --uninstall` commands write to your agent CLI's hook settings file (the hook entry points), while `policies-config.json` is the file you manage directly. The two are separate:
* **Agent CLI settings** — tells the agent to call `failproofai --hook ` on each tool use:
* **Claude Code**: `~/.claude/settings.json` (user), `/.claude/settings.json` (project), `/.claude/settings.local.json` (local)
* **OpenAI Codex**: `~/.codex/hooks.json` (user), `/.codex/hooks.json` (project) — Codex doesn't have a `local` scope
* **GitHub Copilot CLI *(beta)***: `~/.copilot/hooks/failproofai.json` (user), `/.github/hooks/failproofai.json` (project) — Copilot has no `local` scope. Hook entries use Copilot's OS-keyed `bash`/`powershell` command fields with `timeoutSec`; the file carries a top-level `version: 1` marker. Copilot CLI support is **beta** while we verify the `events.jsonl` record schema (which the public docs do not specify) against more real-world sessions. **VS Code Copilot Chat agent mode (Preview)** reads hook configs from `.github/hooks/*.json`, `~/.copilot/hooks/*.json`, and `~/.claude/settings.json` (governed by the `chat.hookFilesLocations` setting) using the same Claude-shaped `{hookSpecificOutput:{permissionDecision:"deny",…}}` contract — the exact paths this `copilot` integration and the `claude` integration (`~/.claude/settings.json`) already write, so `failproofai policies --install --cli copilot` (or `--cli claude`) **already enforces in VS Code agent mode** with no separate `vscode` integration needed (confirmed live from VS Code's discovery logs).
* **Cursor Agent *(beta)***: `~/.cursor/hooks.json` (user), `/.cursor/hooks.json` (project) — Cursor has no `local` scope. Hook entries use the Claude-shaped `{type, command, timeout}` form (no `bash`/`powershell` split), but stored under camelCase event keys (`preToolUse`, `beforeSubmitPrompt`, …) in a flat array per Cursor's [hooks schema](https://cursor.com/docs/hooks); the file carries a top-level `version: 1` marker. The handler canonicalizes camelCase → PascalCase via `CURSOR_EVENT_MAP` so existing built-in policies fire unchanged. Cursor Agent support is **beta** while we verify Cursor's transcript on-disk format (not specified in the public docs) against more real-world installs.
* **OpenCode *(beta)***: `~/.config/opencode/opencode.json` + `~/.config/opencode/plugins/failproofai.mjs` (user), `/.opencode/opencode.json` + `/.opencode/plugins/failproofai.mjs` (project) — OpenCode has no `local` scope. Unlike the other five CLIs, OpenCode has **no external-command hook system**: it loads in-process JS/TS plugins explicitly registered via the `plugin: []` array in `opencode.json` (auto-discovery from `.opencode/plugins/` is **not** how plugins load on opencode v1.14.33). Install drops a small generated plugin shim that subprocess-calls the failproofai binary and translates the binary's Claude-shape JSON response back into plugin semantics: `throw new Error()` for tool-event deny (cancels the tool call), `client.session.prompt(...)` for instruct AND for `Stop` / `SubagentStop` deny (submits the deny reason as the next user message — the only force-retry channel since `session.idle` is notification-only and throwing from it is a no-op), and no-op for allow. The shim canonicalizes both tool names (lowercase → PascalCase via `OPENCODE_TOOL_MAP`) and tool-input arg keys (camelCase → snake\_case via `OPENCODE_TOOL_INPUT_MAP` for `Read` / `Write` / `Edit`, e.g. `filePath` → `file_path`, `oldString` → `old_string`) before forwarding to the binary, so path-checking builtins like `block-read-outside-cwd`, `block-env-files`, and `block-secrets-write` fire unchanged on OpenCode tool calls. Sessions live in opencode's SQLite DB at `~/.local/share/opencode/opencode.db`; the dashboard's session viewer reads them via `opencode db --format json` and `opencode export `. OpenCode support is **beta** while we verify behavior across versions and against more real-world sessions. See the [OpenCode plugins docs](https://opencode.ai/docs/plugins/).
* **Pi *(beta)***: `~/.pi/agent/settings.json` (user), `/.pi/settings.json` (project) — Pi has no `local` scope. Pi loads TypeScript extension packages at startup; the settings file is a flat string array `{"packages": ["./relative/path", …]}`. failproofai writes a single packages-array entry pointing at its bundled `pi-extension/` directory. The extension internally subscribes to Pi's `tool_call` / `user_bash` / `input` / `session_start` events and shells out to `failproofai --hook --cli pi`; the handler canonicalizes underscore\_lower\_snake\_case → PascalCase via `PI_EVENT_MAP` so existing built-in policies fire unchanged. Tool input args are also canonicalized via `PI_TOOL_INPUT_MAP` (Pi's Read / Write / Edit deliver `path` rather than `file_path`; mapping the top-level key lets `block-env-files` and `block-secrets-write` fire — `block-read-outside-cwd` already had a `path` fallback). Pi support is **beta** while Pi's extension API and session-log layout stabilize.
* **Hermes (hermes-agent)**: `~/.hermes/config.yaml` (**user scope only** — Hermes has no project/local config). Hermes is a Slack/Telegram **gateway**, so one install intercepts tool calls from every platform (Slack/Telegram/cli/cron) **and** internal subagents. Hook entries are a `{command, timeout}` pair (timeout in **seconds**) under a `hooks:` map keyed by Hermes's snake\_case events (`pre_tool_call` / `post_tool_call` / `on_session_start` / `on_session_end` / `subagent_stop`); the handler canonicalizes events via `HERMES_EVENT_MAP` and tool names via `HERMES_TOOL_MAP` so built-in policies fire unchanged. The config is edited through a comment-preserving YAML `Document` round-trip so the operator's other settings survive, and install sets `hooks_auto_accept: true` so the headless gateway (no TTY) runs the hooks without a consent prompt. The evaluator emits Hermes's `{"decision":"block","reason"}` stdout contract (Hermes ignores exit codes). **Limitations:** Hermes has no turn-end `Stop` event, so the `require-*-before-stop` builtins never fire for it (inapplicable, not broken); `instruct` degrades to allow-with-logged-note (no additional-context channel); and output-secret redaction (`sanitize-*`) can't rewrite tool output over the shell-hook contract. Hermes is **also** an offline **audit** source — the dashboard reads its gateway sessions directly from `~/.hermes/state.db`.
* **OpenClaw (openclaw gateway)**: `~/.openclaw/openclaw.json` (**user scope only** — OpenClaw has no project/local config). Like Hermes, OpenClaw is a self-hosted multi-channel **gateway**, so one install intercepts tool calls from every channel and its internal subagents. Enforcement runs through OpenClaw's **in-process plugin hooks** (its file-based internal hooks are observation-only and cannot block), so — like OpenCode/Pi — failproofai ships a static `openclaw-plugin/` package that async-spawns the failproofai binary and translates the verdict. Install registers the shipped plugin dir in `openclaw.json`'s `plugins.load.paths[]` and enables it under `plugins.entries.failproofai` (with `hooks.allowConversationAccess: true`, required for the raw-conversation hooks). The evaluator emits a flat `{permission, reason}` verdict and the shim maps it to each hook's native return shape: `before_tool_call → {block:true, blockReason}` (**PreToolUse**), `before_agent_run → {outcome:"block", reason}` (**UserPromptSubmit**), and `before_agent_finalize → {action:"revise", reason}` (**Stop** — a real turn-end gate, so the `require-*-before-stop` builtins **enforce** on OpenClaw, unlike Hermes). Events and tool names canonicalize binary-side via `OPENCLAW_EVENT_MAP` / `OPENCLAW_TOOL_MAP` (`exec→Bash`, `read→Read`, …) so built-in policies fire unchanged; the shim fails open on any spawn/parse/timeout error. OpenClaw is **also** an offline **audit** source — the dashboard reads its JSONL sessions at `~/.openclaw/agents//sessions/.jsonl`.
* **Factory Droid (`droid`)**: `~/.factory/hooks.json` (user), `/.factory/hooks.json` (project) — Factory has no `local` scope. droid ships a Claude-style external-command hook system, but with two quirks verified live against droid v0.171.0: (1) event names live at the **top level** of `hooks.json` — there is **no `"hooks"` wrapper** (droid rejects one); tool events (`PreToolUse`/`PostToolUse`) carry `"matcher": "*"`, non-tool events omit it. (2) Deny is driven by hook **exit code 2 + stderr**, not a JSON decision — the evaluator's `factory` branch returns exit 2 for tool/prompt events and `{decision:"block", reason}` only on the turn-end `Stop` event (droid's sole force-retry channel). Events are already PascalCase (no event map) and the payload is Claude snake\_case; only tool names are canonicalized via `FACTORY_TOOL_MAP` (`Execute→Bash`, `Create→Write`, `FetchUrl→WebFetch`, …). Factory is **also** an offline **audit** source — the dashboard reads its on-disk JSONL sessions at `~/.factory/sessions//.jsonl`.
* **Devin CLI (`devin`, Cognition)**: `~/.config/devin/config.json` (user), `/.devin/config.json` (project) — Devin has no `local` scope. Devin is a **pure Claude-clone** verified live against devin v3000.1.27: it uses the standard Claude `"hooks"`-wrapper schema (writes are merge-preserving so the config file's other keys — `org_id`, `theme_mode`, … — survive), already-PascalCase event names (no event map, no handler branch), and a Claude snake\_case stdin payload (no normalization). The evaluator's `devin` branch denies with `{"decision":"block","reason"}` JSON on stdout at exit 0 for **every** event (verified — the block overrode `--permission-mode dangerous`); on the turn-end `Stop` event the reason carries the MANDATORY-ACTION force-retry wording so the `require-*-before-stop` builtins enforce. Only tool names are canonicalized via `DEVIN_TOOL_MAP` (`exec→Bash`; `tool_input.command` is already canonical). Devin is **also** an offline **audit** source — the dashboard reads its SQLite sessions at `~/.local/share/devin/cli/sessions.db` (each `sessions` row carries a real `working_directory`, so sessions group by project cwd like Claude).
* **Antigravity CLI (`agy`)**: `~/.gemini/config/hooks.json` (user), `/.agents/hooks.json` (project) — Antigravity has no `local` scope. Unlike Factory/Devin, Antigravity has its **own** contract (not a Claude-clone), verified live against agy v1.1.2. `hooks.json` uses a **named-hook** schema: the top-level key is a hook *name* (`"failproofai"`) whose value is an event→handlers map — tool events (`PreToolUse`/`PostToolUse`) wrap handlers in `{matcher:"*", hooks:[…]}`, while `PreInvocation`/`Stop` are **flat** handler arrays (other named hooks are preserved). The stdin payload is **camelCase protojson** (`toolCall:{name,args}`, `conversationId`, `workspacePaths`, `transcriptPath`) — failproofai normalizes it to snake\_case before policies run, and maps `run_command`'s PascalCase args (`CommandLine`/`Cwd`) via `ANTIGRAVITY_TOOL_INPUT_MAP`. The evaluator's `antigravity` branch uses Antigravity's **own** response shapes: `{decision:"deny", reason}` blocks a tool/prompt (exit 0), `{decision:"continue", reason}` on the turn-end `Stop` re-enters the loop (so the `require-*-before-stop` builtins enforce), and `{injectSteps:[{ephemeralMessage}]}` injects an instruction on `PreInvocation` (→ `UserPromptSubmit`). Tool names canonicalize via `ANTIGRAVITY_TOOL_MAP` (`run_command→Bash`, `view_file→Read`, …). Antigravity is **also** an offline **audit** source — the dashboard reads its plain-JSONL transcripts at `~/.gemini/antigravity-cli/brain//.system_generated/logs/transcript_full.jsonl` (conversation index in `conversation_summaries.db`).
* **Goose (codename goose, Block)**: `~/.agents/plugins/failproofai/hooks/hooks.json` (user), `/.agents/plugins/failproofai/hooks/hooks.json` (project) — Goose has no `local` scope. Enforcement uses Goose's **hooks** system, the cross-agent **Open Plugins** spec: the installer just drops the `failproofai` plugin dir and Goose auto-discovers it at startup (self-registering it into `~/.config/goose/config.yaml`). The `hooks.json` uses an Open Plugins schema **with** a top-level `"hooks"` wrapper, and the matcher is **omitted** on every event — a bare `"*"` is an invalid regex that matches nothing (verified live against goose v1.43.0). Event names are already PascalCase (no event map); the stdin payload uses `event`/`working_dir`, which the handler normalizes to `hook_event_name`/`cwd`. The evaluator's `goose` branch denies with `{"decision":"block","reason"}` JSON on stdout at exit 0, honored on the **`PreToolUse`** event only (shipped in goose ≥ v1.37.0) — which fires for the shell tool **and inside delegated subagents**, so it is the single sufficient deny point; any other hook error fails **open**. Goose has **no `Stop` event**, so the `require-*-before-stop` builtins don't apply (as with Hermes). Tool names canonicalize via `GOOSE_TOOL_MAP` (`shell→Bash`, `write→Write`, `todo__todo_write→TodoWrite`, …) and path keys via `GOOSE_TOOL_INPUT_MAP` (`path`/`source` → `file_path`). Goose is **also** an offline **audit** source — the dashboard reads its SQLite sessions at `~/.local/share/goose/sessions/sessions.db` (each `sessions` row carries a real `working_dir`, so sessions group by project cwd like Devin; `--no-session` scratch runs are filtered).
* **`policies-config.json`** — tells failproofai which policies to evaluate and with what params (shared across all agent CLIs)
Pass `--cli claude|codex|copilot|cursor|opencode|pi|hermes|openclaw|factory|devin|antigravity|goose` to target a specific agent (space-separated or repeated for any subset):
```bash theme={null}
failproofai policies --install --cli codex --scope project
failproofai policies --install --cli copilot --scope project
failproofai policies --install --cli cursor --scope project
failproofai policies --install --cli opencode --scope project
failproofai policies --install --cli pi --scope project
failproofai policies --install --cli hermes --scope user
failproofai policies --install --cli openclaw --scope user
failproofai policies --install --cli factory --scope project
failproofai policies --install --cli devin --scope project
failproofai policies --install --cli antigravity --scope project
failproofai policies --install --cli goose --scope project
failproofai policies --install --cli claude codex copilot cursor opencode pi hermes openclaw factory devin antigravity goose
```
When `--cli` is omitted, `failproofai` detects which agent CLIs are installed (`which claude` / `which codex` / `which copilot` / `which cursor-agent` / `which opencode` / `which pi` / `which hermes` / `which openclaw` / `which droid` / `which devin` / `which agy` / `which goose`):
* **One CLI detected** — auto-selects that CLI without prompting.
* **Multiple CLIs detected** in an interactive terminal — shows an arrow-key single-select prompt grouped into a `Detected (N)` section (with an `Install for all N detected` aggregate row + each detected CLI individually) and a `Not installed (M) · install hooks ahead of time` section listing every undetected supported CLI as a forward-install option (↑↓ to move, Enter to select, ^C to quit). The uninstall flow shows only the Detected section.
* **Multiple CLIs detected** in a non-interactive run (CI, no TTY) — installs for all detected CLIs without prompting.
* **None detected** — falls back to `claude`, with a warning that no agent binary was found in PATH; the hook command is still written so it activates as soon as you install one.
You can edit `policies-config.json` directly at any time; changes take effect immediately on the next hook event with no restart needed.
***
## Example: project-level config with team defaults
Commit `.failproofai/policies-config.json` to your repo:
```json theme={null}
{
"enabledPolicies": [
"block-sudo",
"block-rm-rf",
"block-push-master",
"sanitize-api-keys",
"block-env-files"
],
"policyParams": {
"block-push-master": {
"protectedBranches": ["main", "release", "hotfix"]
}
}
}
```
Each developer can then create `.failproofai/policies-config.local.json` (gitignored) for personal overrides without affecting teammates.
# Custom Policies
Source: https://docs.befailproof.ai/custom-policies
Write your own policies in JavaScript - enforce conventions, prevent drift, detect failures, integrate with external systems
Custom policies let you write rules for any agent behavior: enforce project conventions, prevent drift, gate destructive operations, detect stuck agents, or integrate with Slack, approval workflows, and more. They use the same hook event system and `allow`, `deny`, `instruct` decisions as built-in policies.
***
## Quick example
```js theme={null}
// my-policies.js
import { customPolicies, allow, deny, instruct } from "failproofai";
customPolicies.add({
name: "no-production-writes",
description: "Block writes to paths containing 'production'",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Write" && ctx.toolName !== "Edit") return allow();
const path = ctx.toolInput?.file_path ?? "";
if (path.includes("production")) {
return deny("Writes to production paths are blocked");
}
return allow();
},
});
```
Install it:
```bash theme={null}
failproofai policies --install --custom ./my-policies.js
```
***
## Two ways to load custom policies
### Option 1: Convention-based (recommended)
Drop `*policies.{js,mjs,ts}` files into `.failproofai/policies/` and they're automatically loaded — no flags or config changes needed. This works like git hooks: drop a file, it just works.
```
# Project level — committed to git, shared with the team
.failproofai/policies/security-policies.mjs
.failproofai/policies/workflow-policies.mjs
# User level — personal, applies to all projects
~/.failproofai/policies/my-policies.mjs
```
**How it works:**
* Both project and user directories are scanned (union — not first-scope-wins)
* Files are loaded alphabetically within each directory. Prefix with `01-`, `02-` to control order
* Only files matching `*policies.{js,mjs,ts}` are loaded; other files are ignored
* Each file is loaded independently (fail-open per file)
* Works alongside explicit `--custom` and built-in policies
Convention policies are the easiest way to build a quality standard for your org. Commit `.failproofai/policies/` to git and every team member gets the same rules automatically — no per-developer setup needed. As your team discovers new failure modes, add a policy and push. Over time these become a living quality standard that keeps improving with every contribution.
### Option 2: Explicit file path
```bash theme={null}
# Install with a custom policies file
failproofai policies --install --custom ./my-policies.js
# Replace the custom policy paths
failproofai policies --install --custom ./new-policies.js
# Configure multiple explicit files (loaded in flag order)
failproofai policies --install --custom ./security.js --custom ./workflow.js
# Remove all explicit custom policy paths from config
failproofai policies --uninstall --custom
```
Resolved absolute paths are stored in `policies-config.json` as `customPoliciesPaths`. Repeat `--custom` to configure multiple files. Existing configs using the legacy `customPoliciesPath` field continue to work. Files are loaded fresh on every hook event - there is no caching between events.
Each registered policy appears with its own toggle in the dashboard. Switching
a policy off records its source-qualified ID in `disabledCustomPolicies`; the
file and its other policies continue to load, while the disabled policy is
excluded before event matching. Policy names duplicated across files have
independent toggles.
### Using both together
Convention policies and the explicit `--custom` files can coexist. Load order:
1. Explicit `customPoliciesPaths` files (in configured order)
2. Project convention files (`{cwd}/.failproofai/policies/`, alphabetical)
3. User convention files (`~/.failproofai/policies/`, alphabetical)
***
## API
### Import
```js theme={null}
import { customPolicies, allow, deny, instruct } from "failproofai";
```
### `customPolicies.add(hook)`
Registers a policy. Call this as many times as needed for multiple policies in the same file.
```ts theme={null}
customPolicies.add({
name: string; // required - unique identifier
description?: string; // shown in `failproofai policies` output
match?: { events?: HookEventType[] }; // filter by event type; omit to match all
fn: (ctx: PolicyContext) => PolicyResult | Promise;
});
```
### Decision helpers
| Function | Effect | Use when |
| ------------------- | ----------------------------- | --------------------------------------------- |
| `allow()` | Permit the operation silently | The action is safe, no message needed |
| `deny(message)` | Block the operation | The agent should not take this action |
| `instruct(message)` | Add context without blocking | Give the agent extra context to stay on track |
`deny(message)` - the message appears to Claude prefixed with `"Blocked by failproofai:"`. A single `deny` short-circuits all further evaluation.
`instruct(message)` - the message is appended to Claude's context for the current tool call. All `instruct` messages are accumulated and delivered together.
You can append extra guidance to any `deny` or `instruct` message by adding a `hint` field in `policyParams` — no code change needed. This works for custom (`custom/`), project convention (`.failproofai-project/`), and user convention (`.failproofai-user/`) policies too. See [Configuration → hint](/configuration#hint-cross-cutting) for details.
### Informational allow messages
`allow(message)` permits the operation **and** sends an informational message back to Claude. The message is delivered as `additionalContext` in the hook handler's stdout response — the same mechanism used by `instruct`, but semantically different: it's a status update, not a warning.
| Function | Effect | Use when |
| ---------------- | --------------------------------- | ---------------------------------------------------------- |
| `allow(message)` | Permit and send context to Claude | Confirm a check passed, or explain why a check was skipped |
Use cases:
* **Status confirmations:** `allow("All CI checks passed.")` — tells Claude everything is green
* **Fail-open explanations:** `allow("GitHub CLI not installed, skipping CI check.")` — tells Claude why a check was skipped so it has full context
* **Multiple messages accumulate:** if several policies each return `allow(message)`, all messages are joined with newlines and delivered together
```js theme={null}
customPolicies.add({
name: "confirm-branch-status",
match: { events: ["Stop"] },
fn: async (ctx) => {
const cwd = ctx.session?.cwd;
if (!cwd) return allow("No working directory, skipping branch check.");
// ... check branch status ...
if (allPushed) {
return allow("Branch is up to date with remote.");
}
return deny("Unpushed changes detected.");
},
});
```
### `PolicyContext` fields
| Field | Type | Description |
| ----------- | -------------------------------------- | ----------------------------------------------------------- |
| `eventType` | `string` | `"PreToolUse"`, `"PostToolUse"`, `"Notification"`, `"Stop"` |
| `toolName` | `string \| undefined` | The tool being called (e.g. `"Bash"`, `"Write"`, `"Read"`) |
| `toolInput` | `Record \| undefined` | The tool's input parameters |
| `payload` | `Record` | Full raw event payload from Claude Code |
| `session` | `SessionMetadata \| undefined` | Session context (see below) |
### `SessionMetadata` fields
| Field | Type | Description |
| ---------------- | -------- | -------------------------------------------- |
| `sessionId` | `string` | Claude Code session identifier |
| `cwd` | `string` | Working directory of the Claude Code session |
| `transcriptPath` | `string` | Path to the session's JSONL transcript file |
### Event types
| Event | When it fires | `toolInput` contents |
| -------------- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PreToolUse` | Before Claude runs a tool | The tool's input (e.g. `{ command: "..." }` for Bash) |
| `PostToolUse` | After a tool completes | The tool's input + `tool_result` (the output) |
| `Notification` | When Claude sends a notification | `{ message: "...", notification_type: "idle" \| "permission_prompt" \| ... }` - hooks must always return `allow()`, they cannot block notifications |
| `Stop` | When the Claude session ends | Empty |
***
## Evaluation order
Policies are evaluated in this order:
1. Built-in policies (in definition order)
2. Explicit custom policies from `customPoliciesPath` (in `.add()` order)
3. Convention policies from project `.failproofai/policies/` (files alphabetical, `.add()` order within)
4. Convention policies from user `~/.failproofai/policies/` (files alphabetical, `.add()` order within)
The first `deny` short-circuits all subsequent policies. All `instruct` messages are accumulated and delivered together.
***
## Transitive imports
Custom policy files can import local modules using relative paths:
```js theme={null}
// my-policies.js
import { isBlockedPath } from "./utils.js";
import { checkApproval } from "./approval-client.js";
customPolicies.add({
name: "approval-gate",
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const approved = await checkApproval(ctx.toolInput?.command, ctx.session?.sessionId);
return approved ? allow() : deny("Approval required for this command");
},
});
```
All relative imports reachable from the entry file are resolved. This is implemented by rewriting `from "failproofai"` imports to the actual dist path and creating temporary `.mjs` files to ensure ESM compatibility.
***
## Event type filtering
Use `match.events` to limit when a policy fires:
```js theme={null}
customPolicies.add({
name: "require-summary-on-stop",
match: { events: ["Stop"] },
fn: async (ctx) => {
// Only fires when the session ends
// ctx.session.transcriptPath contains the full session log
return allow();
},
});
```
Omit `match` entirely to fire on every event type.
***
## Error handling and failure modes
Custom policies are **fail-open**: errors never block built-in policies or crash the hook handler.
| Failure | Behavior |
| -------------------------------- | ------------------------------------------------------------------------------------ |
| `customPoliciesPath` not set | No explicit custom policies run; convention policies and built-ins continue normally |
| File not found | Warning logged to `~/.failproofai/hook.log`; built-ins continue |
| Syntax/import error (explicit) | Error logged to `~/.failproofai/hook.log`; explicit custom policies skipped |
| Syntax/import error (convention) | Error logged; that file skipped, other convention files still load |
| `fn` throws at runtime | Error logged; that hook treated as `allow`; other hooks continue |
| `fn` takes longer than 10s | Timeout logged; treated as `allow` |
| Convention directory missing | No convention policies run; no error |
To debug custom policy errors, watch the log file:
```bash theme={null}
tail -f ~/.failproofai/hook.log
```
***
## Full example: multiple policies
```js theme={null}
// my-policies.js
import { customPolicies, allow, deny, instruct } from "failproofai";
// Prevent agent from writing to secrets/ directory
customPolicies.add({
name: "block-secrets-dir",
description: "Prevent agent from writing to secrets/ directory",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (!["Write", "Edit"].includes(ctx.toolName ?? "")) return allow();
const path = ctx.toolInput?.file_path ?? "";
if (path.includes("secrets/")) return deny("Writing to secrets/ is not permitted");
return allow();
},
});
// Keep the agent on track: verify tests before committing
customPolicies.add({
name: "remind-test-before-commit",
description: "Keep the agent on track: verify tests pass before committing",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const cmd = ctx.toolInput?.command ?? "";
if (/git\s+commit/.test(cmd)) {
return instruct("Verify all tests pass before committing. Run `bun test` if you haven't already.");
}
return allow();
},
});
// Prevent unplanned dependency changes during freeze
customPolicies.add({
name: "dependency-freeze",
description: "Prevent unplanned dependency changes during freeze period",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const cmd = ctx.toolInput?.command ?? "";
const isInstall = /^(npm install|yarn add|bun add|pnpm add)\s+\S/.test(cmd);
if (isInstall && process.env.DEPENDENCY_FREEZE === "1") {
return deny("Package installs are frozen. Unset DEPENDENCY_FREEZE to allow.");
}
return allow();
},
});
export { customPolicies };
```
***
## Examples
The `examples/` directory contains ready-to-run policy files:
| File | Contents |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `examples/policies-basic.js` | Five starter policies covering common agent failure modes |
| `examples/policies-advanced/index.js` | Advanced patterns: transitive imports, async calls, output scrubbing, and session-end hooks |
| `examples/convention-policies/security-policies.mjs` | Convention-based security policies (block .env writes, prevent git history rewriting) |
| `examples/convention-policies/workflow-policies.mjs` | Convention-based workflow policies (test reminders, audit file writes) |
### Using explicit file examples
```bash theme={null}
failproofai policies --install --custom ./examples/policies-basic.js
```
### Using convention-based examples
```bash theme={null}
# Copy to project level
mkdir -p .failproofai/policies
cp examples/convention-policies/*.mjs .failproofai/policies/
# Or copy to user level
mkdir -p ~/.failproofai/policies
cp examples/convention-policies/*.mjs ~/.failproofai/policies/
```
No install command needed — the files are picked up automatically on the next hook event.
# Dashboard
Source: https://docs.befailproof.ai/dashboard
Monitor agent sessions, review tool calls, and manage policies
The failproofai dashboard is a local web application for monitoring your AI agent sessions and managing policies. See what your agents did while you were away.
***
## Starting the dashboard
```bash theme={null}
failproofai
```
Opens at `http://localhost:8020`.
The dashboard reads local project, session, and failproofai configuration data directly from the filesystem. Optional authenticated features, such as audit reminders and invitations, send the information needed for those requests (including email addresses) to remote APIs.
***
## Pages
### Projects
Lists all Claude Code, OpenAI Codex, GitHub Copilot CLI *(beta)*, Cursor Agent *(beta)*, OpenCode *(beta)*, Pi *(beta)*, Hermes, OpenClaw, Factory Droid, Devin, Antigravity, and Goose projects found on your machine. Claude projects are discovered from `~/.claude/projects/` (or the path set by `CLAUDE_PROJECTS_PATH`); Codex projects are discovered by scanning every transcript under `~/.codex/sessions///
/*.jsonl` and grouping by the `cwd` recorded in each session's first record; Copilot CLI projects are discovered by scanning each `~/.copilot/session-state//workspace.yaml` (configurable via `COPILOT_HOME`) and grouping by its `cwd` field; Cursor Agent projects are discovered by scanning per-session metadata under `~/.cursor/agent-sessions//` (configurable via `CURSOR_HOME`, with `conversations/` and `sessions/` probed as fallbacks) for a `cwd` scalar in `meta.json` / `session.json` / `workspace.yaml`; OpenCode projects are discovered by querying its SQLite DB at `~/.local/share/opencode/opencode.db` via `opencode db --format json` (we read the `session` and `project` tables and group by `project_id`); Pi projects are discovered by scanning per-session JSONL transcripts under `~/.pi/agent/sessions//_.jsonl` (configurable via `PI_SESSIONS_DIR`) and pulling the `cwd` from each session's first record; Hermes gateway sessions are read directly from the SQLite store of every profile — `~/.hermes/state.db` plus `~/.hermes/profiles//state.db` (overridable via `HERMES_HOME`, or `HERMES_DB_PATH` for a single database) — and grouped into `hermes--` projects by profile and `source` (Slack/Telegram/cli/cron — gateway sessions have no cwd); OpenClaw gateway sessions are read from `~/.openclaw/agents//sessions/*.jsonl` and grouped into `openclaw--` projects by agent and channel (also cwd-less); Factory Droid projects are discovered from the JSONL transcripts at `~/.factory/sessions//*.jsonl` and grouped by cwd; Devin projects from its SQLite DB at `~/.local/share/devin/cli/sessions.db` (grouped by each session's `working_directory`); Antigravity projects from the JSONL transcripts at `~/.gemini/antigravity-cli/brain//…/transcript_full.jsonl` and grouped by cwd; and Goose projects from its SQLite DB at `~/.local/share/goose/sessions/sessions.db` (grouped by each session's `working_dir`). A project that has been used by multiple CLIs renders as a single row with all matching badges. Use the **CLI** dropdown above the table to filter by a specific agent CLI; the URL preserves your selection as `?cli=claude|codex|copilot|cursor|opencode|pi|hermes|openclaw|factory|devin|antigravity|goose`.
Hermes and OpenClaw are user-scoped and have no working directory to group by, so they render as a **collapsible folder tree** — profile (or agent) at the top level, its channels beneath — while every cwd-based CLI stays a flat row. Folder rows roll up the session count and most recent activity of everything under them, collapsed folders are remembered between visits, and a keyword search expands whatever it matches.
Each project shows:
* Project name (derived from the folder path)
* A CLI badge — `Claude Code` (orange), `OpenAI Codex` (purple), `GitHub Copilot` (blue), `Cursor Agent` (emerald), `OpenCode` (amber), `Pi` (pink), and/or `Hermes` (indigo)
* Date of most recent session activity
Click a project to see its sessions.
### Sessions
Lists all sessions within a project. Each session shows:
* Session ID
* Start and end timestamps
* Number of tool calls
* Hook activity count (policies that fired)
Use the date range filter and session ID search to narrow the list. Sessions are paginated.
Click a session to open the session viewer.
### Session viewer
The session viewer answers the key question for autonomous agents: what did the agent do, and did it stay on track? A CLI badge beside the header indicates whether the session is a Claude Code, OpenAI Codex, GitHub Copilot CLI, Cursor Agent, OpenCode, Pi, Hermes, OpenClaw, Factory Droid, Devin, Antigravity, or Goose transcript. It shows a timeline of everything that happened in a session:
* **Messages** - Claude's text responses and user prompts
* **Tool calls** - Every tool Claude invoked, with its input and output
* **Policy activity** - For each tool call, which policies fired and what decision they returned
The stats bar at the top shows session duration, total tool calls, and a summary of hook decisions (allow / deny / instruct counts).
Click the **Download Logs** button to export the session. For Claude Code, Codex, Copilot, Cursor, and Pi sessions you get the original on-disk JSONL transcript byte-for-byte; for OpenCode (whose sessions live in SQLite, not on disk) you get a JSON document mirroring the underlying `session` / `messages` / `parts` tables.
### Audit
A personality-driven report of how your agent has actually been behaving across past sessions. Runs the same scan as the `failproofai audit` CLI but renders it as a single-screen shareable poster + four below-the-fold sections:
1. **Poster** — fills the first viewport. Self-contained PNG-capture region with the failproof\_ai wordmark + audit label · archetype index (`№ NN of 08`) + audit date · numeric score (0–100) + percentile rank pill (`top 15%`) · the archetype name (one of `the optimist`, `the cowboy`, `the explorer`, `the goldfish`, `the paranoid architect`, `the precision builder`, `the hammer`, `the ghost`) + 3-keyword strip · `// only N% of agents are this archetype` rarity line · 8×8 pixel sigil tile · `audit yours → failproof.ai` footer. Three share buttons sit just outside the capture box: `post your archetype` (X intent), `share on linkedin`, `download poster`. Capture runs through `html-to-image` so the PNG matches the on-screen render pixel-for-pixel (dashed borders, SVG logo mask, gradients, font metrics — all preserved).
2. **Strengths** — calm ✓ row list of behaviors your agent already does right, derived from the live audit data (clean tool-call rate, no direct pushes to main, zero credential leaks, zero retry storms) — each surfaced only when the relevant policy has a clean record across the audit window.
3. **Quirks** — table of what slipped through, ranked by severity: `when · what slipped + the policy that would've caught it · severity pill · seen`, where the recurrence reads `new` (once), `N× seen` (2–9 times), or `recurring` (10+).
4. **How to improve** — calm row list, one per prescribed policy: policy name in white, one-line description, install command + copy button on the right side. The section header reads `enable all N → projected · ` (the score you'd reach with every fix applied), and its `[install all]` button copies the combined `failproofai policy add a b c …` command for every prescribed policy.
5. **Come back better** — two side-by-side cards. Left: set a reminder (`3d` / `7d` / `14d` / `30d` cadence picker; persists through `/api/auth/reminder` once authed). Right: unlock failproof perks — `invite a friend` opens a modal that takes a comma/space/newline-separated list of friend emails (max 10 per send), POSTs them to `/api/audit/invite`, which forwards to the api-server's `POST /v0/invite`. The api-server sends one email per recipient from `invite@failproof.ai` with the sender Cc'd and `Reply-To` set, so the recipient sees who invited them and the sender gets a copy in their inbox. Anonymous users get routed through the `AuthDialog` first so the sender's email is known before invites go out. Entitlement / perks fulfillment is a follow-up.
Driven by the `failproofai audit` runtime — see [Audit CLI](/cli/audit) for the underlying scan engine, supported flags, and per-transcript cache invariants. The dashboard caches the latest result at `~/.failproofai/audit-dashboard.json` (mode `0600`, single slot, new runs overwrite) so revisits are instant; **both the per-transcript and whole-result caches are rejected on read once they're older than 7 days** so the dashboard never silently serves a week-old result — past the TTL `/audit` falls through to its empty state and prompts a fresh run. Clicking `[ re-audit now ]` near the bottom of the report POSTs `/api/audit/run` with `noCache: true` — re-audit bypasses the per-transcript cache and re-scans every transcript from scratch rather than silently returning the cached result — and the dashboard polls `/api/audit/status` at 1Hz until the run finishes; a sticky pink progress strip pins to the top of the viewport during the run with an elapsed timer, and the fresh result swaps in place on success (no full-page reload; a failed re-audit leaves the prior report intact). On failure the strip turns red with copy keyed off the `RerunError.kind` (`timeout` / `network` / `post_failed`). Empty state (no cache or expired) and zero-sessions state (cache exists but the scan found no transcripts) are surfaced separately.
### Policies
A two-tab page for managing policies and reviewing activity.
* Multi-select which agent CLIs failproofai protects from a single panel — Claude Code, OpenAI Codex, GitHub Copilot, Cursor Agent, OpenCode, Pi, and Hermes all have a row with install status (`Active` / `Detected` / `Inactive`), the user-scope settings path, and a brand-colored accent. Check or uncheck the CLIs you want and click `Apply changes` to install/uninstall the diff in one step. CLIs whose binary is detected on PATH are pre-checked.
* Toggle individual policies on or off with a single click (writes to `~/.failproofai/policies-config.json` — shared across every installed CLI)
* Expand a policy to configure its parameters (for policies that support `policyParams`)
* Set a custom policies file path
* Full paginated history of every hook event that has fired across all sessions
* Filter by decision, event type, CLI (Claude Code / OpenAI Codex / GitHub Copilot *(beta)* / Cursor Agent *(beta)* / OpenCode *(beta)* / Pi *(beta)* / Hermes / OpenClaw / Factory Droid / Devin / Antigravity / Goose), policy name, or session ID
* Each row shows: timestamp, policy name, decision, CLI badge (orange = Claude Code, purple = OpenAI Codex, blue = GitHub Copilot, emerald = Cursor Agent, amber = OpenCode, pink = Pi, indigo = Hermes, teal = OpenClaw, rose = Factory Droid, violet = Devin, cyan = Antigravity, lime = Goose), tool name, session ID, and the reason for deny/instruct decisions
* Click a session ID to open its transcript — the viewer auto-detects which CLI fired the hook (Claude `~/.claude/projects/…`, Codex `~/.codex/sessions/…`, Copilot CLI `~/.copilot/session-state//events.jsonl`, Cursor Agent `~/.cursor/agent-sessions//events.jsonl`, OpenCode `~/.local/share/opencode/opencode.db`, Pi `~/.pi/agent/sessions//.jsonl`, Hermes `~/.hermes/state.db`, OpenClaw `~/.openclaw/agents//sessions/*.jsonl`, Factory Droid `~/.factory/sessions//.jsonl`, Devin `~/.local/share/devin/cli/sessions.db`, Antigravity `~/.gemini/antigravity-cli/brain//…/transcript_full.jsonl`, Goose `~/.local/share/goose/sessions/sessions.db`) and renders the matching CLI badge in the header
***
## Auto-refresh
The dashboard has an auto-refresh toggle in the top navigation. When enabled, the current page refreshes periodically to show new sessions and policy activity as they appear. Essential for monitoring long-running autonomous agent sessions.
***
## Disabling pages
If you only need some parts of the dashboard, set `FAILPROOFAI_DISABLE_PAGES` to a comma-separated list of page names:
```bash theme={null}
FAILPROOFAI_DISABLE_PAGES=policies failproofai
```
Valid values: `policies`, `projects`, `audit`.
***
## Configuring the projects path
By default, the dashboard reads from the standard Claude Code projects directory. Override it for custom setups:
```bash theme={null}
CLAUDE_PROJECTS_PATH=/custom/path/to/projects failproofai
```
***
## Accessing from a non-localhost host
When running the dashboard in **dev mode** (`npm run dev`) and accessing it from a hostname other than `localhost` - for example, a custom domain, a remote IP, or a tunneled URL - you may see a warning like:
```text theme={null}
⚠ Blocked cross-origin request to Next.js dev resource /_next/webpack-hmr from "dashboard.example.com".
```
This is Next.js blocking cross-origin access to its HMR (hot module reload) websocket, which is a dev-only feature. To allow your host, use the `--allowed-origins` flag:
```bash theme={null}
npm run dev -- --allowed-origins dashboard.example.com
```
For multiple hosts or IPs, pass a comma-separated list:
```bash theme={null}
npm run dev -- --allowed-origins dashboard.example.com,192.168.1.5
```
You can also set the `FAILPROOFAI_ALLOWED_DEV_ORIGINS` environment variable instead:
```bash theme={null}
FAILPROOFAI_ALLOWED_DEV_ORIGINS=dashboard.example.com npm run dev
```
This only applies to dev mode. When running `failproofai` (production mode), there is no HMR websocket and no cross-origin dev resource issue.
# Examples
Source: https://docs.befailproof.ai/examples
How to set up hooks for Claude Code and the Agents SDK
Ready-to-use examples for common scenarios. Each one shows how to install and what to expect.
***
## Setting up hooks for Claude Code
Failproof AI integrates with Claude Code via its [hooks system](https://docs.anthropic.com/en/docs/claude-code/hooks). When you run `failproofai policies --install`, it registers hook commands in Claude Code's `settings.json` that fire on every tool call.
```bash theme={null}
npm install -g failproofai
```
```bash theme={null}
failproofai policies --install
```
```bash theme={null}
cat ~/.claude/settings.json | grep failproofai
```
You should see hook entries for `PreToolUse`, `PostToolUse`, `Notification`, and `Stop` events.
```bash theme={null}
claude
```
Policies now run automatically on every tool call. Try asking Claude to run `sudo rm -rf /` - it will be blocked.
***
## Setting up hooks for the Agents SDK
If you're building with the [Agents SDK](https://docs.anthropic.com/en/docs/agents-sdk), you can use the same hook system programmatically.
```bash theme={null}
npm install failproofai
```
Pass hook commands when creating your agent process. The hooks fire the same way as in Claude Code - via stdin/stdout JSON:
```bash theme={null}
failproofai --hook PreToolUse # called before each tool
failproofai --hook PostToolUse # called after each tool
```
```javascript theme={null}
import { customPolicies, allow, deny } from "failproofai";
customPolicies.add({
name: "limit-to-project-dir",
description: "Keep the agent inside the project directory",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
const path = String(ctx.toolInput?.file_path ?? "");
if (path.startsWith("/") && !path.startsWith(ctx.session?.cwd ?? "")) {
return deny("Agent is restricted to the project directory");
}
return allow();
},
});
```
```bash theme={null}
failproofai policies --install --custom ./my-agent-policies.js
```
***
## Block destructive commands
The most common setup - prevent agents from doing irreversible damage.
```bash theme={null}
failproofai policies --install block-sudo block-rm-rf block-force-push block-curl-pipe-sh
```
What this does:
* `block-sudo` - blocks all `sudo` commands
* `block-rm-rf` - blocks recursive file deletion
* `block-force-push` - blocks `git push --force`
* `block-curl-pipe-sh` - blocks piping remote scripts to shell
***
## Prevent secret leakage
Stop agents from seeing or leaking credentials in tool output.
```bash theme={null}
failproofai policies --install sanitize-api-keys sanitize-jwt sanitize-connection-strings sanitize-bearer-tokens
```
These fire on `PostToolUse` - after a tool runs, they scrub the output before the agent sees it.
***
## Get Slack alerts when agents need attention
Use the notification hook to forward idle alerts to Slack.
```javascript theme={null}
import { customPolicies, allow, instruct } from "failproofai";
customPolicies.add({
name: "slack-on-idle",
description: "Alert Slack when the agent is waiting for input",
match: { events: ["Notification"] },
fn: async (ctx) => {
const webhookUrl = process.env.SLACK_WEBHOOK_URL;
if (!webhookUrl) return allow();
const message = String(ctx.payload?.message ?? "Agent is waiting");
const project = ctx.session?.cwd ?? "unknown";
try {
await fetch(webhookUrl, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
text: `*${message}*\nProject: \`${project}\``,
}),
signal: AbortSignal.timeout(5000),
});
} catch {
// never block the agent if Slack is unreachable
}
return allow();
},
});
```
Install it:
```bash theme={null}
SLACK_WEBHOOK_URL=https://hooks.slack.com/... failproofai policies --install --custom ./slack-alerts.js
```
***
## Keep agents on a branch
Prevent agents from switching branches or pushing to protected ones.
```javascript theme={null}
import { customPolicies, allow, deny } from "failproofai";
customPolicies.add({
name: "stay-on-branch",
description: "Prevent the agent from checking out other branches",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const cmd = String(ctx.toolInput?.command ?? "");
if (/git\s+checkout\s+(?!-b)/.test(cmd)) {
return deny("Stay on the current branch. Create a new branch with -b if needed.");
}
return allow();
},
});
```
***
## Require tests before commits
Remind agents to run tests before committing.
```javascript theme={null}
import { customPolicies, allow, instruct } from "failproofai";
customPolicies.add({
name: "test-before-commit",
description: "Remind the agent to run tests before committing",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const cmd = String(ctx.toolInput?.command ?? "");
if (/git\s+commit/.test(cmd)) {
return instruct("Run tests before committing. Use `npm test` or `bun test` first.");
}
return allow();
},
});
```
***
## Lock down a production repo
Commit a project-level config so every developer on your team gets the same policies.
Create `.failproofai/policies-config.json` in your repo:
```json theme={null}
{
"enabledPolicies": [
"block-sudo",
"block-rm-rf",
"block-force-push",
"block-push-master",
"block-env-files",
"sanitize-api-keys",
"sanitize-jwt"
],
"policyParams": {
"block-push-master": {
"protectedBranches": ["main", "release", "production"]
}
}
}
```
Then commit it:
```bash theme={null}
git add .failproofai/policies-config.json
git commit -m "Add failproofai team policies"
```
Every team member who has failproofai installed will automatically pick up these rules.
***
## Build an org-wide quality standard with convention policies
The most impactful setup: commit `.failproofai/policies/` to your repo with policies tailored to your project. Every team member gets them automatically — no install commands, no config changes.
```bash theme={null}
mkdir -p .failproofai/policies
```
```js theme={null}
// .failproofai/policies/team-policies.mjs
import { customPolicies, allow, deny, instruct } from "failproofai";
// Enforce your team's preferred package manager
// (or enable the built-in prefer-package-manager policy instead)
customPolicies.add({
name: "enforce-bun",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
const cmd = String(ctx.toolInput?.command ?? "");
if (/\bnpm\b/.test(cmd)) return deny("Use bun instead of npm.");
return allow();
},
});
// Remind the agent to run tests before committing
customPolicies.add({
name: "test-before-commit",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
if (/git\s+commit/.test(ctx.toolInput?.command ?? "")) {
return instruct("Run tests before committing.");
}
return allow();
},
});
```
```bash theme={null}
git add .failproofai/policies/
git commit -m "Add team quality policies"
```
As your team hits new failure modes, add policies and push. Everyone gets the update on their next `git pull`. These policies become a living quality standard that grows with your team.
***
## More examples
The [`examples/`](https://github.com/failproofai/failproofai/tree/main/examples) directory in the repo contains:
| File | What it shows |
| ---------------------------- | ---------------------------------------------------------------------------------- |
| `policies-basic.js` | Starter policies - block production writes, force-push, piped scripts |
| `policies-notification.js` | Slack alerts for idle notifications and session end |
| `policies-advanced/index.js` | Transitive imports, async hooks, PostToolUse output scrubbing, Stop event handling |
# For agents
Source: https://docs.befailproof.ai/for-agents
Add Failproof AI knowledge to your coding agent in one command. Works with Claude Code, Cursor, Windsurf, and more.
Add the full Failproof AI reference to your coding agent in one command. Works with Claude Code, Cursor, Windsurf, and any other agent that supports skills.
```bash theme={null}
npx skills add https://docs.befailproof.ai
```
`npx skills` detects which agents you have installed and adds the skill in the right format for each one automatically.
## What the skill covers
| Area | What's included |
| --------------- | -------------------------------------------------------------------- |
| Policies | Built-in policy names, event types, parameters, enable/disable |
| Custom policies | `customPolicies.add()`, match filters, `allow`/`deny`/`instruct` API |
| Context object | `ctx.eventType`, `ctx.toolName`, `ctx.toolInput`, `ctx.session` |
| Configuration | `policies-config.json` structure, scope merging, `policyParams` |
| CLI | `failproofai policies --install`, `--uninstall`, `--custom`, scopes |
| Dashboard | Session viewer, policy activity, environment variables |
| Architecture | Hook handler flow, exit codes, stdin/stdout contract |
## Is the skill complete?
Mintlify generates `llms.txt` from all pages in the navigation. The Failproof AI docs cover the full API - every policy, option, and example is included. If you find something missing, the source is at `https://docs.befailproof.ai/llms-full.txt`.
For targeted context, link directly to a specific page:
```bash theme={null}
# Just the custom policies API
npx skills add https://docs.befailproof.ai/custom-policies
# Just the built-in policies
npx skills add https://docs.befailproof.ai/built-in-policies
```
# Getting started
Source: https://docs.befailproof.ai/getting-started
Install failproofai, enable policies, and let your agents run reliably
## Requirements
* **Node.js** >= 20.9.0
* **Bun** >= 1.3.0 (optional - only needed for building from source)
***
## Installation
```bash npm theme={null}
npm install -g failproofai
```
```bash bun theme={null}
bun add -g failproofai
```
***
## Quick start
Policies are rules that run before and after every agent tool call. They catch destructive commands, secret leakage, and other failure modes before they cause damage.
```bash theme={null}
failproofai policies --install
```
This writes hook entries into your installed agent CLIs (Claude Code's `~/.claude/settings.json`, OpenAI Codex's `~/.codex/hooks.json`, GitHub Copilot CLI's `~/.copilot/hooks/failproofai.json`, Cursor Agent's `~/.cursor/hooks.json`, OpenCode's generated plugin shim at `~/.config/opencode/plugins/failproofai.mjs` plus a registration entry in `~/.config/opencode/opencode.json`'s `plugin` array, Pi's `~/.pi/agent/settings.json`, Hermes's `~/.hermes/config.yaml`, OpenClaw's `~/.openclaw/openclaw.json`, Factory Droid's `~/.factory/hooks.json`, Devin CLI's `~/.config/devin/config.json`, Antigravity CLI's `~/.gemini/config/hooks.json`, or Goose's auto-discovered plugin dir at `~/.agents/plugins/failproofai/hooks/hooks.json`). When more than one is present you'll be prompted; pass `--cli claude codex copilot cursor opencode pi hermes openclaw factory devin antigravity goose` (any subset) to skip the prompt.
GitHub Copilot CLI, Cursor Agent, OpenCode, and Pi support are **beta** — install with `--cli copilot`, `--cli cursor`, `--cli opencode`, or `--cli pi`. Hermes (hermes-agent, a Slack/Telegram gateway) installs user-scope with `--cli hermes` and is **also** an offline audit source. OpenClaw (openclaw gateway, a self-hosted multi-channel assistant) installs user-scope with `--cli openclaw` — enforcement runs through its in-process plugin hooks (`before_agent_finalize` is a real turn-end gate, so the `require-*-before-stop` builtins enforce) — and is **also** an offline audit source. Factory Droid (`droid`) installs with `--cli factory` (user + project scope) and is **also** an offline audit source. Devin CLI (`devin`, Cognition) installs with `--cli devin` (user + project scope) and is **also** an offline audit source. Antigravity CLI (`agy`) installs with `--cli antigravity` (user + project scope) and is **also** an offline audit source. Goose (codename goose, Block) installs with `--cli goose` (user + project scope) — the installer just drops a plugin dir at `~/.agents/plugins/failproofai/` that Goose auto-discovers, and it is **also** an offline audit source.
```bash theme={null}
failproofai policies --install --scope project
failproofai policies --install --cli codex --scope project
failproofai policies --install --cli copilot --scope project
failproofai policies --install --cli cursor --scope project
failproofai policies --install --cli opencode --scope project
failproofai policies --install --cli pi --scope project
failproofai policies --install --cli hermes --scope user
failproofai policies --install --cli openclaw --scope user
failproofai policies --install --cli factory --scope project
failproofai policies --install --cli devin --scope project
failproofai policies --install --cli antigravity --scope project
failproofai policies --install --cli goose --scope project
failproofai policies --install block-sudo block-rm-rf sanitize-api-keys
```
```bash theme={null}
failproofai policies
```
Shows every policy, whether it's enabled, and any configured parameters.
```bash theme={null}
failproofai
```
Opens a local dashboard at `http://localhost:8020` where you can browse sessions, inspect tool calls, and manage policies.
Start Claude Code as usual. If the agent tries something risky, failproofai intercepts it automatically. Leave it running unattended and review what happened in the dashboard.
***
## How policies work
Every time an agent runs a tool, Claude Code calls failproofai as a subprocess:
```text theme={null}
Claude Code → failproofai --hook PreToolUse → reads stdin JSON
evaluates policies
writes decision to stdout
```
Each policy returns one of three decisions:
* **allow** - the agent proceeds normally
* **deny** - the action is blocked, the agent is told why
* **instruct** - extra context is added to the agent's prompt
Policies run in your local process. Nothing is sent to a remote service.
***
## Set up team policies with convention-based policies
The fastest way to establish quality standards across your team is the `.failproofai/policies/` convention. Drop policy files into this directory and they're loaded automatically — no flags, no config changes, no install commands.
```bash theme={null}
mkdir -p .failproofai/policies
```
Copy the starter examples or write your own:
```bash theme={null}
cp node_modules/failproofai/examples/convention-policies/*.mjs .failproofai/policies/
```
Or create a new one:
```js theme={null}
// .failproofai/policies/team-policies.mjs
import { customPolicies, allow, deny, instruct } from "failproofai";
customPolicies.add({
name: "test-before-commit",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (ctx.toolName !== "Bash") return allow();
if (/git\s+commit/.test(ctx.toolInput?.command ?? "")) {
return instruct("Run tests before committing.");
}
return allow();
},
});
```
```bash theme={null}
git add .failproofai/policies/
git commit -m "Add team quality policies"
```
Every team member who has failproofai installed picks up these policies automatically. No per-developer setup needed.
Commit `.failproofai/policies/` to your repo so the whole team shares the same standards. As your team discovers new failure modes, add policies and push — everyone gets the update on their next `git pull`. Over time these policies become a living quality standard that keeps improving.
***
## Data storage
All configuration and logs stay on your machine:
| Path | What it stores |
| ------------------------------------------------------------- | ------------------------------------ |
| `~/.failproofai/policies/local-policies/policies-config.json` | Global policy config |
| `~/.failproofai/hook-activity/` | Hook execution history (paged JSONL) |
| `~/.failproofai/logs/` | Debug logs for custom hook errors |
| `.failproofai/policies-config.json` | Per-project config (committed) |
| `.failproofai/policies-config.local.json` | Personal overrides (gitignored) |
***
## Uninstalling
```bash theme={null}
failproofai policies --uninstall
```
Removes hook entries from `~/.claude/settings.json`. Config files in `~/.failproofai/` are kept.
***
## Next steps
Scopes and config file format
All 26 policies with parameters
Write your own policies in JavaScript
Monitor sessions and review policy activity
# Failproof AI
Source: https://docs.befailproof.ai/introduction
FailproofAI gives AI agents 39 built-in failure policies that catch loops, secret leaks, destructive tool calls, and more in a single install.
[](https://www.npmjs.com/package/failproofai)
Hooks and policies for **AI failure handling**, **error recovery**, and **LLM reliability**. Keep your AI agents reliable and running autonomously across **Claude Code**, **OpenAI Codex**, **GitHub Copilot**, **Cursor Agent**, **OpenCode**, **Pi**, **Hermes**, **OpenClaw**, **Factory Droid**, **Devin CLI**, **Antigravity CLI**, and the **Agents SDK**.
AI agents fail in predictable ways. They run destructive commands, leak secrets, drift off-task, get stuck in loops, or push directly to main. Left unattended, small failures cascade into outages, leaked credentials, and lost work.
FailproofAI solves this with **policies**. These rules hook into every agent tool call to **detect failures**, **mitigate them** (block, instruct, sanitize), and **alert you** when something needs attention. A local dashboard lets you review every tool call, agent failure, and recovery action afterward.
Transcripts and policy evaluation stay on your machine. Data is sent only when you explicitly use an online feature, such as authenticated audit reminders or invitations.
## Get started
Block destructive commands, prevent secret leakage, keep agents inside project boundaries, and more. All out of the box.
Write your own rules in JavaScript with a simple allow / deny / instruct API.
See what your agents did while you were away. Browse sessions, inspect tool calls, review where policies fired.
Tune any policy without code. Set allowlists, protected branches, or thresholds per-project or globally.
## Quick start
```bash npm theme={null}
npm install -g failproofai
```
```bash bun theme={null}
bun add -g failproofai
```
```bash theme={null}
failproofai policies --install # enable policies (or skip — `failproofai` will offer to set them up on first run)
failproofai # launch the dashboard
```
See the [Getting started](/getting-started) guide for the full walkthrough.
# Package Aliases
Source: https://docs.befailproof.ai/package-aliases
Registered typosquat-prevention aliases and how they work
## Official package
The canonical npm package is **`failproofai`**:
```bash theme={null}
npm install -g failproofai
# or
bun add -g failproofai
```
***
## Why we own the alias names
Typosquatting is a common supply-chain attack where a malicious actor registers a package name that is one keystroke away from a popular package. Unsuspecting users who mistype the install command end up running attacker-controlled code with full system access - exactly the kind of threat Failproof AI is designed to defend against.
To eliminate this surface, **we pre-emptively own all common misspellings and formatting variants** of `failproofai` on npm. None of these names can be registered by a third party. Each one is a thin proxy that installs and delegates to the real `failproofai` package.
***
## Registered aliases
**Formatting variants** - different ways to write "failproof ai":
| Package | Status |
| --------------- | --------------------- |
| `failproof` | ✅ Published |
| `failproof-ai` | ⏳ Pending npm support |
| `fail-proof-ai` | ⏳ Pending npm support |
| `failproof_ai` | ⏳ Pending npm support |
| `fail_proof_ai` | ⏳ Pending npm support |
| `fail-proofai` | ⏳ Pending npm support |
**`failprof*` typos** - missing one `o` from "proof":
| Package | Status |
| -------------- | --------------------- |
| `failprof` | ✅ Published |
| `failprof-ai` | ✅ Published |
| `failprofai` | ⏳ Pending npm support |
| `fail-prof-ai` | ⏳ Pending npm support |
| `failprof_ai` | ⏳ Pending npm support |
**`faliproof*` typos** - transposed `a` and `i`:
| Package | Status |
| -------------- | --------------------- |
| `faliproof` | ✅ Published |
| `faliproof-ai` | ✅ Published |
| `faliproofai` | ⏳ Pending npm support |
> **Why pending?** npm's spam-prevention policy blocks names that normalize to the same string as an existing package after stripping punctuation and running similarity checks. We have contacted npm support to reserve these names for anti-squatting purposes. They will be activated once approved.
You can verify any published alias is owned by us:
```bash theme={null}
npm info failproof
# Look for: "ExosphereHost Inc." in the maintainers field
```
***
## How the aliases work
Each alias package:
1. Lists `failproofai` as a dependency - so the real package is installed and its binary becomes available
2. Exposes a binary matching its own name (e.g. `failprof-ai`) that proxies all arguments to the `failproofai` binary
The proxy is a two-line Node script; there is no logic, no network calls, and no data collection beyond what `failproofai` itself does.
***
## If you find a name we missed
Open an issue at [failproofai/failproofai](https://github.com/failproofai/failproofai/issues) and we will register it.
# Testing
Source: https://docs.befailproof.ai/testing
Unit tests, E2E tests, and test helpers
failproofai has two test suites: **unit tests** (fast, mocked) and **end-to-end tests** (real subprocess invocations).
***
## Running tests
```bash theme={null}
# Run all unit tests once
bun run test:run
# Run unit tests in watch mode
bun run test
# Run E2E tests (requires setup - see below)
bun run test:e2e
# Type-check without building
bunx tsc --noEmit
# Lint
bun run lint
```
***
## Unit tests
Unit tests live in `__tests__/` and use [Vitest](https://vitest.dev) with `jsdom`.
```text theme={null}
__tests__/
hooks/
builtin-policies.test.ts # Policy logic for each builtin
hooks-config.test.ts # Config loading and scope merging
policy-evaluator.test.ts # Param injection and evaluation order
custom-hooks-registry.test.ts # globalThis registry add/get/clear
custom-hooks-loader.test.ts # ESM loader, transitive imports, error handling
manager.test.ts # install/remove/list operations
components/
sessions-list.test.tsx # Session list component
project-list.test.tsx # Project list component
...
lib/
logger.test.ts
paths.test.ts
date-filters.test.ts
telemetry.test.ts
...
actions/
get-hooks-config.test.ts
get-hook-activity.test.ts
...
contexts/
ThemeContext.test.tsx
AutoRefreshContext.test.tsx
```
### Writing a policy unit test
```typescript theme={null}
import { describe, it, expect, beforeEach } from "vitest";
import { getBuiltinPolicies } from "../../src/hooks/builtin-policies";
import { allow, deny } from "../../src/hooks/policy-types";
describe("block-sudo", () => {
const policy = getBuiltinPolicies().find((p) => p.name === "block-sudo")!;
it("denies sudo commands", () => {
const ctx = {
eventType: "PreToolUse" as const,
payload: {},
toolName: "Bash",
toolInput: { command: "sudo apt install nodejs" },
params: { allowPatterns: [] },
};
expect(policy.fn(ctx)).toEqual(deny("sudo command blocked by failproofai"));
});
it("allows non-sudo commands", () => {
const ctx = {
eventType: "PreToolUse" as const,
payload: {},
toolName: "Bash",
toolInput: { command: "ls -la" },
params: { allowPatterns: [] },
};
expect(policy.fn(ctx)).toEqual(allow());
});
it("allows patterns in allowPatterns", () => {
const ctx = {
eventType: "PreToolUse" as const,
payload: {},
toolName: "Bash",
toolInput: { command: "sudo systemctl status nginx" },
params: { allowPatterns: ["sudo systemctl status"] },
};
expect(policy.fn(ctx)).toEqual(allow());
});
});
```
***
## End-to-end tests
E2E tests invoke the real `failproofai` binary as a subprocess, pipe a JSON payload to stdin, and assert on the stdout output and exit code. This tests the complete integration path that Claude Code uses.
### Setup
E2E tests run the binary directly from the repo source. Before the first run, build the CJS bundle that custom hook files use when they import from `'failproofai'`:
```bash theme={null}
bun build src/index.ts --outdir dist --target node --format cjs
```
Then run the tests:
```bash theme={null}
bun run test:e2e
```
Rebuild `dist/` whenever you change the public hook API (`src/hooks/custom-hooks-registry.ts`, `src/hooks/policy-helpers.ts`, or `src/hooks/policy-types.ts`).
### E2E test structure
```text theme={null}
__tests__/e2e/
helpers/
hook-runner.ts # Spawn the binary, pipe payload JSON, capture exit code + stdout + stderr
fixture-env.ts # Per-test isolated temp directories with config files
payloads.ts # Claude-accurate payload factories for each event type
hooks/
builtin-policies.e2e.test.ts # Each builtin policy with real subprocess
custom-hooks.e2e.test.ts # Custom hook loading and evaluation
config-scopes.e2e.test.ts # Config merging across project/local/global
policy-params.e2e.test.ts # Parameter injection for each parameterized policy
```
### Using the E2E helpers
**`FixtureEnv`** - isolated per-test environment:
```typescript theme={null}
import { createFixtureEnv } from "../helpers/fixture-env";
const env = createFixtureEnv();
// env.cwd - temp dir; pass as payload.cwd to pick up .failproofai/policies-config.json
// env.home - isolated home dir; no real ~/.failproofai leaks in
env.writeConfig({
enabledPolicies: ["block-sudo"],
policyParams: {
"block-sudo": { allowPatterns: ["sudo systemctl status"] },
},
});
```
`createFixtureEnv()` registers `afterEach` cleanup automatically.
**`runHook`** - invoke the binary:
```typescript theme={null}
import { runHook } from "../helpers/hook-runner";
import { Payloads } from "../helpers/payloads";
const result = await runHook(
"PreToolUse",
Payloads.preToolUse.bash("sudo apt install nodejs", env.cwd),
{ homeDir: env.home }
);
expect(result.exitCode).toBe(0);
expect(result.parsed?.hookSpecificOutput?.permissionDecision).toBe("deny");
```
**`Payloads`** - ready-made payload factories:
```typescript theme={null}
Payloads.preToolUse.bash(command, cwd)
Payloads.preToolUse.write(filePath, content, cwd)
Payloads.preToolUse.read(filePath, cwd)
Payloads.postToolUse.bash(command, output, cwd)
Payloads.postToolUse.read(filePath, content, cwd)
Payloads.notification(message, cwd)
Payloads.stop(cwd)
```
### Writing an E2E test
```typescript theme={null}
import { describe, it, expect } from "vitest";
import { createFixtureEnv } from "../helpers/fixture-env";
import { runHook } from "../helpers/hook-runner";
import { Payloads } from "../helpers/payloads";
describe("block-rm-rf (E2E)", () => {
it("denies rm -rf", async () => {
const env = createFixtureEnv();
env.writeConfig({ enabledPolicies: ["block-rm-rf"] });
const result = await runHook(
"PreToolUse",
Payloads.preToolUse.bash("rm -rf /", env.cwd),
{ homeDir: env.home }
);
expect(result.exitCode).toBe(0);
expect(result.parsed?.hookSpecificOutput?.permissionDecision).toBe("deny");
});
it("allows non-recursive rm", async () => {
const env = createFixtureEnv();
env.writeConfig({ enabledPolicies: ["block-rm-rf"] });
const result = await runHook(
"PreToolUse",
Payloads.preToolUse.bash("rm /tmp/file.txt", env.cwd),
{ homeDir: env.home }
);
expect(result.exitCode).toBe(0);
expect(result.stdout).toBe(""); // allow → empty stdout
});
});
```
### E2E response shapes
| Decision | Exit code | stdout |
| ------------------- | --------- | --------------------------------------------------------------------------------------- |
| `PreToolUse` deny | `0` | `{"hookSpecificOutput":{"permissionDecision":"deny","permissionDecisionReason":"..."}}` |
| `PostToolUse` deny | `0` | `{"hookSpecificOutput":{"additionalContext":"Blocked ... because: ..."}}` |
| Instruct (non-Stop) | `0` | `{"hookSpecificOutput":{"additionalContext":"Instruction from failproofai: ..."}}` |
| Stop instruct | `2` | empty stdout; reason in stderr |
| Allow | `0` | empty string |
### Vitest config
E2E tests use `vitest.config.e2e.mts` with:
* `environment: "node"` - no browser globals needed
* `pool: "forks"` - true process isolation (tests spawn subprocesses)
* `testTimeout: 20_000` - 20s per test (binary startup + hook eval)
The `forks` pool is important: thread-based workers share `globalThis`, which can interfere with subprocess-spawning tests. Process-based forks avoid this.
***
## CI
The full CI run (`bun run lint && bunx tsc --noEmit && bun run test:run && bun run build`) is required to pass before merging. The E2E suite runs as a separate CI job in parallel.
See [Contributing](../CONTRIBUTING.md) for the complete pre-merge checklist.