> ## Documentation Index
> Fetch the complete documentation index at: https://docs.befailproof.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Alerts
> AgentEye Alerts documentation.
Alerts evaluate a rule on a schedule and notify you when something crosses a threshold. They turn AgentEye from a passive dashboard into a paging surface for the things that matter: error rate spikes, latency regressions, evaluator score drops, or any custom event you can express as a ClickHouse query.
This guide covers what an alert is, the five trigger types, the four notification channels, the incident lifecycle, and the operational knobs.
***
## Concepts
* **Alert** — the rule. It says *what* to check (the trigger), *how often* (the eval interval), *when to consider it broken* (compound logic), *how urgent* (severity), and *who/where to notify* (channels).
* **Incident** — what happens when an alert fires. One alert has at most one open incident at a time. Repeated breaches update the same incident's evidence. Incidents are resolved by an operator; automatic resolution when the rule stops firing is planned but not currently enabled.
* **Channel** — where a notification goes: email, Slack, generic webhook, or in-dashboard. Each alert can attach any combination.
Alerts and incidents belong to an organization and are shared across that org's operators (in a single-tenant deployment that is simply the built-in `default` org). Incidents can be assigned to an individual operator for triage.
***
## Trigger types
Five kinds, each with its own JSON spec. Pick the one that matches how you want to describe "broken". The dashboard's **new alert** form builds the same spec for you, switching the condition editor to match the trigger type you pick:
### 1. `metric_threshold`
The simplest. Choose a metric from a closed list, an operator, a threshold, and a time window.
| Metric | What it counts |
| ---------------- | -------------------------------------------------------------------- |
| `event_count` | Total events in the window |
| `error_count` | `event_type = 'error'` OR any event with `error_type IS NOT NULL` |
| `error_rate` | `error_count / event_count` (0..1) |
| `p95_latency_ms` | `quantile(0.95)(duration_ms)` across all events with a `duration_ms` |
| `p99_latency_ms` | `quantile(0.99)(duration_ms)` |
| `token_sum` | `SUM(input_tokens + output_tokens)` |
Operators: `>`, `>=`, `<`, `<=`, `==` (also accepted as `gt`, `gte`, `lt`, `lte`, `eq`).
Optional filters: `environment`, `event_type`.
Example spec:
```json theme={null}
{
"metric": "p95_latency_ms",
"filter": { "environment": "production" },
"window_secs": 900,
"op": ">",
"value": 5000
}
```
### 2. `custom_sql`
For anything the preset metrics don't cover. The operator-supplied SQL goes through the same `/queries/run` guard (SELECT/WITH only, single statement, 10 000-row cap) before the dispatcher runs it. Two modes:
* **rows mode** (no `op`/`value`): the alert fires as soon as the query returns at least one row.
* **value mode**: the query must alias one column as `metric_value`; the dispatcher compares the first row's `metric_value` against `value` using `op`.
```json theme={null}
{
"sql": "SELECT count() AS metric_value FROM agenteye.events WHERE event_type = 'error' AND ts >= now() - INTERVAL 1 HOUR",
"op": ">",
"value": 100
}
```
### 3. `evaluation_score`
Reads `agenteye.evaluations` and compares the average of a named score across a window.
```json theme={null}
{
"score_key": "hallucination",
"op": ">",
"value": 0.6,
"window_secs": 3600,
"min_count": 5,
"environment": "production"
}
```
`min_count` guards against single-sample outliers; the dispatcher won't fire until at least N evaluations exist in the window.
### 4. `eval_compound`
Catch a quality regression that only shows up across *several* evaluator scores at once. Where `evaluation_score` watches a single named score, `eval_compound` composes multiple evaluation-score conditions into one alert and combines their results with a chosen logic; so one rule can express "fire if helpfulness drops **or** hallucination climbs", "fire only if helpfulness **and** tool-efficiency both drop", or "fire if **at least 2 of** these three checks breach".
Each condition reads the average of one named score from `agenteye.evaluations` over the shared window and tests it with its own operator and threshold. The boolean results are then combined by the `combinator`:
| Combinator | Logic | Fires when |
| ------------------- | ------ | ------------------------------- |
| `"any"` | OR | at least one condition breaches |
| `"all"` | AND | every condition breaches |
| `{ "at_least": N }` | M-of-N | at least N conditions breach |
```json theme={null}
{
"combinator": { "at_least": 2 },
"conditions": [
{ "score_key": "hallucination", "op": ">", "value": 0.8 },
{ "score_key": "helpfulness", "op": "<", "value": 0.6 },
{ "score_key": "tone", "op": "<", "value": 0.5 }
],
"window_secs": 3600,
"min_count": 5,
"environment": "production"
}
```
* `combinator`: `"any"`, `"all"`, or `{ "at_least": N }`.
* `conditions[]`: each `{ score_key, op, value }`, using the same operators as the other triggers (`>`, `>=`, `<`, `<=`, `==`).
* `window_secs`: the shared lookback applied to every condition (default `3600`).
* `min_count`: per-condition minimum number of evaluations before that condition can breach; a condition with too few samples in the window counts as "not breached" (default `1`).
* `environment`: optional; restricts every condition to one environment.
The notification's evidence records each condition's observed average, the threshold it was tested against, how many evaluations were seen, and whether it breached; so you can see exactly which checks tripped.
### 5. `per_event`
For "any event matching X has landed" alerts. No aggregation; the dispatcher fires as soon as it sees a match in the lookback window.
```json theme={null}
{
"event_type": "error",
"lookback_secs": 60,
"environment": "production",
"tool_name": "bash",
"agent_id": "caa",
"error_type": "TimeoutError",
"message_contains": "vertex-ai timed out"
}
```
All filters are AND-combined; any field you leave out is unconstrained.
| Field | Purpose |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_id` | Restrict to errors from a specific agent (the one shown on the `/errors` row). |
| `error_type` | Restrict to a specific error class (e.g. `TimeoutError`) rather than every error. |
| `message_contains` | Case-insensitive substring match against `payload.message`. Useful for catching one specific failure mode (e.g. `prompt is too long`) without alerting on every error from the same agent. Capped at 200 characters; matched as a literal string, not a pattern. |
Tip: set `lookback_secs` to roughly match the alert's `eval_interval_secs` so you don't double-notify on the same event.
**Shortcut from `/errors`:** each error group's representative row on the errors view (and the session event-detail panel for an error event) has a **+ alert** button that opens `/alerts/new` prefilled with a `per_event` trigger keyed to that row's `event_type` + environment, including a name seeded from the payload's `error_type` when present. You still pick channels and confirm the lookback, but the matcher is filled in for you. Operators need `alerts:write` for the button to appear.
***
## Compound logic (M of N)
Every alert has two integer knobs in addition to the trigger:
* `eval_window`: how many recent evaluations to look at (default 1)
* `min_breaches`: how many of those must breach before the alert fires (default 1)
`1 of 1` (the default) is "fire on the first breach". `3 of 5` means "fire when the rule has breached 3 of the last 5 evaluations", useful for jittery signals where a single bad measurement is noise. The dispatcher maintains a ring buffer per alert; you don't have to manage state.
***
## Eval interval
`eval_interval_secs` controls how often the dispatcher runs your rule. Clamped to `[30, 86400]`. Presets in the dashboard: 1m / 5m / 15m / 1h. Pick an interval matched to how fast the underlying signal moves: a 5-minute error-rate alert evaluated every 15 seconds wastes CPU; a per-event alert needs a short lookback or it'll silently drop events between ticks.
***
## Channels
Each alert can attach any combination of these four. Per-channel credentials (Slack webhook URL, generic webhook URL + signing secret, default email recipients) are configured once in **`/settings`** and referenced by key from each alert. This way one Slack channel can serve many alerts without each one storing its own copy of the webhook URL.
The three external channel kinds (email, Slack, webhook) are also gated by an org-wide kill switch, `alerts.enabled_channels`. When a fired alert attaches a channel kind that is not in this set, the dispatcher skips it and records an `alert_notifications` row with status `skipped_disabled` and target `` (letting you globally pause, say, all Slack delivery without editing every rule). The in-dashboard channel is always allowed. See [Configuration](#configuration).
### Email
Reuses the same SMTP transport that ships OTP login emails. Recipients resolve in order:
1. Per-channel `recipients[]` override (when non-empty).
2. The `alerts.email_default_recipients` setting (an array of email strings).
If SMTP is unconfigured the channel is a no-op; the dispatcher still records an `alert_notifications` row with target `` so the audit trail makes the misconfiguration visible.
### Slack
Sends a Block Kit message to an [incoming webhook URL](https://api.slack.com/messaging/webhooks).
* Default URL: `alerts.slack_default_webhook` (set in `/settings`).
* Per-alert override: set the channel's `webhook_setting_key` to any other URL-typed setting key, e.g. `alerts.slack_oncall`, `alerts.slack_costs`.
The header includes a severity emoji (`:rotating_light:` / `:warning:` / `:bell:`), and the message carries a button that deep-links to the incident page.
### Generic webhook
A JSON-POST integration for PagerDuty, Opsgenie, or your own ingestion endpoint. Body shape:
```json theme={null}
{
"schema_version": 1,
"event": "alert.firing",
"incident": { "id": "uuid", "url": "https://…/incidents/uuid", "opened_at": "2026-…" },
"alert": { "id": "uuid", "name": "errors > 50/hr", "severity": "critical" },
"breach": { "value": 73.0, "summary": "ErrorCount > 50 (observed 73.000)", "evidence": { … } }
}
```
When `alerts.webhook_signing_secret` is set, the request includes an `X-AgentEye-Signature: sha256=` header, the HMAC-SHA256 of the body using the secret. Verify it on the receiving side before trusting the payload.
The header `X-AgentEye-Event` carries `alert.firing` / `alert.test`. (`alert.resolved` is reserved for the planned auto-resolve feature and is not currently emitted.)
### In-dashboard
No external delivery: the alert just writes an `alert_notifications` row that the dashboard's incidents page surfaces. Useful while you're tuning a rule and don't want to spam an external system, or for low-urgency alerts that operators check during normal triage.
***
## Incident lifecycle
```
firing ──ack──▶ acknowledged
│ │
└────────────────────┴───── operator resolve ───▶ resolved
```
* **firing** — the dispatcher just opened the incident or detected another breach. The firing notification is fanned out exactly once (gated by the `notified_firing_at` timestamp on the incident).
* **acknowledged** — an operator pressed *ack* on `/incidents/:id`. The incident is still considered open; subsequent breaches will update its evidence without re-notifying.
* **resolved** — an operator pressed *resolve*. Automatic resolution when the rule stops breaching is planned but not currently enabled, so an open incident stays open until an operator resolves it.
A new incident can re-open on the same alert at any point after the previous one resolved.
**Activity timeline.** Every action on an incident — opened, acknowledged, resolved — is recorded in an append-only activity log and shown on the incident's *activity* timeline, each entry attributed to the operator who performed it (by email) or to **automated** for actions the dispatcher took on its own (auto-open on breach). Acknowledgement is shared: several operators can ack the same incident and each appears as a separate, attributed entry.
The **Incidents** inbox groups open incidents by state and lets you filter by severity and assignee:
Opening an incident shows the breach evidence, assignees and subscribers, the attributed activity timeline, and a comment thread:
***
## Required permissions
Authoring alert *rules* and triaging *incidents* are separate concerns with separate grants, so you can give an on-call rotation incident access without also handing it the ability to rewrite the rules.
* `alerts:read`: view alert rules.
* `alerts:write`: create, edit, delete alert rules, and trigger a test notification.
* `incidents:read`: view incidents.
* `incidents:write`: open manual (ad-hoc) incidents that are not tied to an alert.
* `incidents:ack`: acknowledge, assign, comment on, and resolve incidents.
> **Legacy `alerts:ack`.** Keys and operators that were granted the older `alerts:ack` token keep working: it is honored as `incidents:ack` (and implies `incidents:read`), so existing on-callers retain their access without re-issuing credentials. Issue new grants with the `incidents:*` family.
Grant on API keys (`POST /keys`) and operators (`PUT /users/:id`). The dashboard's `PermGate` locks the relevant buttons when a permission is missing; you'll see `// 403` next to the action.
> **Email-recipient picker.** The alert editor's recipient picker lists your org's members so you can pick them by name. It loads for any operator holding `alerts:read` or `alerts:write`; viewing your team's directory for this purpose does **not** require `users:read`, and the picker returns only member email addresses, never full user records.
***
## Configuration
Environment variables consumed by the dispatcher:
| Var | Default | Purpose |
| -------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------ |
| `ALERT_WORKERS` | `1` | Worker tasks per server instance. Most deployments only need one. |
| `ALERT_CLAIM_BATCH` | `16` | Max alerts a single worker tick processes. |
| `ALERT_POLL_IDLE_SECS` | `5` | Worker sleep when the queue is empty. |
| `ALERT_REQUEST_TIMEOUT_MS` | `15000` | Per-trigger evaluation timeout. |
| `DASHBOARD_URL` | `https://app.befailproof.ai` | Origin used to build the incident magic-link in notifications. Set to your dashboard host. |
After a transient evaluation failure (ClickHouse unreachable, a query timeout), the dispatcher retries the rule with exponential backoff. Once a rule has accumulated **5** consecutive transient failures it is rescheduled at its normal cadence instead of continuing to back off, so a persistently failing rule still keeps being re-evaluated. This ceiling is fixed and is not operator-tunable.
Channel settings (managed from `/settings`, not env):
* `alerts.email_default_recipients` (`email_list`): JSON array of email strings — the default recipients for email channels.
* `alerts.slack_default_webhook` (`url`): default Slack incoming-webhook URL.
* `alerts.webhook_default_url` (`url`): default generic-webhook URL.
* `alerts.webhook_signing_secret` (`secret`): HMAC-SHA256 key. Always returned as `""` in the GET response; type a new value to rotate.
* `alerts.enabled_channels` (`channel_set`): the org-wide set of external channel kinds that are dispatched when an alert fires; defaults to all three (`email`, `slack`, `webhook`). Removing a kind here globally suppresses that channel for every alert without editing each rule. The in-dashboard channel is always delivered and is not affected by this setting.
***
## Verify a new alert
Before relying on a new alert:
1. Save it as enabled with at least one notification channel.
2. Select **test** on the alert detail page and confirm each configured destination receives the synthetic notification.
3. After the first real breach, confirm the incident appears under **Incidents** and its measured value matches the corresponding dashboard query.
Incidents do not resolve automatically when a condition clears. An operator must resolve them from the incident detail page.
***
## Troubleshooting
| Symptom | Likely cause |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Alert never fires | `enabled = false`, or no channels attached, or the underlying CH query returns 0 rows. Use *test* to confirm channels; use `/queries/run` to confirm the metric. |
| Slack notification missing | `alerts.slack_default_webhook` (or the per-alert override key) is unset: check `alert_notifications.target` for `` rows; or the `slack` kind is globally disabled in `alerts.enabled_channels`: check for `alert_notifications` rows with status `skipped_disabled` and target ``. |
| Generic webhook 401 | The recipient is requiring a signature but `alerts.webhook_signing_secret` is unset. Verify on the recipient that the HMAC matches `hmac_sha256(secret, body)`. |
| Email "from all sends failed" | SMTP credentials wrong or the `from` address is rejected by your relay. Same surface that ships OTP emails: if those work, the SMTP transport is fine. |
| Incident reopens repeatedly | The compound knobs are too aggressive: try raising `min_breaches` or `eval_window` so transient spikes don't reopen incidents you resolved. |