Skip to main content
This page defines the vocabulary FailproofAI Observability uses. If a term in another guide is unfamiliar, it’s defined here. You don’t need to read it end to end: skim it, or jump back when you hit a word you want pinned down.

​
The data model

Event The smallest unit of data. One event records a single step your agent took: a tool_use, a model_request, a hook_completed, an error, and so on. Your agent emits events through the Python SDK; they show up live on the Events page. Session One agent run, identified by a session_id. A session is all the events that share that id, rolled up into a single row on the Sessions page and drawn as an execution graph on its detail page. A session usually starts with agent_start and ends with agent_end. Agent A named actor inside a run, identified by an agent_id. A run can involve several agents: a planner that spawns a summarizer sub-agent, for example. Sub-agents carry a parent_id, which is what lets FailproofAI Observability draw them on their own lanes in the execution graph. Environment A label for where the run happened: production, staging, dev. You set it once when you configure the SDK. Almost every dashboard page can filter by environment. Context-window fill The percentage of a model’s context window a response consumed. FailproofAI Observability stamps it on model_response events for models it recognizes, so prompt growth and impending compaction are visible right in the event stream.

​
Quality

Evaluation A quality score for a finished session, produced by a scoring service you run. Evaluations are opt-in: until you connect an evaluator, sessions are recorded but not scored. Each evaluation can carry several named scores (for example helpfulness, factuality, tool_efficiency), each with a short reasoning note. See Evaluation suite. Score key The name of one dimension an evaluator reports, such as helpfulness. Alerts and audits can watch a specific score key over time. Evaluator Your scoring service. FailproofAI Observability POSTs a finished run’s transcript to it and stores the scores it returns. It does not ship a default evaluator; the scoring logic is yours.

​
Finding and fixing failures

Hook A guardrail or side-effect your agent framework runs around a step: a content-safety check, PII redaction, a budget guard. Hooks emit hook_triggered / hook_completed events with an outcome (allow, deny, modify), and get their own observe page. Alert rule A rule that fires when a metric crosses a threshold you set: error rate, p95 latency, token cost, or an evaluator score. When a rule fires, it opens an incident and notifies your chosen channels (email, Slack, webhook, in-dashboard). See Alerts. Incident An open issue created when an alert rule fires. Incidents have a lifecycle (acknowledge, assign, resolve) and an activity timeline that records every action. You can also open one manually. Audit A recurring investigation (hourly to weekly) that mines your logs across sessions for failure patterns you haven’t written a rule for: error clusters, low scores, latency outliers, tool-call loops, and runs that never finished. Where an alert watches a metric you already know about, an audit tells you what to look at next. See Audits. Finding One ranked, evidence-backed result from an audit run. A finding names a pattern, links to the exact sessions behind it, and carries a triage lifecycle (acknowledge, resolve, mute, dismiss). FailproofAI Observability deduplicates findings run-over-run so a known pattern updates instead of piling up. The AI assistant The in-dashboard chat that answers questions about your agents in plain English, over your own data. It is read-only by default; anything it creates (a saved query, a dashboard) is approval-gated, and it can never delete. See AI assistant.

​
Running it

Organization (tenant) An isolated workspace. One FailproofAI Observability instance can host many organizations, each with its own users, keys, and data. Every dashboard URL is scoped under your org slug (/<org-slug>/…). Collector agenteye-collector, the lightweight daemon that runs on each agent machine, batches the events the SDK writes to disk, and ships them to the server. API key A scoped token that authenticates a client against the server. Keys carry granular permissions (for example events:add for the collector, read-only scopes for a dashboard key). See API keys. Server The ingest and API service. It ingests events, stores operational state in your databases, and serves the dashboard and CLI. Dashboard The web UI. Every page is scoped to an organization and reads through the server’s API.

​
Next steps

  • Overview: how these pieces fit together.
  • Observability: the observe surfaces (Events, Sessions, Models, Tools, Hooks, Errors).