Failproof AI Observability is an enterprise product from Failproof AI. Want to see it in action? Request a demo: email nikita@befailproof.ai.
See it in action
Two short videos show the two things teams reach for first: tracing a run, and finding failures automatically. Agent tracing: follow a single run step by step, from goal to tools to final answer. Failproof Audit: let Failproof AI Observability mine your logs across sessions and tell you what to fix.Why teams use it
- See what your agent actually did. Every run becomes a readable, git-style execution graph: which tools ran in parallel, which sub-agents branched off, where it stalled, and what it spent.
- Catch quality regressions automatically. Connect a small scoring service and Failproof AI Observability scores every finished run, so a drop in helpfulness or a spike in hallucinations shows up on its own.
- Find failures you didn’t write a rule for. Recurring audits mine your logs across sessions for error clusters, latency outliers, low scores, and stuck runs, then hand you ranked, evidence-backed findings.
- Get paged when it matters. Threshold rules fire on error rate, latency, cost, or evaluator scores and open incidents you can acknowledge, assign, and resolve.
- Ask questions in plain English. An in-dashboard AI assistant answers “how is quality trending in prod this week?” over your own data. Any change it makes is approval-gated.
- Keep your data. Failproof AI Observability is self-hosted: events, prompts, and analytics stay in infrastructure you control.
What you get
Failproof AI Observability is organized around three ideas (observe, analyze, and admin), mirrored in the dashboard’s left sidebar. Observe (the raw truth of what happened):- Event stream: the live, per-step trail of every run (tool calls, model calls, hooks, errors).
- Sessions: those events rolled up into one row per run, each ready to be scored, with a git-style execution graph.
- Performance metrics: per-surface latency heat-maps and p50/p95/p99 vitals for models, tools, and hooks, so a tail spike stands out from the median.
- Error tracking: one triage surface for everything that went wrong, one click from a firing alert.
- Queries and dashboards: saved SQL over your events and evaluations, charted into shared, org-scoped dashboards.
- Evaluations: quality scores produced by your own evaluator service, with per-score reasoning.
- Audits: recurring investigations that surface failure patterns across sessions.
- Alerts and incidents: threshold rules that page you, plus an incident workflow to triage them.
- CLI: drive your whole deployment from the terminal or a script, and let a coding agent do it for you in plain English.
- AI assistant: ask questions about your agents in plain English, right inside the dashboard.
- REST API: everything the dashboard and CLI do is backed by a REST API you can call directly with a scoped API key — ingest events, query sessions and evaluations, and manage dashboards, alerts, audits, users, and keys, so you can wire Failproof AI Observability into your own tooling.
- API keys: scoped tokens for the collector, the dashboard, and the assistant.
- Users: passwordless, email-based sign-in with an allowlist.
- Settings: per-org configuration, including model context-window overrides.
How the pieces fit
Data flows in one direction, from your agent code to the dashboard: your agent (via the Python SDK) emits events to the agenteye-collector, which ships them to the server, which serves the dashboard. Two optional services round it out — a scoring service (evaluations) and an AI assistant service (the in-dashboard chat).- Python SDK: you add a few
agenteye.event.*calls to your agent; events are buffered locally. - agenteye-collector: a lightweight daemon on each agent machine that batches events and ships them to the server.
- Server: ingests your events, keeps operational state in your own databases, and serves the REST API that the dashboard, CLI, and your own integrations all use.
- Dashboard: where you explore everything.
- Optional services: a scoring service (evaluations), and an AI assistant service (the in-dashboard chat).
Getting Failproof AI Observability
Failproof AI Observability is an enterprise product from Failproof AI, and it works alongside Failproof AI Enforcement — the policy and guardrail product — under the Failproof AI brand. It runs entirely in your own environment. If you don’t have access to the packages yet, request a demo and we’ll get you set up: email nikita@befailproof.ai.Next steps
- Concepts: the Failproof AI Observability vocabulary in one place.
- Observability: follow what your agents do, run by run.
- Security: how Failproof AI Observability keeps your data isolated and in your control.