> ## Documentation Index
> Fetch the complete documentation index at: https://docs.befailproof.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluator SDK
> Build a service that scores Failproof AI sessions synchronously or asynchronously.
An evaluator receives a completed agent session and returns the quality signals you care about: numeric scores, an explanation for each score, and an optional summary. Failproof AI stores these results beside the trace and charts them across agents and environments.
## Set up an evaluator
Install the SDK and the server used to run it.
```bash theme={null}
pip install failproofai-sdk uvicorn
```
Create `evaluator.py`. This example checks whether a session contains any failed tool calls.
```python theme={null}
import os
from failproofai.evaluator import Evaluator, EvalResponse
app = Evaluator(token=os.environ.get("EVALUATOR_TOKEN"))
@app.config
def config():
return {"inactivity_timeout_secs": 1800}
@app.evaluator
def evaluate(req):
tool_errors = sum(
1 for item in req.events
if item.event_type == "tool_result" and item.payload.get("error")
)
return EvalResponse(
scores={"tool_reliability": 1.0 if tool_errors == 0 else 0.0},
reasoning={"tool_reliability": f"{tool_errors} tool errors"},
)
```
Set a shared token, start the evaluator, and confirm its health endpoint responds.
```bash theme={null}
export EVALUATOR_TOKEN=
uvicorn evaluator:app --host 0.0.0.0 --port 8080
```
In another terminal:
```bash theme={null}
curl http://127.0.0.1:8080/health
```
## Connect the evaluator to Failproof AI
1. Deploy the evaluator at an HTTPS URL reachable by Failproof AI Cloud.
2. Configure `EVALUATOR_ENDPOINT` with that URL and set `EVALUATOR_TOKEN` to the same token used by the evaluator. For managed Cloud, contact [support@befailproof.ai](mailto:support@befailproof.ai) to configure the connection.
3. Run an evaluation and confirm its scores appear in Failproof AI.
Open a completed session under **Observe → Sessions** and select **Run evaluation** if it was not evaluated automatically. Review the status, scores, reasoning, and summary in the session's **Evaluation** panel.
Use **Observe → Evaluations** to compare scores across agents or environments. Use **Observe → Metrics** for latency, cost, token, and other numeric measurements.
Start with one session to confirm that the evaluator returned the expected score keys and useful reasoning for that specific run.
Once individual results look correct, use the evaluation dashboard to compare those scores over time and across agents or environments.
A healthy chart should use stable score names; changing a key creates a separate series.
```bash theme={null}
fp evals --since 1h --score tool_reliability:0..1
fp evals --since 24h --aggregate
```
For a self-hosted Cloud instance, automatic evaluation is disabled until `EVALUATOR_ENDPOINT` is set on the server process. Restart the server after changing evaluator environment variables.
The service exposes `GET /health`, `GET /config`, `POST /evaluate`, and optionally `GET /evaluate/{job_id}`. Return `JobPending` for asynchronous work and register `@app.job_lookup` so Failproof AI can poll it.
When a token is configured, all routes except health require the same bearer token that Failproof AI sends as `EVALUATOR_TOKEN`.
## SDK types
| Type | Fields |
| ----------------- | --------------------------------------------------------------------------------------------- |
| `AgentEvent` | `id`, `ts`, `event_type`, `payload` |
| `EvalRequest` | `schema_version`, `session_id`, `agent_id`, `environment`, `started_at`, `ended_at`, `events` |
| `EvalResponse` | `scores`, `reasoning`, `summary` |
| `JobPending` | `job_id`, `next_poll_secs` |
| `EvaluatorConfig` | `inactivity_timeout_secs`, `default_poll_interval_secs` |
## Decorators and routes
| Decorator | Route | Required |
| ----------------- | ------------------------ | --------------------------- |
| `@app.evaluator` | `POST /evaluate` | Yes |
| `@app.job_lookup` | `GET /evaluate/{job_id}` | When returning `JobPending` |
| `@app.config` | `GET /config` | No |
The SDK caps evaluation request bodies at 25 MiB. Unknown request fields are ignored so services remain compatible as the event contract grows.
## Return asynchronous work
Use `JobPending` when evaluation cannot finish inside one request. The job ID is opaque to Failproof AI and must remain resolvable by your service until the result is collected or the server timeout expires.
```python theme={null}
from failproofai.evaluator import EvalRequest, EvalResponse, Evaluator, JobPending
app = Evaluator(token="shared-secret")
@app.evaluator
def start(req: EvalRequest) -> JobPending:
job_id = enqueue(req)
return JobPending(job_id=job_id, next_poll_secs=30)
@app.job_lookup
def lookup(job_id: str):
result = get_result(job_id)
if result is None:
return JobPending(job_id=job_id, next_poll_secs=30)
return EvalResponse(
scores=result.scores,
reasoning=result.reasoning,
summary=result.summary,
)
```
Polling cadence is selected in this order: `JobPending.next_poll_secs`, `EvaluatorConfig.default_poll_interval_secs`, then the server's `EVALUATOR_POLLING_INTERVAL_SECS`. Values are clamped between 1 second and 1 hour. The server's default wall-clock polling cap is one hour.
## Request and response fields
| Field | Type | Notes |
| --------------------------------------- | -------------------------- | ---------------------------------------------------- |
| `EvalRequest.schema_version` | `str` | Currently `"1"`. |
| `session_id`, `agent_id`, `environment` | `str` | Session identity and environment. |
| `started_at` | `datetime` | Timestamp of the first event. |
| `ended_at` | `datetime \| None` | Present when the session emitted an end event. |
| `events` | `list[AgentEvent]` | Full ordered event stream. |
| `AgentEvent.id` | `int` | Backend event row identifier. |
| `AgentEvent.ts` | `datetime` | Event timestamp. |
| `AgentEvent.event_type` | `str` | Event family such as `tool_use`. |
| `AgentEvent.payload` | `dict[str, Any]` | Complete event payload. |
| `EvalResponse.scores` | `dict[str, float] \| None` | Numeric dimensions charted in evaluations. |
| `EvalResponse.reasoning` | `dict[str, str] \| None` | Per-score explanations; keys should mirror `scores`. |
| `EvalResponse.summary` | `str \| None` | Overall evaluation narrative. |
## Server operator settings
Automatic evaluation is deployment-wide and remains disabled when `EVALUATOR_ENDPOINT` is absent.
| Variable | Default | Purpose |
| ---------------------------------- | ------- | ------------------------------------------------ |
| `EVALUATOR_ENDPOINT` | unset | Base URL of the evaluator service. |
| `EVALUATOR_TOKEN` | unset | Bearer token shared with `Evaluator(token=...)`. |
| `EVALUATOR_WORKERS` | `2` | Concurrent dispatcher workers. |
| `EVALUATOR_CLAIM_BATCH` | `4` | Sessions claimed per dispatcher pass. |
| `EVALUATOR_POLLING_INTERVAL_SECS` | `10` | Fallback async polling cadence. |
| `EVALUATOR_REQUEST_TIMEOUT_MS` | `30000` | Per-request evaluator timeout. |
| `EVALUATOR_MAX_ATTEMPTS` | `5` | Delivery attempts before terminal failure. |
| `EVALUATOR_CONFIG_REFRESH_SECS` | `300` | Refresh cadence for `/config`. |
| `EVALUATOR_MAX_POLL_DURATION_SECS` | `3600` | Maximum wall-clock async polling time. |
The server can also constrain which organizations use the deployment-global evaluator. Treat endpoint, token, retry, and organization-gate changes as operator configuration and restart or roll the server after changing them.
## Security and operations
* Put the evaluator behind HTTPS when traffic crosses a trusted network boundary.
* Configure a non-empty bearer token and keep it identical on both services.
* Do not log the token or full sensitive prompts from request payloads.
* Make synchronous handlers idempotent; retries may repeat a request.
* Persist asynchronous job state outside process memory in production.
* Return stable score keys. Renaming a key creates a new chart series rather than changing the old one.
The SDK emits structured lifecycle logs such as `eval received`, `eval responded`, `job lookup`, `config returned`, `auth rejected`, and handler exceptions. It does not configure logging handlers; use the host application's logging configuration.