Evaluator SDK

An evaluator receives a completed agent session and returns the quality signals you care about: numeric scores, an explanation for each score, and an optional summary. Failproof AI stores these results beside the trace and charts them across agents and environments.

Set up an evaluator

Install the evaluator SDK

Install the SDK and the server used to run it.

pip install failproofai-sdk uvicorn

Define what to score

Create evaluator.py. This example checks whether a session contains any failed tool calls.

import os
from failproofai.evaluator import Evaluator, EvalResponse

app = Evaluator(token=os.environ.get("EVALUATOR_TOKEN"))

@app.config
def config():
    return {"inactivity_timeout_secs": 1800}

@app.evaluator
def evaluate(req):
    tool_errors = sum(
        1 for item in req.events
        if item.event_type == "tool_result" and item.payload.get("error")
    )
    return EvalResponse(
        scores={"tool_reliability": 1.0 if tool_errors == 0 else 0.0},
        reasoning={"tool_reliability": f"{tool_errors} tool errors"},
    )

Run and test it locally

Set a shared token, start the evaluator, and confirm its health endpoint responds.

export EVALUATOR_TOKEN=<shared-token>
uvicorn evaluator:app --host 0.0.0.0 --port 8080

In another terminal:

curl http://127.0.0.1:8080/health

Connect the evaluator to Failproof AI

Deploy the evaluator at an HTTPS URL reachable by Failproof AI Cloud.
Configure EVALUATOR_ENDPOINT with that URL and set EVALUATOR_TOKEN to the same token used by the evaluator. For managed Cloud, contact support@befailproof.ai to configure the connection.
Run an evaluation and confirm its scores appear in Failproof AI.

Dashboard
CLI

Open a completed session under Observe → Sessions and select Run evaluation if it was not evaluated automatically. Review the status, scores, reasoning, and summary in the session’s Evaluation panel.Use Observe → Evaluations to compare scores across agents or environments. Use Observe → Metrics for latency, cost, token, and other numeric measurements.Start with one session to confirm that the evaluator returned the expected score keys and useful reasoning for that specific run.

A session detail view showing evaluation scores and reasoning beside its trace.

Once individual results look correct, use the evaluation dashboard to compare those scores over time and across agents or environments.

A quality dashboard charting evaluator scores over time.

A healthy chart should use stable score names; changing a key creates a separate series.

fp evals --since 1h --score tool_reliability:0..1
fp evals --since 24h --aggregate

For a self-hosted Cloud instance, automatic evaluation is disabled until EVALUATOR_ENDPOINT is set on the server process. Restart the server after changing evaluator environment variables. The service exposes GET /health, GET /config, POST /evaluate, and optionally GET /evaluate/{job_id}. Return JobPending for asynchronous work and register @app.job_lookup so Failproof AI can poll it. When a token is configured, all routes except health require the same bearer token that Failproof AI sends as EVALUATOR_TOKEN.

SDK types

Type	Fields
`AgentEvent`	`id`, `ts`, `event_type`, `payload`
`EvalRequest`	`schema_version`, `session_id`, `agent_id`, `environment`, `started_at`, `ended_at`, `events`
`EvalResponse`	`scores`, `reasoning`, `summary`
`JobPending`	`job_id`, `next_poll_secs`
`EvaluatorConfig`	`inactivity_timeout_secs`, `default_poll_interval_secs`

Decorators and routes

Decorator	Route	Required
`@app.evaluator`	`POST /evaluate`	Yes
`@app.job_lookup`	`GET /evaluate/{job_id}`	When returning `JobPending`
`@app.config`	`GET /config`	No

The SDK caps evaluation request bodies at 25 MiB. Unknown request fields are ignored so services remain compatible as the event contract grows.

Return asynchronous work

Use JobPending when evaluation cannot finish inside one request. The job ID is opaque to Failproof AI and must remain resolvable by your service until the result is collected or the server timeout expires.

from failproofai.evaluator import EvalRequest, EvalResponse, Evaluator, JobPending

app = Evaluator(token="shared-secret")

@app.evaluator
def start(req: EvalRequest) -> JobPending:
    job_id = enqueue(req)
    return JobPending(job_id=job_id, next_poll_secs=30)

@app.job_lookup
def lookup(job_id: str):
    result = get_result(job_id)
    if result is None:
        return JobPending(job_id=job_id, next_poll_secs=30)
    return EvalResponse(
        scores=result.scores,
        reasoning=result.reasoning,
        summary=result.summary,
    )

Polling cadence is selected in this order: JobPending.next_poll_secs, EvaluatorConfig.default_poll_interval_secs, then the server’s EVALUATOR_POLLING_INTERVAL_SECS. Values are clamped between 1 second and 1 hour. The server’s default wall-clock polling cap is one hour.

Request and response fields

Field	Type	Notes
`EvalRequest.schema_version`	`str`	Currently `"1"`.
`session_id`, `agent_id`, `environment`	`str`	Session identity and environment.
`started_at`	`datetime`	Timestamp of the first event.
`ended_at`	`datetime \| None`	Present when the session emitted an end event.
`events`	`list[AgentEvent]`	Full ordered event stream.
`AgentEvent.id`	`int`	Backend event row identifier.
`AgentEvent.ts`	`datetime`	Event timestamp.
`AgentEvent.event_type`	`str`	Event family such as `tool_use`.
`AgentEvent.payload`	`dict[str, Any]`	Complete event payload.
`EvalResponse.scores`	`dict[str, float] \| None`	Numeric dimensions charted in evaluations.
`EvalResponse.reasoning`	`dict[str, str] \| None`	Per-score explanations; keys should mirror `scores`.
`EvalResponse.summary`	`str \| None`	Overall evaluation narrative.

Server operator settings

Automatic evaluation is deployment-wide and remains disabled when EVALUATOR_ENDPOINT is absent.

Variable	Default	Purpose
`EVALUATOR_ENDPOINT`	unset	Base URL of the evaluator service.
`EVALUATOR_TOKEN`	unset	Bearer token shared with `Evaluator(token=...)`.
`EVALUATOR_WORKERS`	`2`	Concurrent dispatcher workers.
`EVALUATOR_CLAIM_BATCH`	`4`	Sessions claimed per dispatcher pass.
`EVALUATOR_POLLING_INTERVAL_SECS`	`10`	Fallback async polling cadence.
`EVALUATOR_REQUEST_TIMEOUT_MS`	`30000`	Per-request evaluator timeout.
`EVALUATOR_MAX_ATTEMPTS`	`5`	Delivery attempts before terminal failure.
`EVALUATOR_CONFIG_REFRESH_SECS`	`300`	Refresh cadence for `/config`.
`EVALUATOR_MAX_POLL_DURATION_SECS`	`3600`	Maximum wall-clock async polling time.

The server can also constrain which organizations use the deployment-global evaluator. Treat endpoint, token, retry, and organization-gate changes as operator configuration and restart or roll the server after changing them.

Security and operations

Put the evaluator behind HTTPS when traffic crosses a trusted network boundary.
Configure a non-empty bearer token and keep it identical on both services.
Do not log the token or full sensitive prompts from request payloads.
Make synchronous handlers idempotent; retries may repeat a request.
Persist asynchronous job state outside process memory in production.
Return stable score keys. Renaming a key creates a new chart series rather than changing the old one.

The SDK emits structured lifecycle logs such as eval received, eval responded, job lookup, config returned, auth rejected, and handler exceptions. It does not configure logging handlers; use the host application’s logging configuration.

Connect your agents

Reference

HTTP API

Set up an evaluator

Connect the evaluator to Failproof AI

SDK types

Decorators and routes

Return asynchronous work

Request and response fields

Server operator settings

Security and operations

​Set up an evaluator

​Connect the evaluator to Failproof AI

​SDK types

​Decorators and routes

​Return asynchronous work

​Request and response fields

​Server operator settings

​Security and operations

Set up an evaluator

Connect the evaluator to Failproof AI

SDK types

Decorators and routes

Return asynchronous work

Request and response fields

Server operator settings

Security and operations