Evaluator SDK

The Evaluator SDK runs evaluations on your own infrastructure. Your worker registers its evaluations with Failproof AI, claims sessions as they finish, scores them, and submits the results, all over outbound HTTPS: nothing connects in to it. Use it for what hosted Python cannot do — LLM judges, model calls, packages, secrets, and network access. Its results appear beside hosted ones on the evaluations page, tagged customer. It ships in failproofai-sdk, under failproofai_sdk.evaluator; importing the tracing SDK does not load it.

pip install failproofai-sdk

Write evaluations

from failproofai_sdk.evaluator import ConditionResult, EvalResult, Evaluator, Metric, Score

app = Evaluator(name="customer-production", version="2026.08.1")


@app.eval(
    "tool_efficiency",
    version="1.0.0",
    labels=["tools", "deterministic"],
    when=lambda session: ConditionResult(session.count("tool_use") > 0, "no_tool_calls"),
)
def tool_efficiency(session):
    calls = session.events_of_type("tool_use")
    distinct = {e.payload.get("tool_name") for e in calls if e.payload.get("tool_name")}
    value = len(distinct) / len(calls)
    return EvalResult(
        score=Score(value, passed=value >= 0.7),
        metrics={"tool_call_count": Metric(len(calls), unit="events")},
        reasoning=f"{len(distinct)} distinct tools across {len(calls)} calls",
    )


@app.eval(
    "answer_relevance",
    version="judge-v1",
    labels=["llm_judge", "relevance"],
    when=lambda session: ConditionResult(
        session.count("human_input") > 0 and session.count("model_response") > 0,
        "no_exchange",
    ),
    timeout_seconds=30,
)
async def answer_relevance(session):
    question = session.events_of_type("human_input")[-1].payload.get("response")
    answer = session.events_of_type("model_response")[-1].payload.get("content")
    value, reasoning = await ask_judge(question, answer)  # your LLM call: a 0-1 score and why
    return EvalResult(score=Score(value, passed=value >= 0.7), reasoning=reasoning)


if __name__ == "__main__":
    app.run_from_env()

@app.eval(key, version=...) registers an evaluation. The key is what its results chart under; change the version whenever the logic changes, and each result keeps the version that produced it. One worker holds up to 100 evaluations.
result_kind is "score" unless you say otherwise. For a "metric" or "assertion" evaluation, name one metrics or assertions entry after the key: that entry is its result.
when decides whether a session applies. Return ConditionResult(False, "<reason>") to skip one, and the reason is recorded.
An evaluation can be a plain function or async, and timeout_seconds bounds it.
Payload keys — tool_name, response, and content above — are whatever your agents send, so read them off a real session.

Run the worker

Put a key with the evaluations:run permission, created under Administration → Keys, in FAILPROOFAI_EVALUATOR_TOKEN — set it from your secret store rather than typing it into a command — and start the worker:

FAILPROOFAI_EVALUATOR_URL=https://app.befailproof.ai python evaluator.py

Without the __main__ block, python -m failproofai_sdk.evaluator evaluator:app does the same.

Variable	Default	Purpose
`FAILPROOFAI_EVALUATOR_URL`	required	Where Failproof AI is: `https://app.befailproof.ai` for Cloud. HTTPS unless it points at loopback
`FAILPROOFAI_EVALUATOR_TOKEN`	required	A key with `evaluations:run`
`FAILPROOFAI_EVALUATOR_WORKER_ID`	`<hostname>-<pid>`	Names this worker
`FAILPROOFAI_EVALUATOR_CONCURRENCY`	`1`	Sessions this worker scores at once
`FAILPROOFAI_EVALUATOR_REQUEST_TIMEOUT_SECONDS`	`30`	Timeout for each request to Failproof AI
`FAILPROOFAI_EVALUATOR_DRAIN_TIMEOUT_SECONDS`	`60`	How long a stopping worker waits for runs in flight
`FAILPROOFAI_EVALUATOR_ALLOW_INSECURE_HTTP`	`false`	Allow plain HTTP to a URL that is not loopback — see the warning below
`FAILPROOFAI_EVALUATOR_MODULE`	none	The `module:attribute` for `python -m failproofai_sdk.evaluator`

FAILPROOFAI_EVALUATOR_ALLOW_INSECURE_HTTP sends everything in cleartext. The worker carries FAILPROOFAI_EVALUATOR_TOKEN as an Authorization: Bearer header on every request, and the transcripts it fetches are the sessions themselves — so anyone on the path reads both, and the token they read runs evaluations until you rotate it. Use it only on an isolated development network. Everywhere else the URL must be HTTPS; loopback needs no flag.

Result types

Type	Fields
`Score`	`value` (0 to 1), `passed`, `unit` (default `ratio`), `display_value`, `description`
`Metric`	`value`, `unit`, `display_value`, `description`
`Assertion`	`passed`, `description`
`EvalResult`	`score`, `metrics`, `assertions`, `reasoning`, `summary`, `labels`
`ConditionResult`	`applicable`, `reason_code`

An EvalResult carries at least one score, metric, or assertion, and at most 25, each under a unique key.

The session

Field or method	Gives you
`session_id`, `agent_id`, `environment`	The session’s identity
`started_at`, `ended_at`	When it started and ended
`event_count`, `events`	The full, ordered transcript
`count(event_type)`	How many events of that type it holds
`events_of_type(event_type)`	Those events, in order

Each event carries id, ts, event_type, and payload.

The legacy evaluator

The earlier Evaluator SDK — an HTTP service Failproof AI called at EVALUATOR_ENDPOINT, answering /evaluate and polled through JobPending — is retired. Build new evaluators on this worker; operators of a self-hosted instance running a legacy service can keep it through the transition.

Connect your agents

Jev reference

Reference

HTTP API

Write evaluations

Run the worker

Result types

The session

The legacy evaluator

​Write evaluations

​Run the worker

​Result types

​The session

​The legacy evaluator

Write evaluations

Run the worker

Result types

The session

The legacy evaluator