- a score from 0 to 1, optionally marked passed or failed
- a metric, such as a count, a duration, or a cost, with its unit
- an assertion, which passed or did not
Two kinds of evaluator
Hosted Python is deliberately small: one expression, no imports, no network. Anything that needs a model — an LLM judge scoring whether an answer was relevant, say — runs in your own worker instead. Neither kind needs an inbound connection: workers claim finished sessions and submit results over outbound HTTPS.
Each organization evaluates its own agents
Evaluations belong to the organization that defines them. Each organization on an instance writes its own — its own checks, conditions, thresholds, and labels — versions and deploys them without affecting any other, and sees only its own results. Filter those results by agent, environment, evaluation, and time, or ask the assistant about them.From first draft to live scores
1
Write it
Describe what to measure and let the assistant draft it, or write it yourself. See Write an evaluation.
2
Test it
Run it against real sessions before it goes live; nothing is stored. See Test an evaluation.
3
Deploy and version it
Deploy an immutable version, publish new ones as it evolves, and roll back to an earlier one. See Deploy and version.
4
Read the results
Chart scores over time, compare agents and environments, and ask the assistant. See Read evaluation results.