- a score from 0 to 1, optionally marked passed or failed
- a metric, such as a count, a duration, or a cost, with its unit
- an assertion, which passed or did not
Two kinds of evaluator
Hosted evaluations come in three shapes, and the assistant picks between them for you:
Code costs nothing to run. The other two cost a model call per session, so give them a condition that narrows them to the sessions the question is actually about.
Your own worker is still where an evaluation goes when it needs something we do not host: a package, a secret, your own network, or a model you run yourself. Neither kind needs an inbound connection: workers claim finished sessions and submit results over outbound HTTPS.
Each organization evaluates its own agents
Evaluations belong to the organization that defines them. Each organization on an instance writes its own — its own checks, conditions, thresholds, and labels — versions and deploys them without affecting any other, and sees only its own results. Filter those results by agent, environment, evaluation, and time, or ask the assistant about them.From first draft to live scores
1
Write it
Describe what to measure and let the assistant draft it, or write it yourself. See Write an evaluation.
2
Test it
Run it against real sessions before it goes live; nothing is stored. See Test an evaluation.
3
Deploy and version it
Deploy an immutable version, publish new ones as it evolves, and roll back to an earlier one. See Deploy and version.
4
Read the results
Chart scores over time, compare agents and environments, and ask the assistant. See Read evaluation results.