Use Jev - Failproof AI

Jev helps at two points in an agent run: score a finished session against known answers, or review a tool call in the context of what you asked the agent to do.

Jev evals
Jev policies

Use a Jev eval when a finished session can be scored against a question with a few known answers, such as “Did the customer ask for a refund? Answer yes or no.” It helps you find patterns across sessions.

Create an eval

In the Cloud dashboard, open Analyze → eval authoring → new eval. Enter one fixed-answer question, select draft, and check that it chose a classifier score. Test it on real sessions, then deploy it.

The shared eval authoring form where you describe a question, review the draft, and deploy it. This screenshot shows a code draft; use a fixed-answer question for Jev.

Read the scores

After a new session completes, open Observe → Evaluations or use the Cloud CLI:

fp evals --since 7d
fp evals --aggregate --since 7d

The CLI reads scores; creating a Jev eval currently uses the dashboard. See Jev evaluations for question types and examples.

Use Jev policy review when a string-matching policy needs the context of your request to decide whether a tool call is safe. Start in observe mode so you can inspect Jev’s answers while your installed policies still decide each call.Jev’s checks come from a pack; Failproof AI ships none. Until you install them, Jev asks nothing, even when it is configured:

failproofai policies add FailproofAI/jev-policies

Set up Cloud Jev

In the Cloud dashboard, open Administration → Keys and create a key with the machine preset. Use it with failproofai config as shown in the quickstart. On a machine without an existing Jev configuration, this enables Cloud Jev in observe mode. Check the connection with:

failproofai jev status
failproofai jev test

Use your own endpoint

In the local dashboard, open Settings → Jev. Choose the provider, paste its token, select observe, and turn Jev on.

The local Jev settings panel with a provider, token field, and observe mode selected.

Or configure and test your endpoint from a terminal:

failproofai jev --url https://api.typesafe.ai/v1 --mode observe --key-stdin < ~/typesafe.key
failproofai jev test

Ask a hooked agent to use its file-reading tool on README.md. Confirm that tool call appears in the session, then inspect it under Policies → Activity in the local dashboard. Once the observe results look right, Jev policies explains when to enforce. For provider details and configuration, see the integration reference.

​Create an eval

​Read the scores

​Set up Cloud Jev

​Use your own endpoint

Create an eval

Read the scores

Set up Cloud Jev

Use your own endpoint