> ## Documentation Index
> Fetch the complete documentation index at: https://docs.befailproof.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Jev evaluator (bring your own key)
> Let TypeSafe's Jev classifier judge your agents' tool calls above a hard regex floor, through your own Jev endpoint and key.
Regex policies match strings. They cannot tell `rm -rf build/` that you asked for from `rm -rf ~` that slipped into a plan, so they block too much in one place and too little in another. **Jev**, TypeSafe's classifier, reads the call against what you actually asked for and answers a set of yes/no questions about it in one fast request.
With your own Jev endpoint and key configured, Failproof AI asks Jev about each tool call **alongside** the regex policies, never instead of them:
* A **hard** policy's deny is final. Jev cannot clear it. Every policy is hard unless it is explicitly marked reviewable and names the Jev checks that cover it, so a custom, pack or Cloud policy that says nothing is hard, and the always-on self-protection guard is always hard.
* A **reviewable** policy's deny may be cleared, but only when Jev was asked about the exact concern that policy covers and answered "nothing here" or "the user asked for this". A check that finds the concern real, when the user did not ask for the call, keeps the deny — even when its own verdict is only a warning, because before a tool call a warning does not stop the agent. And when that check is one that can deny (secret exposure, credential exfiltration, destructive deletion, …), nothing is cleared on that call.
* A block can still become a **warning** when the call is a step of the task you gave and reaches no further: Jev softens its own deny to a warning, and that warning — naming what is actually wrong with the call — replaces the policy's block.
* Jev can also warn or deny on its own, for harm no regex describes.
* If Jev cannot answer (timeout, rate limit, server error, no credits, an unexpected model version), that call gets the regex result, exactly as without Jev.
* Jev never makes a call more permissive than your policies alone unless it read the whole call and was asked about the exact concern. Anything less — a call too big to send whole, a suspected injection — withdraws the clearances and keeps every deny.
Without a Jev config nothing changes: hooks run the regex policies exactly as they always have. The config is the whole opt-in.
On FailproofAI Cloud? You do not need a key of your own: a machine connected with a key that carries `jev:evaluate` can use Jev on your organization's plan. See [Jev through FailproofAI Cloud](/policies/jev-cloud).
## Choose a provider
Jev is reachable through five routes. Bring a key for any one of them.
| Provider | `--provider` | Endpoint | Default model | Notes |
| --------------------- | ------------ | --------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| TypeSafe | `typesafe` | `api.typesafe.ai/v1/systemone` | `jev-1.13.0` | Exact version pin. |
| OpenRouter | `openrouter` | `openrouter.ai/api/v1/systemone` | `typesafe/jev-1.13` | Requests are routed to zero-data-retention endpoints only, with no fallback to another provider. Reports a dated version such as `typesafe/jev-1.13-20260917`. |
| Vercel AI Gateway | `vercel` | `ai-gateway.vercel.sh/typesafe/v1/systemone` | `typesafe-ai/jev` | Names Jev only by an alias, so the answering version is recorded as unverified. |
| Cloudflare Workers AI | `cloudflare` | `api.cloudflare.com/client/v4/accounts//ai/run` | `typesafe/jev` | Needs `--account-id`. About six calls a second per key were measured before HTTP 429. |
| Your own endpoint | `custom` | `/systemone` | `jev-1.13.0` | Any endpoint that accepts TypeSafe's request body and reports which model answered. `https` only; plain `http://localhost` is accepted in shadow mode only. |
With Vercel's own bring-your-own-key feature, a failed request is silently retried with Vercel's credentials. If you need every call billed to, and seen by, your own TypeSafe account only, use TypeSafe directly.
## Set it up
One command, the endpoint and the key:
```bash theme={null}
failproofai jev --url https://api.typesafe.ai/v1 --key-stdin < ~/typesafe.key
```
### The URL picks the provider
You do not have to name the provider: the URL's **host** is which one it is.
| URL host | Provider | Also needs |
| ---------------------- | ------------ | ---------------------------------- |
| `api.typesafe.ai` | `typesafe` | — |
| `openrouter.ai` | `openrouter` | — |
| `ai-gateway.vercel.sh` | `vercel` | — |
| `api.cloudflare.com` | `cloudflare` | `--account-id <32-hex-account-id>` |
| any other host | `custom` | — the URL you gave is the base URL |
Three things follow from that:
* **A URL that is the provider's own API writes no override.** `--url https://api.typesafe.ai/v1` produces exactly the config `--provider typesafe` would have. Give a different path or host on a known provider and it is stored as the base URL, as `--base-url` would store it.
* **`--provider` still overrides the inference**, which is how you reach a proxy that speaks a provider's API from a host of your own: `--url https://jev-proxy.internal/v1 --provider typesafe`.
* **A `--provider` that contradicts the host is refused**, not guessed at. `--provider openrouter --url https://api.typesafe.ai/v1` writes nothing and says why: the two spellings disagree about where your key is about to be sent. The same pair is refused from `jev setup --base-url` and from the dashboard's Jev settings. (`--provider custom` is not a contradiction — it means "treat this URL as itself" — except on Cloudflare's host, whose per-account endpoint a custom route cannot reach.)
`--url` is validated exactly as the `baseUrl` in the config file is, and refused in the same words: `https`, or plain `http://localhost` in shadow mode only.
### The key
Pipe it in with `--key-stdin`, or run the command in a terminal without it and paste the key at a masked prompt. Either way it goes straight into the config file and is never printed back.
```bash theme={null}
failproofai jev --url https://api.typesafe.ai/v1 --key-stdin < ~/typesafe.key
```
```bash theme={null}
failproofai jev --url https://openrouter.ai/api/v1 --key-stdin < ~/openrouter.key
```
```bash theme={null}
failproofai jev --url https://ai-gateway.vercel.sh/typesafe/v1 \
--key-stdin < ~/vercel-gateway.key
```
```bash theme={null}
failproofai jev --url https://api.cloudflare.com/client/v4 \
--account-id <32-hex-account-id> --key-stdin < ~/cloudflare.token
```
```bash theme={null}
failproofai jev --url https://jev.internal.example.com/v1 --key-stdin < ~/jev.key
```
`failproofai jev setup` takes the same flags and is the longhand for all of it: `setup --provider ` where you would rather name the provider than the URL.
### `--token`, and what it costs
`--token ` puts the key on the command line, which is the fastest way to configure a machine and the only spelling that leaves the key anywhere but the config file:
```bash theme={null}
failproofai jev --url https://openrouter.ai/api/v1 --token
```
A command-line argument is in your shell's history file afterwards, and while the command runs it is in the process list — readable from `/proc` by anything running as you. `setup` says so every time `--token` is used. Prefer `--key-stdin` on a machine you share, in a recorded session, or anywhere the history file is synced; rotate a key you have passed this way if it matters.
`--token`, `--key-stdin` and `--key-from-env` are mutually exclusive: give one.
Then send one small live request to check the key, the endpoint and which Jev answered:
```bash theme={null}
failproofai jev test
```
```text theme={null}
failproofai jev test ok · 523 ms
provider cloudflare
model asked typesafe/jev
answered by jev-1.13.0 (Jev 1.13 family — verified)
latency 523 ms — within the 3000 ms timeout
```
`jev test` exits 1, and says so in its title, when the answer arrives after the timeout (every hook would fall back to regex as `timeout`) or answers its check question wrongly.
Hooks read the config on every tool call, so it applies from the next one. There is nothing to restart, with or without the daemon.
## Check what it is doing
```bash theme={null}
failproofai jev status
failproofai jev status --json
```
`status` shows the provider, endpoint, model, mode, the config file and its permissions, and never the key. Below that it summarizes recent activity: how many calls Jev evaluated, how often it fell back to regex and why, its latency, and which reviewable policies it cleared.
## Shadow mode
`enforce` is the default. To watch Jev without letting it change any decision, switch to `shadow`: Jev is still asked and its verdicts are recorded, but the regex result is what is enforced.
```bash theme={null}
failproofai jev setup --mode shadow
failproofai jev setup --mode enforce
failproofai jev setup --mode off
```
`off` keeps the config — the endpoint and the key — and stops asking Jev: hooks run the regex policies exactly as without a config, and `failproofai jev status` says "off (switched off)". Switch back with `--mode shadow` or `--mode enforce`.
Re-running `setup` for the same provider keeps the stored key, so a mode switch is one flag. Switching provider starts over and asks for that provider's key. So does a `--base-url` that moves requests to a different host: a stored key is only sent to the host it was given for, or to its provider's own API.
## The config file
Everything lives in one file, `~/.failproofai/jev.json`, written by `setup`:
```json theme={null}
{
"provider": "cloudflare",
"apiKey": "",
"accountId": "<32-hex-account-id>",
"mode": "enforce",
"timeoutMs": 3000
}
```
| Field | Meaning |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `provider` | `typesafe`, `openrouter`, `vercel`, `cloudflare` or `custom` — or `failproofai`, whose key comes from the FailproofAI Cloud connection instead of this file (see [Jev through FailproofAI Cloud](/policies/jev-cloud)). |
| `apiKey` | Sent as `Authorization: Bearer `. |
| `baseUrl` | Required for `custom`; replaces the provider's API base otherwise. Must be `https`. Plain `http` to `localhost` is accepted only with `mode: shadow`: nothing authenticates a local port, so while your proxy is down any process on the machine, including the agent being judged, could answer in its place. |
| `accountId` | Cloudflare only: 32 lowercase hex characters. |
| `model` | Replaces the provider's default model id. A versioned id must name Jev 1.13. A value shaped like an API key is refused (and not repeated back), so a key pasted into `--model` is never stored or sent as the model. |
| `timeoutMs` | How long a tool call waits for Jev before using the regex result. 100–10000, default 3000. |
| `mode` | `enforce` (default), `shadow`, or `off` (keep the config, run no Jev). |
Three rules protect it:
* **Owner-only.** It is written with permissions `0600`. A copy that any other user or group can read or write is **refused**, and hooks fall back to regex until you run `chmod 600 ~/.failproofai/jev.json` or `setup` again. The directory is checked too: `~/.failproofai` must not be **writable** by anyone else, because whoever can write there can replace the file whatever its own permissions are. `setup` takes those write bits off if it finds them. `failproofai jev status` says when a config has been refused and shows the endpoint the file names: someone else could have changed it, so check it is yours before you `chmod`. Re-running `setup` on such a file carries its stored key only to the provider's own API; any other endpoint it names needs the key again (`--key-stdin`), or `--base-url default` to send requests back to the provider.
* **Global only.** A repository cannot turn Jev on, point it at another endpoint or pick its model: a `.failproofai/jev.json` inside a project is ignored, and the provider, URL, model and account id are read only from that file — never from the environment, which a repository's agent settings can set. (`FAILPROOFAI_HOME` is not a way around that: it moves the whole failproofai directory, your policies included, rather than redirecting Jev on its own.)
* **The key alone may come from the environment.** If the file has no `apiKey`, `FAILPROOFAI_JEV_API_KEY` supplies it for that session (`setup --key-from-env` writes such a file). It never replaces a key the file holds, and it cannot turn Jev on without the file. Where the variable is not set, Jev is simply off for that shell: `failproofai jev status` says so, exits 0 and leaves the config alone (`status --json` reports `"status": "key-missing"` with `"reason": "no-env-key"`). The `failproofaid` daemon does not see your shell's environment, so on a machine set up with `failproofai config`, keep the key in the file.
## Which Jev answers
Failproof AI's decision thresholds were calibrated on Jev 1.13, so an answer is used only when it comes from that family: `jev-1.13.x`, or OpenRouter's `typesafe/jev-1.13-`. Where a provider names Jev only by an alias and reports no version (Vercel, and Cloudflare when it does not say), the answer is used and recorded as unverified. A `custom` endpoint must report the model that answered; the one exception is an unversioned `--model` name you configured for it, which, echoed back, is recorded as unverified in the same way. An answer reporting any other version, or a `custom` answer reporting none, is not used: that call falls back to regex with the reason `model-mismatch`.
## When Jev cannot answer
Each of these falls back to the regex result for that call and is recorded with its reason, which `failproofai jev status` totals:
| Reason | Cause |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `timeout` | No answer within `timeoutMs`. |
| `http-429` | The provider rate-limited the key. |
| `rate-limited` | Failproof AI's own limiter held the call back before sending it: 5 requests a second, in bursts of up to 5, and none for a moment after the provider answers `429`. Not the provider. |
| `http-500`, `http-502`, `http-503`, … | A server error at the provider. The exact status is recorded. |
| `out-of-credits` | HTTP 402: the provider account has no credits left. |
| `provider-refused` | HTTP 402 from Cloudflare reading "Model execution failed (Payment error)": the provider declined to run the model on this request. Usually not billing, so topping up will not move it. |
| `http-401`, `http-403` | The key was refused. |
| `http-404` | Nothing is served at `/systemone`, so the base URL is wrong — `/systemone` is appended to it, and every provider serves it at its version root. `failproofai jev models` shows what the endpoint does serve. |
| `network` | The endpoint could not be reached. |
| `http-301`, `http-302`, `http-307`, `http-308` | The endpoint answered with a redirect. Redirects are never followed, so the answer only ever comes from the URL in your config; set `--base-url` to the final URL. |
| `malformed` | The endpoint answered, but not with a Jev answer — a body that is not JSON, or one with no answers in it. |
| `cloudflare-error`, `cloudflare-incomplete` | Cloudflare's envelope reported a failure, or a job that had not finished. |
| `model-mismatch` | A Jev version other than 1.13 answered, or a `custom` endpoint did not say which model answered. |
| `request-cut` | **Not an outage.** Jev answered; it was shown only part of the call, so its answer cleared nothing. See [When Jev answered, but not on the whole call](#when-jev-answered-but-not-on-the-whole-call). |
`failproofai jev status` can show a few rarer reasons too, such as `upstream-error` (the answer carried the provider's own error) or `config`, and totals any reason it cannot name as `other`.
`request-cut` is in this table because `failproofai jev status` totals it with the rest, and because it too leaves every deny standing. It is the one reason here that says nothing about your provider: the request arrived and Jev answered it. Unlike every row above it, that answer still counts — Jev's own deny or warning applies on top of the regex result rather than being discarded. So a run of them means calls are reaching the evaluator too big to send whole, not that your endpoint is unwell, and topping up credits or changing the URL will not move the number.
## When Jev answered, but not on the whole call
Two more things can happen, and neither is Jev failing to answer. Both are about how much of the call, or of the conversation, fitted into one request.
**Part of the call itself did not fit.** A tool call is sent inside a fixed budget, and an outsized one — a very large `Write`, a huge MCP body, a command padded out to the cap — is sent with what fitted. Jev still answers, and its answer still counts: its own deny or warning applies as usual. What it cannot do is **clear** anything, because a verdict given on part of a call is not a verdict on the call. So every policy deny stands, and the call is recorded as a fallback with the reason `request-cut`, which `failproofai jev status` totals alongside the reasons above. The rule this gives you: making a call bigger can cost it its clearances, and can never buy one.
**A message did not fit.** A long prompt you pasted, the agent's last message, or a prompt this evaluator's own store had already capped. **Nothing changes**: the call is judged, cleared and recorded exactly as any other, and it is not counted as a fallback. The length of what you type never decides a verdict, and a cut cannot manufacture consent: where a prompt arrived already capped, "you did not ask for this" stops being a conclusion that can be drawn from it at all, rather than becoming one.
The line between the two is who wrote the text. The call is the agent's, and a rule that let its length subtract severity would be a rule the agent can use; your prompt is yours, and treating its length as a signal only ever punished pasting a spec or a stack trace.
## What leaves the machine
For each tool call Jev evaluates, one request goes to your provider, carrying:
* the tool call itself, with secrets such as API keys, bearer tokens and `KEY=` assignments redacted;
* the recent prompts you typed, with text your agent's harness added removed;
* the agent's last message before your latest prompt, labelled as agent-written;
* facts computed locally, such as whether a path is inside the project — the one the session was in at its first reviewed call, [pinned for the session](/reference/jev-intent#the-project-root) — and the current git branch.
It goes only to the endpoint in your config, under your key.
## Turn it off
```bash theme={null}
failproofai jev remove
```
This deletes `~/.failproofai/jev.json`. From the next tool call, hooks run the regex policies exactly as before. The per-session stores under `~/.failproofai/state/semantic/` (recorded prompts in `sessions/`, project roots in `roots/`) are left in place and age out. To stop asking Jev but keep the config, use `failproofai jev setup --mode off` instead.
## Command reference
| Command | Outcome |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `failproofai jev --url --key-stdin` | Configure it in one command; the provider comes from the URL's host |
| `failproofai jev --url --token ` | Same, with the key on the command line — your history and the process list see it |
| `failproofai jev setup --provider --key-stdin` | Write the config from a key piped on stdin |
| `failproofai jev setup --provider ` | Same, asking for the key at a masked prompt |
| `failproofai jev setup --key-from-env` | Store no key; read `FAILPROOFAI_JEV_API_KEY` per session |
| `failproofai jev setup --mode shadow` | Switch mode (`enforce`, `shadow` or `off`), keeping the stored key |
| `failproofai jev setup --model ` / `--base-url ` | Override the model or API base; `default` clears the override |
| `failproofai jev setup --timeout-ms ` | Change the per-call budget |
| `failproofai jev status [--json]` | Configuration, permissions and recent activity; never the key |
| `failproofai jev test [--json]` | One live request: latency and the version that answered |
| `failproofai jev models [--provider ] [--url ] [--json]` | The model ids that endpoint's `/models` reports, marking the configured one |
| `failproofai jev remove` | Delete the config; Jev is off |