> ## Documentation Index > Fetch the complete documentation index at: https://docs.befailproof.ai/llms.txt > Use this file to discover all available pages before exploring further. # Deployment > AgentEye Deployment documentation. This guide covers deploying the AgentEye server and dashboard in production. *** ## Architecture Overview ``` [ AI agent machines ] [ Your infrastructure ] Python SDK | writes JSONL +----------------------+ v +--->| PostgreSQL 15+ | agenteye-collector --HTTP--+ | | (relational store) | | | +----------------------+ v | +--------+ | +----------------------+ | Server |<------+--->| ClickHouse 24+ | +--------+ | | (events / analytics) | ^ | +----------------------+ API | | | | +----------------------+ +-----------+ +- - >| Redis 7+ (optional) | | Dashboard | +----------------------+ +-----------+ ``` * **Server**: Rust HTTP service; receives event batches, writes them to ClickHouse, and maintains relational state in PostgreSQL. * **Dashboard**: Next.js web app; reads and writes exclusively through the server API. * **agenteye-collector**: deployed on agent machines, not the server host. * **Postgres 15+**: REQUIRED. (Raised from 14 in the multi-tenant release; the org-membership schema uses a column-list `ON DELETE SET NULL` foreign key, which is Postgres 15+. Upgrade Postgres before deploying this version.) Stores OLTP state: `api_keys`, `users`, `sessions`, `evaluation_jobs` (queue), `dashboards`, `saved_queries`, `otp_codes`, plus the multi-tenant tables `orgs`, `org_memberships`, `org_settings`. * **ClickHouse 24+**: REQUIRED. The analytics store for every ingested event. Engine: `ReplacingMergeTree`, partitioned by month, ordered by `(session_id, ts, dedup_key)`. The server connects via `CLICKHOUSE_URL`; the bundled `deploy/base/clickhouse/` ships a perf-tuned single-node configuration. **Multi-tenant requirement:** the bundled config enables SQL access management + `users_without_row_policies_can_read_rows=false` so the server can create one read-only ClickHouse user + row policy per organization (the engine-enforced isolation boundary for the SQL editor and AI agent). If you supply your own ClickHouse config, carry these settings over (see `deploy/base/clickhouse/configmap.yaml`). * **Redis 7+**: *optional* shared cache + rate-limit backend. Server and dashboard both connect via `REDIS_URL`. If absent, both degrade gracefully to Postgres-only paths. See **Redis (optional cache)** below. *** ## Server ### Pull the image ```bash theme={null} echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin docker pull ghcr.io/agenteye-enterprise/server:beta-latest ``` > Current builds publish under `beta-latest`; `latest` is assigned only to stable releases. For production, pin a specific `:v` tag; see [Available Image Tags](#available-image-tags). ### Environment variables | Variable | Required | Default | Description | | --------------------------------------------- | ---------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | Yes | none | Postgres DSN. Standard libpq connection string format with scheme `postgres://`. Supports `?sslmode=require` and other libpq parameters. The password must not contain `/`, `+`, or `=`; use `openssl rand -hex` to generate URL-safe passwords. | | `ADMIN_KEY` | No | none | Bootstrap admin API key. Upserted with all permissions on every startup. Rotate by changing the value and restarting. | | `LISTEN_ADDR` | No | `0.0.0.0:8080` | TCP address to bind | | `MAX_BODY_BYTES` | No | `134217728` (128 MB) | Maximum request body size | | `ADMIN_EMAIL` | No | none | Bootstrap admin user email. Upserted with all permissions on every startup and marked protected: cannot be disabled or have permissions modified via the dashboard/API. To rotate the bootstrap admin, change `ADMIN_EMAIL` and restart; the new email is upserted as protected, and the previous one retains its protection until manually cleared in the database. | | `ALLOWED_EMAILS` | No | none (all blocked) | Comma-separated list of allowed emails for user creation and login. Supports exact addresses (`user@example.com`) and domain wildcards (`*@example.com`). If unset, no users can be created or log in. **First-boot seed only**: seeds the default org's allowlist on first boot; thereafter each org's [`//settings`](#operational-settings) page is the source of truth and changing this env var has no effect. | | `SMTP_HOST` | No | none | SMTP server hostname for sending OTP emails. If unset, OTP codes are logged to stdout instead. | | `SMTP_PORT` | No | `587` | SMTP server port | | `SMTP_USERNAME` | No | none | SMTP authentication username | | `SMTP_PASSWORD` | No | none | SMTP authentication password | | `SMTP_FROM` | No | none | Sender email address for OTP emails | | `SMTP_TLS` | No | STARTTLS | STARTTLS is used unless you explicitly turn it off: `false` or `0` sends plaintext (no TLS); any other value — including unset — enables STARTTLS. | | `DASHBOARD_URL` | No | built-in default | Dashboard origin used to build both the OTP-email magic link and the incident magic-links in alert notifications. If unset it falls back to a built-in default (and, for OTP only, to the dashboard-derived request origin first). Set this for split-domain setups so both email and Slack/incident links point at your dashboard. See **Email magic-link URL** below; most operators do not need to set this. | | `SESSION_TTL_SECS` | No | `86400` (24 h) | Dashboard session duration in seconds. **First-boot seed only**: edit per org via [`//settings`](#operational-settings) after the first deploy. | | `OTP_TTL_SECS` | No | `600` (10 min) | OTP code validity period in seconds. **First-boot seed only**: edit per org via [`//settings`](#operational-settings) after the first deploy. | | `REDIS_URL` | No | none | Optional shared cache + rate-limit backend, e.g. `redis://redis:6379/0`. When set, the server caches authenticated API-key lookups, the dashboard's `/models` aggregate, the sessions list, and the env-list facet; it also moves OTP-request rate limiting off Postgres COUNT and onto Redis INCR. If unset or unreachable, the server runs without the cache (the OTP limit falls back to Postgres, every other cache call falls through to the source of truth). See **Redis (optional cache)** below. | | `CLICKHOUSE_URL` | **Yes** | none | Base URL of the ClickHouse instance, e.g. `http://clickhouse:8123`. The server applies its events schema to this database on every startup and refuses to boot if it can't reach ClickHouse. See **ClickHouse (required analytics store)** below. | | `CLICKHOUSE_DATABASE` | No | `agenteye` | ClickHouse database (schema) name. The server creates it on startup if it doesn't exist. | | `ORG_CH_SECRET` | No (single-tenant) / **Yes (multi-org)** | dev default | HMAC key from which each organization's per-tenant ClickHouse password is derived. The SQL editor and AI agent's `run_query` execute as the org's own read-only ClickHouse user, whose row policy enforces tenant isolation in the engine. Single-tenant deployments boot fine on the built-in dev default; **before provisioning a second org you MUST set a strong, stable value**, because the `agenteye-orgctl org create` CLI refuses to run on the built-in dev default. Rotating it orphans every org's ClickHouse user until the next startup re-provisions them (the boot-time reconcile heals this automatically). Keep it secret and unchanged across replicas. Org provisioning itself is operator-only; see **Organizations (multi-tenancy)** below. | | `DEFAULT_ORG_NAME` | No | `Default` | Display name seeded for the built-in default org. **First-boot seed only**, and only while the org still carries its freshly-migrated generic identity, applied on startup, then ignored. Once you rename the org (`agenteye-orgctl org rename`) the rename is authoritative and this env var has no further effect. | | `DEFAULT_ORG_SLUG` | No | `default` | URL slug for the built-in default org, the dashboard path it lives at (`//…`). Same first-boot-only / pristine-only semantics as `DEFAULT_ORG_NAME`. Must be 1-40 lowercase alphanumerics with single internal hyphens and not a [reserved word](#organizations-multi-tenancy); an invalid value is ignored (the org keeps `default`). Lets a single-tenant install present as e.g. `/acme` instead of `/default` without any post-deploy CLI step. | | `RUST_LOG` | No | `info` | Log verbosity (`debug`, `warn`, `error`, `agenteye_server=trace`) | | `EVALUATOR_ENDPOINT` | No | none | Base URL of your evaluator service (e.g. `http://evaluator:9000`). When unset the entire evaluation pipeline is a no-op; no queue rows are written, no workers run. See [Evaluation Suite](/agenteye/evaluation-suite). | | `EVALUATOR_TOKEN` | No | none | Sent as `Authorization: Bearer ` to the evaluator. **Must equal the same value the evaluator service is configured with.** Optional only if your evaluator is configured with no token. | | `EVALUATOR_WORKERS` | No | `2` | Concurrency: number of worker tasks per server instance that dispatch evaluations. Safe to run across multiple horizontally-scaled servers. | | `EVALUATOR_CLAIM_BATCH` | No | `4` | Maximum number of evaluations a single worker claims per tick. Batches are dispatched **concurrently**, so total concurrency on your evaluator endpoint is `EVALUATOR_WORKERS × EVALUATOR_CLAIM_BATCH`. | | `EVALUATOR_POLL_IDLE_SECS` | No | `2` | How long a worker sleeps between dispatch attempts when nothing is due. | | `EVALUATOR_POLLING_INTERVAL_SECS` | No | `10` | Final fallback cadence (seconds) for `GET /evaluate/{id}` polls when the evaluator does not return a per-response `next_poll_secs` and does not advertise a `default_poll_interval_secs` from `GET /config`. | | `EVALUATOR_REQUEST_TIMEOUT_MS` | No | `30000` | Per-HTTP-request timeout against the evaluator (milliseconds). | | `EVALUATOR_MAX_ATTEMPTS` | No | `5` | After this many failed attempts an evaluation is recorded as terminal `error` (or `timeout` if the failures were request timeouts). | | `EVALUATOR_CONFIG_REFRESH_SECS` | No | `300` (5 min) | How often the server re-fetches `GET /config` from the evaluator. | | `EVALUATOR_MAX_POLL_DURATION_SECS` | No | `3600` (1 h) | Maximum wallclock time a session may remain in the polling queue before AgentEye terminates it as `timeout`. Guards against an evaluator that returns `pending` forever. | | `ALERT_WORKERS` | No | `1` | Concurrency: number of worker tasks per server instance that evaluate alert rules. See [Alerts](/agenteye/alerts). | | `ALERT_CLAIM_BATCH` | No | `16` | Maximum number of alerts a single worker claims per tick. | | `ALERT_POLL_IDLE_SECS` | No | `5` | How long an alerts worker sleeps when the queue is empty. | | `ALERT_REQUEST_TIMEOUT_MS` | No | `15000` | Per-trigger evaluation timeout (ClickHouse queries + outbound channel HTTP). | | `ALERT_MAX_ATTEMPTS` | No | `5` | Consecutive transient failures before an alert reschedules at its normal cadence instead of exponential backoff. | | `AUDIT_WORKERS` | No | `1` | Concurrency: number of worker tasks per server instance that execute audits. See [Audits](/agenteye/audits). | | `AUDIT_CLAIM_BATCH` | No | `1` | Maximum number of due audits a single worker claims per tick. An agentic investigation is one long loop, so the default is 1. | | `AUDIT_POLL_IDLE_SECS` | No | `30` | How long an audits worker sleeps when no audit is due. | | `AUDIT_REQUEST_TIMEOUT_MS` | No | `30000` | Per-policy-query timeout against ClickHouse (milliseconds). | | `AUDIT_LLM_TIMEOUT_MS` | No | `1440000` | Timeout for the agentic investigation call to the AI assistant service. A full agent loop runs for minutes; keep this ABOVE the agent's own `AGENTEYE_AUDIT_TIMEOUT_MS` so the agent returns its partial findings before the server gives up. | | `AUDIT_MAX_ATTEMPTS` | No | `5` | Consecutive transient failures before an audit reschedules at its normal cadence instead of exponential backoff. | | `AGENTEYE_AGENT_URL` / `AGENTEYE_AGENT_TOKEN` | No | — | The audit's agentic investigation calls the AI-assistant `agent` service, **reusing the same connection as the assistant** — so set these two on the **server** as well (the bundled manifests/compose do). Both set ⇒ audits run the AI investigation; either unset ⇒ audits run **policy-only** (the deterministic SQL policy pass still runs), regardless of the per-audit `llm_enabled` flag. The agent must also have an LLM configured — see [assistant.md](/agenteye/assistant). | **AI assistant service — audit + sandbox settings.** The agentic investigation and its in-pod Python sandbox are tuned on the **agent service** (not the server), all on the `AGENTEYE_AUDIT_*` prefix and all optional: | Variable | Default | Meaning | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- | | `AGENTEYE_AUDIT_MAX_STEPS` | `200` | Max agent turns per investigation. | | `AGENTEYE_AUDIT_TIMEOUT_MS` | `1200000` | Wall-clock for one investigation (20 min). Must stay **below** the server's `AUDIT_LLM_TIMEOUT_MS`. | | `AGENTEYE_AUDIT_MAX_CONCURRENCY` | `1` | Concurrent investigations per agent pod (separate from the chat assistant's budget). | | `AGENTEYE_AUDIT_SANDBOX_TIMEOUT_MS` / `_MEM_MB` / `_CPU_SECS` / `_OUTPUT_MAX_BYTES` / `_SCRIPT_MAX_BYTES` | `20000` / `768` / `10` / `64000` / `64000` | Per-script limits for the bubblewrap sandbox. | **Sandbox platform requirement.** The audit code sandbox runs the model's Python inside a bubblewrap jail, which needs **unprivileged user namespaces**. The agent pod must allow the `clone()` flags — set `seccompProfile: Unconfined` (k8s) or `security_opt: [seccomp:unconfined]` (compose) on the agent. Where the node kernel disables unprivileged user namespaces (e.g. some GKE COS images), the sandbox **preflight fails and the auditor degrades to SQL-only automatically** — no error, just a `sandbox_available: false` on the agent's `/health`. ### Run Set `DATABASE_URL` in your environment, then pass it through to the container: ```bash theme={null} docker run -d --restart unless-stopped \ --name agenteye-server \ -e DATABASE_URL="$DATABASE_URL" \ -e ADMIN_KEY="$ADMIN_KEY" \ -p 8080:8080 \ ghcr.io/agenteye-enterprise/server:beta-latest ``` The server runs database migrations automatically on startup; no separate migration step needed. ### Health check ``` GET /health # liveness - always {"status":"ok"} once the process is up GET /ready # readiness - 200 when Postgres + ClickHouse are reachable, else 503 ``` No authentication required. Use `/health` for **liveness** probes and `/ready` for **readiness** / load-balancer probes. `/ready` checks the hard dependencies the server cannot serve without (Postgres + ClickHouse), so a server that is running but cannot reach its database is taken out of rotation and shows as `NotReady`; Redis is reported but never fails readiness. On the bundled Kubernetes manifests the readiness probe already points at `/ready` and liveness stays on `/health`. See [enterprise-docs/health-monitoring.md](/agenteye/health-monitoring) for the full picture, including opt-in Kubernetes-native pod-failure alerting to Slack. ### Email magic-link URL OTP login emails contain a one-tap **open the dashboard** button. Clicking it lands the user on `/login?token=&email=
`; the dashboard exchanges that pair for a session and redirects to the app, with no manual code re-entry. The server resolves the dashboard origin used to build the link in three tiers: 1. **`X-AgentEye-Dashboard-Url` header**: set automatically by the dashboard's `/api/auth/otp/request` proxy from its own public origin. In a same-origin deployment (server and dashboard share a host behind one ingress that forwards proxy headers), **no configuration is required**. 2. **`DASHBOARD_URL` env var**: set this if your dashboard is reachable on a different origin than the one the server's OTP request endpoint sees (split `api.example.com` / `app.example.com`), or if your ingress doesn't propagate the public host into the dashboard pod (so `request.nextUrl.origin` would otherwise resolve to a wildcard bind like `0.0.0.0:3000`). Example: `DASHBOARD_URL=https://app.example.com`. 3. **Default**: `https://app.befailproof.ai`, used only if neither of the above is present. The header value is validated: only `https://*` and loopback (`http://localhost*`, `http://127.0.0.1*`) origins are accepted, and wildcard bind addresses (`0.0.0.0`, `[::]`) are rejected even with the `https://` scheme. Anything else falls through to tier 2. Set it on a running cluster with a one-liner; no file, no kustomize rebuild: ```bash theme={null} kubectl set env deployment/server -n agenteye \ DASHBOARD_URL=https://app.example.com ``` This triggers a rollout; the new pods pick the value up on first request. Note that the override lives only on the Deployment; a subsequent `kustomize build | kubectl apply` against the overlay will wipe it unless you add the same env var to your overlay's `server-env.yaml` patch. *** ## Dashboard ### Pull the image ```bash theme={null} docker pull ghcr.io/agenteye-enterprise/dashboard:beta-latest ``` ### Environment variables | Variable | Required | Default | Description | | ----------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AGENTEYE_SERVER_URL` | Yes | none | Base URL of the server, e.g. `http://localhost:8080` | | `AGENTEYE_API_KEY` | Yes | none | API key the dashboard uses to authenticate to the server. Needs all permissions (admin key recommended). | | `AE_LOG_LEVEL` | No | `info` | Server-side log verbosity: `debug`, `info`, `warn`, `error`. Set to `debug` to see upstream-request/response lines and session-validation traces when diagnosing issues. | | `AE_LOG_JSON` | No | auto | `1` forces JSON-per-line output; `0` forces human-readable output. When unset, JSON is enabled automatically if `NODE_ENV=production`. JSON is recommended in production so logs parse cleanly with `jq` or a log aggregator. | | `AE_ANALYTICS_DISABLED` | No | none | Set to `1`/`true` to disable the dashboard's anonymous product-usage telemetry. See [Telemetry & privacy](#telemetry--privacy) below. | | `REDIS_URL` | No | none | Optional shared cache backend, e.g. `redis://redis:6379/0`. When set, the dashboard caches `validateSession()` results across replicas and shares the Next.js fetch cache for the latency-aggregate / env-list proxy routes. Edge-side OTP request and verify rate limits also use Redis when present (falling open if Redis is unreachable; the server-side limit is the security backstop). See **Redis (optional cache)** below. | | `AGENTEYE_AGENT_URL` | No | none | Base URL of the optional AI-assistant `agent` service, e.g. `http://agent:9100`. **Leave it unset to hide the assistant entirely**: no assistant bubble appears in the dashboard. See [enterprise-docs/assistant.md](/agenteye/assistant). | | `AGENTEYE_AGENT_TOKEN` | No | none | Shared secret the dashboard presents to the `agent` service. Must match the `AGENTEYE_AGENT_TOKEN` configured on the agent. See [enterprise-docs/assistant.md](/agenteye/assistant). | ### Run ```bash theme={null} docker run -d --restart unless-stopped \ --name agenteye-dashboard \ -e AGENTEYE_SERVER_URL="http://your-server-host:8080" \ -e AGENTEYE_API_KEY="$ADMIN_KEY" \ -p 3000:3000 \ ghcr.io/agenteye-enterprise/dashboard:beta-latest ``` ### Telemetry & privacy The dashboard sends **anonymous product-usage analytics** to Exosphere's analytics service (PostHog): which dashboard pages are viewed and a handful of UI actions such as creating an API key or re-evaluating a session. This usage signal informs which features are prioritised. * **No agent, session, or event data ever leaves your infrastructure.** Only dashboard UI usage is reported. Page URLs are stripped of identifiers before sending, and operators are identified only by an opaque internal id, never by email. * Telemetry is **enabled by default**. To turn it off completely, set `AE_ANALYTICS_DISABLED=1` on the dashboard container and restart. * Analytics are sent to the dashboard's own `/ingest` path, which the dashboard reverse-proxies to PostHog (`https://us.i.posthog.com`). Keeping requests first-party means browser ad-blockers don't drop them. The **dashboard container** needs outbound access to PostHog; if it's blocked, telemetry silently does nothing and the dashboard is unaffected. *** ## AI Assistant (optional) An in-dashboard AI assistant lets your team ask questions of their agent data in plain language (summarising sessions, drafting SQL for the `/queries` editor, and turning saved queries into dashboard tiles) without leaving the dashboard. It runs as a separate internal `agent` container (on the Claude Agent SDK) that only the dashboard can reach, and stays **disabled until you configure an LLM endpoint**. To enable it you set, on the `agent` service, an LLM connection (**Portkey** via `PORTKEY_API_KEY` + a model-catalog slug `AGENTEYE_AGENT_MODEL=@/`, direct Anthropic via `ANTHROPIC_API_KEY`, another gateway via `ANTHROPIC_BASE_URL`, or Bedrock/Vertex), a **dedicated** data key, and a shared `AGENTEYE_AGENT_TOKEN` matching the dashboard. Dashboard users additionally need the `agent:use` permission. For the assistant's data key you don't mint anything by hand: pick a random secret, set it as `AGENTEYE_API_KEY` on the `agent` **and** as `AGENT_API_KEY` on the `server`, and the server seeds it on startup with a fixed permission set. Its data access is read-only (`events:read`, `evaluations:read`, `dashboards:read`, `queries:read`), and it additionally holds approval-gated authoring scopes (`dashboards:write`, `queries:write`, `queries:run`) so it can draft and validate saved queries and build dashboard tiles on the user's behalf; all SQL still runs through the org's read-only ClickHouse role, so this widens what the assistant can author, not what data it can reach. The scopes are fixed in code and cannot be widened by configuration. That key is protected; it can't be disabled or regenerated via the API, only rotated by changing the value and restarting. Never reuse the admin/dashboard key for this. Full setup, the complete environment-variable reference, telemetry options, and the security model are in **[enterprise-docs/assistant.md](/agenteye/assistant)**. *** ## ClickHouse (required analytics store) ClickHouse keeps your dashboards responsive at high event volumes and lets the `/queries` SQL editor join across events, evaluations, and sessions in a single store. It is the required canonical store for every ingested event, every terminal evaluation outcome, and the derived per-session aggregates. PostgreSQL holds the relational / mutable-state tables (api\_keys, users, otp\_codes, evaluation\_jobs, dashboards, saved\_queries); the analytical surface lives in ClickHouse so the dashboard's rollups and your own SQL queries can scan and join it natively, without cross-database round-trips. The server refuses to boot without `CLICKHOUSE_URL`. ### Schema Three ClickHouse objects are created at server startup, all idempotent (`CREATE IF NOT EXISTS`): * **`agenteye.events`**: `ReplacingMergeTree(ingested_at)`, partitioned by `toYYYYMM(ts)`, ordered by `(session_id, ts, dedup_key)`. Duplicate inserts (collector retries) collapse to a single row at merge time; the server computes a deterministic SHA-256 `dedup_key` for every event so retries are safe. * **`agenteye.evaluations`**: `ReplacingMergeTree(ingested_at)`, partitioned by `toYYYYMM(finished_at)`, ordered by `(session_id, finished_at, dedup_key)`. Written once per terminal evaluation outcome by the evaluator pipeline. Same dedup-key model as `events`. * **`agenteye.agent_sessions`**: a **VIEW** over `agenteye.events`, not a physical table. Every column is derived (`started_at = min(ts)`, `last_event_at = max(ts)`, `ended_at = max(if event_type='agent_end', ts, NULL)`, `event_count = count()`, etc.). No per-event upsert and no separate backfill; the view auto-reflects whatever is in `events`. For backwards-compat with saved queries that reference `analytics.evaluations` / `analytics.sessions`, the server also creates an `analytics` ClickHouse database with views over the `agenteye.*` tables; `analytics.events`, `analytics.evaluations`, `analytics.agent_sessions`, `analytics.sessions` all resolve correctly. ### Configuration The bundled docker-compose and `deploy/base/clickhouse/` ship a ClickHouse service tuned for AgentEye's workload: * 2 GiB requested / 4 GiB limit memory in the shipped base overlay (sized to fit small POC/staging nodes); production customers should overlay up — the recommended floor is 2c / 4Gi request, 6c / 8Gi limit. `max_server_memory_usage_to_ram_ratio=0.9` * 5 GiB mark cache + 8 GiB uncompressed cache * `background_pool_size=16`, `background_merges_mutations_concurrency_ratio=2` * MergeTree: `parts_to_throw_insert=3000`, `parts_to_delay_insert=1500`, `non_replicated_deduplication_window=1000` * `local_io_method=auto` (io\_uring on supported kernels) * `fsync_metadata=0`: acceptable because of at-least-once ingest + ReplacingMergeTree dedup * `query_log` enabled with 30-day TTL; `query_thread_log` removed (expensive at high QPS) * `max_execution_time=30` for user-side queries * 100 GiB PVC at the StatefulSet template (customer overlays SHOULD override to a fast SSD storage class for production) ### Backups Your full dataset is captured nightly in a single restorable archive, so a cluster or storage loss is recoverable. ClickHouse is backed up automatically by the daily `agenteye-backup` CronJob, which dumps both PostgreSQL and ClickHouse in one pass. ClickHouse is read over its HTTP API: `agenteye.events` and `agenteye.evaluations` are dumped in ClickHouse-native format (the views and row policies are recreated by the server on startup, so the table data is the complete picture) and bundled with the Postgres dump into a single compressed archive uploaded to your object storage. The destination bucket and cloud credentials are configured per overlay. See the **Backups** section of [enterprise-docs/kubernetes-deployment.md](/agenteye/kubernetes-deployment) for upload configuration and restore steps. *** ## Redis (optional cache) Redis is an **optional** shared cache + rate-limit backend used by the server and dashboard. With Redis deployed and `REDIS_URL` set on both services: * **Server** caches authenticated API-key lookups, the `/events/environments` + `/evaluations/environments` lists, the `/events/latency_aggregate` rollup (the heaviest query the dashboard polls), the `/sessions` list, and switches OTP-request rate limiting from a Postgres `COUNT(*)` to a Redis `INCR + EXPIRE`. * **Dashboard** caches `validateSession()` results so the 10-20 authed API calls a typical page load issues all share one upstream session check. It also rate-limits OTP-request and OTP-verify at the dashboard edge. **Both services degrade gracefully if Redis is unreachable.** Every cache call returns `Err` within a bounded timeout and the caller falls back to the source of truth (Postgres on the server, the upstream Rust server on the dashboard). OTP rate limiting falls back to the Postgres `COUNT(*)` path on the server (the security property is preserved); the dashboard's edge OTP limit fails open while the server-side limit still holds. Redis being down degrades latency, not correctness. ### Configuration The docker-compose bundle already includes a Redis service and wires `REDIS_URL=redis://redis:6379/0` into the server and dashboard. To use an external Redis, set `REDIS_URL` to your endpoint and remove the `redis` service from the compose file. ### Memory + persistence The bundled Redis image runs with `--appendonly yes --appendfsync everysec --maxmemory 256mb --maxmemory-policy allkeys-lru`. AOF persistence means the cache survives container restarts; `everysec` is the right durability/perf balance because losing the last second of cache writes is harmless. LRU eviction caps memory growth. ### When NOT to deploy Redis * Single-instance dev/QA. The in-process caches on the server alone deliver most of the per-replica benefit; Redis adds the cross-replica sharing that single-instance setups don't need. * Air-gapped installs where the operational cost of running one more service outweighs the latency win. *** ## Docker Compose (recommended) A `docker-compose.yml` is available in the `agenteye-enterprise/releases` repo. It brings up Postgres, the server, and the dashboard with a single command. ```bash theme={null} GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download \ --repo agenteye-enterprise/releases \ --pattern 'docker-compose.yml' \ --dir ./agenteye cd agenteye ``` **Override defaults via `.env`:** ``` # Use URL-safe passwords (no /, +, or = characters). # Generate with: openssl rand -hex 24 POSTGRES_PASSWORD=your-db-password ADMIN_KEY=your-admin-secret # Dashboard authentication ADMIN_EMAIL=admin@yourcompany.com ALLOWED_EMAILS=*@yourcompany.com # SMTP for OTP emails (omit to log OTP codes to stdout) # SMTP_HOST=smtp.yourprovider.com # SMTP_PORT=587 # SMTP_USERNAME=your-smtp-user # SMTP_PASSWORD=your-smtp-password # SMTP_FROM=noreply@yourcompany.com RUST_LOG=info ``` ```bash theme={null} docker compose up -d ``` **Stop (keeps data volume):** ```bash theme={null} docker compose down ``` **Stop and wipe all data:** ```bash theme={null} docker compose down -v ``` *** ## Operational settings A small set of operational knobs that used to be pinned by env vars are now editable per organization from the dashboard's **`//settings`** page; each org configures its own. Changes take effect within seconds, with no restart and no redeploy. | Setting | Bootstrap env var | What it controls | | --------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Allowed sign-ins | `ALLOWED_EMAILS` | Emails (or `*@domain.com` wildcards) permitted to receive an OTP and be added as users | | Default user permissions | `DEFAULT_USER_PERMISSIONS` | Comma-separated permission tokens preselected when an admin opens **+ new user**. Each token must be one of the strings listed under [API key permissions](/agenteye/api-keys). Defaults to the `standard` preset: read-only access plus the everyday on-call actions (trigger re-evaluations, run queries, ack incidents, use the assistant). | | Session lifetime | `SESSION_TTL_SECS` | How long a dashboard login stays valid before re-auth. The dashboard re-checks the upstream session every 5 seconds, so a permission update on `//users` takes effect on the affected user's next request, with no relogin. | | One-time-code lifetime | `OTP_TTL_SECS` | How long an OTP / magic-link stays usable | | Alert notification channels | `ALERTS_ENABLED_CHANNELS` | Comma-separated list of channel kinds the alert dispatcher is allowed to use: `email`, `slack`, `webhook`. Per-alert configuration is still authored on `//alerts/`, but the dispatcher filters every outbound delivery through this set; a channel disabled here short-circuits with a `skipped_disabled` audit row. The `dashboard` channel (the local audit insert) is always allowed. Defaults to all three on. | ### How the bootstrap works Settings are stored per organization in `org_settings`. On first boot, the server seeds the default org's missing rows from the matching env var (or a sensible default if the env var is unset). After that, **the stored value is the source of truth and the env var is ignored**; changing the env var on a later restart will not affect a live org's value, and additional orgs start from defaults and configure their own. This means: * For a fresh deploy, set the env vars as shown above and the default org reads them on first boot. * To change a value later, log into the dashboard and edit it under `//settings`. The change applies within seconds across all server replicas; no restart needed. * A startup log line records what got seeded vs. what was already present, so you can confirm the bootstrap took effect: ```text theme={null} INFO settings bootstrap: seeded default-org row key=allowed_sign_ins env_var=ALLOWED_EMAILS seeded_from_env=true ``` #### Sign-in semantics across organizations A session and an OTP are global to the user, not to a single org, so two rules reconcile per-org settings at sign-in time: * **Session / OTP lifetime**: the strictest (shortest) lifetime among the orgs the user belongs to wins. * **Allowed sign-ins**: the gate ORs every org's allowlist together with org membership: a user may request an OTP if any org's allowlist admits their email **or** they are already a member of any org. ### Permissions Access to a `//settings` page is gated by two permissions: * `settings:read`: see the page and current values. * `settings:write`: save changes. The bootstrap admin user (seeded from `ADMIN_EMAIL`) gets both automatically along with every other permission. Grant them to other users from `//users` as needed. *** ## Organizations (multi-tenancy) A single deployment can serve multiple isolated **organizations** (tenants); every row of data belongs to exactly one org and isolation is enforced in the database engine. A single-tenant install needs nothing here; all data lives in a built-in `default` org. (You can give that org a friendlier name and URL slug, so it lives at e.g. `/acme` instead of `/default`, by setting `DEFAULT_ORG_NAME` / `DEFAULT_ORG_SLUG` before the first boot, or by renaming it anytime with `agenteye-orgctl org rename`.) **Tenant provisioning is operator-only.** Organizations and their memberships are created and managed with the **`agenteye-orgctl`** CLI, which ships **inside the server image** (alongside `agenteye-server`) and runs **inside the existing server pod**; there is **no separate pod/Job, no HTTP API, and no dashboard button**. It reuses the server's `DATABASE_URL`, `CLICKHOUSE_URL`, and `ORG_CH_SECRET`. ```bash theme={null} # Docker Compose - exec into the running server service: docker compose exec server agenteye-orgctl org create --slug acme --name "Acme Corp" docker compose exec server agenteye-orgctl member add --org acme --email alice@acme.example --set admin # Kubernetes - exec into the running server Deployment: kubectl -n agenteye exec deploy/server -- agenteye-orgctl org list ``` Available verbs: `org create | list | rename | delete | purge` and `member add | list | update | remove`, with builtin permission sets `admin`, `standard`, and `read-only`. Added members get an OTP on first dashboard login. **Before creating a second org:** set a strong, stable `ORG_CH_SECRET` (the `org create` command refuses to run on the built-in dev default) and ensure Postgres is **15+**. **Unchanged:** per-org API keys are still minted in the dashboard/API by org members; only the org + member lifecycle moved to the CLI. Full command reference and a worked example: **[enterprise-docs/tenant-management.md](/agenteye/tenant-management)**. *** ## Context-window fill Each `model_response` event shows a **context-fill pill** — input plus output tokens as a percentage of that model's context window. The bands are `healthy` (0–24%), `watch` (25–49%), `compacting` (50–74%), and `reset context` (75–100%). AgentEye resolves common model IDs automatically, so no initial configuration is required. Every model an organization sends appears under **Settings → model context windows**. Users with `settings:write` can override its window or add a private/proxy model (0–1,000,000 tokens); `0` means “unknown” and suppresses the pill. Changes apply to newly ingested events. Users with `settings:read` can view the list. New events get the fill from the moment you upgrade. To also populate **historical** events (and the per-model list) for an existing deployment, run the one-off backfill — it ships inside the server image (like `agenteye-orgctl`) and runs in the existing server pod: ```bash theme={null} # preview (prints the per-org mutation, changes nothing): kubectl -n agenteye exec deploy/server -- agenteye-backfill-context-window --dry-run # apply: kubectl -n agenteye exec deploy/server -- agenteye-backfill-context-window # docker compose: docker compose exec server agenteye-backfill-context-window ``` It is idempotent (safe to re-run) and re-uses `DATABASE_URL` / `CLICKHOUSE_URL` / `REDIS_URL` from the pod. Re-run it after editing model windows if you want existing events recomputed. *** ## Production Considerations * **Postgres**: Use a managed Postgres service or a dedicated instance with regular backups. The `DATABASE_URL` supports all standard libpq parameters, including `sslmode=require` for encrypted connections. * **TLS**: Put the server and dashboard behind a reverse proxy (nginx, Caddy, Traefik) that terminates TLS. * **Firewall**: The server port (default 8080) should only be reachable from collector machines and the dashboard host, not the public internet. * **Admin key**: Set `ADMIN_KEY` to a strong random secret. After bootstrapping, create dedicated scoped keys for collectors and the dashboard rather than using the admin key everywhere. * **Image tags**: Pin to the version in your release manifests (for example, `server:v0.0.1-beta.48`) in production rather than a floating tag to avoid unintended upgrades. Current beta builds publish under `beta-latest`; `latest` is assigned only to stable releases. * **Health monitoring**: On Kubernetes the readiness probe uses `/ready` (Postgres + ClickHouse reachability) while liveness stays on `/health`. For fleet-wide "is AgentEye itself up?" alerting to Slack, enable the opt-in Robusta add-on; see [enterprise-docs/health-monitoring.md](/agenteye/health-monitoring). *** ## Available Image Tags | Tag | Description | | ------------- | ------------------------------------------------------------------ | | `latest` | Latest stable release | | `beta-latest` | Latest pre-release (beta) | | `v` | Pinned version, e.g. `v0.0.1-beta.48` (recommended for production) |