> ## Documentation Index
> Fetch the complete documentation index at: https://docs.befailproof.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes Deployment Guide

> AgentEye Kubernetes Deployment Guide documentation.

This guide deploys the full AgentEye stack onto a dedicated Kubernetes cluster:

* **ClickHouse 24.8** -- the canonical events and evaluations analytics store (StatefulSet with 100Gi persistent volume). Required: the server refuses to start without it.
* **PostgreSQL 16** -- relational/metadata store for organizations, API keys, users, dashboards, saved queries, and auth (StatefulSet with 50Gi persistent volume)
* **Redis 7.2** -- optional shared cache and rate-limit backend; the server and dashboard degrade gracefully if it is unavailable
* **AgentEye Server** -- Rust API for event ingestion, analytics, and key management (2 replicas)
* **AgentEye Dashboard** -- Next.js web UI (2 replicas)
* **AI assistant (agent service)** -- optional read-only in-dashboard assistant on port 9100; inert until an LLM endpoint is configured
* **Traefik (public)** -- ingress controller for collector traffic, mTLS-protected
* **Traefik (dashboard)** -- ingress controller for the dashboard, VPN/IP-allowlist-only
* **cert-manager** -- TLS certificates and mTLS CA
* **Backup CronJob** -- daily combined dump of PostgreSQL + ClickHouse at 03:00 UTC
* **Cert Renewal Monitor** -- alerts when client certificates are near expiry

**Estimated time:** 60--90 minutes for a first deployment.

For the managed deployment model where Exosphere handles all of this on your behalf, see [enterprise-docs/managed-deployment.md](/agenteye/managed-deployment).

***

## Prerequisites

Run each verification command before starting. Every check must pass.

| Requirement                      | Minimum                                       | Verification Command                                         | Expected                                                                  |
| -------------------------------- | --------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------- |
| Kubernetes cluster               | 1.27+                                         | `kubectl version`                                            | Server Version >= v1.27                                                   |
| Kustomize (bundled with kubectl) | Kustomize v1.14+ (ships inside kubectl 1.27+) | `kubectl kustomize --help`                                   | Prints usage text                                                         |
| Helm                             | v3                                            | `helm version`                                               | `Version:"v3.x.x"`                                                        |
| cluster-admin RBAC               | --                                            | `kubectl auth can-i create namespaces`                       | `yes`                                                                     |
| Default StorageClass             | --                                            | `kubectl get storageclass`                                   | At least one row marked `(default)`                                       |
| LoadBalancer support             | --                                            | Cloud-dependent (EKS, GKE, AKS all support this by default)  | --                                                                        |
| GitHub PAT                       | --                                            | `echo $AGENTEYE_TOKEN`                                       | Non-empty (see [enterprise-docs/github-token.md](/agenteye/github-token)) |
| openssl                          | --                                            | `openssl version`                                            | OpenSSL 1.x or 3.x                                                        |
| Cloud storage bucket             | --                                            | For PostgreSQL + ClickHouse backups (S3, GCS, or Azure Blob) | --                                                                        |

**Cluster sizing:** Minimum 3 nodes, 4 vCPU / 8 GB RAM each. See [enterprise-docs/managed-deployment.md](/agenteye/managed-deployment) for full requirements.

### Run all checks at once

```bash theme={null}
echo "--- Prerequisites Check ---"
kubectl version --client -o yaml 2>/dev/null | grep -q gitVersion && echo "PASS: kubectl" || echo "FAIL: kubectl not found"
helm version --short 2>/dev/null | grep -q v3 && echo "PASS: helm v3" || echo "FAIL: helm v3 not found"
kubectl auth can-i create namespaces 2>/dev/null | grep -q yes && echo "PASS: cluster-admin" || echo "FAIL: no cluster-admin"
kubectl get storageclass 2>/dev/null | grep -q default && echo "PASS: default StorageClass" || echo "FAIL: no default StorageClass"
[ -n "$AGENTEYE_TOKEN" ] && echo "PASS: AGENTEYE_TOKEN set" || echo "FAIL: AGENTEYE_TOKEN not set"
openssl version >/dev/null 2>&1 && echo "PASS: openssl" || echo "FAIL: openssl not found"
echo "---"
```

### Deployment shape

The **ingest endpoint** is served on a hostname you control (e.g. `ingest.your-company.example`). cert-manager requests a publicly-trusted TLS certificate from Let's Encrypt over HTTP-01, so collectors verify the server cert against the system trust store, with no per-customer CA pinning.

The **dashboard endpoint** works the same way: it is served on a second hostname you control (e.g. `agenteye.your-company.example`) pointing at the dashboard Traefik LoadBalancer, and cert-manager issues its Let's Encrypt certificate through that LoadBalancer. Browsers get a trusted certificate with no warning.

> **Certificate issuance and renewal validate over HTTP-01**, so both LoadBalancers must be reachable from the public internet on port 80. If you need to IP-restrict the dashboard LoadBalancer, coordinate a DNS-01 solver with support first — otherwise renewals fail silently and the certificate expires.

***

## Get the Manifests

```bash theme={null}
git clone https://x:${AGENTEYE_TOKEN}@github.com/agenteye-enterprise/releases.git
cd releases/deploy
```

**Test it:**

```bash theme={null}
ls base/kustomization.yaml
```

Expected: the file exists. If it doesn't, the clone failed -- check your `AGENTEYE_TOKEN`.

**Directory structure:**

```
deploy/
  base/           Shared Kustomize base (all K8s resources)
  overlays/       Cluster-specific overrides (image tags, hostnames, resources)
  third-party/    Helm values for Traefik, cert-manager, and (opt-in) Robusta health monitoring
```

The **base** contains every resource needed for a full deployment, including Let's Encrypt certificates for the two public hostnames you configure in Phase 3.1. An **overlay** patches the base for a specific environment (e.g. custom image tags, resource limits, env wiring). The **third-party** directory contains Helm values files for external infrastructure.

> **Health monitoring (optional):** the server's readiness probe already reflects Postgres + ClickHouse health, and `third-party/robusta/` adds opt-in Kubernetes-native pod-failure alerting to Slack. See [enterprise-docs/health-monitoring.md](/agenteye/health-monitoring).

***

## Phase 1 -- Third-Party Infrastructure (\~30 min)

### 1.1 Install cert-manager

cert-manager manages TLS certificates for HTTPS and the private CA used for mTLS client certificates.

```bash theme={null}
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.install=true
```

**Test it:**

```bash theme={null}
kubectl get pods -n cert-manager
```

Expected: 3 pods all `Running` -- `cert-manager`, `cert-manager-cainjector`, `cert-manager-webhook`.

```bash theme={null}
kubectl get crds | grep cert-manager
```

Expected: at least `certificates.cert-manager.io`, `clusterissuers.cert-manager.io`, `issuers.cert-manager.io`.

**If it fails:** Pods in `CrashLoopBackOff` usually means CRDs weren't installed. Re-run with `--set crds.install=true`. If webhook pods fail readiness, wait 30 seconds and check again -- they can take a moment to start.

***

### 1.2 Install Traefik -- Public Ingest Controller

This Traefik instance handles collector traffic on an **external** LoadBalancer. It terminates TLS and enforces mTLS (client certificate verification) on the ingest endpoint.

```bash theme={null}
helm repo add traefik https://traefik.github.io/charts
helm repo update

helm install traefik-public traefik/traefik \
  --namespace traefik-public --create-namespace \
  --version 39.0.8 \
  -f third-party/traefik/values-public.yaml
```

**Test it:**

```bash theme={null}
kubectl get pods -n traefik-public
```

Expected: 1 pod `Running`.

```bash theme={null}
kubectl get ingressclass traefik-public
```

Expected: the IngressClass exists (it is not the default class).

**If it fails:** Check `kubectl describe pod -n traefik-public <pod-name>` for image pull errors or resource constraints.

***

### 1.3 Install Traefik -- Dashboard Controller

This Traefik instance serves the dashboard on a dedicated LoadBalancer, restricted by IP allowlist.

> **Two allowlist mechanisms ship for this instance.** This guide uses `values-dashboard.yaml`, which restricts access with the portable `service.loadBalancerSourceRanges` field. A parallel `values-internal.yaml` is also provided for AWS environments that prefer the `service.beta.kubernetes.io/aws-load-balancer-source-ranges` annotation instead. Pick one and use it consistently; the steps below assume `values-dashboard.yaml`.

**Before installing**, edit `third-party/traefik/values-dashboard.yaml` to set the allowed source IPs. The `loadBalancerSourceRanges` field controls which IPs can reach the dashboard. By default it is set to `0.0.0.0/0` (all IPs); restrict it to your VPN, office, or known egress IPs.

#### Whitelist a single IP

```yaml theme={null}
service:
  loadBalancerSourceRanges:
    - "<YOUR_VPN_IP>/32"
```

#### Whitelist multiple IPs

Add one entry per IP or CIDR block. A `/32` suffix matches a single IPv4 address; a CIDR block (e.g. `/24`) matches a range. You can mix individual IPs and ranges freely:

```yaml theme={null}
service:
  loadBalancerSourceRanges:
    - "203.0.113.10/32"       # office gateway
    - "203.0.113.11/32"       # backup office gateway
    - "198.51.100.0/24"       # VPN pool
    - "192.0.2.50/32"         # on-call engineer home IP
```

Tips when maintaining the list:

* Keep one entry per line and add a short `#` comment identifying each IP's owner or purpose; this is what future operators use to decide whether an entry is still needed.
* Always use CIDR notation. A bare IP like `203.0.113.10` is rejected by the cloud provider; use `203.0.113.10/32`.
* For IPv6 ranges, use the equivalent `/128` (single address) or larger CIDR, e.g. `2001:db8::1/128`. Not all cloud providers support IPv6 source ranges; check your provider's LoadBalancer documentation.
* The list is an **OR**: traffic is allowed if the source matches any entry.

After editing the file, proceed to `helm install` below. If the controller is already installed, run `helm upgrade` with the same flags, or patch the Service at runtime (next section).

#### Update the whitelist at runtime

You can change the allowed IPs without a Helm upgrade by patching the Service directly. **The patch replaces the entire list**; always include every IP you want to keep, not just the new one.

To replace the list with a new set of IPs:

```bash theme={null}
kubectl patch svc traefik-dashboard -n traefik-dashboard \
  -p '{"spec":{"loadBalancerSourceRanges":["203.0.113.10/32","198.51.100.0/24","192.0.2.50/32"]}}'
```

To safely **append** an IP without losing existing entries, read the current list first, then patch with the combined set:

```bash theme={null}
# 1. Show the current allowlist
kubectl get svc traefik-dashboard -n traefik-dashboard \
  -o jsonpath='{.spec.loadBalancerSourceRanges}'

# 2. Patch with the full list including the new IP
kubectl patch svc traefik-dashboard -n traefik-dashboard \
  -p '{"spec":{"loadBalancerSourceRanges":["<EXISTING_IP_1>/32","<EXISTING_IP_2>/32","<NEW_IP>/32"]}}'
```

> Runtime patches are not persisted back to `values-dashboard.yaml`. To keep the change across future Helm upgrades, also update the values file and commit it.

Then install:

```bash theme={null}
helm install traefik-dashboard traefik/traefik \
  --namespace traefik-dashboard --create-namespace \
  --version 39.0.8 \
  -f third-party/traefik/values-dashboard.yaml
```

**Test it:**

```bash theme={null}
kubectl get pods -n traefik-dashboard
```

Expected: 1 pod `Running`.

```bash theme={null}
kubectl get ingressclass traefik-dashboard
```

Expected: the IngressClass exists.

***

### 1.4 Wait for LoadBalancers

Both Traefik instances need external IPs before proceeding.

```bash theme={null}
kubectl get svc -n traefik-public
kubectl get svc -n traefik-dashboard
```

**Test it:** Both services show an `EXTERNAL-IP` (not `<pending>`).

If still pending, watch for assignment:

```bash theme={null}
kubectl get svc -n traefik-public -w
```

Press `Ctrl+C` once the IP appears. IP assignment typically takes 2--5 minutes.

**If it fails:** `<pending>` after 10 minutes usually means the cloud provider can't provision a LoadBalancer. Check: subnet tags (EKS requires `kubernetes.io/role/elb`), VPC configuration, service quotas, and that the correct internal LB annotation is set for the internal instance.

***

## Phase 2 -- Create Secrets (\~10 min)

All secrets are created manually before deploying the application. This ensures sensitive values never appear in manifest files.

### 2.1 Create the namespace

```bash theme={null}
kubectl create namespace agenteye
```

**Test it:**

```bash theme={null}
kubectl get namespace agenteye
```

Expected: status `Active`.

***

### 2.2 Image pull secret

This secret authenticates with `ghcr.io` to pull the AgentEye container images. See [enterprise-docs/github-token.md](/agenteye/github-token) for how to generate your PAT.

```bash theme={null}
kubectl create secret docker-registry agenteye-image-pull \
  --namespace agenteye \
  --docker-server=ghcr.io \
  --docker-username=agenteye-enterprise \
  --docker-password="${AGENTEYE_TOKEN}"
```

**Test it:**

```bash theme={null}
kubectl get secret agenteye-image-pull -n agenteye -o jsonpath='{.type}'
```

Expected: `kubernetes.io/dockerconfigjson`.

**Test it (deep)** -- verify the token can actually pull images:

Use the `server` image tag pinned in your overlay's `kustomization.yaml` (currently `v0.0.1-beta.48` in both the bundled `acme` overlay and base deployment). Substitute the tag below for the one you are deploying so this check does not drift across releases:

```bash theme={null}
kubectl run test-pull \
  --image=ghcr.io/agenteye-enterprise/server:v0.0.1-beta.48 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"agenteye-image-pull"}]}}' \
  --restart=Never -n agenteye --command -- echo ok

# Wait a few seconds for the pull, then:
kubectl logs test-pull -n agenteye
kubectl delete pod test-pull -n agenteye
```

Expected: `ok` printed in the logs.

**If it fails:** `ErrImagePull` or `401 Unauthorized` means the PAT is invalid or lacks `read:packages` scope. Re-check [enterprise-docs/github-token.md](/agenteye/github-token).

***

### 2.3 PostgreSQL credentials

```bash theme={null}
POSTGRES_PASSWORD=$(openssl rand -hex 24)

kubectl create secret generic agenteye-postgres \
  --namespace agenteye \
  --from-literal=POSTGRES_USER=postgres \
  --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \
  --from-literal=POSTGRES_DB=agenteye
```

> **Important:** We use `-hex` (not `-base64`) to generate the password. Base64 output can contain `+`, `/`, and `=` which break the `DATABASE_URL` connection string. See [enterprise-docs/troubleshooting.md](/agenteye/troubleshooting) for details.

> **Store `POSTGRES_PASSWORD` in your secrets manager immediately.** You will need it if you ever restore from a backup or connect to the database directly.

**Test it:**

```bash theme={null}
kubectl get secret agenteye-postgres -n agenteye
```

Expected: the secret exists.

```bash theme={null}
kubectl get secret agenteye-postgres -n agenteye \
  -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d | wc -c
```

Expected: `48` (24 hex bytes = 48 characters).

***

### 2.4 Admin API key

```bash theme={null}
ADMIN_KEY=$(openssl rand -hex 32)

kubectl create secret generic agenteye-admin-key \
  --namespace agenteye \
  --from-literal=ADMIN_KEY="${ADMIN_KEY}"
```

The admin key is the bootstrap credential. The server upserts it on every startup with all permissions. Use it to create scoped collector keys in Phase 7. See [enterprise-docs/api-keys.md](/agenteye/api-keys) for the full permissions model.

> **Store `ADMIN_KEY` in your secrets manager immediately.**

**Test it:**

```bash theme={null}
kubectl get secret agenteye-admin-key -n agenteye
```

Expected: the secret exists.

***

### 2.5 Auth configuration (dashboard login)

The dashboard uses email + OTP for user login. Without this secret the server still starts and the `ADMIN_KEY` API path keeps working, but **no user can log in via the UI**.

All keys are referenced as `optional: true` in the base manifest, so partial secrets (or no secret at all) are fine; the server falls back to the documented defaults. Bundling everything into one `agenteye-auth` secret keeps the auth surface rotatable in a single place.

```bash theme={null}
kubectl create secret generic agenteye-auth \
  --namespace agenteye \
  --from-literal=ADMIN_EMAIL="admin@yourcompany.com" \
  --from-literal=ALLOWED_EMAILS="*@yourcompany.com" \
  --from-literal=SMTP_HOST="smtp.yourprovider.com" \
  --from-literal=SMTP_PORT="587" \
  --from-literal=SMTP_USERNAME="your-smtp-user" \
  --from-literal=SMTP_PASSWORD="your-smtp-password" \
  --from-literal=SMTP_FROM="noreply@yourcompany.com" \
  --from-literal=SMTP_TLS="starttls" \
  --from-literal=DEFAULT_ORG_NAME="Acme Corp" \
  --from-literal=DEFAULT_ORG_SLUG="acme"
```

| Key                                                                     | Purpose                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ADMIN_EMAIL`                                                           | Bootstrap admin user. Upserted on every startup with all permissions and protected from deletion/permission edits via the dashboard. Without it, no admin is seeded and first login is impossible.                                                                                                                                                                                              |
| `ALLOWED_EMAILS`                                                        | Comma-separated allowlist. Supports exact addresses (`user@example.com`) and domain wildcards (`*@example.com`). Without it, **no user can log in or be created**.                                                                                                                                                                                                                              |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM` | SMTP relay for sending OTP codes. If `SMTP_HOST` is unset, OTP codes are logged to the server's stdout instead of emailed (useful for first-boot smoke tests). Provide all SMTP keys together for real email delivery.                                                                                                                                                                          |
| `SMTP_TLS`                                                              | One of `starttls` (default), `tls`, or `none`.                                                                                                                                                                                                                                                                                                                                                  |
| `DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG`                                  | Optional. Give the built-in `default` organization a friendly display name and URL slug so it lives at e.g. `/acme` instead of `/default`. Applied on **first boot only**; once you rename the org with `agenteye-orgctl org rename` (see §7.6) these are ignored. The slug must be 1--40 lowercase alphanumerics with single internal hyphens. Leave both unset to keep the generic `default`. |

> **Store the SMTP credentials in your secrets manager.**

**Test it:**

```bash theme={null}
kubectl get secret agenteye-auth -n agenteye \
  -o jsonpath='{.data}' | grep -o '"[A-Z_]*"' | sort -u
```

Expected: the keys you populated appear in the output.

***

### 2.6 Multi-tenant org isolation key (optional)

Skip this for a single-tenant deployment; the server runs on a built-in dev default and serves the one `default` org fine. **Before you create a second organization**, set a strong, stable `ORG_CH_SECRET`: each org's ClickHouse password is derived as `HMAC(ORG_CH_SECRET, org_id)`, so the publicly-known dev default would yield publicly-derivable per-org credentials. The `agenteye-orgctl org create` command (see [§7.6 Provision organizations](#76-provision-organizations-multi-tenant)) refuses to run while the server is still on the built-in dev default.

```bash theme={null}
kubectl create secret generic agenteye-org-ch-secret \
  --namespace agenteye \
  --from-literal=ORG_CH_SECRET="$(openssl rand -base64 48)"

# Roll the server so it picks up the new value.
kubectl -n agenteye rollout restart deployment/server
```

The server reads this via an **optional** `secretKeyRef`, so a single-tenant cluster that never creates it still boots normally. Keep the value **stable and identical across all replicas**; rotating it invalidates every org's derived ClickHouse password until the boot-time reconcile re-provisions the users (a rolling restart with the value consistent everywhere heals it). See `deploy/base/server/secret.example.yaml`.

> **Store `ORG_CH_SECRET` in your secrets manager and don't rotate it casually.**

***

### 2.7 Verify all secrets

```bash theme={null}
kubectl get secrets -n agenteye -o custom-columns='NAME:.metadata.name,TYPE:.type'
```

Expected output (among any default secrets):

```
NAME                    TYPE
agenteye-admin-key      Opaque
agenteye-auth           Opaque
agenteye-image-pull     kubernetes.io/dockerconfigjson
agenteye-postgres       Opaque
agenteye-org-ch-secret  Opaque   # only if you completed §2.6 (multi-tenant)
```

The four core secrets (`agenteye-admin-key`, `agenteye-auth`, `agenteye-image-pull`, `agenteye-postgres`) must be present before continuing. `agenteye-org-ch-secret` is only required for multi-tenant deployments (see §2.6).

***

## Phase 3 -- Deploy the Application (\~5 min)

### 3.1 Configure the public hostnames

cert-manager needs the ingest and dashboard hostnames before it can request their Let's Encrypt certificates. Copy the template and set both:

```bash theme={null}
cp base/certificates/domain.env.example base/certificates/domain.env
# Edit base/certificates/domain.env and set:
#   INGEST_DOMAIN=ingest.your-company.example      (resolves to the public Traefik LB)
#   DASHBOARD_DOMAIN=agenteye.your-company.example (resolves to the dashboard Traefik LB)
```

`domain.env` is gitignored; it stays local to each deployment. The kustomize build fails loudly if either key is missing.

> **DNS must resolve first.** You don't have to point DNS at the LBs yet (they don't exist until Phase 1.2 is complete), but ACME issuance in step 3.2 will retry until each hostname resolves to its LoadBalancer. You can either set DNS now (using the LB hostnames captured in Phase 1.4) or proceed and add the records in Phase 4.

***

### 3.2 Apply manifests

Apply the base directly for a fresh install, or an overlay if you've cut one for this environment (overlays just pin image tags, env vars, and resource limits; they inherit the base's certs and routing):

```bash theme={null}
kubectl apply -k base/
# or
kubectl apply -k overlays/<your-env>/
```

The overlay includes the base automatically; apply one, not both.

***

### 3.3 Wait for pods

```bash theme={null}
kubectl wait --for=condition=Ready pod -l 'app in (server,dashboard,postgres,clickhouse)' \
  -n agenteye --timeout=180s
```

The wait is scoped to the core data-plane pods. The optional `agent` (AI assistant) and `redis` pods come up alongside them; the assistant stays inert until you supply its LLM endpoint (see [enterprise-docs/assistant.md](/agenteye/assistant)), and Redis is a best-effort cache, so neither needs to be Ready for the platform to serve traffic.

**Test it:**

```bash theme={null}
kubectl get pods -n agenteye
```

Expected (the optional `agent` and `redis` pods also appear and reach `Running`):

```
NAME                         READY   STATUS    RESTARTS   AGE
agent-xxxxxxxxxx-xxxxx       1/1     Running   0          ...
clickhouse-0                 1/1     Running   0          ...
dashboard-xxxxxxxxxx-xxxxx   1/1     Running   0          ...
dashboard-xxxxxxxxxx-xxxxx   1/1     Running   0          ...
postgres-0                   1/1     Running   0          ...
redis-0                      1/1     Running   0          ...
server-xxxxxxxxxx-xxxxx      1/1     Running   0          ...
server-xxxxxxxxxx-xxxxx      1/1     Running   0          ...
```

**If it fails:**

| Pod Status         | Likely Cause                                   | Debug Command                                            |
| ------------------ | ---------------------------------------------- | -------------------------------------------------------- |
| `ImagePullBackOff` | Bad image pull secret or PAT                   | `kubectl describe pod <name> -n agenteye`                |
| `CrashLoopBackOff` | Bad environment variables (e.g. DATABASE\_URL) | `kubectl logs <name> -n agenteye`                        |
| `Pending`          | Insufficient CPU/memory or no nodes            | `kubectl describe pod <name> -n agenteye` (check Events) |

***

### 3.4 Verify storage

```bash theme={null}
kubectl get pvc -n agenteye
```

Expected, both with status `Bound`:

| PVC                            | Capacity | Backs                                           |
| ------------------------------ | -------- | ----------------------------------------------- |
| `postgres-data-postgres-0`     | `50Gi`   | PostgreSQL relational/metadata store            |
| `clickhouse-data-clickhouse-0` | `100Gi`  | ClickHouse events + evaluations analytics store |

A `redis-data-redis-0` PVC (1Gi) also appears for the optional cache.

**If it fails:** `Pending` means no StorageClass can provision the volume. Check `kubectl get storageclass` and ensure a default exists. For production, overlay the ClickHouse volume onto a fast SSD StorageClass (e.g. gp3 on AWS, pd-ssd on GCP); compaction throughput suffers on slow disks.

***

### 3.5 Verify certificates

```bash theme={null}
kubectl get certificates -n agenteye
```

Expected: 3 certificates, all `Ready: True`:

| Name            | Issuer             | Purpose                                                               |
| --------------- | ------------------ | --------------------------------------------------------------------- |
| `mtls-ca`       | `selfsigned`       | Private CA for issuing mTLS client certs (10-year validity)           |
| `ingest-tls`    | `letsencrypt-prod` | Public TLS certificate for the ingest endpoint (90-day, auto-renewed) |
| `dashboard-tls` | `letsencrypt-prod` | Public TLS certificate for the dashboard (90-day, auto-renewed)       |

**If `ingest-tls` or `dashboard-tls` is not Ready:**

`kubectl describe certificate <name> -n agenteye` and read the Events. The common causes:

* **DNS not yet pointing at the LB.** Let's Encrypt resolves the hostname and hits port 80 to validate — `INGEST_DOMAIN` must resolve to the public LB, `DASHBOARD_DOMAIN` to the dashboard LB. Until the CNAME/Alias propagates, the order stays `pending`. Once DNS is correct, cert-manager retries automatically (no need to delete the Certificate).
* **Hostname not substituted.** If `dnsNames` still reads `INGEST_DOMAIN_PLACEHOLDER` / `DASHBOARD_DOMAIN_PLACEHOLDER`, you skipped step 3.1 -- create `base/certificates/domain.env` and re-apply.
* **Dashboard Traefik can't serve the challenge** (`dashboard-tls` only). The dashboard Traefik instance must be installed with the bundled values file (Phase 1.2), which enables the scoped Ingress provider that serves cert-manager's HTTP-01 solver. An instance installed without it leaves the challenge unroutable and the order `pending` forever.

**If `mtls-ca` is not Ready:** cert-manager itself is unhealthy. Re-check the cert-manager pods from step 1.1.

***

### 3.6 Verify CronJobs

```bash theme={null}
kubectl get cronjobs -n agenteye
```

Expected:

| Name                 | Schedule       | Purpose                                          |
| -------------------- | -------------- | ------------------------------------------------ |
| `agenteye-backup`    | `0 3 * * *`    | Daily Postgres + ClickHouse backup at 03:00 UTC  |
| `cert-renewal-check` | `0 3,15 * * *` | Certificate expiry alerts at 03:00 and 15:00 UTC |

***

### 3.7 Verify server started correctly

```bash theme={null}
kubectl logs -n agenteye -l app=server --tail=20
```

**Test it:** Look for a startup line indicating the server is listening on port 8080. There should be no database connection errors (the server requires both PostgreSQL and ClickHouse to be reachable before it reports Ready).

**If it fails:** The most common cause is a `POSTGRES_PASSWORD` containing URL-unsafe characters that break the `DATABASE_URL`. See [enterprise-docs/troubleshooting.md](/agenteye/troubleshooting).

***

### 3.8 Verify dashboard connected to server

```bash theme={null}
kubectl logs -n agenteye -l app=dashboard --tail=20
```

**Test it:** Look for `Ready` in the output with no `ECONNREFUSED` or similar errors.

**If it fails:** Check that the `server` Service exists (`kubectl get svc server -n agenteye`) and that `AGENTEYE_SERVER_URL` is set to `http://server:8080` in the dashboard deployment.

***

## Phase 4 -- Network Access (\~5 min)

### 4.1 Retrieve LoadBalancer addresses

```bash theme={null}
PUBLIC_IP=$(kubectl get svc -n traefik-public \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

INTERNAL_IP=$(kubectl get svc -n traefik-dashboard \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
```

> On AWS EKS, LoadBalancers return a hostname instead of an IP. Replace `.ip` with `.hostname` in the commands above.

**Test it:**

```bash theme={null}
echo "Public  (ingest):    $PUBLIC_IP"
echo "Internal (dashboard): $INTERNAL_IP"
```

Both must be non-empty.

***

### 4.2 Point DNS at the LoadBalancers

Create DNS records so the hostnames from `base/certificates/domain.env` resolve to their LoadBalancers — `INGEST_DOMAIN` to the **public** Traefik LB, `DASHBOARD_DOMAIN` to the **dashboard** Traefik LB:

* **AWS Route 53:** `A` record with `Alias = Yes`, target = the LB hostname. Don't use plain A → IP; ELB IPs rotate.
* **Any other provider:** `CNAME` from the hostname to the LB hostname.

Verify:

```bash theme={null}
dig +short ingest.your-company.example
dig +short agenteye.your-company.example
```

Should return the same addresses as `$PUBLIC_IP` and `$INTERNAL_IP` respectively (or, on EKS, resolve to the same `*.elb.amazonaws.com` hostnames).

Once DNS resolves, cert-manager finishes the pending ACME orders from Phase 3.5 within a minute. Re-run `kubectl get certificates -n agenteye` until both `ingest-tls` and `dashboard-tls` show `Ready: True`.

***

### 4.3 Reach the ingest endpoint

The public ingest endpoint enforces mutual TLS, so every request (including `/health`) must present a client certificate. You issue your first client certificate in Phase 5; if you have one already, verify reachability now:

```bash theme={null}
curl -s --cert issued/<cluster-name>/client.crt \
        --key issued/<cluster-name>/client.key \
        https://ingest.your-company.example/health
```

Expected: `{"status":"ok"}`. No `-k` is needed -- the server certificate chains to a public CA for `INGEST_DOMAIN`, so it validates against the system trust store. Reach the ingest endpoint by its `INGEST_DOMAIN` hostname (which matches the issued certificate), not by the raw LoadBalancer IP/hostname.

The dashboard endpoint is served on `DASHBOARD_DOMAIN` with a publicly-trusted certificate and is not behind mTLS, so no `-k` and no client certificate are needed:

```bash theme={null}
curl -s https://agenteye.your-company.example/ -o /dev/null -w '%{http_code}\n'
```

Reach the dashboard by its hostname, not the raw LB address — the certificate is bound to `DASHBOARD_DOMAIN`, so the raw address shows a certificate-name mismatch.

**If it fails:** If `curl` hangs, check that the LB is reachable from your machine (VPN, security groups, firewall rules). A `certificate required` handshake error on the ingest hostname means no client certificate was presented; complete Phase 5 first. A TLS validation error on the ingest hostname means the server cert hasn't finished issuing; go back to Phase 3.5 and resolve the issue there.

***

## Phase 5 -- Issue mTLS Client Certificates (\~10 min per cluster)

Collectors authenticate with **two factors**: a client certificate (transport layer, proves the request comes from an authorized cluster) and an API key (application layer, proves the request is from a collector with `events:add` permission). A leaked key is useless without the cert; a stolen cert is useless without a valid key.

### 5.1 Issue a certificate

Each cluster running collectors needs its own client certificate. From the manifests directory:

```bash theme={null}
cd base/certificates/client-certs
./issue-client-cert.sh <cluster-name>
```

Replace `<cluster-name>` with a meaningful identifier (e.g. `us-east-1-prod`, `staging`).

**Test it:** The script prints `==> Done!` and lists the output files.

```bash theme={null}
kubectl get certificate mtls-client-<cluster-name> -n agenteye
```

Expected: `Ready: True`.

Output files in `issued/<cluster-name>/`:

| File                         | Purpose                                                    |
| ---------------------------- | ---------------------------------------------------------- |
| `client.crt`                 | Client certificate (90-day validity)                       |
| `client.key`                 | Client private key                                         |
| `ca.crt`                     | CA certificate for server verification                     |
| `collector-mtls-secret.yaml` | Ready-to-apply Kubernetes Secret for the collector cluster |

***

### 5.1b Alternative delivery: AWS Secrets Manager

If the consumer of the cert is a Kubernetes Pod that needs `client.crt` and `client.key` on disk -- the typical case when you run the agenteye-collector as a sidecar in your application pod -- push the cert bundle into AWS Secrets Manager. The application pod then mounts it via the [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/) with IRSA, and certificate rotation is fully hands-off.

```bash theme={null}
cd base/certificates/client-certs
export AWS_REGION=us-east-1     # region where your workload runs
./issue-client-cert.sh <cluster-name> --save-to aws-secrets-manager
```

On re-run (renewal), the script calls `PutSecretValue` on the same secret, so the ARN and name stay stable. The CSI Driver picks up the new version on its next rotation poll and rewrites the files inside the pod.

**Prerequisites:**

* `aws` CLI v2 authenticated to your AWS account.
* `jq` installed.
* `AWS_REGION` environment variable set.
* IAM permissions on your caller identity (scope `Resource` to `arn:aws:secretsmanager:<region>:<account>:secret:agenteye/mtls-client/*`):
  * `secretsmanager:CreateSecret`
  * `secretsmanager:DescribeSecret`
  * `secretsmanager:PutSecretValue`
  * `secretsmanager:TagResource`

**What the script does in this mode:**

| Step | Action                                                                                                                                                                                                                                      |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1    | Issues / re-extracts the cert via cert-manager (same as default mode).                                                                                                                                                                      |
| 2    | Calls `DescribeSecret` on `agenteye/mtls-client/<cluster-name>` to decide create-vs-update.                                                                                                                                                 |
| 3    | On first run: `CreateSecret` with a three-key JSON payload (`client.crt`, `client.key`, `ca.crt`), tagged `AgentEyeCluster=<cluster-name>`. On subsequent runs: `PutSecretValue` to publish a new version; tag refreshed via `TagResource`. |
| 4    | Deletes `issued/<cluster-name>/` only after a successful upload. On any failure, the directory is preserved so you can retry.                                                                                                               |

**If the secret is scheduled for deletion**, the script fails with a clear error telling you to run `aws secretsmanager restore-secret --secret-id agenteye/mtls-client/<cluster-name>` before retrying.

For full pod wiring (SecretProviderClass, IRSA setup, rotation behavior, troubleshooting) see [enterprise-docs/single-pod-deployment.md](/agenteye/single-pod-deployment).

***

### 5.2 Verify the certificate works

Test the issued certificate against the mTLS ingress:

```bash theme={null}
curl -sk --cert issued/<cluster-name>/client.crt \
     --key issued/<cluster-name>/client.key \
     https://${PUBLIC_IP}/health
```

Expected: `{"status":"ok"}`

**If it fails:**

| Error                  | Cause                              | Fix                                                                                                    |
| ---------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `certificate required` | Cert not being presented           | Check file paths in the `curl` command                                                                 |
| `bad certificate`      | CA mismatch                        | Verify `mtls-ca-issuer` issued the cert: `kubectl describe certificate mtls-client-<name> -n agenteye` |
| `connection refused`   | Wrong hostname or LB not reachable | Check `/etc/hosts` or DNS                                                                              |

***

### 5.3 Deliver to the collector cluster

Send `collector-mtls-secret.yaml` to the team operating the collector cluster. They apply it:

```bash theme={null}
kubectl apply -f collector-mtls-secret.yaml -n <collector-namespace>
```

Then configure the collector to mount the secret and use the cert paths:

```json theme={null}
{
  "tls_cert": "/etc/agenteye/tls/client.crt",
  "tls_key": "/etc/agenteye/tls/client.key"
}
```

See [enterprise-docs/collector-installation.md](/agenteye/collector-installation) for complete collector setup including Kubernetes volume mounts.

**Test it (in the collector cluster):**

```bash theme={null}
kubectl get secret agenteye-collector-mtls -n <collector-namespace>
```

Expected: the secret exists with 3 data keys (`client.crt`, `client.key`, `ca.crt`).

***

### 5.4 Certificate lifecycle

| Property             | Value                                          |
| -------------------- | ---------------------------------------------- |
| Client cert validity | 90 days                                        |
| Auto-renewal         | cert-manager renews 15 days before expiry      |
| CA validity          | 10 years                                       |
| Expiry alerts        | CronJob alerts 30 days before expiry (Phase 6) |

cert-manager auto-renews the certificate on the **AgentEye cluster**, but the renewed cert must be re-delivered to the collector cluster. Re-run `issue-client-cert.sh` and re-apply `collector-mtls-secret.yaml` before the old cert expires.

If you are using `--save-to aws-secrets-manager` (see § 5.1b), re-run the same command. The script calls `PutSecretValue` on the same secret; pods mounting the secret via the Secrets Store CSI Driver pick up the new version on their next rotation poll (default: every hour), with no pod restart required.

***

### 5.5 Revoke a certificate

To immediately block a cluster's collector access:

```bash theme={null}
kubectl delete certificate mtls-client-<cluster-name> -n agenteye
```

**Test it:** The `curl` command from step 5.2 now fails with a TLS handshake error.

***

## Phase 6 -- Certificate Renewal Monitoring (\~2 min)

A built-in CronJob runs every 12 hours (03:00 and 15:00 UTC) and checks all client certificates labeled `agenteye.io/cert-type=mtls-client`. It alerts when any certificate is within 30 days of expiry.

### 6.1 Enable Slack notifications (optional)

```bash theme={null}
kubectl create secret generic cert-renewal-notify-config \
  --namespace agenteye \
  --from-literal=SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
```

Without this secret, the CronJob still runs and logs certificate status to stdout.

**Test it:**

```bash theme={null}
kubectl get secret cert-renewal-notify-config -n agenteye
```

Expected: the secret exists.

***

### 6.2 Test the CronJob

```bash theme={null}
kubectl create job --from=cronjob/cert-renewal-check test-cert-check -n agenteye

kubectl wait --for=condition=Complete job/test-cert-check -n agenteye --timeout=60s

kubectl logs -n agenteye -l job-name=test-cert-check
```

Expected: a list of certificates with their expiry status. If the Slack webhook is configured, check the Slack channel for the alert message.

**If it fails:** Check RBAC -- the CronJob's ServiceAccount needs `get, list` permissions on cert-manager Certificate resources. Verify with: `kubectl describe role cert-renewal-check -n agenteye`.

Clean up the test job:

```bash theme={null}
kubectl delete job test-cert-check -n agenteye
```

***

## Phase 7 -- Verify End-to-End

This phase confirms the entire pipeline works: health check, key creation, event ingestion, and dashboard display.

> **Note:** The examples below reach the ingest endpoint by its raw LoadBalancer address (`${PUBLIC_IP}`) for convenience, which is why they pass `-k`; the server certificate is bound to `INGEST_DOMAIN`, not to the LB IP, so the hostname check is skipped. The ingest endpoint enforces mutual TLS on **every** path, so every call must also present a client certificate (`--cert`/`--key`). To validate the public certificate as well, target `https://ingest.your-company.example/...` instead of `${PUBLIC_IP}` and drop the `-k`.

### 7.1 Health check

```bash theme={null}
curl -sk --cert issued/<cluster-name>/client.crt \
        --key issued/<cluster-name>/client.key \
        https://${PUBLIC_IP}/health
```

Expected: `{"status":"ok"}` with HTTP 200.

***

### 7.2 Create scoped collector keys

The admin key is for bootstrap and management. Create dedicated `events:add` keys for collectors:

```bash theme={null}
COLLECTOR_KEY=$(openssl rand -hex 32)

curl -sk -X POST https://${PUBLIC_IP}/keys \
  --cert issued/<cluster-name>/client.crt \
  --key issued/<cluster-name>/client.key \
  -H "Authorization: Bearer ${ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-collector",
    "key": "'"${COLLECTOR_KEY}"'",
    "permissions": ["events:add"]
  }'
```

**Test it:** Response includes `"id"`, `"name": "prod-collector"`, `"permissions": ["events:add"]`, `"created_at"`.

**Test it:** Verify the key appears in the key list:

```bash theme={null}
curl -sk https://${PUBLIC_IP}/keys \
  --cert issued/<cluster-name>/client.crt \
  --key issued/<cluster-name>/client.key \
  -H "Authorization: Bearer ${ADMIN_KEY}"
```

Expected: `prod-collector` appears in the response.

See [enterprise-docs/api-keys.md](/agenteye/api-keys) for the full key management reference.

***

### 7.3 Ingest a test event

```bash theme={null}
echo '{"session_id":"test","agent_id":"smoke-test","type":"test","timestamp":"2026-04-20T00:00:00Z"}' \
  | curl -sk --cert issued/<cluster-name>/client.crt \
         --key issued/<cluster-name>/client.key \
         -H "Authorization: Bearer ${COLLECTOR_KEY}" \
         -H "Content-Type: application/x-ndjson" \
         --data-binary @- \
         https://${PUBLIC_IP}/events
```

Expected: `{"accepted":1,"skipped":0}` with HTTP 200.

**If it fails:**

| HTTP Status         | Cause                                                   |
| ------------------- | ------------------------------------------------------- |
| 401                 | Invalid or missing API key                              |
| 403                 | Key lacks `events:add` permission                       |
| TLS handshake error | Client certificate issue -- see Phase 5 troubleshooting |

***

### 7.4 Verify event appears in dashboard

Open `https://agenteye.your-company.example` (your `DASHBOARD_DOMAIN`) in a browser. The certificate is publicly trusted, so there is no warning.

> If the dashboard LoadBalancer is restricted by IP allowlist and you can't connect, verify your IP is allowed:
>
> ```bash theme={null}
> kubectl get svc traefik-dashboard -n traefik-dashboard -o jsonpath='{.spec.loadBalancerSourceRanges}'
> ```
>
> Keep in mind that Let's Encrypt renews the dashboard certificate over HTTP-01 on port 80, and source ranges apply to the whole LoadBalancer — before restricting it to corporate ranges, coordinate a DNS-01 solver with support or renewals fail silently.

**Test it:** The smoke-test event should appear in the event list with session `test` and agent `smoke-test`.

**If it fails:** Check dashboard logs (`kubectl logs -n agenteye -l app=dashboard --tail=50`). Verify `AGENTEYE_SERVER_URL` and `AGENTEYE_API_KEY` are set correctly.

***

### 7.5 Test the backup CronJob

```bash theme={null}
kubectl create job --from=cronjob/agenteye-backup test-backup -n agenteye

kubectl wait --for=condition=Complete job/test-backup -n agenteye --timeout=300s

kubectl logs -n agenteye -l job-name=test-backup
```

Expected: `Backup created: agenteye-YYYYMMDD-HHMMSS.tar.gz (NNN)` in the logs; the archive bundles the Postgres dump and the ClickHouse tables.

> The S3 upload step ships wired into the CronJob and runs whenever `BACKUP_BUCKET` is set (the base ships a default bucket value). It is skipped only when `BACKUP_BUCKET` is empty or literally `PLACEHOLDER`. Point it at your own bucket and grant the `agenteye-backup` ServiceAccount write access before relying on it (see Backups section below).

Clean up:

```bash theme={null}
kubectl delete job test-backup -n agenteye
```

***

### 7.6 Provision organizations (multi-tenant)

Skip this for a single-tenant deployment; all data lives in the built-in `default` org and nothing here is required.

If you are running multiple isolated tenants, organizations and their memberships are created with the **`agenteye-orgctl`** CLI. It ships **inside the server image** (alongside `agenteye-server`) and you run it **inside the existing `server` Deployment with `kubectl exec`; there is no separate pod, Job, or Deployment, and no HTTP API or dashboard button for tenant lifecycle.** Running it in the server pod means it reuses the pod's `DATABASE_URL`, `CLICKHOUSE_URL`, and the `ORG_CH_SECRET` from §2.6.

> **Prerequisite:** complete §2.6 first. `org create` refuses to run while the server is still on the built-in dev `ORG_CH_SECRET`, and the per-org ClickHouse user it provisions depends on that secret being strong and stable.

**Create an org and add its first admin:**

```bash theme={null}
kubectl -n agenteye exec deploy/server -- \
  agenteye-orgctl org create --slug acme --name "Acme Corp"

kubectl -n agenteye exec deploy/server -- \
  agenteye-orgctl member add --org acme --email alice@acme.example --set admin
```

The new member receives an OTP on first dashboard login and then works entirely in the UI under the org's URL prefix (e.g. `/acme/...`).

**Other commands** (run the same `kubectl -n agenteye exec deploy/server -- agenteye-orgctl …` way):

| Command                                                                             | What it does                                                                  |
| ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `org list`                                                                          | List organizations and their state.                                           |
| `org rename --slug <slug> --name <new name>`                                        | Rename an org (slug unchanged).                                               |
| `org delete --slug <slug>`                                                          | Soft-delete + drop the org's ClickHouse user; **data retained**.              |
| `org purge --slug <slug>`                                                           | Irreversible data wipe; org must be `delete`d first; never the `default` org. |
| `member list --org <slug>`                                                          | List members and their permissions.                                           |
| `member update --org <slug> --email <email> [--set ...] [--add ...] [--remove ...]` | Change a member's permissions.                                                |
| `member remove --org <slug> --email <email>`                                        | Remove a member from the org.                                                 |

Builtin permission sets are `admin`, `standard`, and `read-only`. **Per-org API keys are still minted in the dashboard/API by org members (§7.2 shows the keys API); only the org + member lifecycle is operator-only.** Full reference and a worked example: [enterprise-docs/tenant-management.md](/agenteye/tenant-management).

***

## Post-Deployment Checklist

Use this checklist to confirm everything is working. Every item should be checked before handing off to collectors.

* [ ] All pods `Running` in `agenteye` namespace
* [ ] PostgreSQL PVC bound (50Gi) and ClickHouse PVC bound (100Gi)
* [ ] All 3 certificates `Ready: True`
* [ ] Both LoadBalancer IPs assigned
* [ ] DNS or `/etc/hosts` configured and resolving
* [ ] `/health` returns HTTP 200
* [ ] mTLS cert test passes (`curl` with client cert to `/health`)
* [ ] Scoped collector key created and tested
* [ ] Test event ingested (`accepted: 1`)
* [ ] Event visible in dashboard
* [ ] Client certificates issued for each collector cluster
* [ ] Backup CronJob tested manually
* [ ] Cert renewal CronJob tested manually
* [ ] Slack webhook for cert alerts configured (optional)
* [ ] Backup bucket configured in overlay (see below)
* [ ] Admin key and Postgres password stored in secrets manager

***

## Backups

A single `agenteye-backup` CronJob runs daily at 03:00 UTC. It dumps **both** stores: PostgreSQL (relational state) and ClickHouse (the `events` + `evaluations` analytics tables), into one compressed archive in the pod, then uploads it to the object storage you wire in your overlay.

Each run produces one object, `agenteye-<timestamp>.tar.gz`, which unpacks to:

```
postgres.sql        # pg_dump of the relational database
events.sql          # ClickHouse events table DDL
events.native       # ClickHouse events data (Native format)
evaluations.sql     # ClickHouse evaluations table DDL
evaluations.native  # ClickHouse evaluations data
```

ClickHouse is read over its HTTP API (the same endpoint the server uses), so the job needs no ClickHouse client. Only the two physical tables are dumped; the server recreates all views (`agent_sessions`, the `analytics.*` aliases) and row policies on startup, so those tables are the complete picture.

### Configure cloud upload

The backup CronJob ships with the S3 upload step (`aws s3 cp`) already wired in, and the base sets a default `BACKUP_BUCKET` you must override with your own bucket.

**On AWS:** you only set `BACKUP_BUCKET` to your bucket and grant the `agenteye-backup` ServiceAccount write access via IRSA — no script change needed.

**On GCP / Azure:** you must *replace* the shipped `aws s3 cp` line in the CronJob script with the matching command below — do not just add yours, because the leftover `aws s3 cp` runs under `set -eu` and fails the job.

| Cloud                 | Upload Command                                                                    |
| --------------------- | --------------------------------------------------------------------------------- |
| **AWS S3**            | `aws s3 cp /tmp/${FILENAME} s3://${BACKUP_BUCKET}/${FILENAME}` (ships by default) |
| **GCP Cloud Storage** | `gsutil cp /tmp/${FILENAME} gs://${BACKUP_BUCKET}/${FILENAME}`                    |
| **Azure Blob**        | `az storage blob upload -f /tmp/${FILENAME} -c backups -n ${FILENAME}`            |

One object per run means a bucket lifecycle rule (e.g. "delete after 30 days") prunes whole backups cleanly.

The `agenteye-backup` ServiceAccount needs IAM permissions to write to the bucket. On AWS use IRSA; on GKE Workload Identity; on AKS Pod Identity. Annotate the ServiceAccount in your overlay accordingly.

The dump is buffered in a 20 GiB `emptyDir` at `/tmp`; the ClickHouse `events` dump dominates its size. If your data is larger, raise the volume's `sizeLimit` in your overlay.

### Manual backup

```bash theme={null}
kubectl create job --from=cronjob/agenteye-backup manual-backup -n agenteye
```

### Restore from backup

1. Download the archive from your bucket and unpack it:
   ```bash theme={null}
   tar xzf agenteye-<timestamp>.tar.gz
   # -> postgres.sql, events.sql, events.native, evaluations.sql, evaluations.native
   ```

2. Scale the server down to stop writes during the restore:
   ```bash theme={null}
   kubectl scale deployment/server -n agenteye --replicas=0
   ```

3. Restore PostgreSQL:
   ```bash theme={null}
   kubectl cp postgres.sql agenteye/postgres-0:/tmp/postgres.sql
   kubectl exec -n agenteye postgres-0 -- bash -c 'psql -U postgres -d agenteye < /tmp/postgres.sql'
   ```

4. Restore ClickHouse. The schema already exists (the server creates it on startup; on a bare cluster, apply the `.sql` DDL files first). Re-inserting is safe; `ReplacingMergeTree` collapses duplicates on merge:
   ```bash theme={null}
   kubectl cp events.native      agenteye/clickhouse-0:/tmp/events.native
   kubectl cp evaluations.native agenteye/clickhouse-0:/tmp/evaluations.native
   kubectl exec -n agenteye clickhouse-0 -- sh -c \
     'clickhouse-client --query "INSERT INTO agenteye.events FORMAT Native"      < /tmp/events.native'
   kubectl exec -n agenteye clickhouse-0 -- sh -c \
     'clickhouse-client --query "INSERT INTO agenteye.evaluations FORMAT Native" < /tmp/evaluations.native'
   ```

5. Scale the server back up:
   ```bash theme={null}
   kubectl scale deployment/server -n agenteye --replicas=2
   ```

6. **Test it:** hit `/health` and confirm the dashboard shows historical events and evaluations.

***

## Upgrading

1. **Pull the latest manifests:**
   ```bash theme={null}
   git pull
   ```

2. **Check release notes** for breaking changes or new environment variables.

   Upgrading from a pre-`v0.0.1-beta.6` deploy: dashboard login now uses email + OTP and reads its config from a new `agenteye-auth` secret. Create it per §2.5 before applying, or no user will be able to log in via the UI. API-key traffic is unaffected either way.

3. **Update image tags** in your overlay's `kustomization.yaml` if applicable:
   ```yaml theme={null}
   images:
     - name: ghcr.io/agenteye-enterprise/server
       newTag: <new-version>
     - name: ghcr.io/agenteye-enterprise/dashboard
       newTag: <new-version>
   ```

4. **Take a pre-upgrade backup:**
   ```bash theme={null}
   kubectl create job --from=cronjob/agenteye-backup pre-upgrade-backup -n agenteye
   kubectl wait --for=condition=Complete job/pre-upgrade-backup -n agenteye --timeout=120s
   ```

5. **Apply:**
   ```bash theme={null}
   kubectl apply -k overlays/<your-overlay>/
   ```

6. **Test it -- monitor rollout:**
   ```bash theme={null}
   kubectl rollout status deployment/server -n agenteye
   kubectl rollout status deployment/dashboard -n agenteye
   ```

7. **Test it -- health check:**
   ```bash theme={null}
   curl -sk https://${PUBLIC_IP}/health
   ```
   Expected: `{"status":"ok"}`

8. **Test it -- dashboard:** Open in browser, verify it loads and shows existing events.

### Rollback

If something is wrong after upgrading:

```bash theme={null}
kubectl rollout undo deployment/server -n agenteye
kubectl rollout undo deployment/dashboard -n agenteye
```

**Test it:** `/health` returns 200, dashboard loads and shows events.

***

## Creating Overlays for New Environments

To deploy to a new cluster, create a new overlay by copying the existing `acme` overlay as a template:

```bash theme={null}
cp -r overlays/acme overlays/<new-env>
```

Files to customize in `overlays/<new-env>/`:

| File                            | What to Change                   |
| ------------------------------- | -------------------------------- |
| `kustomization.yaml`            | Image tags (`newTag` values)     |
| `patches/server-env.yaml`       | `DATABASE_URL` sslmode if needed |
| `patches/resource-limits.yaml`  | CPU and memory requests/limits   |
| `patches/cert-renewal-env.yaml` | Slack webhook URL                |

**Test it -- dry run:** Render the manifests without applying to verify correctness:

```bash theme={null}
kubectl kustomize overlays/<new-env>/
```

Inspect the output for correct hostnames, image tags, and resource values. Then apply:

```bash theme={null}
kubectl apply -k overlays/<new-env>/
```

***

## Architecture Reference

### Kubernetes resources created

| Kind                | Name                 | Namespace | Purpose                                                                      |
| ------------------- | -------------------- | --------- | ---------------------------------------------------------------------------- |
| Namespace           | `agenteye`           | --        | All AgentEye resources                                                       |
| StatefulSet         | `clickhouse`         | agenteye  | ClickHouse 24.8 with 100Gi PVC -- canonical events + evaluations store       |
| StatefulSet         | `postgres`           | agenteye  | PostgreSQL 16 with 50Gi PVC -- relational/metadata store                     |
| StatefulSet         | `redis`              | agenteye  | Redis 7.2 with 1Gi PVC -- optional shared cache                              |
| Deployment          | `server`             | agenteye  | Rust/Axum API (2 replicas, pod anti-affinity)                                |
| Deployment          | `dashboard`          | agenteye  | Next.js UI (2 replicas)                                                      |
| Deployment          | `agent`              | agenteye  | Optional AI assistant (1 replica); inert until an LLM endpoint is configured |
| Service             | `clickhouse`         | agenteye  | ClusterIP, port 8123 (HTTP)                                                  |
| Service             | `postgres`           | agenteye  | ClusterIP, port 5432                                                         |
| Service             | `redis`              | agenteye  | ClusterIP, port 6379                                                         |
| Service             | `server`             | agenteye  | ClusterIP, port 8080                                                         |
| Service             | `dashboard`          | agenteye  | ClusterIP, port 3000                                                         |
| Service             | `agent`              | agenteye  | ClusterIP, port 9100 (internal-only)                                         |
| PodDisruptionBudget | `clickhouse`         | agenteye  | Keeps at least one ClickHouse pod available during disruptions               |
| NetworkPolicy       | `agent`              | agenteye  | Restricts ingress to the AI assistant service                                |
| Certificate         | `mtls-ca`            | agenteye  | Self-signed CA (10-year, ECDSA P-256)                                        |
| Issuer              | `mtls-ca-issuer`     | agenteye  | Issues client certs from the CA                                              |
| Certificate         | `ingest-tls`         | agenteye  | Server TLS for ingest endpoint (90-day, Let's Encrypt)                       |
| Certificate         | `dashboard-tls`      | agenteye  | Server TLS for dashboard (90-day, Let's Encrypt)                             |
| IngressRoute        | `ingest`             | agenteye  | Routes `/events`, `/health`, `/keys` via traefik-public with mTLS            |
| IngressRoute        | `dashboard`          | agenteye  | Routes `/` via traefik-dashboard (no mTLS)                                   |
| TLSOption           | `mtls-ingest`        | agenteye  | Requires and verifies client certificates                                    |
| CronJob             | `agenteye-backup`    | agenteye  | Daily Postgres + ClickHouse backup at 03:00 UTC                              |
| CronJob             | `cert-renewal-check` | agenteye  | Certificate expiry alerts, twice daily                                       |
| ServiceAccount      | `agenteye-backup`    | agenteye  | IAM-annotatable SA for backup bucket access                                  |
| ServiceAccount      | `cert-renewal-check` | agenteye  | SA for cert-manager Certificate read access                                  |
| Role                | `cert-renewal-check` | agenteye  | `get, list` on cert-manager Certificates                                     |
| RoleBinding         | `cert-renewal-check` | agenteye  | Binds role to ServiceAccount                                                 |

### PostgreSQL sizing and tuning

The base manifests ship a single-replica PostgreSQL 16 StatefulSet with a tuned `postgresql.conf` mounted from a ConfigMap, so the dashboard's hot read paths get a real buffer cache and `work_mem` budget out of the box rather than the image defaults.

**Default resources (acme overlay):**

| Resource | Request                               | Limit |
| -------- | ------------------------------------- | ----- |
| CPU      | 1                                     | 4     |
| Memory   | 4 Gi                                  | 8 Gi  |
| Storage  | 50 Gi PVC (RWO, default StorageClass) | --    |

**Default `postgresql.conf` highlights** (full file in `postgres/configmap.yaml`):

| Setting                      | Value                | Rationale                                                    |
| ---------------------------- | -------------------- | ------------------------------------------------------------ |
| `shared_buffers`             | `2GB`                | \~25% of the 8 Gi limit                                      |
| `effective_cache_size`       | `6GB`                | \~75% of the limit                                           |
| `work_mem`                   | `32MB`               | Headroom for dashboard sorts (events / evaluations history)  |
| `maintenance_work_mem`       | `512MB`              | Faster `CREATE INDEX`, `VACUUM`                              |
| `max_connections`            | `100`                | Server pool (10 × 2 replicas) + readonly pool (4) + headroom |
| `random_page_cost`           | `1.1`                | SSD-backed PVC                                               |
| `jit`                        | `off`                | OLTP-shaped workload                                         |
| `shared_preload_libraries`   | `pg_stat_statements` | Per-query statistics                                         |
| `log_min_duration_statement` | `250ms`              | Surface slow queries in pod logs                             |

**Override for a different pod size:** patch the `postgres-config` ConfigMap in your overlay. Keep `shared_buffers` around 25% of the memory limit and `effective_cache_size` around 75%. After editing, restart the pod (`kubectl -n agenteye rollout restart statefulset/postgres`) for the new `postgresql.conf` to take effect.

**Inspect slow queries** once the pod is running. `pg_stat_statements` aggregates per-query timing, so this is the fastest way to find what is loading the database. Its views live in the `extensions` schema (not `public`) so they stay hidden from the read-only role that backs the dashboard's ad-hoc query workbench:

```bash theme={null}
kubectl exec -n agenteye postgres-0 -- psql -U postgres -d agenteye -c \
  "SELECT round(total_exec_time::numeric, 0) AS total_ms, calls,
          round(mean_exec_time::numeric, 1) AS mean_ms, query
   FROM extensions.pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;"
```

### Environment variables

**Server:**

| Variable                                                                            | Source                                                                    | Value                                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL`                                                                      | Constructed from `agenteye-postgres` secret                               | Built at deploy time from secret values (see `server/deployment.yaml`). Points at the relational/metadata store.                                                                                                                                                                                                                                                                                |
| `CLICKHOUSE_URL`                                                                    | Deployment manifest                                                       | `http://clickhouse:8123` -- **required**; the canonical events + evaluations store. The server refuses to start without it.                                                                                                                                                                                                                                                                     |
| `CLICKHOUSE_DATABASE`                                                               | Deployment manifest                                                       | `agenteye`                                                                                                                                                                                                                                                                                                                                                                                      |
| `REDIS_URL`                                                                         | Deployment manifest                                                       | `redis://redis:6379/0` -- optional cache and rate-limit backend. If Redis is unreachable, auth and OTP fall back to PostgreSQL (graceful degradation).                                                                                                                                                                                                                                          |
| `ADMIN_KEY`                                                                         | `agenteye-admin-key` secret                                               | Bootstrap admin credential                                                                                                                                                                                                                                                                                                                                                                      |
| `AGENT_API_KEY`                                                                     | `agenteye-agent` secret (optional)                                        | Seeds the protected `dashboard-assistant` key the AI assistant's agent service uses (the same value the agent reads as `AGENTEYE_API_KEY`). Unset means the key isn't seeded and the assistant stays off. See [enterprise-docs/assistant.md](/agenteye/assistant).                                                                                                                              |
| `AGENTEYE_AGENT_URL`                                                                | Deployment manifest                                                       | `http://agent:9100` -- where the **server** reaches the agent service to run the audit's agentic investigation (the same connection the dashboard uses). Unset ⇒ audits run policy-only. See [enterprise-docs/assistant.md](/agenteye/assistant).                                                                                                                                               |
| `AGENTEYE_AGENT_TOKEN`                                                              | `agenteye-agent` secret (optional)                                        | Shared secret the server presents to the agent for audit investigation calls; must match the agent's `AGENTEYE_AGENT_TOKEN`. Unset ⇒ audits run policy-only.                                                                                                                                                                                                                                    |
| `ADMIN_EMAIL`                                                                       | `agenteye-auth` secret (optional)                                         | Bootstrap admin user email, upserted as protected on every startup                                                                                                                                                                                                                                                                                                                              |
| `ALLOWED_EMAILS`                                                                    | `agenteye-auth` secret (optional)                                         | Comma-separated allowlist for user creation/login; supports `*@domain` wildcards                                                                                                                                                                                                                                                                                                                |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM`, `SMTP_TLS` | `agenteye-auth` secret (optional)                                         | SMTP relay for OTP delivery; if `SMTP_HOST` is unset, OTPs are logged to stdout                                                                                                                                                                                                                                                                                                                 |
| `DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG`                                              | `agenteye-auth` secret (optional)                                         | First-boot-only display name and URL slug for the built-in `default` org; ignored after an org rename. Slug must be 1--40 lowercase alphanumerics with single internal hyphens. See §2.5.                                                                                                                                                                                                       |
| `EVALUATOR_ENDPOINT`, `EVALUATOR_TOKEN`                                             | `agenteye-evaluator` secret (optional)                                    | Wire the evaluation pipeline. With the secret absent the pipeline is a no-op. See [enterprise-docs/evaluation-suite.md](/agenteye/evaluation-suite).                                                                                                                                                                                                                                            |
| `SESSION_TTL_SECS`                                                                  | Default                                                                   | `86400` (24 h)                                                                                                                                                                                                                                                                                                                                                                                  |
| `OTP_TTL_SECS`                                                                      | Default                                                                   | `600` (10 min)                                                                                                                                                                                                                                                                                                                                                                                  |
| `ORG_CH_SECRET`                                                                     | `agenteye-org-ch-secret` secret (optional; **required before a 2nd org**) | HMAC key from which each organization's per-tenant ClickHouse password is derived. Single-tenant clusters run on the built-in dev default; **set a strong, stable value in your own secret before provisioning a second organization** (see §2.6), and keep it identical across all server replicas; rotating it orphans every org's ClickHouse user until the next startup re-provisions them. |
| `RUST_LOG`                                                                          | Deployment manifest                                                       | `info`                                                                                                                                                                                                                                                                                                                                                                                          |
| `LISTEN_ADDR`                                                                       | Default                                                                   | `0.0.0.0:8080`                                                                                                                                                                                                                                                                                                                                                                                  |
| `MAX_BODY_BYTES`                                                                    | Default                                                                   | `134217728` (128 MB)                                                                                                                                                                                                                                                                                                                                                                            |

> **Multi-tenant ClickHouse.** Per-org isolation requires SQL access management on ClickHouse; the bundled `deploy/base/clickhouse/` config enables it (access management + `users_without_row_policies_can_read_rows=false`) so the server can create one read-only ClickHouse user and row policy per organization. If you replace that config, carry those settings over.

**Dashboard:**

| Variable               | Source                             | Value                                                                                                                                                                                                        |
| ---------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `AGENTEYE_SERVER_URL`  | Deployment manifest                | `http://server:8080`                                                                                                                                                                                         |
| `AGENTEYE_API_KEY`     | `agenteye-admin-key` secret        | Same as `ADMIN_KEY`                                                                                                                                                                                          |
| `AGENTEYE_AGENT_URL`   | Deployment manifest                | `http://agent:9100` -- where the dashboard reaches the optional AI assistant                                                                                                                                 |
| `AGENTEYE_AGENT_TOKEN` | `agenteye-agent` secret (optional) | Shared secret the dashboard presents to the AI assistant; the assistant stays inert until its LLM endpoint is configured. See [enterprise-docs/assistant.md](/agenteye/assistant).                           |
| `REDIS_URL`            | Deployment manifest                | `redis://redis:6379/0` -- optional; caches `validateSession()` results across replicas and shares the Next.js fetch cache. Unset => every authed request falls through to the server (correct, just slower). |
| `HOSTNAME`             | Deployment manifest                | `0.0.0.0`                                                                                                                                                                                                    |

### Traffic flow

```
Collector Clusters                          AgentEye Cluster
┌─────────────────┐                    ┌──────────────────────────────────────┐
│ Collector Pod    │                    │                                      │
│ + client cert    │──POST /events────►│ Traefik Public (mTLS) ──► Server    │
│ + API key        │   (port 443)      │                              │       │
└─────────────────┘                    │                              ▼       │
                                       │                          PostgreSQL  │
Browser (VPN)                          │                              ▲       │
┌─────────────────┐                    │                              │       │
│ Developer        │──HTTPS──────────►│ Traefik Internal ────► Dashboard     │
└─────────────────┘   (port 443)      │                                      │
                                       └──────────────────────────────────────┘
```

***

## Troubleshooting

The most useful commands for deployment-specific issues:

```bash theme={null}
# Pod details and events
kubectl describe pod -n agenteye <pod-name>

# Application logs
kubectl logs -n agenteye -l app=server --tail=200
kubectl logs -n agenteye -l app=dashboard --tail=200

# Certificate status
kubectl describe certificate -n agenteye <cert-name>

# Recent cluster events
kubectl get events -n agenteye --sort-by=.lastTimestamp
```

For a comprehensive list of common issues and solutions, see [enterprise-docs/troubleshooting.md](/agenteye/troubleshooting).

***

## Related Documentation

| Guide                                                      | Description                                                                    |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [Managed Deployment](/agenteye/managed-deployment)         | Setup guide for managed deployment on your Kubernetes cluster                  |
| [Getting Started](/agenteye/getting-started)               | End-to-end walkthrough with Docker Compose                                     |
| [Deployment](/agenteye/deployment)                         | Docker-based deployment, env vars, configuration                               |
| [Tenant Management](/agenteye/tenant-management)           | Provision organizations & members with the operator-only `agenteye-orgctl` CLI |
| [GitHub Token Setup](/agenteye/github-token)               | Generate your GitHub PAT to access artifacts                                   |
| [Collector Installation](/agenteye/collector-installation) | All collector install methods                                                  |
| [Python SDK](/agenteye/python-sdk)                         | Full SDK API reference                                                         |
| [API Keys](/agenteye/api-keys)                             | Creating and managing API keys                                                 |
| [Troubleshooting](/agenteye/troubleshooting)               | Common issues and fixes                                                        |

***

## Support

Email `support@exosphere.host` for help with deployment.