Kubernetes Deployment Guide

This guide deploys the full AgentEye stack onto a dedicated Kubernetes cluster:

ClickHouse 24.8 — the canonical events and evaluations analytics store (StatefulSet with 100Gi persistent volume). Required: the server refuses to start without it.
PostgreSQL 16 — relational/metadata store for organizations, API keys, users, dashboards, saved queries, and auth (StatefulSet with 50Gi persistent volume)
Redis 7.2 — optional shared cache and rate-limit backend; the server and dashboard degrade gracefully if it is unavailable
AgentEye Server — Rust API for event ingestion, analytics, and key management (2 replicas)
AgentEye Dashboard — Next.js web UI (2 replicas)
AI assistant (agent service) — optional read-only in-dashboard assistant on port 9100; inert until an LLM endpoint is configured
Traefik (public) — ingress controller for collector traffic, mTLS-protected
Traefik (dashboard) — ingress controller for the dashboard, VPN/IP-allowlist-only
cert-manager — TLS certificates and mTLS CA
Backup CronJob — daily combined dump of PostgreSQL + ClickHouse at 03:00 UTC
Cert Renewal Monitor — alerts when client certificates are near expiry

Estimated time: 60—90 minutes for a first deployment. For the managed deployment model where Exosphere handles all of this on your behalf, see enterprise-docs/managed-deployment.md.

Prerequisites

Run each verification command before starting. Every check must pass.

Requirement	Minimum	Verification Command	Expected
Kubernetes cluster	1.27+	`kubectl version`	Server Version >= v1.27
Kustomize (bundled with kubectl)	Kustomize v1.14+ (ships inside kubectl 1.27+)	`kubectl kustomize --help`	Prints usage text
Helm	v3	`helm version`	`Version:"v3.x.x"`
cluster-admin RBAC	—	`kubectl auth can-i create namespaces`	`yes`
Default StorageClass	—	`kubectl get storageclass`	At least one row marked `(default)`
LoadBalancer support	—	Cloud-dependent (EKS, GKE, AKS all support this by default)	—
GitHub PAT	—	`echo $AGENTEYE_TOKEN`	Non-empty (see enterprise-docs/github-token.md)
openssl	—	`openssl version`	OpenSSL 1.x or 3.x
Cloud storage bucket	—	For PostgreSQL + ClickHouse backups (S3, GCS, or Azure Blob)	—

Cluster sizing: Minimum 3 nodes, 4 vCPU / 8 GB RAM each. See enterprise-docs/managed-deployment.md for full requirements.

Run all checks at once

echo "--- Prerequisites Check ---"
kubectl version --client -o yaml 2>/dev/null | grep -q gitVersion && echo "PASS: kubectl" || echo "FAIL: kubectl not found"
helm version --short 2>/dev/null | grep -q v3 && echo "PASS: helm v3" || echo "FAIL: helm v3 not found"
kubectl auth can-i create namespaces 2>/dev/null | grep -q yes && echo "PASS: cluster-admin" || echo "FAIL: no cluster-admin"
kubectl get storageclass 2>/dev/null | grep -q default && echo "PASS: default StorageClass" || echo "FAIL: no default StorageClass"
[ -n "$AGENTEYE_TOKEN" ] && echo "PASS: AGENTEYE_TOKEN set" || echo "FAIL: AGENTEYE_TOKEN not set"
openssl version >/dev/null 2>&1 && echo "PASS: openssl" || echo "FAIL: openssl not found"
echo "---"

Deployment shape

The ingest endpoint is served on a hostname you control (e.g. ingest.your-company.example). cert-manager requests a publicly-trusted TLS certificate from Let’s Encrypt over HTTP-01, so collectors verify the server cert against the system trust store, with no per-customer CA pinning. The dashboard endpoint works the same way: it is served on a second hostname you control (e.g. agenteye.your-company.example) pointing at the dashboard Traefik LoadBalancer, and cert-manager issues its Let’s Encrypt certificate through that LoadBalancer. Browsers get a trusted certificate with no warning.

Certificate issuance and renewal validate over HTTP-01, so both LoadBalancers must be reachable from the public internet on port 80. If you need to IP-restrict the dashboard LoadBalancer, coordinate a DNS-01 solver with support first — otherwise renewals fail silently and the certificate expires.

Get the Manifests

git clone https://x:${AGENTEYE_TOKEN}@github.com/agenteye-enterprise/releases.git
cd releases/deploy

Test it:

ls base/kustomization.yaml

Expected: the file exists. If it doesn’t, the clone failed — check your AGENTEYE_TOKEN. Directory structure:

deploy/
  base/           Shared Kustomize base (all K8s resources)
  overlays/       Cluster-specific overrides (image tags, hostnames, resources)
  third-party/    Helm values for Traefik, cert-manager, and (opt-in) Robusta health monitoring

The base contains every resource needed for a full deployment, including Let’s Encrypt certificates for the two public hostnames you configure in Phase 3.1. An overlay patches the base for a specific environment (e.g. custom image tags, resource limits, env wiring). The third-party directory contains Helm values files for external infrastructure.

Health monitoring (optional): the server’s readiness probe already reflects Postgres + ClickHouse health, and third-party/robusta/ adds opt-in Kubernetes-native pod-failure alerting to Slack. See enterprise-docs/health-monitoring.md.

Phase 1 — Third-Party Infrastructure (~30 min)

1.1 Install cert-manager

cert-manager manages TLS certificates for HTTPS and the private CA used for mTLS client certificates.

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.install=true

Test it:

kubectl get pods -n cert-manager

Expected: 3 pods all Running — cert-manager, cert-manager-cainjector, cert-manager-webhook.

kubectl get crds | grep cert-manager

Expected: at least certificates.cert-manager.io, clusterissuers.cert-manager.io, issuers.cert-manager.io. If it fails: Pods in CrashLoopBackOff usually means CRDs weren’t installed. Re-run with --set crds.install=true. If webhook pods fail readiness, wait 30 seconds and check again — they can take a moment to start.

1.2 Install Traefik — Public Ingest Controller

This Traefik instance handles collector traffic on an external LoadBalancer. It terminates TLS and enforces mTLS (client certificate verification) on the ingest endpoint.

helm repo add traefik https://traefik.github.io/charts
helm repo update

helm install traefik-public traefik/traefik \
  --namespace traefik-public --create-namespace \
  --version 39.0.8 \
  -f third-party/traefik/values-public.yaml

Test it:

kubectl get pods -n traefik-public

Expected: 1 pod Running.

kubectl get ingressclass traefik-public

Expected: the IngressClass exists (it is not the default class). If it fails: Check kubectl describe pod -n traefik-public <pod-name> for image pull errors or resource constraints.

1.3 Install Traefik — Dashboard Controller

This Traefik instance serves the dashboard on a dedicated LoadBalancer, restricted by IP allowlist.

Two allowlist mechanisms ship for this instance. This guide uses values-dashboard.yaml, which restricts access with the portable service.loadBalancerSourceRanges field. A parallel values-internal.yaml is also provided for AWS environments that prefer the service.beta.kubernetes.io/aws-load-balancer-source-ranges annotation instead. Pick one and use it consistently; the steps below assume values-dashboard.yaml.

Before installing, edit third-party/traefik/values-dashboard.yaml to set the allowed source IPs. The loadBalancerSourceRanges field controls which IPs can reach the dashboard. By default it is set to 0.0.0.0/0 (all IPs); restrict it to your VPN, office, or known egress IPs.

Whitelist a single IP

service:
  loadBalancerSourceRanges:
    - "<YOUR_VPN_IP>/32"

Whitelist multiple IPs

Add one entry per IP or CIDR block. A /32 suffix matches a single IPv4 address; a CIDR block (e.g. /24) matches a range. You can mix individual IPs and ranges freely:

service:
  loadBalancerSourceRanges:
    - "203.0.113.10/32"       # office gateway
    - "203.0.113.11/32"       # backup office gateway
    - "198.51.100.0/24"       # VPN pool
    - "192.0.2.50/32"         # on-call engineer home IP

Tips when maintaining the list:

Keep one entry per line and add a short # comment identifying each IP’s owner or purpose; this is what future operators use to decide whether an entry is still needed.
Always use CIDR notation. A bare IP like 203.0.113.10 is rejected by the cloud provider; use 203.0.113.10/32.
For IPv6 ranges, use the equivalent /128 (single address) or larger CIDR, e.g. 2001:db8::1/128. Not all cloud providers support IPv6 source ranges; check your provider’s LoadBalancer documentation.
The list is an OR: traffic is allowed if the source matches any entry.

After editing the file, proceed to helm install below. If the controller is already installed, run helm upgrade with the same flags, or patch the Service at runtime (next section).

Update the whitelist at runtime

You can change the allowed IPs without a Helm upgrade by patching the Service directly. The patch replaces the entire list; always include every IP you want to keep, not just the new one. To replace the list with a new set of IPs:

kubectl patch svc traefik-dashboard -n traefik-dashboard \
  -p '{"spec":{"loadBalancerSourceRanges":["203.0.113.10/32","198.51.100.0/24","192.0.2.50/32"]}}'

To safely append an IP without losing existing entries, read the current list first, then patch with the combined set:

# 1. Show the current allowlist
kubectl get svc traefik-dashboard -n traefik-dashboard \
  -o jsonpath='{.spec.loadBalancerSourceRanges}'

# 2. Patch with the full list including the new IP
kubectl patch svc traefik-dashboard -n traefik-dashboard \
  -p '{"spec":{"loadBalancerSourceRanges":["<EXISTING_IP_1>/32","<EXISTING_IP_2>/32","<NEW_IP>/32"]}}'

Runtime patches are not persisted back to values-dashboard.yaml. To keep the change across future Helm upgrades, also update the values file and commit it.

Then install:

helm install traefik-dashboard traefik/traefik \
  --namespace traefik-dashboard --create-namespace \
  --version 39.0.8 \
  -f third-party/traefik/values-dashboard.yaml

Test it:

kubectl get pods -n traefik-dashboard

Expected: 1 pod Running.

kubectl get ingressclass traefik-dashboard

Expected: the IngressClass exists.

1.4 Wait for LoadBalancers

Both Traefik instances need external IPs before proceeding.

kubectl get svc -n traefik-public
kubectl get svc -n traefik-dashboard

Test it: Both services show an EXTERNAL-IP (not <pending>). If still pending, watch for assignment:

kubectl get svc -n traefik-public -w

Press Ctrl+C once the IP appears. IP assignment typically takes 2—5 minutes. If it fails: <pending> after 10 minutes usually means the cloud provider can’t provision a LoadBalancer. Check: subnet tags (EKS requires kubernetes.io/role/elb), VPC configuration, service quotas, and that the correct internal LB annotation is set for the internal instance.

Phase 2 — Create Secrets (~10 min)

All secrets are created manually before deploying the application. This ensures sensitive values never appear in manifest files.

2.1 Create the namespace

kubectl create namespace agenteye

Test it:

kubectl get namespace agenteye

Expected: status Active.

2.2 Image pull secret

This secret authenticates with ghcr.io to pull the AgentEye container images. See enterprise-docs/github-token.md for how to generate your PAT.

kubectl create secret docker-registry agenteye-image-pull \
  --namespace agenteye \
  --docker-server=ghcr.io \
  --docker-username=agenteye-enterprise \
  --docker-password="${AGENTEYE_TOKEN}"

Test it:

kubectl get secret agenteye-image-pull -n agenteye -o jsonpath='{.type}'

Expected: kubernetes.io/dockerconfigjson. Test it (deep) — verify the token can actually pull images: Use the server image tag pinned in your overlay’s kustomization.yaml (currently v0.0.1-beta.48 in both the bundled acme overlay and base deployment). Substitute the tag below for the one you are deploying so this check does not drift across releases:

kubectl run test-pull \
  --image=ghcr.io/agenteye-enterprise/server:v0.0.1-beta.48 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"agenteye-image-pull"}]}}' \
  --restart=Never -n agenteye --command -- echo ok

# Wait a few seconds for the pull, then:
kubectl logs test-pull -n agenteye
kubectl delete pod test-pull -n agenteye

Expected: ok printed in the logs. If it fails: ErrImagePull or 401 Unauthorized means the PAT is invalid or lacks read:packages scope. Re-check enterprise-docs/github-token.md.

2.3 PostgreSQL credentials

POSTGRES_PASSWORD=$(openssl rand -hex 24)

kubectl create secret generic agenteye-postgres \
  --namespace agenteye \
  --from-literal=POSTGRES_USER=postgres \
  --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \
  --from-literal=POSTGRES_DB=agenteye

Important: We use -hex (not -base64) to generate the password. Base64 output can contain +, /, and = which break the DATABASE_URL connection string. See enterprise-docs/troubleshooting.md for details.

Store POSTGRES_PASSWORD in your secrets manager immediately. You will need it if you ever restore from a backup or connect to the database directly.

Test it:

kubectl get secret agenteye-postgres -n agenteye

Expected: the secret exists.

kubectl get secret agenteye-postgres -n agenteye \
  -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d | wc -c

Expected: 48 (24 hex bytes = 48 characters).

2.4 Admin API key

ADMIN_KEY=$(openssl rand -hex 32)

kubectl create secret generic agenteye-admin-key \
  --namespace agenteye \
  --from-literal=ADMIN_KEY="${ADMIN_KEY}"

The admin key is the bootstrap credential. The server upserts it on every startup with all permissions. Use it to create scoped collector keys in Phase 7. See enterprise-docs/api-keys.md for the full permissions model.

Store ADMIN_KEY in your secrets manager immediately.

Test it:

kubectl get secret agenteye-admin-key -n agenteye

Expected: the secret exists.

The dashboard uses email + OTP for user login. Without this secret the server still starts and the ADMIN_KEY API path keeps working, but no user can log in via the UI. All keys are referenced as optional: true in the base manifest, so partial secrets (or no secret at all) are fine; the server falls back to the documented defaults. Bundling everything into one agenteye-auth secret keeps the auth surface rotatable in a single place.

kubectl create secret generic agenteye-auth \
  --namespace agenteye \
  --from-literal=ADMIN_EMAIL="admin@yourcompany.com" \
  --from-literal=ALLOWED_EMAILS="*@yourcompany.com" \
  --from-literal=SMTP_HOST="smtp.yourprovider.com" \
  --from-literal=SMTP_PORT="587" \
  --from-literal=SMTP_USERNAME="your-smtp-user" \
  --from-literal=SMTP_PASSWORD="your-smtp-password" \
  --from-literal=SMTP_FROM="noreply@yourcompany.com" \
  --from-literal=SMTP_TLS="starttls" \
  --from-literal=DEFAULT_ORG_NAME="Acme Corp" \
  --from-literal=DEFAULT_ORG_SLUG="acme"

Key	Purpose
`ADMIN_EMAIL`	Bootstrap admin user. Upserted on every startup with all permissions and protected from deletion/permission edits via the dashboard. Without it, no admin is seeded and first login is impossible.
`ALLOWED_EMAILS`	Comma-separated allowlist. Supports exact addresses (`user@example.com`) and domain wildcards (`@example.com`). Without it, no user can log in or be created*.
`SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM`	SMTP relay for sending OTP codes. If `SMTP_HOST` is unset, OTP codes are logged to the server’s stdout instead of emailed (useful for first-boot smoke tests). Provide all SMTP keys together for real email delivery.
`SMTP_TLS`	One of `starttls` (default), `tls`, or `none`.
`DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG`	Optional. Give the built-in `default` organization a friendly display name and URL slug so it lives at e.g. `/acme` instead of `/default`. Applied on first boot only; once you rename the org with `agenteye-orgctl org rename` (see §7.6) these are ignored. The slug must be 1—40 lowercase alphanumerics with single internal hyphens. Leave both unset to keep the generic `default`.

Store the SMTP credentials in your secrets manager.

Test it:

kubectl get secret agenteye-auth -n agenteye \
  -o jsonpath='{.data}' | grep -o '"[A-Z_]*"' | sort -u

Expected: the keys you populated appear in the output.

2.6 Multi-tenant org isolation key (optional)

Skip this for a single-tenant deployment; the server runs on a built-in dev default and serves the one default org fine. Before you create a second organization, set a strong, stable ORG_CH_SECRET: each org’s ClickHouse password is derived as HMAC(ORG_CH_SECRET, org_id), so the publicly-known dev default would yield publicly-derivable per-org credentials. The agenteye-orgctl org create command (see §7.6 Provision organizations) refuses to run while the server is still on the built-in dev default.

kubectl create secret generic agenteye-org-ch-secret \
  --namespace agenteye \
  --from-literal=ORG_CH_SECRET="$(openssl rand -base64 48)"

# Roll the server so it picks up the new value.
kubectl -n agenteye rollout restart deployment/server

The server reads this via an optional secretKeyRef, so a single-tenant cluster that never creates it still boots normally. Keep the value stable and identical across all replicas; rotating it invalidates every org’s derived ClickHouse password until the boot-time reconcile re-provisions the users (a rolling restart with the value consistent everywhere heals it). See deploy/base/server/secret.example.yaml.

Store ORG_CH_SECRET in your secrets manager and don’t rotate it casually.

2.7 Verify all secrets

kubectl get secrets -n agenteye -o custom-columns='NAME:.metadata.name,TYPE:.type'

Expected output (among any default secrets):

NAME                    TYPE
agenteye-admin-key      Opaque
agenteye-auth           Opaque
agenteye-image-pull     kubernetes.io/dockerconfigjson
agenteye-postgres       Opaque
agenteye-org-ch-secret  Opaque   # only if you completed §2.6 (multi-tenant)

The four core secrets (agenteye-admin-key, agenteye-auth, agenteye-image-pull, agenteye-postgres) must be present before continuing. agenteye-org-ch-secret is only required for multi-tenant deployments (see §2.6).

Phase 3 — Deploy the Application (~5 min)

3.1 Configure the public hostnames

cert-manager needs the ingest and dashboard hostnames before it can request their Let’s Encrypt certificates. Copy the template and set both:

cp base/certificates/domain.env.example base/certificates/domain.env
# Edit base/certificates/domain.env and set:
#   INGEST_DOMAIN=ingest.your-company.example      (resolves to the public Traefik LB)
#   DASHBOARD_DOMAIN=agenteye.your-company.example (resolves to the dashboard Traefik LB)

domain.env is gitignored; it stays local to each deployment. The kustomize build fails loudly if either key is missing.

DNS must resolve first. You don’t have to point DNS at the LBs yet (they don’t exist until Phase 1.2 is complete), but ACME issuance in step 3.2 will retry until each hostname resolves to its LoadBalancer. You can either set DNS now (using the LB hostnames captured in Phase 1.4) or proceed and add the records in Phase 4.

3.2 Apply manifests

Apply the base directly for a fresh install, or an overlay if you’ve cut one for this environment (overlays just pin image tags, env vars, and resource limits; they inherit the base’s certs and routing):

kubectl apply -k base/
# or
kubectl apply -k overlays/<your-env>/

The overlay includes the base automatically; apply one, not both.

3.3 Wait for pods

kubectl wait --for=condition=Ready pod -l 'app in (server,dashboard,postgres,clickhouse)' \
  -n agenteye --timeout=180s

The wait is scoped to the core data-plane pods. The optional agent (AI assistant) and redis pods come up alongside them; the assistant stays inert until you supply its LLM endpoint (see enterprise-docs/assistant.md), and Redis is a best-effort cache, so neither needs to be Ready for the platform to serve traffic. Test it:

kubectl get pods -n agenteye

Expected (the optional agent and redis pods also appear and reach Running):

NAME                         READY   STATUS    RESTARTS   AGE
agent-xxxxxxxxxx-xxxxx       1/1     Running   0          ...
clickhouse-0                 1/1     Running   0          ...
dashboard-xxxxxxxxxx-xxxxx   1/1     Running   0          ...
dashboard-xxxxxxxxxx-xxxxx   1/1     Running   0          ...
postgres-0                   1/1     Running   0          ...
redis-0                      1/1     Running   0          ...
server-xxxxxxxxxx-xxxxx      1/1     Running   0          ...
server-xxxxxxxxxx-xxxxx      1/1     Running   0          ...

If it fails:

Pod Status	Likely Cause	Debug Command
`ImagePullBackOff`	Bad image pull secret or PAT	`kubectl describe pod <name> -n agenteye`
`CrashLoopBackOff`	Bad environment variables (e.g. DATABASE_URL)	`kubectl logs <name> -n agenteye`
`Pending`	Insufficient CPU/memory or no nodes	`kubectl describe pod <name> -n agenteye` (check Events)

3.4 Verify storage

kubectl get pvc -n agenteye

Expected, both with status Bound:

PVC	Capacity	Backs
`postgres-data-postgres-0`	`50Gi`	PostgreSQL relational/metadata store
`clickhouse-data-clickhouse-0`	`100Gi`	ClickHouse events + evaluations analytics store

A redis-data-redis-0 PVC (1Gi) also appears for the optional cache. If it fails: Pending means no StorageClass can provision the volume. Check kubectl get storageclass and ensure a default exists. For production, overlay the ClickHouse volume onto a fast SSD StorageClass (e.g. gp3 on AWS, pd-ssd on GCP); compaction throughput suffers on slow disks.

3.5 Verify certificates

kubectl get certificates -n agenteye

Expected: 3 certificates, all Ready: True:

Name	Issuer	Purpose
`mtls-ca`	`selfsigned`	Private CA for issuing mTLS client certs (10-year validity)
`ingest-tls`	`letsencrypt-prod`	Public TLS certificate for the ingest endpoint (90-day, auto-renewed)
`dashboard-tls`	`letsencrypt-prod`	Public TLS certificate for the dashboard (90-day, auto-renewed)

If ingest-tls or dashboard-tls is not Ready: kubectl describe certificate <name> -n agenteye and read the Events. The common causes:

DNS not yet pointing at the LB. Let’s Encrypt resolves the hostname and hits port 80 to validate — INGEST_DOMAIN must resolve to the public LB, DASHBOARD_DOMAIN to the dashboard LB. Until the CNAME/Alias propagates, the order stays pending. Once DNS is correct, cert-manager retries automatically (no need to delete the Certificate).
Hostname not substituted. If dnsNames still reads INGEST_DOMAIN_PLACEHOLDER / DASHBOARD_DOMAIN_PLACEHOLDER, you skipped step 3.1 — create base/certificates/domain.env and re-apply.
Dashboard Traefik can’t serve the challenge (dashboard-tls only). The dashboard Traefik instance must be installed with the bundled values file (Phase 1.2), which enables the scoped Ingress provider that serves cert-manager’s HTTP-01 solver. An instance installed without it leaves the challenge unroutable and the order pending forever.

If mtls-ca is not Ready: cert-manager itself is unhealthy. Re-check the cert-manager pods from step 1.1.

3.6 Verify CronJobs

kubectl get cronjobs -n agenteye

Expected:

Name	Schedule	Purpose
`agenteye-backup`	`0 3 * * *`	Daily Postgres + ClickHouse backup at 03:00 UTC
`cert-renewal-check`	`0 3,15 * * *`	Certificate expiry alerts at 03:00 and 15:00 UTC

3.7 Verify server started correctly

kubectl logs -n agenteye -l app=server --tail=20

Test it: Look for a startup line indicating the server is listening on port 8080. There should be no database connection errors (the server requires both PostgreSQL and ClickHouse to be reachable before it reports Ready). If it fails: The most common cause is a POSTGRES_PASSWORD containing URL-unsafe characters that break the DATABASE_URL. See enterprise-docs/troubleshooting.md.

3.8 Verify dashboard connected to server

kubectl logs -n agenteye -l app=dashboard --tail=20

Test it: Look for Ready in the output with no ECONNREFUSED or similar errors. If it fails: Check that the server Service exists (kubectl get svc server -n agenteye) and that AGENTEYE_SERVER_URL is set to http://server:8080 in the dashboard deployment.

Phase 4 — Network Access (~5 min)

4.1 Retrieve LoadBalancer addresses

PUBLIC_IP=$(kubectl get svc -n traefik-public \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

INTERNAL_IP=$(kubectl get svc -n traefik-dashboard \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

On AWS EKS, LoadBalancers return a hostname instead of an IP. Replace .ip with .hostname in the commands above.

Test it:

echo "Public  (ingest):    $PUBLIC_IP"
echo "Internal (dashboard): $INTERNAL_IP"

Both must be non-empty.

4.2 Point DNS at the LoadBalancers

Create DNS records so the hostnames from base/certificates/domain.env resolve to their LoadBalancers — INGEST_DOMAIN to the public Traefik LB, DASHBOARD_DOMAIN to the dashboard Traefik LB:

AWS Route 53: A record with Alias = Yes, target = the LB hostname. Don’t use plain A → IP; ELB IPs rotate.
Any other provider: CNAME from the hostname to the LB hostname.

Verify:

dig +short ingest.your-company.example
dig +short agenteye.your-company.example

Should return the same addresses as $PUBLIC_IP and $INTERNAL_IP respectively (or, on EKS, resolve to the same *.elb.amazonaws.com hostnames). Once DNS resolves, cert-manager finishes the pending ACME orders from Phase 3.5 within a minute. Re-run kubectl get certificates -n agenteye until both ingest-tls and dashboard-tls show Ready: True.

4.3 Reach the ingest endpoint

The public ingest endpoint enforces mutual TLS, so every request (including /health) must present a client certificate. You issue your first client certificate in Phase 5; if you have one already, verify reachability now:

curl -s --cert issued/<cluster-name>/client.crt \
        --key issued/<cluster-name>/client.key \
        https://ingest.your-company.example/health

Expected: {"status":"ok"}. No -k is needed — the server certificate chains to a public CA for INGEST_DOMAIN, so it validates against the system trust store. Reach the ingest endpoint by its INGEST_DOMAIN hostname (which matches the issued certificate), not by the raw LoadBalancer IP/hostname. The dashboard endpoint is served on DASHBOARD_DOMAIN with a publicly-trusted certificate and is not behind mTLS, so no -k and no client certificate are needed:

curl -s https://agenteye.your-company.example/ -o /dev/null -w '%{http_code}\n'

Reach the dashboard by its hostname, not the raw LB address — the certificate is bound to DASHBOARD_DOMAIN, so the raw address shows a certificate-name mismatch. If it fails: If curl hangs, check that the LB is reachable from your machine (VPN, security groups, firewall rules). A certificate required handshake error on the ingest hostname means no client certificate was presented; complete Phase 5 first. A TLS validation error on the ingest hostname means the server cert hasn’t finished issuing; go back to Phase 3.5 and resolve the issue there.

Phase 5 — Issue mTLS Client Certificates (~10 min per cluster)

Collectors authenticate with two factors: a client certificate (transport layer, proves the request comes from an authorized cluster) and an API key (application layer, proves the request is from a collector with events:add permission). A leaked key is useless without the cert; a stolen cert is useless without a valid key.

5.1 Issue a certificate

Each cluster running collectors needs its own client certificate. From the manifests directory:

cd base/certificates/client-certs
./issue-client-cert.sh <cluster-name>

Replace <cluster-name> with a meaningful identifier (e.g. us-east-1-prod, staging). Test it: The script prints ==> Done! and lists the output files.

kubectl get certificate mtls-client-<cluster-name> -n agenteye

Expected: Ready: True. Output files in issued/<cluster-name>/:

File	Purpose
`client.crt`	Client certificate (90-day validity)
`client.key`	Client private key
`ca.crt`	CA certificate for server verification
`collector-mtls-secret.yaml`	Ready-to-apply Kubernetes Secret for the collector cluster

5.1b Alternative delivery: AWS Secrets Manager

If the consumer of the cert is a Kubernetes Pod that needs client.crt and client.key on disk — the typical case when you run the agenteye-collector as a sidecar in your application pod — push the cert bundle into AWS Secrets Manager. The application pod then mounts it via the Secrets Store CSI Driver with IRSA, and certificate rotation is fully hands-off.

cd base/certificates/client-certs
export AWS_REGION=us-east-1     # region where your workload runs
./issue-client-cert.sh <cluster-name> --save-to aws-secrets-manager

On re-run (renewal), the script calls PutSecretValue on the same secret, so the ARN and name stay stable. The CSI Driver picks up the new version on its next rotation poll and rewrites the files inside the pod. Prerequisites:

aws CLI v2 authenticated to your AWS account.
jq installed.
AWS_REGION environment variable set.
IAM permissions on your caller identity (scope Resource to arn:aws:secretsmanager:<region>:<account>:secret:agenteye/mtls-client/*):
- secretsmanager:CreateSecret
- secretsmanager:DescribeSecret
- secretsmanager:PutSecretValue
- secretsmanager:TagResource

What the script does in this mode:

Step	Action
1	Issues / re-extracts the cert via cert-manager (same as default mode).
2	Calls `DescribeSecret` on `agenteye/mtls-client/<cluster-name>` to decide create-vs-update.
3	On first run: `CreateSecret` with a three-key JSON payload (`client.crt`, `client.key`, `ca.crt`), tagged `AgentEyeCluster=<cluster-name>`. On subsequent runs: `PutSecretValue` to publish a new version; tag refreshed via `TagResource`.
4	Deletes `issued/<cluster-name>/` only after a successful upload. On any failure, the directory is preserved so you can retry.

If the secret is scheduled for deletion, the script fails with a clear error telling you to run aws secretsmanager restore-secret --secret-id agenteye/mtls-client/<cluster-name> before retrying. For full pod wiring (SecretProviderClass, IRSA setup, rotation behavior, troubleshooting) see enterprise-docs/single-pod-deployment.md.

5.2 Verify the certificate works

Test the issued certificate against the mTLS ingress:

curl -sk --cert issued/<cluster-name>/client.crt \
     --key issued/<cluster-name>/client.key \
     https://${PUBLIC_IP}/health

Expected: {"status":"ok"} If it fails:

Error	Cause	Fix
`certificate required`	Cert not being presented	Check file paths in the `curl` command
`bad certificate`	CA mismatch	Verify `mtls-ca-issuer` issued the cert: `kubectl describe certificate mtls-client-<name> -n agenteye`
`connection refused`	Wrong hostname or LB not reachable	Check `/etc/hosts` or DNS

5.3 Deliver to the collector cluster

Send collector-mtls-secret.yaml to the team operating the collector cluster. They apply it:

kubectl apply -f collector-mtls-secret.yaml -n <collector-namespace>

Then configure the collector to mount the secret and use the cert paths:

{
  "tls_cert": "/etc/agenteye/tls/client.crt",
  "tls_key": "/etc/agenteye/tls/client.key"
}

See enterprise-docs/collector-installation.md for complete collector setup including Kubernetes volume mounts. Test it (in the collector cluster):

kubectl get secret agenteye-collector-mtls -n <collector-namespace>

Expected: the secret exists with 3 data keys (client.crt, client.key, ca.crt).

5.4 Certificate lifecycle

Property	Value
Client cert validity	90 days
Auto-renewal	cert-manager renews 15 days before expiry
CA validity	10 years
Expiry alerts	CronJob alerts 30 days before expiry (Phase 6)

cert-manager auto-renews the certificate on the AgentEye cluster, but the renewed cert must be re-delivered to the collector cluster. Re-run issue-client-cert.sh and re-apply collector-mtls-secret.yaml before the old cert expires. If you are using --save-to aws-secrets-manager (see § 5.1b), re-run the same command. The script calls PutSecretValue on the same secret; pods mounting the secret via the Secrets Store CSI Driver pick up the new version on their next rotation poll (default: every hour), with no pod restart required.

5.5 Revoke a certificate

To immediately block a cluster’s collector access:

kubectl delete certificate mtls-client-<cluster-name> -n agenteye

Test it: The curl command from step 5.2 now fails with a TLS handshake error.

Phase 6 — Certificate Renewal Monitoring (~2 min)

A built-in CronJob runs every 12 hours (03:00 and 15:00 UTC) and checks all client certificates labeled agenteye.io/cert-type=mtls-client. It alerts when any certificate is within 30 days of expiry.

6.1 Enable Slack notifications (optional)

kubectl create secret generic cert-renewal-notify-config \
  --namespace agenteye \
  --from-literal=SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

Without this secret, the CronJob still runs and logs certificate status to stdout. Test it:

kubectl get secret cert-renewal-notify-config -n agenteye

Expected: the secret exists.

6.2 Test the CronJob

kubectl create job --from=cronjob/cert-renewal-check test-cert-check -n agenteye

kubectl wait --for=condition=Complete job/test-cert-check -n agenteye --timeout=60s

kubectl logs -n agenteye -l job-name=test-cert-check

Expected: a list of certificates with their expiry status. If the Slack webhook is configured, check the Slack channel for the alert message. If it fails: Check RBAC — the CronJob’s ServiceAccount needs get, list permissions on cert-manager Certificate resources. Verify with: kubectl describe role cert-renewal-check -n agenteye. Clean up the test job:

kubectl delete job test-cert-check -n agenteye

Phase 7 — Verify End-to-End

This phase confirms the entire pipeline works: health check, key creation, event ingestion, and dashboard display.

Note: The examples below reach the ingest endpoint by its raw LoadBalancer address (${PUBLIC_IP}) for convenience, which is why they pass -k; the server certificate is bound to INGEST_DOMAIN, not to the LB IP, so the hostname check is skipped. The ingest endpoint enforces mutual TLS on every path, so every call must also present a client certificate (--cert/--key). To validate the public certificate as well, target https://ingest.your-company.example/... instead of ${PUBLIC_IP} and drop the -k.

7.1 Health check

curl -sk --cert issued/<cluster-name>/client.crt \
        --key issued/<cluster-name>/client.key \
        https://${PUBLIC_IP}/health

Expected: {"status":"ok"} with HTTP 200.

7.2 Create scoped collector keys

The admin key is for bootstrap and management. Create dedicated events:add keys for collectors:

COLLECTOR_KEY=$(openssl rand -hex 32)

curl -sk -X POST https://${PUBLIC_IP}/keys \
  --cert issued/<cluster-name>/client.crt \
  --key issued/<cluster-name>/client.key \
  -H "Authorization: Bearer ${ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-collector",
    "key": "'"${COLLECTOR_KEY}"'",
    "permissions": ["events:add"]
  }'

Test it: Response includes "id", "name": "prod-collector", "permissions": ["events:add"], "created_at". Test it: Verify the key appears in the key list:

curl -sk https://${PUBLIC_IP}/keys \
  --cert issued/<cluster-name>/client.crt \
  --key issued/<cluster-name>/client.key \
  -H "Authorization: Bearer ${ADMIN_KEY}"

Expected: prod-collector appears in the response. See enterprise-docs/api-keys.md for the full key management reference.

7.3 Ingest a test event

echo '{"session_id":"test","agent_id":"smoke-test","type":"test","timestamp":"2026-04-20T00:00:00Z"}' \
  | curl -sk --cert issued/<cluster-name>/client.crt \
         --key issued/<cluster-name>/client.key \
         -H "Authorization: Bearer ${COLLECTOR_KEY}" \
         -H "Content-Type: application/x-ndjson" \
         --data-binary @- \
         https://${PUBLIC_IP}/events

Expected: {"accepted":1,"skipped":0} with HTTP 200. If it fails:

HTTP Status	Cause
401	Invalid or missing API key
403	Key lacks `events:add` permission
TLS handshake error	Client certificate issue — see Phase 5 troubleshooting

7.4 Verify event appears in dashboard

Open https://agenteye.your-company.example (your DASHBOARD_DOMAIN) in a browser. The certificate is publicly trusted, so there is no warning.

If the dashboard LoadBalancer is restricted by IP allowlist and you can’t connect, verify your IP is allowed:
kubectl get svc traefik-dashboard -n traefik-dashboard -o jsonpath='{.spec.loadBalancerSourceRanges}'
Keep in mind that Let’s Encrypt renews the dashboard certificate over HTTP-01 on port 80, and source ranges apply to the whole LoadBalancer — before restricting it to corporate ranges, coordinate a DNS-01 solver with support or renewals fail silently.

Test it: The smoke-test event should appear in the event list with session test and agent smoke-test. If it fails: Check dashboard logs (kubectl logs -n agenteye -l app=dashboard --tail=50). Verify AGENTEYE_SERVER_URL and AGENTEYE_API_KEY are set correctly.

7.5 Test the backup CronJob

kubectl create job --from=cronjob/agenteye-backup test-backup -n agenteye

kubectl wait --for=condition=Complete job/test-backup -n agenteye --timeout=300s

kubectl logs -n agenteye -l job-name=test-backup

Expected: Backup created: agenteye-YYYYMMDD-HHMMSS.tar.gz (NNN) in the logs; the archive bundles the Postgres dump and the ClickHouse tables.

The S3 upload step ships wired into the CronJob and runs whenever BACKUP_BUCKET is set (the base ships a default bucket value). It is skipped only when BACKUP_BUCKET is empty or literally PLACEHOLDER. Point it at your own bucket and grant the agenteye-backup ServiceAccount write access before relying on it (see Backups section below).

Clean up:

kubectl delete job test-backup -n agenteye

7.6 Provision organizations (multi-tenant)

Skip this for a single-tenant deployment; all data lives in the built-in default org and nothing here is required. If you are running multiple isolated tenants, organizations and their memberships are created with the agenteye-orgctl CLI. It ships inside the server image (alongside agenteye-server) and you run it inside the existing server Deployment with kubectl exec; there is no separate pod, Job, or Deployment, and no HTTP API or dashboard button for tenant lifecycle. Running it in the server pod means it reuses the pod’s DATABASE_URL, CLICKHOUSE_URL, and the ORG_CH_SECRET from §2.6.

Prerequisite: complete §2.6 first. org create refuses to run while the server is still on the built-in dev ORG_CH_SECRET, and the per-org ClickHouse user it provisions depends on that secret being strong and stable.

Create an org and add its first admin:

kubectl -n agenteye exec deploy/server -- \
  agenteye-orgctl org create --slug acme --name "Acme Corp"

kubectl -n agenteye exec deploy/server -- \
  agenteye-orgctl member add --org acme --email alice@acme.example --set admin

The new member receives an OTP on first dashboard login and then works entirely in the UI under the org’s URL prefix (e.g. /acme/...). Other commands (run the same kubectl -n agenteye exec deploy/server -- agenteye-orgctl … way):

Command	What it does
`org list`	List organizations and their state.
`org rename --slug <slug> --name <new name>`	Rename an org (slug unchanged).
`org delete --slug <slug>`	Soft-delete + drop the org’s ClickHouse user; data retained.
`org purge --slug <slug>`	Irreversible data wipe; org must be `delete`d first; never the `default` org.
`member list --org <slug>`	List members and their permissions.
`member update --org <slug> --email <email> [--set ...] [--add ...] [--remove ...]`	Change a member’s permissions.
`member remove --org <slug> --email <email>`	Remove a member from the org.

Builtin permission sets are admin, standard, and read-only. Per-org API keys are still minted in the dashboard/API by org members (§7.2 shows the keys API); only the org + member lifecycle is operator-only. Full reference and a worked example: enterprise-docs/tenant-management.md.

Post-Deployment Checklist

Use this checklist to confirm everything is working. Every item should be checked before handing off to collectors.

Backups

A single agenteye-backup CronJob runs daily at 03:00 UTC. It dumps both stores: PostgreSQL (relational state) and ClickHouse (the events + evaluations analytics tables), into one compressed archive in the pod, then uploads it to the object storage you wire in your overlay. Each run produces one object, agenteye-<timestamp>.tar.gz, which unpacks to:

postgres.sql        # pg_dump of the relational database
events.sql          # ClickHouse events table DDL
events.native       # ClickHouse events data (Native format)
evaluations.sql     # ClickHouse evaluations table DDL
evaluations.native  # ClickHouse evaluations data

ClickHouse is read over its HTTP API (the same endpoint the server uses), so the job needs no ClickHouse client. Only the two physical tables are dumped; the server recreates all views (agent_sessions, the analytics.* aliases) and row policies on startup, so those tables are the complete picture.

Configure cloud upload

The backup CronJob ships with the S3 upload step (aws s3 cp) already wired in, and the base sets a default BACKUP_BUCKET you must override with your own bucket. On AWS: you only set BACKUP_BUCKET to your bucket and grant the agenteye-backup ServiceAccount write access via IRSA — no script change needed. On GCP / Azure: you must replace the shipped aws s3 cp line in the CronJob script with the matching command below — do not just add yours, because the leftover aws s3 cp runs under set -eu and fails the job.

Cloud	Upload Command
AWS S3	`aws s3 cp /tmp/${FILENAME} s3://${BACKUP_BUCKET}/${FILENAME}` (ships by default)
GCP Cloud Storage	`gsutil cp /tmp/${FILENAME} gs://${BACKUP_BUCKET}/${FILENAME}`
Azure Blob	`az storage blob upload -f /tmp/${FILENAME} -c backups -n ${FILENAME}`

One object per run means a bucket lifecycle rule (e.g. “delete after 30 days”) prunes whole backups cleanly. The agenteye-backup ServiceAccount needs IAM permissions to write to the bucket. On AWS use IRSA; on GKE Workload Identity; on AKS Pod Identity. Annotate the ServiceAccount in your overlay accordingly. The dump is buffered in a 20 GiB emptyDir at /tmp; the ClickHouse events dump dominates its size. If your data is larger, raise the volume’s sizeLimit in your overlay.

Manual backup

kubectl create job --from=cronjob/agenteye-backup manual-backup -n agenteye

Restore from backup

Download the archive from your bucket and unpack it:

tar xzf agenteye-<timestamp>.tar.gz
# -> postgres.sql, events.sql, events.native, evaluations.sql, evaluations.native

Scale the server down to stop writes during the restore:

kubectl scale deployment/server -n agenteye --replicas=0

Restore PostgreSQL:

kubectl cp postgres.sql agenteye/postgres-0:/tmp/postgres.sql
kubectl exec -n agenteye postgres-0 -- bash -c 'psql -U postgres -d agenteye < /tmp/postgres.sql'

Restore ClickHouse. The schema already exists (the server creates it on startup; on a bare cluster, apply the .sql DDL files first). Re-inserting is safe; ReplacingMergeTree collapses duplicates on merge:

kubectl cp events.native      agenteye/clickhouse-0:/tmp/events.native
kubectl cp evaluations.native agenteye/clickhouse-0:/tmp/evaluations.native
kubectl exec -n agenteye clickhouse-0 -- sh -c \
  'clickhouse-client --query "INSERT INTO agenteye.events FORMAT Native"      < /tmp/events.native'
kubectl exec -n agenteye clickhouse-0 -- sh -c \
  'clickhouse-client --query "INSERT INTO agenteye.evaluations FORMAT Native" < /tmp/evaluations.native'

Scale the server back up:

kubectl scale deployment/server -n agenteye --replicas=2

Test it: hit /health and confirm the dashboard shows historical events and evaluations.

Upgrading

Pull the latest manifests:
```
git pull
```
Check release notes for breaking changes or new environment variables. Upgrading from a pre-v0.0.1-beta.6 deploy: dashboard login now uses email + OTP and reads its config from a new agenteye-auth secret. Create it per §2.5 before applying, or no user will be able to log in via the UI. API-key traffic is unaffected either way.

Update image tags in your overlay’s kustomization.yaml if applicable:

images:
  - name: ghcr.io/agenteye-enterprise/server
    newTag: <new-version>
  - name: ghcr.io/agenteye-enterprise/dashboard
    newTag: <new-version>

Take a pre-upgrade backup:

kubectl create job --from=cronjob/agenteye-backup pre-upgrade-backup -n agenteye
kubectl wait --for=condition=Complete job/pre-upgrade-backup -n agenteye --timeout=120s

Apply:

kubectl apply -k overlays/<your-overlay>/

Test it — monitor rollout:

kubectl rollout status deployment/server -n agenteye
kubectl rollout status deployment/dashboard -n agenteye

Test it — health check:
```
curl -sk https://${PUBLIC_IP}/health
```
Expected: {"status":"ok"}
Test it — dashboard: Open in browser, verify it loads and shows existing events.

Rollback

If something is wrong after upgrading:

kubectl rollout undo deployment/server -n agenteye
kubectl rollout undo deployment/dashboard -n agenteye

Test it: /health returns 200, dashboard loads and shows events.

Creating Overlays for New Environments

To deploy to a new cluster, create a new overlay by copying the existing acme overlay as a template:

cp -r overlays/acme overlays/<new-env>

Files to customize in overlays/<new-env>/:

File	What to Change
`kustomization.yaml`	Image tags (`newTag` values)
`patches/server-env.yaml`	`DATABASE_URL` sslmode if needed
`patches/resource-limits.yaml`	CPU and memory requests/limits
`patches/cert-renewal-env.yaml`	Slack webhook URL

Test it — dry run: Render the manifests without applying to verify correctness:

kubectl kustomize overlays/<new-env>/

Inspect the output for correct hostnames, image tags, and resource values. Then apply:

kubectl apply -k overlays/<new-env>/

Architecture Reference

Kubernetes resources created

Kind	Name	Namespace	Purpose
Namespace	`agenteye`	—	All AgentEye resources
StatefulSet	`clickhouse`	agenteye	ClickHouse 24.8 with 100Gi PVC — canonical events + evaluations store
StatefulSet	`postgres`	agenteye	PostgreSQL 16 with 50Gi PVC — relational/metadata store
StatefulSet	`redis`	agenteye	Redis 7.2 with 1Gi PVC — optional shared cache
Deployment	`server`	agenteye	Rust/Axum API (2 replicas, pod anti-affinity)
Deployment	`dashboard`	agenteye	Next.js UI (2 replicas)
Deployment	`agent`	agenteye	Optional AI assistant (1 replica); inert until an LLM endpoint is configured
Service	`clickhouse`	agenteye	ClusterIP, port 8123 (HTTP)
Service	`postgres`	agenteye	ClusterIP, port 5432
Service	`redis`	agenteye	ClusterIP, port 6379
Service	`server`	agenteye	ClusterIP, port 8080
Service	`dashboard`	agenteye	ClusterIP, port 3000
Service	`agent`	agenteye	ClusterIP, port 9100 (internal-only)
PodDisruptionBudget	`clickhouse`	agenteye	Keeps at least one ClickHouse pod available during disruptions
NetworkPolicy	`agent`	agenteye	Restricts ingress to the AI assistant service
Certificate	`mtls-ca`	agenteye	Self-signed CA (10-year, ECDSA P-256)
Issuer	`mtls-ca-issuer`	agenteye	Issues client certs from the CA
Certificate	`ingest-tls`	agenteye	Server TLS for ingest endpoint (90-day, Let’s Encrypt)
Certificate	`dashboard-tls`	agenteye	Server TLS for dashboard (90-day, Let’s Encrypt)
IngressRoute	`ingest`	agenteye	Routes `/events`, `/health`, `/keys` via traefik-public with mTLS
IngressRoute	`dashboard`	agenteye	Routes `/` via traefik-dashboard (no mTLS)
TLSOption	`mtls-ingest`	agenteye	Requires and verifies client certificates
CronJob	`agenteye-backup`	agenteye	Daily Postgres + ClickHouse backup at 03:00 UTC
CronJob	`cert-renewal-check`	agenteye	Certificate expiry alerts, twice daily
ServiceAccount	`agenteye-backup`	agenteye	IAM-annotatable SA for backup bucket access
ServiceAccount	`cert-renewal-check`	agenteye	SA for cert-manager Certificate read access
Role	`cert-renewal-check`	agenteye	`get, list` on cert-manager Certificates
RoleBinding	`cert-renewal-check`	agenteye	Binds role to ServiceAccount

PostgreSQL sizing and tuning

The base manifests ship a single-replica PostgreSQL 16 StatefulSet with a tuned postgresql.conf mounted from a ConfigMap, so the dashboard’s hot read paths get a real buffer cache and work_mem budget out of the box rather than the image defaults. Default resources (acme overlay):

Resource	Request	Limit
CPU	1	4
Memory	4 Gi	8 Gi
Storage	50 Gi PVC (RWO, default StorageClass)	—

Default postgresql.conf highlights (full file in postgres/configmap.yaml):

Setting	Value	Rationale
`shared_buffers`	`2GB`	~25% of the 8 Gi limit
`effective_cache_size`	`6GB`	~75% of the limit
`work_mem`	`32MB`	Headroom for dashboard sorts (events / evaluations history)
`maintenance_work_mem`	`512MB`	Faster `CREATE INDEX`, `VACUUM`
`max_connections`	`100`	Server pool (10 × 2 replicas) + readonly pool (4) + headroom
`random_page_cost`	`1.1`	SSD-backed PVC
`jit`	`off`	OLTP-shaped workload
`shared_preload_libraries`	`pg_stat_statements`	Per-query statistics
`log_min_duration_statement`	`250ms`	Surface slow queries in pod logs

Override for a different pod size: patch the postgres-config ConfigMap in your overlay. Keep shared_buffers around 25% of the memory limit and effective_cache_size around 75%. After editing, restart the pod (kubectl -n agenteye rollout restart statefulset/postgres) for the new postgresql.conf to take effect. Inspect slow queries once the pod is running. pg_stat_statements aggregates per-query timing, so this is the fastest way to find what is loading the database. Its views live in the extensions schema (not public) so they stay hidden from the read-only role that backs the dashboard’s ad-hoc query workbench:

kubectl exec -n agenteye postgres-0 -- psql -U postgres -d agenteye -c \
  "SELECT round(total_exec_time::numeric, 0) AS total_ms, calls,
          round(mean_exec_time::numeric, 1) AS mean_ms, query
   FROM extensions.pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;"

Environment variables

Server:

Variable	Source	Value
`DATABASE_URL`	Constructed from `agenteye-postgres` secret	Built at deploy time from secret values (see `server/deployment.yaml`). Points at the relational/metadata store.
`CLICKHOUSE_URL`	Deployment manifest	`http://clickhouse:8123` — required; the canonical events + evaluations store. The server refuses to start without it.
`CLICKHOUSE_DATABASE`	Deployment manifest	`agenteye`
`REDIS_URL`	Deployment manifest	`redis://redis:6379/0` — optional cache and rate-limit backend. If Redis is unreachable, auth and OTP fall back to PostgreSQL (graceful degradation).
`ADMIN_KEY`	`agenteye-admin-key` secret	Bootstrap admin credential
`AGENT_API_KEY`	`agenteye-agent` secret (optional)	Seeds the protected `dashboard-assistant` key the AI assistant’s agent service uses (the same value the agent reads as `AGENTEYE_API_KEY`). Unset means the key isn’t seeded and the assistant stays off. See enterprise-docs/assistant.md.
`AGENTEYE_AGENT_URL`	Deployment manifest	`http://agent:9100` — where the server reaches the agent service to run the audit’s agentic investigation (the same connection the dashboard uses). Unset ⇒ audits run policy-only. See enterprise-docs/assistant.md.
`AGENTEYE_AGENT_TOKEN`	`agenteye-agent` secret (optional)	Shared secret the server presents to the agent for audit investigation calls; must match the agent’s `AGENTEYE_AGENT_TOKEN`. Unset ⇒ audits run policy-only.
`ADMIN_EMAIL`	`agenteye-auth` secret (optional)	Bootstrap admin user email, upserted as protected on every startup
`ALLOWED_EMAILS`	`agenteye-auth` secret (optional)	Comma-separated allowlist for user creation/login; supports `*@domain` wildcards
`SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM`, `SMTP_TLS`	`agenteye-auth` secret (optional)	SMTP relay for OTP delivery; if `SMTP_HOST` is unset, OTPs are logged to stdout
`DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG`	`agenteye-auth` secret (optional)	First-boot-only display name and URL slug for the built-in `default` org; ignored after an org rename. Slug must be 1—40 lowercase alphanumerics with single internal hyphens. See §2.5.
`EVALUATOR_ENDPOINT`, `EVALUATOR_TOKEN`	`agenteye-evaluator` secret (optional)	Wire the evaluation pipeline. With the secret absent the pipeline is a no-op. See enterprise-docs/evaluation-suite.md.
`SESSION_TTL_SECS`	Default	`86400` (24 h)
`OTP_TTL_SECS`	Default	`600` (10 min)
`ORG_CH_SECRET`	`agenteye-org-ch-secret` secret (optional; required before a 2nd org)	HMAC key from which each organization’s per-tenant ClickHouse password is derived. Single-tenant clusters run on the built-in dev default; set a strong, stable value in your own secret before provisioning a second organization (see §2.6), and keep it identical across all server replicas; rotating it orphans every org’s ClickHouse user until the next startup re-provisions them.
`RUST_LOG`	Deployment manifest	`info`
`LISTEN_ADDR`	Default	`0.0.0.0:8080`
`MAX_BODY_BYTES`	Default	`134217728` (128 MB)

Multi-tenant ClickHouse. Per-org isolation requires SQL access management on ClickHouse; the bundled deploy/base/clickhouse/ config enables it (access management + users_without_row_policies_can_read_rows=false) so the server can create one read-only ClickHouse user and row policy per organization. If you replace that config, carry those settings over.

Dashboard:

Variable	Source	Value
`AGENTEYE_SERVER_URL`	Deployment manifest	`http://server:8080`
`AGENTEYE_API_KEY`	`agenteye-admin-key` secret	Same as `ADMIN_KEY`
`AGENTEYE_AGENT_URL`	Deployment manifest	`http://agent:9100` — where the dashboard reaches the optional AI assistant
`AGENTEYE_AGENT_TOKEN`	`agenteye-agent` secret (optional)	Shared secret the dashboard presents to the AI assistant; the assistant stays inert until its LLM endpoint is configured. See enterprise-docs/assistant.md.
`REDIS_URL`	Deployment manifest	`redis://redis:6379/0` — optional; caches `validateSession()` results across replicas and shares the Next.js fetch cache. Unset => every authed request falls through to the server (correct, just slower).
`HOSTNAME`	Deployment manifest	`0.0.0.0`

Traffic flow

Collector Clusters                          AgentEye Cluster
┌─────────────────┐                    ┌──────────────────────────────────────┐
│ Collector Pod    │                    │                                      │
│ + client cert    │──POST /events────►│ Traefik Public (mTLS) ──► Server    │
│ + API key        │   (port 443)      │                              │       │
└─────────────────┘                    │                              ▼       │
                                       │                          PostgreSQL  │
Browser (VPN)                          │                              ▲       │
┌─────────────────┐                    │                              │       │
│ Developer        │──HTTPS──────────►│ Traefik Internal ────► Dashboard     │
└─────────────────┘   (port 443)      │                                      │
                                       └──────────────────────────────────────┘

Troubleshooting

The most useful commands for deployment-specific issues:

# Pod details and events
kubectl describe pod -n agenteye <pod-name>

# Application logs
kubectl logs -n agenteye -l app=server --tail=200
kubectl logs -n agenteye -l app=dashboard --tail=200

# Certificate status
kubectl describe certificate -n agenteye <cert-name>

# Recent cluster events
kubectl get events -n agenteye --sort-by=.lastTimestamp

For a comprehensive list of common issues and solutions, see enterprise-docs/troubleshooting.md.

Guide	Description
Managed Deployment	Setup guide for managed deployment on your Kubernetes cluster
Getting Started	End-to-end walkthrough with Docker Compose
Deployment	Docker-based deployment, env vars, configuration
Tenant Management	Provision organizations & members with the operator-only `agenteye-orgctl` CLI
GitHub Token Setup	Generate your GitHub PAT to access artifacts
Collector Installation	All collector install methods
Python SDK	Full SDK API reference
API Keys	Creating and managing API keys
Troubleshooting	Common issues and fixes

Support

Email support@exosphere.host for help with deployment.

Managed Deployment on Your Kubernetes Cluster

Single-Pod Deployment: Collector + Application Sidecar on EKS

​Prerequisites

​Run all checks at once

​Deployment shape

​Get the Manifests

​Phase 1 — Third-Party Infrastructure (~30 min)

​1.1 Install cert-manager

​1.2 Install Traefik — Public Ingest Controller

​1.3 Install Traefik — Dashboard Controller

​Whitelist a single IP

​Whitelist multiple IPs

​Update the whitelist at runtime

​1.4 Wait for LoadBalancers

​Phase 2 — Create Secrets (~10 min)

​2.1 Create the namespace

​2.2 Image pull secret

​2.3 PostgreSQL credentials

​2.4 Admin API key

​2.5 Auth configuration (dashboard login)

​2.6 Multi-tenant org isolation key (optional)

​2.7 Verify all secrets

​Phase 3 — Deploy the Application (~5 min)

​3.1 Configure the public hostnames

​3.2 Apply manifests

​3.3 Wait for pods

​3.4 Verify storage

​3.5 Verify certificates

​3.6 Verify CronJobs

​3.7 Verify server started correctly

​3.8 Verify dashboard connected to server

​Phase 4 — Network Access (~5 min)

​4.1 Retrieve LoadBalancer addresses

​4.2 Point DNS at the LoadBalancers

​4.3 Reach the ingest endpoint

​Phase 5 — Issue mTLS Client Certificates (~10 min per cluster)

​5.1 Issue a certificate

​5.1b Alternative delivery: AWS Secrets Manager

​5.2 Verify the certificate works

​5.3 Deliver to the collector cluster

​5.4 Certificate lifecycle

​5.5 Revoke a certificate

​Phase 6 — Certificate Renewal Monitoring (~2 min)

​6.1 Enable Slack notifications (optional)

​6.2 Test the CronJob

​Phase 7 — Verify End-to-End

​7.1 Health check

​7.2 Create scoped collector keys

​7.3 Ingest a test event

​7.4 Verify event appears in dashboard

​7.5 Test the backup CronJob

​7.6 Provision organizations (multi-tenant)

​Post-Deployment Checklist

​Backups

​Configure cloud upload

​Manual backup

​Restore from backup

​Upgrading

​Rollback

​Creating Overlays for New Environments

​Architecture Reference

​Kubernetes resources created

​PostgreSQL sizing and tuning

​Environment variables

​Traffic flow

​Troubleshooting

​Related Documentation

​Support

Prerequisites

Run all checks at once

Deployment shape

Get the Manifests

Phase 1 — Third-Party Infrastructure (~30 min)

1.1 Install cert-manager

1.2 Install Traefik — Public Ingest Controller

1.3 Install Traefik — Dashboard Controller

Whitelist a single IP

Whitelist multiple IPs

Update the whitelist at runtime

1.4 Wait for LoadBalancers

Phase 2 — Create Secrets (~10 min)

2.1 Create the namespace

2.2 Image pull secret

2.3 PostgreSQL credentials

2.4 Admin API key

2.5 Auth configuration (dashboard login)

2.6 Multi-tenant org isolation key (optional)

2.7 Verify all secrets

Phase 3 — Deploy the Application (~5 min)

3.1 Configure the public hostnames

3.2 Apply manifests

3.3 Wait for pods

3.4 Verify storage

3.5 Verify certificates

3.6 Verify CronJobs

3.7 Verify server started correctly

3.8 Verify dashboard connected to server

Phase 4 — Network Access (~5 min)

4.1 Retrieve LoadBalancer addresses

4.2 Point DNS at the LoadBalancers

4.3 Reach the ingest endpoint

Phase 5 — Issue mTLS Client Certificates (~10 min per cluster)

5.1 Issue a certificate

5.1b Alternative delivery: AWS Secrets Manager

5.2 Verify the certificate works

5.3 Deliver to the collector cluster

5.4 Certificate lifecycle

5.5 Revoke a certificate

Phase 6 — Certificate Renewal Monitoring (~2 min)

6.1 Enable Slack notifications (optional)

6.2 Test the CronJob

Phase 7 — Verify End-to-End

7.1 Health check

7.2 Create scoped collector keys

7.3 Ingest a test event

7.4 Verify event appears in dashboard

7.5 Test the backup CronJob

7.6 Provision organizations (multi-tenant)

Post-Deployment Checklist

Backups

Configure cloud upload

Manual backup

Restore from backup

Upgrading

Rollback

Creating Overlays for New Environments

Architecture Reference

Kubernetes resources created

PostgreSQL sizing and tuning

Environment variables

Traffic flow

Troubleshooting

Related Documentation

Support