Skip to content

Host it: Kubernetes

For shared use and production. The server as a Deployment, Redis alongside, sandbox pods created on demand and disposed when each run ends. No CRDs, no operators — stock Kubernetes objects. The manifests ship in the repo under deploy/k8s/, numbered in apply order:

deploy/k8s/
├── 1-namespace.yaml             # agentsmith
├── 2-rbac.yaml                  # ServiceAccount + Role + RoleBinding (pod create/delete)
├── 3-configmap.yaml             # the bootstrap agentsmith.yml (persistence + secret names)
├── 4-secret-template.yaml       # keys, tokens, REDIS_URL — fill in, never commit real values
├── 5-service-redis.yaml
├── 6-deployment-redis.yaml
├── 7-service-server.yaml
├── 8-deployment-server.yaml     # init-container migrate + the server
├── 9-ingress.yaml
├── 10-deployment-dashboard.yaml
├── 11-service-dashboard.yaml
├── 12-pvc-persistence.yaml
├── examples/                    # air-gapped, path-mode, egress proxy; not applied by the command below
└── sandbox-isolation/           # opt-in: sandbox namespace, its RBAC, the sandbox NetworkPolicy

kubectl apply -f deploy/k8s/
kubectl -n agentsmith rollout status deployment/agentsmith-server

What this gets you

  • The server survives node failures because Kubernetes reschedules it, and the run history survives everything because it lives in a relational database (SQLite on a PVC by default; point persistence: at PostgreSQL, MySQL or SQL Server for anything shared, and the same init-container migrates it). To keep the database password out of the ConfigMap, set AGENTSMITH_PERSISTENCE_PROVIDER and AGENTSMITH_PERSISTENCE_CONNECTION as environment variables on both the server and the migrate init-container, for example from the Secret. Set together, they replace the persistence: block; one without the other is refused with a startup finding.
  • Sandbox pods created per repo and toolchain image per run, then deleted. They run in the agentsmith namespace beside the server. The server's ServiceAccount may create and delete pods and read the ResourceQuota there. That is the whole RBAC story. Moving sandboxes to a namespace of their own, with a network policy, is an opt-in step: see Isolating sandboxes.
  • Real capacity control: a ResourceQuota on the namespace sandboxes run in turns "too many runs" into a FIFO queue instead of failures (see the capacity section below).
  • The dashboard as its own Deployment + Service, port 3000 behind the Service.

The shape of the server pod

The load-bearing details from 8-deployment-server.yaml, so you know what you're looking at:

  • An init-container runs agentsmith database migrate --config /app/config/agentsmith.yml before the server starts. Migrations are applied exactly there — the server never migrates its own database on startup, deliberately. It shares the persistence volume with the server (and for an external DB it migrates over the connection string instead).
  • The server container listens on 8081 (/health, which answers for liveness and readiness alike and stays 200 while a subsystem is degraded, so the pod that reports the fault is never pulled out of the service). The body lists each background subsystem with its state and reason, and the startup preflight — the same checks as agent-smith doctor — runs warn-only in the background and reports there too; a degraded tracker shows up there instead of blocking startup.
  • The ConfigMap mounted at /app/config/agentsmith.yml carries the bootstrap slice only, meaning persistence: and secrets:. Everything else (agents, trackers, repos, projects, and the global settings) lives in the database and is edited in the dashboard's Config studio. Putting a full catalog into this ConfigMap does nothing, because the server reads those two blocks and ignores the rest. See Where configuration lives.
  • Secrets come from the agentsmith-secrets Secret (REDIS_URL, provider keys, tracker tokens, webhook secrets, optional Slack/Teams tokens). The ConfigMap names them, the Secret holds the values.
  • Redis takes a password when the Secret has one: set redis-password, and put the same value into redis-url as redis:6379,password=…. Each sandbox reaches Redis as its own ACL user limited to its job's keys, which only protects anything once Redis has a password. Without one, Redis runs as before and the server shows a startup notice. See Requiring a Redis password.
  • Every run executes inside the server pod. In-cluster the sandbox backend is Kubernetes (SANDBOX_TYPE says so explicitly; left unset, the in-cluster service host decides): the server creates each run's sandbox pods in its own namespace and removes them when the run ends. With sandbox isolation turned on, K8S_SANDBOX_NAMESPACE names another namespace and SANDBOX_REDIS_URL gives those pods Redis's full name; the reference manifests set neither. That's why the quota math below counts a run's sandboxes, and why the Role grants pods and nothing else.

  • The images are one release: holgerleichsenring/agent-smith-server, holgerleichsenring/agent-smith-cli (the migrate init-container) and holgerleichsenring/agentsmith-dashboard carry the same tag in the manifests. The sandbox-agent image (holgerleichsenring/agent-smith-sandbox-agent) needs no pin: its tag is derived from the release the server is. Pin it through Configuration → Deployment (deployment.version) or sandbox.agent_version only to run a different tag on purpose; a pin is reported as an advisory finding, never refused.

  • Skills need no pin: every release embeds the catalog it was tested with. The skills volume is an emptyDir the embedded catalog materializes into at startup.

First configuration

The init-container creates the schema. It does not seed configuration, so a fresh deployment comes up with an empty catalog.

Fill it either from an existing file, running the CLI image as a one-shot pod, or through the dashboard:

kubectl -n agentsmith port-forward svc/agentsmith-dashboard 3000:3000

Then switch the rail to Configuration and build the catalog. When you script the import instead, call dotnet AgentSmith.Cli.dll config import ... directly rather than the agentsmith entrypoint. The entrypoint drops privileges with gosu, and that fights a read only, non root securityContext.

Webhooks

The Ingress (9-ingress.yaml) routes your public hostname to the server Service. Point your tracker webhooks at https://agent-smith.your-domain.example/webhook — endpoint details and per-tracker setup on the webhooks page. The webhook shared secrets (GITHUB_WEBHOOK_SECRET, GITLAB_WEBHOOK_TOKEN, AZDO_WEBHOOK_SECRET) go into the Secret next to the API keys.

Sandbox pods

The server creates one pod per repo and toolchain image per run. Contexts of one repo that share an image share one pod, sized to the largest resource envelope among them; a repo whose contexts need two different images gets two pods. Each pod has an init-container that copies the sandbox-agent binary into a shared emptyDir, then the main toolchain container starts and the agent binary takes over the entrypoint. The toolchain image comes from, in order: projects.<name>.sandbox.toolchain_image, projects.<name>.sandbox.images.<language>, the context's stack.image in .agentsmith/contexts/<name>/context.yaml, a built-in per-language table, and finally a generic image that carries git and no toolchain (with a warning that nothing can be built there).

A repository is cloned inside its own sandbox, so the image has to carry git. One that doesn't fails the checkout with an error that says so, instead of a bare exit code.

You don't pre-create anything. Sizing is pipeline-aware: code-changing pipelines use the repo's declared stack.resources (clamped to a hard ceiling), scans and other non-build pipelines get a light fixed profile. When a run finishes (success, failure, cancel) the pods are deleted; a force-killed cancel releases them immediately. A run doesn't provision every repo in the project either: the ScopeRepos step reads the ticket first and spawns sandboxes only for the affected repos.

Pods are labelled with the deployment that owns them. The owner identity is derived from the Redis endpoint; SANDBOX_OWNER_ID names it explicitly. A corpse reaper deletes this deployment's pods whose run is no longer live, on a timer and again at admission, so a crashed server doesn't leave pods holding quota. Two deployments in one namespace never touch each other's pods.

What a sandbox pod runs as

By default a sandbox pod runs as the toolchain image's own user, which is root for every built-in image, with the runtime's default capabilities.

Hardening is a setting for the whole installation: sandbox.hardening_enabled: true in agentsmith.yml, or Harden sandboxes under Configuration → Settings → Sandbox. It applies after a server restart. With it on, a sandbox pod runs as uid and gid 1000, with every Linux capability dropped, the RuntimeDefault seccomp profile, no privilege escalation and no service-account token. Both containers, the agent loader and the toolchain, get the same settings. /root is a writable emptyDir and HOME points at it, so feed configs and git identity are staged where they always were. Git is told that the root-owned /work is a safe directory.

Check before you turn it on:

  • Which projects install system packages in their context's prerequisites (apt-get install, npm install -g, corepack enable). Those fail as uid 1000, and the failure says so. Set run_as_root for them first, or move the install into their toolchain image.
  • Whether a toolchain image ships files under /root. The mount hides them; such a project needs run_as_root too.
  • Most images have no user entry for uid 1000. Tools that look the user up, such as ssh, fail; clone and push go over https and are not affected.

A project opts back to root like this:

projects:
  legacy-app:
    sandbox:
      run_as_root: true

or with run sandboxes as root in the project's Sandbox tab. Its pods then run as the image's own user with the runtime's default capabilities. Seccomp, no escalation and no token stay. While hardening is on, every run of that project shows a sandbox-runs-as-root warning in its preflight, so the choice stays visible. While hardening is off, run_as_root changes nothing.

Isolating sandboxes (opt-in)

Off by default. kubectl apply -f deploy/k8s/ leaves sandbox pods in agentsmith and creates no network policy, so updating Agent Smith changes nothing here. Isolation is a separate step, and so is hardening (sandbox.hardening_enabled, above); either works without the other.

deploy/k8s/sandbox-isolation/ holds three manifests and a README with the full steps:

  • 1-namespace-sandboxes.yaml: the namespace agentsmith-sandboxes. A pod there cannot reference the server's Secret.
  • 2-rbac-sandboxes.yaml: a Role and RoleBinding that give the server's ServiceAccount the same pod and ResourceQuota grants in that namespace.
  • 3-networkpolicy-sandbox.yaml: selects every pod in agentsmith-sandboxes. A sandbox accepts no inbound connection, and reaches only cluster DNS, Redis on 6379, and ports 80 and 443 on public addresses. Private ranges (10/8, 172.16/12, 192.168/16), shared address space (100.64/10), link-local including the metadata service (169.254/16), loopback and Azure's 168.63.129.16 are excepted, so a sandbox cannot reach the server, the dashboard, another sandbox or cloud metadata.

Before you apply the folder, create the sandbox Secrets, image pull secrets and the ResourceQuota in agentsmith-sandboxes, and add the key sandbox-redis-url to agentsmith-secrets (apply-k8s-secret.sh writes it; with a redis-password the URL carries ,password=… as redis-url does). Fill in the allowance section at the top of the network policy: a project that clones from a private git host, restores from a private feed, scans a private target or needs another port fails at clone or restore until its address is allowed there. The air-gapped example reaches nothing until it is filled. The same section covers NodeLocal DNS, OpenShift's DNS and how to keep sandboxes off the server's public ingress.

Then apply the folder and add two variables to the server container, as the README shows:

kubectl apply -f deploy/k8s/sandbox-isolation/

            - name: K8S_SANDBOX_NAMESPACE
              value: "agentsmith-sandboxes"
            - name: SANDBOX_REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: agentsmith-secrets
                  key: sandbox-redis-url

A pod in another namespace cannot resolve the short name redis, so both are needed: with the namespace set and SANDBOX_REDIS_URL unset, startup reports a blocking finding and no sandbox is spawned. Sandbox pods move on the next run; a project secret left only in agentsmith stops resolving.

The policy does nothing unless the cluster's network plugin enforces NetworkPolicy. After applying, check from a sandbox that a run still clones and restores, and that the server service and 169.254.169.254 are not reachable. Domain allowlists need an egress proxy; the egress proxy example sets one up.

Sending sandbox traffic through an egress proxy

Off by default, and it needs sandbox isolation turned on first. deploy/k8s/examples/sandbox-egress-proxy.yaml runs a squid proxy in agentsmith with a domain allowlist, and replaces the sandbox NetworkPolicy with one that lets a sandbox reach only DNS, Redis and the proxy on 3128. It has the same name and namespace as the policy in sandbox-isolation/3-networkpolicy-sandbox.yaml on purpose: policies add up, so a second object would leave 80 and 443 open beside the proxy.

To turn it on:

  1. Edit the allowlist in the example's ConfigMap. It ships with the public git hosts, nuget.org, npm, PyPI, the Go module proxy and crates.io. Add your private git host, feeds and registries. The allowlist is yours; nothing else adds to it.
  2. Set SANDBOX_EGRESS_PROXY on the server, for example http://agentsmith-egress-proxy.agentsmith.svc.cluster.local:3128. The server then gives every sandbox toolchain HTTP_PROXY, HTTPS_PROXY, their lower-case forms and a NO_PROXY for loopback and in-cluster names.
  3. Apply the example after deploy/k8s/sandbox-isolation/, and again after each later apply of that folder, which puts the public web rule back.

A host that is not on the list shows up in the step output as the tool's own proxy error, a 403 from squid. Squid's access log is the record of what was refused. HTTPS is tunnelled, not intercepted: the allowlist judges the host name of each connection.

What does not go through the proxy: render_reference, whose browser guard dials targets directly, so renders fail under this policy and a deployment that needs them keeps the public rule. JVM builds (they read JAVA_TOOL_OPTIONS or settings.xml) and Node's built-in fetch ignore the proxy variables and fail closed.

Which registries a toolchain image may come from

The context's stack.image is written by a model, so it is held to a registry boundary before it is used:

sandbox:
  allowed_registries: ["mcr.microsoft.com/", "ghcr.io/", "registry.acme-org.example/"]
  allow_docker_hub_library: false

allowed_registries is a list of image-reference prefixes. Left empty it means the built-in default, mcr.microsoft.com/ and ghcr.io/. allow_docker_hub_library decides whether an official Docker Hub image with no namespace (node:20-bookworm) is trusted. Unset, it follows the list: trusted while you name no registries, refused once you do, because a named list is a narrowing. A stack.image outside the boundary is skipped with a warning and the chain falls through to the next source. An image you name yourself in toolchain_image or images is your choice and is not checked.

Entries and images are compared the way Docker reads a reference: an image with no registry host lives on docker.io, so sbtscala/ and docker.io/sbtscala/ are the same entry and both trust sbtscala/scala-sbt:…. A bare docker.io/ trusts all of Docker Hub. On a server both settings are edited in the dashboard under Configuration → Settings → Sandbox → Trusted image registries, which also shows the boundary in force; a saved change applies to the next stack.image decision without a restart.

Pulling from a private registry

sandbox:
  image_pull_secrets: [acme-registry-pull]

image_pull_secrets names Kubernetes image pull secrets you created in the namespace sandbox pods run in (agentsmith by default, agentsmith-sandboxes with sandbox isolation). They go on every sandbox pod, init container included, so the agent image and the toolchain image can come from different credentialed registries. The list is global; a project cannot name its own. Kubernetes only: the Docker backend pulls without credentials and says so when a pull fails.

Credentials inside the sandbox

A build step that needs a credential (a private feed, a CLI login) gets it from a Kubernetes Secret you own. The project names the reference; the value never passes through Agent Smith, Redis, the context file or the model:

projects:
  todolist:
    sandbox:
      secrets:
        env:
          FEED_TOKEN: "todolist-build:feed-token"     # secretName:key
        files:
          - mount: /secrets/signing.key
            secret: todolist-build
            key: signing-key

The Secret lives in the namespace sandbox pods run in: agentsmith by default, agentsmith-sandboxes with sandbox isolation. With isolation on, a reference to a Secret that only exists in the server's namespace, agentsmith-secrets among them, does not resolve and the pod does not start. env entries become environment variables from a secretKeyRef; files entries are mounted read-only at mount. The command that uses them lives in the context's prerequisites. The run preflight checks the declaration without touching the cluster: a reference without a single :, a duplicate or a clashing mount is named before anything starts. Once the pod is up, a second check asserts by name that every declared variable and file arrived, and fails naming the missing one, never its value. On the Docker and in-process backends nothing is injected, and the preflight reports "NOT INJECTED" as a warning instead of failing.

Updating

An update changes no infrastructure: sandbox pods stay in the namespace they run in, and the sandbox namespace and network policy are only created when you apply deploy/k8s/sandbox-isolation/ yourself. If you turned isolation on, keep the two server variables in your copy of the Deployment; the shipped 8-deployment-server.yaml does not carry them.

Bump the tag in the manifests' images (server, the migrate init-container, the dashboard), then:

kubectl apply -f deploy/k8s/
kubectl -n agentsmith rollout status deployment/agentsmith-server

RollingUpdate is the default strategy. The migrate init-container applies any new migrations before the new server accepts traffic. Runs execute in the server pod, so a run still going on the replaced pod ends with it and the corpse reaper removes its sandbox pods; a run parked on a question holds no pod and resumes on the new one. Roll out between runs when you can.

Resources

The server itself is cheap on CPU — it waits on LLM calls and shuffles events — but it is not cheap on memory: ASP.NET + SignalR + EF + the skills catalog + live event streams need room. Give the server pod a request of at least 512Mi and a limit of 1–1.5Gi. Below that it OOMKills under normal load, and every OOM-restart reaps the in-flight run (surfacing as a bogus "cancelled"), truncates the durable event trail, and leaves the run's sandbox pods to the corpse reaper. The startup preflight WARNs when the pod's memory ceiling is under the 512Mi floor. Remember the namespace ResourceQuota counts the server's request/limit too.

The interesting sizing is per run: a run's pods are its sandboxes, one per repo and toolchain image. Build sandboxes default to a 1Gi request with a 4Gi limit as the OOM guard. Keep requests honest, not minimal — see the warning below.

Capacity quota: count requests, not limits

The capacity probe reads the namespace ResourceQuota and admits a run only when its whole footprint (one sandbox per repo and toolchain image) still fits. It compares only the quota keys present in status.hard — so the quota's shape decides what "capacity" means. A run that doesn't fit is queued (strict FIFO, one entry per ticket, visible amber in the dashboard with its position) and launched when capacity frees.

Quota the namespace on requests, not limits. Requests are what the scheduler packs nodes by — i.e. what the cluster actually provisions and what costs money. A quota on limits.memory reserves the theoretical worst case for a pod's whole runtime: five default pods "use" 20Gi of quota while their real reservation is a fraction of that, and runs queue behind capacity nobody is consuming.

Worked example for an 8-CPU / 20Gi-class cluster (adjust to yours; leave headroom for the server, Redis, dashboard, and system pods):

apiVersion: v1
kind: ResourceQuota
metadata:
  name: agentsmith-capacity
  namespace: agentsmith    # agentsmith-sandboxes with sandbox isolation
spec:
  hard:
    requests.cpu: "6"        # ~75% of 8 CPUs — headroom for the platform pods
    requests.memory: 14Gi    # ~70% of 20Gi — same reasoning
    pods: "12"               # the deterministic backpressure knob

No limits.* keys: limits stay on the pods purely as the OOM guard, they no longer count against capacity. The pods key is the deterministic backpressure knob — the Kubernetes analog of Docker's max_concurrent_sandboxes.

That analogy is the whole relationship: max_concurrent_sandboxes in the sandbox: settings (maxConcurrentSandboxes on the wire and in the dashboard, MaxConcurrentSandboxes in the stored document and the C# model, SANDBOX_MAX_CONCURRENT as the environment-variable fallback for an empty store) is read by the Docker capacity probe only. Here it does nothing — this quota is what bounds your sandboxes. See where the bound is set.

Two warnings:

  • Keep requests honest, not minimal. Node-pressure eviction kills Burstable pods ranked by usage-above-request first. A build sandbox declared at 512Mi that peaks at 3–4Gi during dotnet build is the prime eviction victim — that resurrects the "sandbox vanished" failure class. The build-sandbox default stays at a 1Gi request with a 4Gi limit as the OOM guard.
  • The quota lives in your cluster config, not in this repo. Applying the requests-based quota is an operator step in whatever repo manages your namespace; size it together with the server pod's own requests in deploy/k8s/8-deployment-server.yaml, which the quota counts too.

Each finished run shows its reserved capacity-time (memory request × pod lifetime, in Gi·minutes) next to the LLM cost on the run detail page — reservation, not measured consumption — so you can see whether a run was expensive in tokens or in pods. More on the capacity page.

Next