Enable and operate agent code sandboxes

Prepare nodes, object storage and networking, turn on the isolated workspaces where agents run code, control who can use them, and turn them off again.

Agentic Friendly

The agent sandbox gives agents an isolated workspace in which they run commands, install packages and, optionally, serve a live app preview. Each workspace is a pod in the coreai-sandboxes namespace that runs inside a Kata Containers micro-VM. The sandbox is off by default.

This playbook is for the platform engineer who prepares the cluster, turns the sandbox on and operates it. Once the site enables it, every tenant can use it; a platform administrator can turn it off for every tenant, and a tenant administrator for their own organization, without a render. See 10. Turn the sandbox off without a render.

What the platform deploys

ComponentParent application and waveWhat it installs
kata-deploycommon, wave 5A DaemonSet that installs the Kata Containers runtime on sandbox nodes, and the kata-qemu RuntimeClass that pins Kata pods to those nodes and charges the VM overhead to each pod
agent-sandboxcoreai, wave 10The upstream agent-sandbox controller in agent-sandbox-system, its four CRDs, the coreai-sandboxes namespace, a ResourceQuota and LimitRange, and an ingress NetworkPolicy
llm-backendcoreai, wave 11The SandboxTemplate and SandboxWarmPool, the reaper CronJob, the preview proxy, and, on Rook-Ceph, the sandbox-s3 bucket claim
portalcoreai, wave 8The Build chat mode at /app/vibe and, with previews, the preview host suffix

Inside agent-sandbox, the controller namespace syncs at wave -2 and the CRDs at wave -1, before the controller and the sandbox namespace. A single push-manifests therefore installs everything in the right order.

The BSQAI API creates one sandbox claim per chat. The claim takes a ready pod from the warm pool, and the pool starts a replacement. When a session is idle or reaches its maximum lifetime, the reaper saves the workspace to the tenant's object storage bucket and stops the pod. The next message in the chat restores the workspace into a new pod. Session lifetimes, workspace size limits and preview availability are tenant settings, not platform variables.

Sandbox pods run as a non-root user with all Linux capabilities dropped, the RuntimeDefault seccomp profile, and no Kubernetes service-account token. The agent-sandbox controller has cluster-wide permissions on pods, services, persistent volume claims and network policies. It writes only in coreai-sandboxes and, for its leader lease, in agent-sandbox-system.

Recommended path

Prepare nodes → Mirror artifacts → Prepare object storage → Prepare the network → Size the namespace → Prepare app preview hosts → Turn on the sandbox → Verify

1. Prepare sandbox nodes

Kata runs each workspace as a lightweight virtual machine, so sandbox nodes need hardware virtualization: /dev/kvm on bare metal, or nested virtualization on cloud VMs. On Azure, choose a VM size that exposes nested virtualization, such as the Dsv5 series.

The platform installs Kata with the kata-deploy DaemonSet in kube-system, only on nodes that carry the sandbox placement label. Sandbox pods select the same label and tolerate a NoSchedule taint with the same key and value, so taint the nodes to keep other workloads off them.

SettingOverride variableDefault
Placement label keyKATA_DEPLOY_PLACEMENT_NODE_SELECTOR_KEYworkload
Placement label valueKATA_DEPLOY_PLACEMENT_NODE_SELECTOR_VALUEsandbox
Kata RuntimeClassKATA_DEPLOY_RUNTIME_CLASS_NAMEkata-qemu

Label and taint each sandbox node, for example:

kubectl label node <node> workload=sandbox
kubectl taint node <node> workload=sandbox:NoSchedule

The warm pool keeps LLM_BACKEND_SANDBOX_WARMPOOL_REPLICAS pods running at all times, two by default. A node pool that can scale to zero therefore keeps at least one node running while the sandbox is enabled. Include the warm pool pods when sizing the pool.

2. Mirror the artifacts

The platform pulls every sandbox artifact from the platform registry and downloads nothing from the internet at deploy time. Mirror these artifacts of the release before enabling the sandbox on a disconnected or restricted site:

ArtifactDefault registry path
Kata Containers Helm chart<registry>/coreai/helm/kata-deploy:4.2.0
Kata Containers installer image<registry>/coreai/platform/kata-containers/kata-deploy:4.2.0
agent-sandbox controller image<registry>/coreai/platform/agent-sandbox/agent-sandbox-controller:v1.0.5
Sandbox runtime image<registry>/coreai/backend/sandbox-runtime at the digest pinned by the release

The repository prefixes follow COMMON_REGISTRY_HELM_REPOSITORY, COMMON_REGISTRY_PLATFORM_REPOSITORY and COMMON_REGISTRY_BACKEND_REPOSITORY. See Artifact delivery with Harbor for the restricted-network path.

The sandbox runtime image is about 1.1 GB, because it carries a headless Chromium browser, Playwright and fonts, so that agents can take screenshots without internet access. Plan space for it in the mirror and on the disk of each sandbox node. Each sandbox node pulls the image before its first sandbox pod starts.

3. Prepare object storage

The BSQAI API saves a workspace in the bucket its session records when it starts:

  • For a tenant, the tenant's own <tenantId>-data bucket, under sandbox/. The tenant operator creates it when the tenant keeps the Object storage bucket service, see Provision a tenant. A tenant without a ready storage bucket cannot start a sandbox.
  • For the Global organization, the sandbox-workspaces bucket, set by LLM_BACKEND_SANDBOX_BUCKET.

Prepare both:

  • With rook-ceph, the platform creates sandbox-workspaces through the sandbox-s3 ObjectBucketClaim, which exists on every Rook-Ceph site, also while the sandbox is off. The BSQAI API reaches tenant buckets with its own Ceph user; there is nothing to prepare.
  • With external-s3, create sandbox-workspaces before the first render and give the BSQAI API account read, write, list and delete access to it. The BSQAI API reaches tenant buckets with the tenant operator's provisioner account, so enable the tenant operator and set its BSQAI_TENANT_OPERATOR_S3_* pair. See External S3 object storage.

4. Prepare the network

The sandbox network policy controls what workspaces can reach. LLM_BACKEND_SANDBOX_EGRESS_MODE selects the outbound traffic of every sandbox pod on the site:

ModeOutbound traffic
internet (default)HTTPS (TCP 443) to public IPv4 addresses, plus DNS to the cluster DNS pods. Agents can install packages from public registries such as PyPI and npm.
blockedNone. Agents cannot install packages at run time.

internet excludes the private ranges 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, the shared address space 100.64.0.0/10 that holds Scaleway Kapsule pods, the link-local range 169.254.0.0/16 that holds the cloud metadata endpoint, and multicast, so workspaces cannot reach cluster services, other pods, private networks or a cloud metadata endpoint. Sandbox pods do not use the platform [proxy] settings: on a site that reaches the internet only through an HTTP proxy, internet mode gives agents no package access.

The mode applies to the whole site; tenants cannot choose another one. A tenant whose data must not leave the cluster through the sandbox has its sandbox turned off instead, see 10. Turn the sandbox off without a render.

Configure the cluster so that the policy works as intended:

  • Network policy engine. The cluster must enforce Kubernetes network policies; without an engine, nothing restricts sandbox traffic. Allow rules add up, so a cluster-wide policy that allows all outbound traffic also removes the sandbox restrictions. Calico is available as a separate opt-in, see Network policies.

  • Cluster DNS labels. Workspaces may query only the DNS pods that [network_policy] names: dns_namespace, dns_pod_label_key and dns_pod_label_value, with defaults kube-system, k8s-app and kube-dns. Canonical Kubernetes labels CoreDNS k8s-app=coredns; set NETWORK_POLICY_DNS_POD_LABEL_VALUE=coredns there. The platform's other network policies use the same settings.

  • Cilium socket load balancing. Cilium's socket-level load balancer translates Service addresses in the host kernel's socket calls, which a process inside a Kata VM never makes. A workspace's DNS query then leaves the pod addressed to the DNS Service, and the sandbox policy, which allows only the DNS pods, drops it. On Cilium with socket load balancing or kube-proxy replacement enabled, set socketLB.hostNamespaceOnly=true (bpf-lb-sock-hostns-only: "true" in the cilium-config ConfigMap) and restart the Cilium agents, as the Cilium Kata Containers guide requires. Check whether a cluster needs it from a Cilium agent:

    kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
      cilium-dbg status --verbose | grep "Socket LB"

    Socket LB: Enabled with Socket LB Coverage: Full needs the setting; Hostns-only means it is in place. Canonical Kubernetes manages Cilium itself, and its Cilium annotations do not include this option, so check the coverage again after a Kubernetes upgrade.

5. Size the sandbox namespace

Three values decide how many sandboxes run at once:

  • the pod size, AGENT_SANDBOX_SIZING_POD_*: 250m CPU and 512Mi memory requested, 1 CPU and 2Gi memory limit;
  • the Kata VM overhead, KATA_DEPLOY_SIZING_OVERHEAD_*: 250m CPU and 160Mi memory, added to the requests and limits of every sandbox pod;
  • the namespace quota, AGENT_SANDBOX_SIZING_QUOTA_*.

The default quota fits 300 pods: pods=300, 150 CPU and 197Gi memory in requests, 375 CPU and 647Gi memory in limits. Warm pool pods count against it, and all tenants share it. When the quota is full, the BSQAI API cannot start a new session until the reaper stops an idle one.

The quota is only a ceiling. The node pool decides how many sandboxes actually run: a pod that the quota admits but no node can hold stays Pending, and the session fails when the BSQAI API stops waiting for it. Set the node pool's autoscaler maximum so that it holds the quota, or lower the quota to what the pool holds.

To size the quota for N pods, round up these values:

SettingFormulaN = 100 with defaults
AGENT_SANDBOX_SIZING_QUOTA_PODSN100
AGENT_SANDBOX_SIZING_QUOTA_REQUESTS_CPUN × (pod CPU request + overhead CPU)50
AGENT_SANDBOX_SIZING_QUOTA_REQUESTS_MEMORYN × (pod memory request + overhead memory)66Gi
AGENT_SANDBOX_SIZING_QUOTA_LIMITS_CPUN × (pod CPU limit + overhead CPU)125
AGENT_SANDBOX_SIZING_QUOTA_LIMITS_MEMORYN × (pod memory limit + overhead memory)216Gi

Size the node pool for the request totals. Each sandbox pod also requests 256Mi and is limited to 2Gi of ephemeral storage on its node.

Tune the overhead from measured use with kubectl top pod on a sandbox node. The LimitRange rejects a single pod larger than the pod limits, so a change to AGENT_SANDBOX_SIZING_POD_* applies to the quota, the LimitRange and the SandboxTemplate together.

6. Prepare app preview hosts

App previews are off by default. With LLM_BACKEND_SANDBOX_PREVIEW_ENABLED=true, the platform serves each preview at <chat-id>.sandbox.<base-domain>; LLM_BACKEND_SANDBOX_PREVIEW_HOST_PREFIX changes sandbox. Preview hosts use the same Envoy Gateway HTTPS listener as the rest of the platform. A *.<base-domain> wildcard certificate covers only one DNS label, so these hosts need their own *.sandbox.<base-domain> name, in DNS and in the certificate:

  • When cert-manager issues the platform certificate, the platform adds the name to it, whichever DNS_PROVIDER the site uses, including the self-signed certificate of DNS_PROVIDER=none. Where External DNS manages the zone, the platform also publishes the name on the gateway.
  • With operator-managed DNS or custom TLS, create a *.sandbox.<base-domain> record that resolves to the shared gateway, and include the name in the supplied certificate. See Manage DNS and TLS yourself.

Preview hosts are reachable from any network that reaches the gateway, but the BSQAI API preview proxy admits only browsers that hold a preview session. The Portal obtains a single-use ticket for the chat, and the proxy exchanges it for a session cookie that it checks on every request. Only the Portal origin can embed a preview, and the proxy does not forward the sign-in and Portal session cookies to the app in the sandbox. A preview session lasts 15 minutes by default. Signing out of the Portal ends the user's previews, and an open preview stops when its session ends.

7. Turn on the sandbox

Set the three switches in platform.env, and the optional preview and egress settings:

KATA_DEPLOY_COMPONENT_ENABLED=true
AGENT_SANDBOX_COMPONENT_ENABLED=true
LLM_BACKEND_SANDBOX_ENABLED=true

# Optional
LLM_BACKEND_SANDBOX_PREVIEW_ENABLED=true
LLM_BACKEND_SANDBOX_PREVIEW_HOST_PREFIX=sandbox
LLM_BACKEND_SANDBOX_EGRESS_MODE=internet

The render stops when LLM_BACKEND_SANDBOX_ENABLED=true and either component is off. Keep all three set for every later render: a render without them removes Kata from the nodes, and Argo CD prunes the sandbox controller together with every workspace.

Render and publish the manifests:

bsqai apply all -r
bsqai push-manifests "Enable the agent sandbox"

Argo CD syncs kata-deploy in the common application, then agent-sandbox, the Portal and the BSQAI API in the coreai application. The agent-sandbox application skips client-side validation, because the upstream CRDs carry validation rules that client-side validation rejects.

8. Verify the sandbox

Confirm that the controller, the CRDs and the warm pool are in place. Warm workspaces wait unassigned, two by default, and run only when Kata works on a sandbox node:

kubectl get runtimeclass kata-qemu
kubectl -n agent-sandbox-system get deployment agent-sandbox-controller
kubectl get crd sandboxes.agents.x-k8s.io sandboxclaims.extensions.agents.x-k8s.io \
  sandboxtemplates.extensions.agents.x-k8s.io sandboxwarmpools.extensions.agents.x-k8s.io
kubectl -n coreai-sandboxes get pods -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n coreai-sandboxes describe resourcequota sandbox-quota

With egress mode internet, resolve a public name from one of the unassigned pods:

kubectl -n coreai-sandboxes exec <warm-pool-pod> -- \
  python3 -c 'import socket; print(socket.gethostbyname("pypi.org"))'

The command prints an IP address. A Temporary failure in name resolution error points to the DNS labels or the Cilium setting in 4. Prepare the network.

Then run an end-to-end check as a Portal user:

  1. Open https://portal.<base-domain>/app/vibe and ask the agent to create a small web app.
  2. Confirm that the generated files appear and that kubectl -n coreai-sandboxes get sandboxclaims lists a claim for the chat.
  3. With previews enabled, ask the agent to start the app. The preview loads in the Portal over HTTPS without a certificate warning.
  4. Leave the chat idle past the tenant's idle timeout, 15 minutes by default. Confirm that the pod stops and that objects for the chat appear under sandbox/<tenantId>/<chat-id>/ in the tenant's <tenantId>-data bucket.
  5. Send another message in the chat. Confirm that the files are restored in a new pod.

9. Troubleshooting

SymptomLikely causeCheck
The render stops with Invalid agent sandbox configurationA required component is off, the preview prefix is not a DNS label, or the egress mode is unknownThe CLI message names the variable to set
The render stops with LLM_BACKEND_SANDBOX_PREVIEW_ENABLED is true but LLM_BACKEND_SANDBOX_ENABLED is falsePreviews are on without the sandboxTurn the sandbox on, or previews off
Warm pool pods stay PendingNo ready sandbox node, a missing taint toleration, or a full node poolkubectl -n coreai-sandboxes describe pod <pod> and the cluster autoscaler events
Pod creation is Forbidden with exceeded quotaThe namespace quota is fullkubectl -n coreai-sandboxes describe resourcequota sandbox-quota, then resize as in step 5
Sessions fail while sandbox pods wait in PendingThe quota admits more pods than the node pool holdsThe cluster autoscaler maximum of the sandbox node pool; see step 5
Sandbox calls in one tenant fail with tenant_data_binding_unavailableThe tenant has no ready storage bucket, or, on external S3, the BSQAI API has no tenant bucket accessThe tenant's Object storage bucket service and Tenant bucket refused lines in the BSQAI API log; on external S3, the tenant operator and its BSQAI_TENANT_OPERATOR_S3_* pair
Idle sandboxes are never stopped and the quota stays fullThe reaper cannot save workspaces: missing bucket, missing permissions on sandbox-workspaces or the tenant bucket, or an S3 certificate that the platform CA bundle does not trustLogs of the latest *-sandbox-reaper job in the BSQAI API namespace
Pods fail with a runtime handler errorThe RuntimeClass handler does not match the shim that Kata installedkubectl get runtimeclass and the kata-deploy pod logs in kube-system
Kata pods fail on a nodeThe node does not expose /dev/kvmThe VM size and the kata-deploy pod logs on that node
Package installs fail in the sandboxEgress is blocked, DNS is dropped, or the site reaches the internet only through a proxyLLM_BACKEND_SANDBOX_EGRESS_MODE, the DNS check in step 8, and the site firewall
Screenshots fail in the sandbox, or text in them renders as boxesThe pinned runtime image is older than the release and has no browser or fontsThe digest that kubectl -n coreai-sandboxes get sandboxtemplates -o jsonpath='{..image}' prints, against [sandbox] runtime_image in the release's src/manifests/source/coreai/llm-backend/variables.toml
Users see that the sandbox is turned offA platform or tenant administrator turned it offGET /v1/sandbox/capabilities as a member of the tenant; see step 10
The preview shows a certificate errorThe certificate lacks *.<preview_host_prefix>.<base-domain>The subject alternative names of the served certificate
The preview host does not resolveThe wildcard DNS record is missingnslookup test.<preview_host_prefix>.<base-domain>
The preview returns 404The per-chat HTTPRoute is not accepted by the gatewaykubectl -n coreai-sandboxes get httproutes and the route status.parents conditions
The preview asks to reload after sign-out or after some minutesExpected behavior: the preview session expired or was revoked, and the Portal requests a new ticket on retryThe tenant's preview session lifetime, preview_session_ttl_seconds in the sandbox configuration
Sandbox code reaches cluster servicesThe cluster does not enforce NetworkPolicyThe network policy engine in use; see step 4

10. Turn the sandbox off without a render

Besides the site variables, the sandbox has two switches in the Portal. Both are on by default, and the sandbox runs only while both are on.

SwitchWho sets itApplies toWhere
Platform switchA platform administratorEvery tenant, the Global organization includedAdministration → Platform → Configuration → Code sandbox
Tenant switchA tenant administrator with tenant.updateTheir own organizationAdministration → Tenant → Code sandbox

While the platform switch is off, no tenant can use the sandbox and no tenant administrator can turn it back on. Resetting a tenant's configuration keeps its tenant switch.

When either switch turns off:

  • The BSQAI API refuses new sandbox work, through the Portal and the API alike, with the error code sandbox_disabled. Agent tools report that the sandbox is turned off.
  • Open app previews close at once: every tenant's for the platform switch, the tenant's own for the tenant switch.
  • The reaper saves the workspace of every affected running sandbox to object storage and stops its pod on its next run, within about two minutes.
  • The Portal hides Build. Users can still open earlier Build chats and download the files they produced.

Turning the switch back on restores the sandbox; the next message in a chat restores its workspace into a new pod.

Each change is written to the BSQAI API log with the acting user and the old and new value: action=platform.configuration.update for the platform switch, action=sandbox.settings.update for the tenant settings. To check the state of a tenant, call GET /v1/sandbox/capabilities as one of its members.

11. Turn the sandbox off for the site

Drain the sessions first, then remove the sandbox in two renders. Only the reaper saves a running workspace before its pod stops, and the first render removes the reaper, so a session still running at that point can lose changes made since its last save.

  1. Turn the platform switch off, as in 10. Turn the sandbox off without a render. Keep every component enabled, so the reaper keeps running. Users can no longer start sandbox work, and the reaper saves and stops every running sandbox on its next run.

  2. Wait until no session is left. Idle warm pool pods stay, but carry no claim, so check the claims rather than the pods:

    kubectl -n coreai-sandboxes get sandboxclaims

    The command returns no resources. In the logs of the latest *-sandbox-reaper job in the BSQAI API namespace, each session ends with reaped sandbox; a reap failed, will retry next run line means that a workspace is not saved yet, so wait for the next run and fix the cause first, see 9. Troubleshooting.

  3. Set LLM_BACKEND_SANDBOX_ENABLED=false and LLM_BACKEND_SANDBOX_PREVIEW_ENABLED=false, then run bsqai apply all -r and bsqai push-manifests. The Portal hides Build, and the BSQAI API removes its sandbox template, warm pool, reaper and preview proxy.

  4. Wait until kubectl -n coreai-sandboxes get sandboxes,pods returns no resources.

  5. Set AGENT_SANDBOX_COMPONENT_ENABLED=false and KATA_DEPLOY_COMPONENT_ENABLED=false, then render and push again.

The platform switch stays off. To turn the sandbox on again later, enable the components as in 7. Turn on the sandbox, then turn the platform switch back on.

Disabling agent-sandbox prunes the controller, its CRDs and the coreai-sandboxes namespace. Deleting the CRDs deletes every remaining sandbox resource. Disabling kata-deploy removes the RuntimeClass and the Kata installer.

Saved workspaces stay in place on both providers, in the tenant buckets and in sandbox-workspaces. On Rook-Ceph, the platform keeps the sandbox-s3 bucket claim while the sandbox is off, so turning the sandbox on again restores existing chats. Delete the sandbox/ objects when the saved workspaces are no longer needed.

12. Upgrade the sandbox components

Each platform release pins the Kata Containers chart, the agent-sandbox controller and CRDs, and the sandbox runtime image. The BSQAI API in the same release uses a client that matches those CRDs. Upgrade them only through a platform release, and do not override a component version on its own.

Mirror the new versions of the artifacts in step 2 before the upgrade. Do not prune the controller or the CRDs during an upgrade while sandbox resources exist.

On this page