Enable and operate agent code sandboxes
Prepare nodes, object storage and networking, turn on the isolated workspaces where agents run code, control who can use them, and turn them off again.
The agent sandbox gives agents an isolated workspace in which they run commands, install packages and, optionally, serve a live app preview. Each workspace is a pod in the coreai-sandboxes namespace that runs inside a Kata Containers micro-VM. The sandbox is off by default.
This playbook is for the platform engineer who prepares the cluster, turns the sandbox on and operates it. Once the site enables it, every tenant can use it; a platform administrator can turn it off for every tenant, and a tenant administrator for their own organization, without a render. See 10. Turn the sandbox off without a render.
What the platform deploys
| Component | Parent application and wave | What it installs |
|---|---|---|
kata-deploy | common, wave 5 | A DaemonSet that installs the Kata Containers runtime on sandbox nodes, and the kata-qemu RuntimeClass that pins Kata pods to those nodes and charges the VM overhead to each pod |
agent-sandbox | coreai, wave 10 | The upstream agent-sandbox controller in agent-sandbox-system, its four CRDs, the coreai-sandboxes namespace, a ResourceQuota and LimitRange, and an ingress NetworkPolicy |
llm-backend | coreai, wave 11 | The SandboxTemplate and SandboxWarmPool, the reaper CronJob, the preview proxy, and, on Rook-Ceph, the sandbox-s3 bucket claim |
portal | coreai, wave 8 | The Build chat mode at /app/vibe and, with previews, the preview host suffix |
Inside agent-sandbox, the controller namespace syncs at wave -2 and the CRDs at wave -1, before the controller and the sandbox namespace. A single push-manifests therefore installs everything in the right order.
The BSQAI API creates one sandbox claim per chat. The claim takes a ready pod from the warm pool, and the pool starts a replacement. When a session is idle or reaches its maximum lifetime, the reaper saves the workspace to the tenant's object storage bucket and stops the pod. The next message in the chat restores the workspace into a new pod. Session lifetimes, workspace size limits and preview availability are tenant settings, not platform variables.
Sandbox pods run as a non-root user with all Linux capabilities dropped, the RuntimeDefault seccomp profile, and no Kubernetes service-account token. The agent-sandbox controller has cluster-wide permissions on pods, services, persistent volume claims and network policies. It writes only in coreai-sandboxes and, for its leader lease, in agent-sandbox-system.
Recommended path
Prepare nodes → Mirror artifacts → Prepare object storage → Prepare the network → Size the namespace → Prepare app preview hosts → Turn on the sandbox → Verify
1. Prepare sandbox nodes
Kata runs each workspace as a lightweight virtual machine, so sandbox nodes need hardware virtualization: /dev/kvm on bare metal, or nested virtualization on cloud VMs. On Azure, choose a VM size that exposes nested virtualization, such as the Dsv5 series.
The platform installs Kata with the kata-deploy DaemonSet in kube-system, only on nodes that carry the sandbox placement label. Sandbox pods select the same label and tolerate a NoSchedule taint with the same key and value, so taint the nodes to keep other workloads off them.
| Setting | Override variable | Default |
|---|---|---|
| Placement label key | KATA_DEPLOY_PLACEMENT_NODE_SELECTOR_KEY | workload |
| Placement label value | KATA_DEPLOY_PLACEMENT_NODE_SELECTOR_VALUE | sandbox |
| Kata RuntimeClass | KATA_DEPLOY_RUNTIME_CLASS_NAME | kata-qemu |
Label and taint each sandbox node, for example:
kubectl label node <node> workload=sandbox
kubectl taint node <node> workload=sandbox:NoScheduleThe warm pool keeps LLM_BACKEND_SANDBOX_WARMPOOL_REPLICAS pods running at all times, two by default. A node pool that can scale to zero therefore keeps at least one node running while the sandbox is enabled. Include the warm pool pods when sizing the pool.
2. Mirror the artifacts
The platform pulls every sandbox artifact from the platform registry and downloads nothing from the internet at deploy time. Mirror these artifacts of the release before enabling the sandbox on a disconnected or restricted site:
| Artifact | Default registry path |
|---|---|
| Kata Containers Helm chart | <registry>/coreai/helm/kata-deploy:4.2.0 |
| Kata Containers installer image | <registry>/coreai/platform/kata-containers/kata-deploy:4.2.0 |
| agent-sandbox controller image | <registry>/coreai/platform/agent-sandbox/agent-sandbox-controller:v1.0.5 |
| Sandbox runtime image | <registry>/coreai/backend/sandbox-runtime at the digest pinned by the release |
The repository prefixes follow COMMON_REGISTRY_HELM_REPOSITORY, COMMON_REGISTRY_PLATFORM_REPOSITORY and COMMON_REGISTRY_BACKEND_REPOSITORY. See Artifact delivery with Harbor for the restricted-network path.
The sandbox runtime image is about 1.1 GB, because it carries a headless Chromium browser, Playwright and fonts, so that agents can take screenshots without internet access. Plan space for it in the mirror and on the disk of each sandbox node. Each sandbox node pulls the image before its first sandbox pod starts.
3. Prepare object storage
The BSQAI API saves a workspace in the bucket its session records when it starts:
- For a tenant, the tenant's own
<tenantId>-databucket, undersandbox/. The tenant operator creates it when the tenant keeps the Object storage bucket service, see Provision a tenant. A tenant without a ready storage bucket cannot start a sandbox. - For the Global organization, the
sandbox-workspacesbucket, set byLLM_BACKEND_SANDBOX_BUCKET.
Prepare both:
- With
rook-ceph, the platform createssandbox-workspacesthrough thesandbox-s3ObjectBucketClaim, which exists on every Rook-Ceph site, also while the sandbox is off. The BSQAI API reaches tenant buckets with its own Ceph user; there is nothing to prepare. - With
external-s3, createsandbox-workspacesbefore the first render and give the BSQAI API account read, write, list and delete access to it. The BSQAI API reaches tenant buckets with the tenant operator's provisioner account, so enable the tenant operator and set itsBSQAI_TENANT_OPERATOR_S3_*pair. See External S3 object storage.
4. Prepare the network
The sandbox network policy controls what workspaces can reach. LLM_BACKEND_SANDBOX_EGRESS_MODE selects the outbound traffic of every sandbox pod on the site:
| Mode | Outbound traffic |
|---|---|
internet (default) | HTTPS (TCP 443) to public IPv4 addresses, plus DNS to the cluster DNS pods. Agents can install packages from public registries such as PyPI and npm. |
blocked | None. Agents cannot install packages at run time. |
internet excludes the private ranges 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, the shared address space 100.64.0.0/10 that holds Scaleway Kapsule pods, the link-local range 169.254.0.0/16 that holds the cloud metadata endpoint, and multicast, so workspaces cannot reach cluster services, other pods, private networks or a cloud metadata endpoint. Sandbox pods do not use the platform [proxy] settings: on a site that reaches the internet only through an HTTP proxy, internet mode gives agents no package access.
The mode applies to the whole site; tenants cannot choose another one. A tenant whose data must not leave the cluster through the sandbox has its sandbox turned off instead, see 10. Turn the sandbox off without a render.
Configure the cluster so that the policy works as intended:
-
Network policy engine. The cluster must enforce Kubernetes network policies; without an engine, nothing restricts sandbox traffic. Allow rules add up, so a cluster-wide policy that allows all outbound traffic also removes the sandbox restrictions. Calico is available as a separate opt-in, see Network policies.
-
Cluster DNS labels. Workspaces may query only the DNS pods that
[network_policy]names:dns_namespace,dns_pod_label_keyanddns_pod_label_value, with defaultskube-system,k8s-appandkube-dns. Canonical Kubernetes labels CoreDNSk8s-app=coredns; setNETWORK_POLICY_DNS_POD_LABEL_VALUE=corednsthere. The platform's other network policies use the same settings. -
Cilium socket load balancing. Cilium's socket-level load balancer translates Service addresses in the host kernel's socket calls, which a process inside a Kata VM never makes. A workspace's DNS query then leaves the pod addressed to the DNS Service, and the sandbox policy, which allows only the DNS pods, drops it. On Cilium with socket load balancing or kube-proxy replacement enabled, set
socketLB.hostNamespaceOnly=true(bpf-lb-sock-hostns-only: "true"in thecilium-configConfigMap) and restart the Cilium agents, as the Cilium Kata Containers guide requires. Check whether a cluster needs it from a Cilium agent:kubectl -n kube-system exec ds/cilium -c cilium-agent -- \ cilium-dbg status --verbose | grep "Socket LB"Socket LB: EnabledwithSocket LB Coverage: Fullneeds the setting;Hostns-onlymeans it is in place. Canonical Kubernetes manages Cilium itself, and its Cilium annotations do not include this option, so check the coverage again after a Kubernetes upgrade.
5. Size the sandbox namespace
Three values decide how many sandboxes run at once:
- the pod size,
AGENT_SANDBOX_SIZING_POD_*:250mCPU and512Mimemory requested,1CPU and2Gimemory limit; - the Kata VM overhead,
KATA_DEPLOY_SIZING_OVERHEAD_*:250mCPU and160Mimemory, added to the requests and limits of every sandbox pod; - the namespace quota,
AGENT_SANDBOX_SIZING_QUOTA_*.
The default quota fits 300 pods: pods=300, 150 CPU and 197Gi memory in requests, 375 CPU and 647Gi memory in limits. Warm pool pods count against it, and all tenants share it. When the quota is full, the BSQAI API cannot start a new session until the reaper stops an idle one.
The quota is only a ceiling. The node pool decides how many sandboxes actually run: a pod that the quota admits but no node can hold stays Pending, and the session fails when the BSQAI API stops waiting for it. Set the node pool's autoscaler maximum so that it holds the quota, or lower the quota to what the pool holds.
To size the quota for N pods, round up these values:
| Setting | Formula | N = 100 with defaults |
|---|---|---|
AGENT_SANDBOX_SIZING_QUOTA_PODS | N | 100 |
AGENT_SANDBOX_SIZING_QUOTA_REQUESTS_CPU | N × (pod CPU request + overhead CPU) | 50 |
AGENT_SANDBOX_SIZING_QUOTA_REQUESTS_MEMORY | N × (pod memory request + overhead memory) | 66Gi |
AGENT_SANDBOX_SIZING_QUOTA_LIMITS_CPU | N × (pod CPU limit + overhead CPU) | 125 |
AGENT_SANDBOX_SIZING_QUOTA_LIMITS_MEMORY | N × (pod memory limit + overhead memory) | 216Gi |
Size the node pool for the request totals. Each sandbox pod also requests 256Mi and is limited to 2Gi of ephemeral storage on its node.
Tune the overhead from measured use with kubectl top pod on a sandbox node. The LimitRange rejects a single pod larger than the pod limits, so a change to AGENT_SANDBOX_SIZING_POD_* applies to the quota, the LimitRange and the SandboxTemplate together.
6. Prepare app preview hosts
App previews are off by default. With LLM_BACKEND_SANDBOX_PREVIEW_ENABLED=true, the platform serves each preview at <chat-id>.sandbox.<base-domain>; LLM_BACKEND_SANDBOX_PREVIEW_HOST_PREFIX changes sandbox. Preview hosts use the same Envoy Gateway HTTPS listener as the rest of the platform. A *.<base-domain> wildcard certificate covers only one DNS label, so these hosts need their own *.sandbox.<base-domain> name, in DNS and in the certificate:
- When cert-manager issues the platform certificate, the platform adds the name to it, whichever
DNS_PROVIDERthe site uses, including the self-signed certificate ofDNS_PROVIDER=none. Where External DNS manages the zone, the platform also publishes the name on the gateway. - With operator-managed DNS or custom TLS, create a
*.sandbox.<base-domain>record that resolves to the shared gateway, and include the name in the supplied certificate. See Manage DNS and TLS yourself.
Preview hosts are reachable from any network that reaches the gateway, but the BSQAI API preview proxy admits only browsers that hold a preview session. The Portal obtains a single-use ticket for the chat, and the proxy exchanges it for a session cookie that it checks on every request. Only the Portal origin can embed a preview, and the proxy does not forward the sign-in and Portal session cookies to the app in the sandbox. A preview session lasts 15 minutes by default. Signing out of the Portal ends the user's previews, and an open preview stops when its session ends.
7. Turn on the sandbox
Set the three switches in platform.env, and the optional preview and egress settings:
KATA_DEPLOY_COMPONENT_ENABLED=true
AGENT_SANDBOX_COMPONENT_ENABLED=true
LLM_BACKEND_SANDBOX_ENABLED=true
# Optional
LLM_BACKEND_SANDBOX_PREVIEW_ENABLED=true
LLM_BACKEND_SANDBOX_PREVIEW_HOST_PREFIX=sandbox
LLM_BACKEND_SANDBOX_EGRESS_MODE=internetThe render stops when LLM_BACKEND_SANDBOX_ENABLED=true and either component is off. Keep all three set for every later render: a render without them removes Kata from the nodes, and Argo CD prunes the sandbox controller together with every workspace.
Render and publish the manifests:
bsqai apply all -r
bsqai push-manifests "Enable the agent sandbox"Argo CD syncs kata-deploy in the common application, then agent-sandbox, the Portal and the BSQAI API in the coreai application. The agent-sandbox application skips client-side validation, because the upstream CRDs carry validation rules that client-side validation rejects.
8. Verify the sandbox
Confirm that the controller, the CRDs and the warm pool are in place. Warm workspaces wait unassigned, two by default, and run only when Kata works on a sandbox node:
kubectl get runtimeclass kata-qemu
kubectl -n agent-sandbox-system get deployment agent-sandbox-controller
kubectl get crd sandboxes.agents.x-k8s.io sandboxclaims.extensions.agents.x-k8s.io \
sandboxtemplates.extensions.agents.x-k8s.io sandboxwarmpools.extensions.agents.x-k8s.io
kubectl -n coreai-sandboxes get pods -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n coreai-sandboxes describe resourcequota sandbox-quotaWith egress mode internet, resolve a public name from one of the unassigned pods:
kubectl -n coreai-sandboxes exec <warm-pool-pod> -- \
python3 -c 'import socket; print(socket.gethostbyname("pypi.org"))'The command prints an IP address. A Temporary failure in name resolution error points to the DNS labels or the Cilium setting in 4. Prepare the network.
Then run an end-to-end check as a Portal user:
- Open
https://portal.<base-domain>/app/vibeand ask the agent to create a small web app. - Confirm that the generated files appear and that
kubectl -n coreai-sandboxes get sandboxclaimslists a claim for the chat. - With previews enabled, ask the agent to start the app. The preview loads in the Portal over HTTPS without a certificate warning.
- Leave the chat idle past the tenant's idle timeout, 15 minutes by default. Confirm that the pod stops and that objects for the chat appear under
sandbox/<tenantId>/<chat-id>/in the tenant's<tenantId>-databucket. - Send another message in the chat. Confirm that the files are restored in a new pod.
9. Troubleshooting
| Symptom | Likely cause | Check |
|---|---|---|
The render stops with Invalid agent sandbox configuration | A required component is off, the preview prefix is not a DNS label, or the egress mode is unknown | The CLI message names the variable to set |
The render stops with LLM_BACKEND_SANDBOX_PREVIEW_ENABLED is true but LLM_BACKEND_SANDBOX_ENABLED is false | Previews are on without the sandbox | Turn the sandbox on, or previews off |
Warm pool pods stay Pending | No ready sandbox node, a missing taint toleration, or a full node pool | kubectl -n coreai-sandboxes describe pod <pod> and the cluster autoscaler events |
Pod creation is Forbidden with exceeded quota | The namespace quota is full | kubectl -n coreai-sandboxes describe resourcequota sandbox-quota, then resize as in step 5 |
Sessions fail while sandbox pods wait in Pending | The quota admits more pods than the node pool holds | The cluster autoscaler maximum of the sandbox node pool; see step 5 |
Sandbox calls in one tenant fail with tenant_data_binding_unavailable | The tenant has no ready storage bucket, or, on external S3, the BSQAI API has no tenant bucket access | The tenant's Object storage bucket service and Tenant bucket refused lines in the BSQAI API log; on external S3, the tenant operator and its BSQAI_TENANT_OPERATOR_S3_* pair |
| Idle sandboxes are never stopped and the quota stays full | The reaper cannot save workspaces: missing bucket, missing permissions on sandbox-workspaces or the tenant bucket, or an S3 certificate that the platform CA bundle does not trust | Logs of the latest *-sandbox-reaper job in the BSQAI API namespace |
| Pods fail with a runtime handler error | The RuntimeClass handler does not match the shim that Kata installed | kubectl get runtimeclass and the kata-deploy pod logs in kube-system |
| Kata pods fail on a node | The node does not expose /dev/kvm | The VM size and the kata-deploy pod logs on that node |
| Package installs fail in the sandbox | Egress is blocked, DNS is dropped, or the site reaches the internet only through a proxy | LLM_BACKEND_SANDBOX_EGRESS_MODE, the DNS check in step 8, and the site firewall |
| Screenshots fail in the sandbox, or text in them renders as boxes | The pinned runtime image is older than the release and has no browser or fonts | The digest that kubectl -n coreai-sandboxes get sandboxtemplates -o jsonpath='{..image}' prints, against [sandbox] runtime_image in the release's src/manifests/source/coreai/llm-backend/variables.toml |
| Users see that the sandbox is turned off | A platform or tenant administrator turned it off | GET /v1/sandbox/capabilities as a member of the tenant; see step 10 |
| The preview shows a certificate error | The certificate lacks *.<preview_host_prefix>.<base-domain> | The subject alternative names of the served certificate |
| The preview host does not resolve | The wildcard DNS record is missing | nslookup test.<preview_host_prefix>.<base-domain> |
| The preview returns 404 | The per-chat HTTPRoute is not accepted by the gateway | kubectl -n coreai-sandboxes get httproutes and the route status.parents conditions |
| The preview asks to reload after sign-out or after some minutes | Expected behavior: the preview session expired or was revoked, and the Portal requests a new ticket on retry | The tenant's preview session lifetime, preview_session_ttl_seconds in the sandbox configuration |
| Sandbox code reaches cluster services | The cluster does not enforce NetworkPolicy | The network policy engine in use; see step 4 |
10. Turn the sandbox off without a render
Besides the site variables, the sandbox has two switches in the Portal. Both are on by default, and the sandbox runs only while both are on.
| Switch | Who sets it | Applies to | Where |
|---|---|---|---|
| Platform switch | A platform administrator | Every tenant, the Global organization included | Administration → Platform → Configuration → Code sandbox |
| Tenant switch | A tenant administrator with tenant.update | Their own organization | Administration → Tenant → Code sandbox |
While the platform switch is off, no tenant can use the sandbox and no tenant administrator can turn it back on. Resetting a tenant's configuration keeps its tenant switch.
When either switch turns off:
- The BSQAI API refuses new sandbox work, through the Portal and the API alike, with the error code
sandbox_disabled. Agent tools report that the sandbox is turned off. - Open app previews close at once: every tenant's for the platform switch, the tenant's own for the tenant switch.
- The reaper saves the workspace of every affected running sandbox to object storage and stops its pod on its next run, within about two minutes.
- The Portal hides Build. Users can still open earlier Build chats and download the files they produced.
Turning the switch back on restores the sandbox; the next message in a chat restores its workspace into a new pod.
Each change is written to the BSQAI API log with the acting user and the old and new value: action=platform.configuration.update for the platform switch, action=sandbox.settings.update for the tenant settings. To check the state of a tenant, call GET /v1/sandbox/capabilities as one of its members.
11. Turn the sandbox off for the site
Drain the sessions first, then remove the sandbox in two renders. Only the reaper saves a running workspace before its pod stops, and the first render removes the reaper, so a session still running at that point can lose changes made since its last save.
-
Turn the platform switch off, as in 10. Turn the sandbox off without a render. Keep every component enabled, so the reaper keeps running. Users can no longer start sandbox work, and the reaper saves and stops every running sandbox on its next run.
-
Wait until no session is left. Idle warm pool pods stay, but carry no claim, so check the claims rather than the pods:
kubectl -n coreai-sandboxes get sandboxclaimsThe command returns no resources. In the logs of the latest
*-sandbox-reaperjob in the BSQAI API namespace, each session ends withreaped sandbox; areap failed, will retry next runline means that a workspace is not saved yet, so wait for the next run and fix the cause first, see 9. Troubleshooting. -
Set
LLM_BACKEND_SANDBOX_ENABLED=falseandLLM_BACKEND_SANDBOX_PREVIEW_ENABLED=false, then runbsqai apply all -randbsqai push-manifests. The Portal hides Build, and the BSQAI API removes its sandbox template, warm pool, reaper and preview proxy. -
Wait until
kubectl -n coreai-sandboxes get sandboxes,podsreturns no resources. -
Set
AGENT_SANDBOX_COMPONENT_ENABLED=falseandKATA_DEPLOY_COMPONENT_ENABLED=false, then render and push again.
The platform switch stays off. To turn the sandbox on again later, enable the components as in 7. Turn on the sandbox, then turn the platform switch back on.
Disabling agent-sandbox prunes the controller, its CRDs and the coreai-sandboxes namespace. Deleting the CRDs deletes every remaining sandbox resource. Disabling kata-deploy removes the RuntimeClass and the Kata installer.
Saved workspaces stay in place on both providers, in the tenant buckets and in sandbox-workspaces. On Rook-Ceph, the platform keeps the sandbox-s3 bucket claim while the sandbox is off, so turning the sandbox on again restores existing chats. Delete the sandbox/ objects when the saved workspaces are no longer needed.
12. Upgrade the sandbox components
Each platform release pins the Kata Containers chart, the agent-sandbox controller and CRDs, and the sandbox runtime image. The BSQAI API in the same release uses a client that matches those CRDs. Upgrade them only through a platform release, and do not override a component version on its own.
Mirror the new versions of the artifacts in step 2 before the upgrade. Do not prune the controller or the CRDs during an upgrade while sandbox resources exist.