Skip to main content
← Back

AI Agent Sandbox at KubeCon: 9.65 Million Wasm Workloads, Sandboxed WebGPU, and a Draft AGENTS.md

The November 11, 2026 wasmCloud community call was streamed from the KubeCon + CloudNativeCon NA floor in Salt Lake City. Bailey Hayes showed why WebAssembly makes a dense AI agent sandbox: one 64-core Arm host served 9.65 million scale-to-zero HTTP workloads, a Kubernetes cluster ran a million Codex agent deployments compiled to Wasm, and 17 open-weight models shared one NVIDIA L4 through sandboxed WebGPU. She also reported six Security Slam badges, and Victor Adossi asked for feedback on a draft AGENTS.md for coding agents.

Key Takeaways​

  • 9.65 million scale-to-zero workloads on one host. On a 64-core Arm Neoverse V3 machine that Arm provided for the KubeCon workshop, the density benchmark served 9.65 million scale-to-zero HTTP components and kept 1.42 million warm at 82 KiB each, with a 1.4 ms request p99 at the top of the ramp. Bailey Hayes had expected a p99 of 100 to 500 ms
  • A million agents through Kubernetes. Bailey turned the same box into a worker node and scheduled one million workload deployments of Codex compiled to Wasm (243 KB each), talking to a mock OpenAI server. The hard parts were Kubernetes problems: etcd, storage for agent session state, reconciliation, and not overloading the API server
  • 17 models on one GPU, no slicing. Using Cosmonic-funded NVIDIA L4 spot instances, Bailey loaded 17 open-weight models onto one GPU without MIG, MPS, or DRA. She said hardware slicing usually tops out at seven slices even on an H100
  • About 4% for the sandbox, measured on a 7B model. For token generation with Qwen2.5 7B, running WebGPU inside a Wasm component cost 4.0% against native WebGPU. The GPU dashboard shows bigger gaps on smaller models (13.7% at 1.5B, 40.5% at 0.5B) and a much larger gap between WebGPU and llama.cpp's CUDA backend, which is still under investigation
  • The bigger gap is upstream. Bailey expects proposed WebGPU features, such as subgroups and faster matrix multiplication, to narrow the distance to CUDA over the next year. Server-side WebGPU is an open area for contributors
  • Six Security Slam badges and OSPS Baseline Level 3. wasmCloud earned all six badges, is ready for the EU Cyber Resilience Act, and now meets OSPS Baseline Level 3 across its repos. The 80% code coverage requirement made CI slower but has already caught real regressions. Bailey was due to accept the award on stage the next morning
  • A draft AGENTS.md is open for feedback. Victor Adossi's draft PR #5965 collects the guidance he keeps giving coding agents, from commit style to PR descriptions, and points to published skills for Rust, WIT, components, wash, and Helm
  • Always-loaded rules vs. skills. Yordis Prieto argued for incremental disclosure and for measuring results by reviewing commit history for repeated mistakes. Bailey wants the guidance to stay vendor-neutral so it works across many agents

Chapters​

Meeting Notes​

Live from KubeCon NA​

Bailey Hayes opened the call on 11/11, which was also Veterans Day in the US, from the show floor at KubeCon + CloudNativeCon NA 2026 in Salt Lake City. Eric Gregory, Mendy Berger, and Liam Randall were working the booth behind her. On Monday, Bailey and Mendy had given their WebGPU talk at Cloud Native AI + Inference Day. Later that afternoon, Bailey and Eric would run the hands-on workshop on sandboxing AI agents on Kubernetes. Jeremy Fleitz shared the benchmark dashboards while Bailey talked.

Bailey said that everyone at the conference was talking about sandboxing, but few people could say what it means. At the booth, the team had been starting from the NIST definition of a sandbox and explaining from first principles why WebAssembly is one of the best sandboxing technologies. Density matters most to teams that want to sandbox a million agents.

Density: 9.65 Million Workloads on One Arm Host​

For the workshop, Arm lent the project a 64-core Neoverse V3 host with 128 GB of memory. The first run used the host without Kubernetes, to measure the raw performance of the wasmCloud host. The density dashboard showed the record run from November 8, with one wash-runtime process serving a wasi:http@0.3 handler:

MetricResult
Warm components1.42M (kept running and answering HTTP)
Scale-to-zero workloads9.65M (each started on its request)
Memory per warm component82 KiB
Request p99 at the top1.4 ms, across 1.42M warm components
Cold starts per second179k (scale-to-zero, peak step)

On the call Bailey rounded the warm figure to 1.5 million. She had expected a p99 of 100 to 500 ms and planned to tune it down, so 1.4 ms surprised her. Other talks at the show were counting workloads in the thousands, and one keynote showed 100,000. Bailey allowed that this compares apples to oranges, but said wasmCloud is counting in millions. The numbers continue the density work Victor Adossi presented on the November 4 call, where one 5-CPU, 2 GiB host held 163,750 scale-to-zero components. For how warm pools and per-instance concurrency are sized, see concurrency and connections, and the wasmCloud 2.12 release post for the pool changes behind these numbers.

A Million Sandboxed Agents on Kubernetes​

The workshop demo goes further: sandboxing a million agents in Kubernetes. Bailey made the same Arm box a worker node, put the control plane on a separate node, and scheduled one million workload deployments of Codex compiled to Wasm. Each agent sends a chat message to an OpenAI-compatible endpoint. Bailey used a mock OpenAI server so she wouldn't spend her budget on tokens. The agent component is 243 KB, bigger than the HTTP handler, and the results were similar.

Every problem she hit was a Kubernetes problem rather than a wasmCloud one. She had to deal with etcd, storage for each agent's session state, reconciliation, and keeping the API server from being overwhelmed. Workloads reach the cluster as custom resources that the runtime operator places on hosts. Workload security explains the isolation each one gets.

Sandboxed WebGPU: 17 Models on One NVIDIA L4​

Next, Bailey showed the GPU dashboard. Cosmonic paid for the hardware, and Bailey used NVIDIA L4 spot instances because they are cheap. A spot node can be taken back with two minutes' notice, which is fine for a benchmark. She tuned the wasi:webgpu implementation in the host over several iterations. The work is in an unmerged PR of about eight patches (the dashboard references PR #5898).

She loaded 17 open-weight models onto one L4, the small Qwen, Llama, and similar models people run on laptops, and served them all at the same time. She didn't use MIG, MPS, DRA, or any other way of slicing the GPU. In Kubernetes, each model ran as its own workload deployment, 17 in all, and the host shared GPU resources fairly between them.

Why WebGPU? A component that uses WebGPU never holds a real pointer to the device. A native GPU kernel holds real, shared GPU resources. Bailey said hardware slicing usually tops out at seven slices even on an H100, and many GPUs allow only four. Vendor APIs can sandbox the GPU, but they are device-specific and lock you in. Browsers have already solved multi-tenant GPU sandboxing at large scale on consumer hardware with WebGPU. And raw performance may not be the only goal. There are many more models than LLMs, such as a computer-vision model on a factory floor that checks whether a mold came out right. Running many models in one cluster without breaking the bank is valuable.

The dashboard compares three ways to run the same qwen2.5-7b model on the L4:

RunPrompt processing (pp512)Generation (tg128)
Vendor (llama.cpp CUDA)3,466 tok/s53.6 tok/s
WebGPU, native binary24.5 tok/s33.4 tok/s
WebGPU in a Wasm component24.3 tok/s32.1 tok/s

The cost of the sandbox is the step from native WebGPU to Wasm. For this 7B model it was 0.7% for prompt processing (within noise) and 4.0% for generation. For generation on smaller models the step was larger: 13.7% at 1.5B and 40.5% at 0.5B. Bailey thinks the overhead can be reduced. The dashboard notes that the gap between CUDA and WebGPU is under investigation, because llama.cpp's WebGPU backend may leave work on the CPU. So that gap is an upper bound on what WebGPU costs.

Bailey said the main problem is in upstream WebGPU, not in wasmCloud. Mendy pointed her to Google's work on faster matrix multiplication and subgroups, which aren't in the W3C WebGPU spec yet. Bailey said Google's slides report about a 10x improvement on a branch, and Google is pushing those features toward standardization. Bailey expects the gap to CUDA to narrow over the next year. Most of the audience in her talk were low-level inference engineers new to WebAssembly. She hopes some will join the community, because server-side WebGPU has plenty of interesting work. The October 14 call covered related work: llama.cpp is merging support for compiling to WebAssembly and WASI WebGPU. The runtime overview lists the wasi-webgpu host feature, and host interface configuration shows how to turn it on.

Security Slam: Six Badges and OSPS Baseline Level 3​

Answering a question in chat about hardening, Bailey announced that wasmCloud had earned all six badges in the CNCF Security Slam and would receive an award the next morning. The project is now ready for the EU Cyber Resilience Act. An incubating project is expected to meet OSPS Baseline Level 2. wasmCloud now meets Level 3, the highest level, across all its repos: the Go and TypeScript SDKs, wasmcloud.com, and wasmCloud itself. The work started in April and covered CI security, attestations, signing, and SBOMs (see verify wasmCloud releases and SBOMs). It follows the Security Slam prep on the October 21 call and the Level 3 target announced on November 4.

The downside is that every repo needed 80% code coverage, which slowed CI. Bailey said it was worth it, because the tests have already caught real regressions and bugs, and she now feels braver about making sweeping changes. She said the call would usually be short when streamed from the show floor, but there was a lot to share. She was busy until her 4 PM workshop and would catch up on reviews when she got home Friday.

A Draft AGENTS.md for Coding Agents​

Victor Adossi shared draft PR #5965, which adds an AGENTS.md to the wasmCloud repo. The goal is to make it easier to contribute with coding agents and to make agents more likely to open the right PR on the first try. It follows the AI contribution plan from the October 7 call, which called for a small AGENTS.md. Victor said an AGENTS.md is a coarse tool, but it works per repo. He wrote down what he usually adds to his prompts when he works on wasmCloud PRs (and sometimes JCO PRs). The 170-line draft covers:

  • Commits: feature branches, Conventional Commits titles under 70 characters, small semantic commits, git commit -s for the DCO, and rebasing instead of merging main
  • Pull requests: a short, readable body without "Validation" sections or test command lists unless asked, while the test policy still applies
  • Code and docs: no reviewer-facing or conversation-history comments in code, no run-on blocks, and semantic line breaks in docs
  • Tooling and CI: cargo xtask, Just over make, and scripts that CI calls and that are easy to run locally, instead of logic inlined in a workflow
  • Skills: pointers to published skills in cosmonic-labs/skills (rust-development, wit-interface-design, webassembly-component-development, wash, helm-charts) and from the Bytecode Alliance

Where agents install skills depends on each person's setup, for example .claude/skills for global Claude skills. Victor noted the trade-off: everything in AGENTS.md uses context, and agents pay less attention to early context as a session goes on.

Always-Loaded Rules vs. Skills​

Yordis Prieto, reviewing the PR live, recommended incremental disclosure: point to where the information is instead of putting it all in the root file, and move the guidance to .agents/skills. Victor agreed that skills are the natural place for language-specific guidance, and the Rust rules can move there. But some rules are worth the cost of always being loaded. One example is the rule against putting the same validation checklist in every PR description, because it adds nothing for a reviewer unless something unusual happened. He said context windows are large now, and the only way to know whether the guidance works is to try it.

Yordis said the evidence he had in mind was about context files, not system prompts. Vendors, Anthropic among them, now suggest deleting these files and checking whether you still need them, especially with newer models like Fable and Opus 5.5. He suggested adding metrics. His team no longer squashes commits, so they can go through contributors' commit history and look for the same mistake repeated. Moving guidance into skills also makes activation cheaper, because tools now search for skills by description. A root file, in contrast, is loaded into every context. Victor replied that vendor guidance has to be rechecked with every batch of models. The rules in the draft are ones he still has to give recent versions of Fable and Codex, and he found Fable usually does a little better than Codex.

Vendor-Neutral Guidance and an AI-Forward Project​

Bailey said she would try the draft. She runs many agents at once and wants variety in reviews: if one model wrote a change, she tests it with another. She has also been trying newer agents such as Pi and Hermes. A lot of her setup lives in her user folder for one agent, and she has opened bad PRs because her Codex setup didn't know what her Claude setup knew. Guidance that works across many agents is a priority for her. Since the field changes so fast, measuring what works today is the best the project can do.

She and Victor have also been expanding CONTRIBUTING.md with OpenSSF and OSPS security guidance. Bailey said LLMs pick up both it and the project's threat model every time. When she tries something risky, such as swapping out the allocator, the agent now warns her. She called wasmCloud an AI-forward project and said this is starting to improve its quality. She closed by promising a rundown next week of what the team learned at KubeCon and any changes the project might want to make.

WebAssembly News and Updates​

In webassembly news this week, KubeCon NA was all about sandboxing, and WebAssembly offers both density and isolation. GPU sharing is the next frontier. The WebGPU standard already lets browsers share GPUs among untrusted tenants. The wasi-webgpu proposal brings that model to server-side components, with the runtime provided by the wasi-gfx project. The features that would let WebGPU compete with CUDA, such as subgroups and better matrix multiplication, are making their way through W3C standardization. On the security side, the OpenSSF OSPS Baseline and CNCF's Security Slam give projects public milestones to work toward. Coding-agent guidance like AGENTS.md and published skills is quickly becoming a normal part of how open source projects take contributions.

What is wasmCloud?​

wasmCloud is a CNCF project for building applications out of WebAssembly components and running them across cloud, edge, and Kubernetes clusters. The Wasm component model lets you write business logic in Rust, Go, Python, TypeScript, C#, Java, and more. The platform supplies capabilities like HTTP, messaging, key-value storage, blob storage, secrets, GPU access, and observability through standard interfaces and a pluggable host plugin architecture backed by Wasmtime. wash is the developer shell for building, running, deploying, and debugging. The runtime operator schedules Wasm workloads on Kubernetes the same way you schedule containers, with WASI Preview 3 support on by default. Together they give you a production platform for WebAssembly on Kubernetes and at the edge.

Topic Deep Dive: An AI Agent Sandbox That Scales to Millions​

An AI agent sandbox has to do two things that usually pull against each other: isolate every agent completely, and stay cheap enough to run thousands or millions of them. Containers and microVMs isolate well, but each sandbox costs megabytes of memory and a noticeable startup time. A WebAssembly component starts with no access to files, the network, or credentials, and gets only what its manifest grants. On this call that cost was 82 KiB per warm component, and 1.4 million warm components still answered at a 1.4 ms p99. Scale-to-zero raises the ceiling further, because an idle agent holds no instance at all. That is how one host reached 9.65 million workloads, and how a Kubernetes cluster ran a million Codex agents compiled to Wasm.

The GPU work extends the same model to inference. With wasi:webgpu, a component asks for GPU access as a capability and never holds a device pointer. The host can then share one physical GPU among many workloads (17 models on one L4 in this benchmark) without vendor-specific slicing. To set this up, start with concurrency and connections for sizing warm pools, then workload security for the isolation model and network access and socket isolation for what an agent can reach. Autoscaling with HPA and KEDA covers scaling workloads up and down. For small clusters and edge nodes, see lightweight deployments.

Who Should Watch This​

Platform engineers planning agent sandboxes on Kubernetes should start with the density results at 1:40 and the million-agent Kubernetes run at 4:45. AI infrastructure and inference engineers will want the WebGPU benchmark from 5:57 and the overhead discussion at 11:57. Maintainers and contributors who use coding agents can jump to Victor Adossi's AGENTS.md walkthrough at 16:49 and Yordis Prieto's case for measuring it at 25:11.

Up Next​

Next week, Bailey plans a rundown of what the team learned at KubeCon, the hot topics, and any changes the project might want to make. Expect more feedback on the draft AGENTS.md (#5965), progress on the WebGPU tuning PR, and a write-up of the workshop demos on the website.

Next call: KubeCon Recap: AI Agent Sandbox for a Million Agents, 17 Models on One GPU, Credential Brokering

Get Involved​

wasmCloud is a CNCF project and contributions are welcome. Join the community:

Full Transcript​

Read the complete transcript with speaker labels and timestamps:

Read the full transcript →