Transcript: AI Agent Sandbox at KubeCon: 9.65 Million Wasm Workloads, Sandboxed WebGPU, and a Draft AGENTS.md
Transcript
Bailey Hayes 0:09
Hello and welcome to wasmCloud's community call on November 11, 11/11. So happy 11/11 day, and also, as Jeremy reminded me, happy Veterans Day for those in the U.S. We are live, as you can see, from the KubeCon North America floor in Salt Lake City. Behind me, you can see Eric and Mendy, and there's Liam. He's very good at getting more people to stop by our booths than anybody else on the planet. He's got mad skills, so you should definitely watch that. It's been a really great conference so far. Mendy and I gave a talk on Monday that had really great reception and feedback. I want to put some more information on that on our webpage, like how to do the demo that I made for wasmCloud, and then be able to run all of Mendy's different isolated types of demos. I'm going to share my screen and walk you through just a few of those really fast. Oh, I can't do that because it's the new laptop. RIP. Well, good news is that it's all on this web page. Let me give you the ability to share. All right, you should be able to go now. Yeah, if you don't mind pulling up the density one.
Bailey Hayes 1:40
Okay. Couple things. So later today we've got a wasmCloud workshop, and Arm gave us a box to run on that's pretty incredible. It's one of their Neoverse V3s, so basically it's got a ton of cores and a ton of RAM. You know RAM is really hard to come by right now, so I really appreciated them hooking us up. This run isn't in Kubernetes, so it's just showing the raw performance of the wasmCloud host. I was able to schedule 9.65 million workloads. These are HTTP components, so they're not doing a ton of work, but that's still a crazy number. And then concurrently, I can run 1.5 million of them at an insanely good P99. Yeah, 1.4 milliseconds was our P99 there, which is just stupid. I did not expect that. I was expecting like 100 milliseconds, 500 milliseconds, and then I was gonna be like, "Okay, let's bring it down and tune it." And instead, we just have an incredible P99 number for concurrent workloads served. And at literally all these other talks that I've been going to here, they're showing, look at the thousands that we're running. And then Google's keynote, those were really bright people, I really admire them, and we are a bit comparing apples to oranges. Their thing was like, look at our 100,000. And we're over here like, hey, check out our millions. So it's been pretty fun and exciting on that front.
And then the other thing I'll say is, everybody here is talking about sandboxing, but I don't think they really know what that means. So we've been doing a lot of bringing up the NIST definition of what a sandbox is, and talking through first principles and why WebAssembly as a sandboxing technology is truly one of the best ones for a lot of reasons. Density here is a really big one for people who are trying to sandbox a million agents, and that is what we're going to be demoing later today.
Bailey Hayes 4:45
It's sandboxing a million Codex agents in Kubernetes, and all the challenges I had there were really Kubernetes challenges: from etcd, to having better volumes for storage to save the agent session state, to reconciliation, to not trying to destroy the API server, et cetera, et cetera. But I was able to tune this exact same box to become a worker node in my Kubernetes cluster. I put our control plane over on a different node, and so was able to schedule a million workload deployments through Kubernetes of Codex that we compiled to Wasm. So it's a legit agent. And what that benchmark does is, the agent says, "I'm gonna say hey in chat." I'm gonna chat over the OpenAI endpoint. And we gave it a mock OpenAI server because, while people are willing to donate compute right now, nobody's given me free NVIDIA hardware yet, and I didn't want to bust my budget on tokens. So our mock OpenAI server is like, you're good, homie, and then it just keeps going through the bench. Similar results. Not as small and compact as this little HTTP component, obviously, but still tiny, 243 kilobytes, which is pretty nice, and doing real work, actually what an agent would be doing, going through the OpenAI API for inferencing. So I'm pretty excited about that one. I'm obviously still cooking, trying to play around with slides and get it all right. It's in five hours, so plenty of time, I'm gonna tell myself.
Bailey Hayes 5:57
And then let's talk about the WebGPU demo that we did. Jeremy, do you mind going to the GPU tab here? It's in the same website. So Cosmonic contributed this hardware, and what I did is I used spot instances because I'm cheap, unlike Arm, who gave us that really amazing box. With a spot instance, the way it works is it's sort of ephemeral. You're able to get a cheaper GPU node, but they can take it away at any time, and they only have to give you two minutes' notice. But for our benchmark, that's actually totally fine, right? If they take it away from me, I'm gonna go find another one, spin it up again, and see what I can get to. And I tuned several loops here, several iterations on our WebGPU implementation inside our host. There's a draft PR with like eight patches or something like that to get it all the way through. But it was a major performance boost just having the opportunity to sit there and tune on this NVIDIA L4, which is not very nice NVIDIA hardware, but I was still able to get it pretty far.
And one of the coolest things that I was able to do is I loaded a bunch of open-weight models onto this one GPU device. I got up to 17 of those, and they were what you would typically pull down for your laptop, like small Qwen and Llama models. I tried to grab all the ones that I thought people would recognize, and then I stuffed them all in this exact same GPU without having to do MIG or MPS or DRA or any kind of slicing of the GPU. This is literally just one GPU device able to serve all of those models concurrently. Was it fast, though? Well, that's why we've got this benchmark. I got it faster than it was, but I wouldn't say that it was as fast as I wanted it to be. Now, in my benchmarking, what I found is that when you're running Wasm plus WebGPU... So what we want to do is be able to load the model in a way that's sandboxed, right? Oh, I see Aditya, glad to see you here. Hey Yordis, good to see you too. So basically, if you do Wasm plus WebGPU, it's pretty interesting because it means that we have an approach that will sandbox the GPU. A lot of people here are really into sandboxing right now at this conference, even though I'm not sure they necessarily know what that means, but they are into it, definitely for the compute. They all agree. You are always invited, Yordis, and you should try to get a talk, and then you get a free pass. That's my move, that's what I always do. But anyways, if you use WebGPU, you are basically making sure that you never actually have a real pointer to the actual device, right? With a GPU, you're compiling a GPU kernel when you're making a model, and that is holding on to real shared GPU resources.
Bailey Hayes 9:31
And so even when you decide to make a multi-tenant GPU, aka you slice it up into slices, most of the time the biggest number of slices that seems to be possible right now, even if you have a massive H100, is like seven. For a lot of the others, the quota is like four. We got to 17. They're stuck on having to find a way to box it very tightly, and they're still not sandboxing it. Now, there are vendor APIs that basically man-in-the-middle sandbox it. NVIDIA's got their own, but it's obviously very device-dependent, very much vendor locked-in. But with WebGPU, the browser's already solved this. How do you do multi-tenant GPU sandboxing at a massive scale on consumer hardware? You do that with WebGPU. And so maybe raw performance isn't the most important thing here. Maybe it's more important that I am able to run lots of different models, because there are so many more models than just these LLMs, right? There might be a model for doing computer vision running directly on my factory floor, and I want to see if the mold came out correctly as I'm building different types of processes in my factory. So I think the universe of models is very vast and large, and being able to run lots of those in your Kubernetes cluster at scale without breaking the bank is pretty interesting. So when we do this bench, what you can see is the effect of adding Wasm plus WebGPU, so adding a little bit of sandboxing to how we multi-tenant a workload deployment. In this case, I had 17 different workload deployments running in Kubernetes, running on this box, and using our host I was able to come up with fair sharing of how much resources I want to give each.
Bailey Hayes 11:57
So that's the benefit of doing Wasm plus WebGPU here with wasmCloud, and our overhead for just adding wasmCloud in the middle here, with Wasm versus native WebGPU implementations, was only 4%. I think we can also narrow that down, but that's a very small overhead to get sandboxing and to be able to make that multi-tenant. So I was really happy with that, and that really means that the problem is upstream in WebGPU, not necessarily in our project. How do you make WebGPU and CUDA a little bit more competitive directly with each other?
Bailey Hayes 12:56
And Mendy pointed out a number of resources to me on this, and I found Google slides where they're adding a better way to do matrix multiplication, and doing subgroups, and a couple of things that aren't standardized yet in the WebGPU spec, WebGPU obviously being a web standard in the W3C. However, they have a branch where it works, and they're pushing this forward to be standardized, and it's like a 10x improvement. So I'm expecting over the next year we're going to really see this gap between WebGPU and native things like CUDA really close. And at that point, it's like, why aren't you sandboxing your GPU if you're running at a massive scale? So anyways, that part was pretty cool, and we've had some good questions. A lot of people in that room were not at all familiar with WebAssembly, because they were low-level inferencing people. But I'm really hopeful that we'll see some people join our community and start building some more things. I think improving WebGPU itself, especially on the server side, is a huge opportunity. So if that's something people are interested in, I think there's a wealth of fun things to do there. And now I'm going to read Yordis's chat, about an agentic, Wasm-based harness. We have been doing a lot of hardening.
Bailey Hayes 13:58
I forgot to mention that tomorrow wasmCloud is going to get an award, because we kicked butt at our Security Slam and we got all six badges. We are now officially EU Cyber Resilience Act ready. We are at Baseline Level 2 for everything, which is what's expected for an incubating project, but actually we've been doing Baseline 3 for all of our stuff now. So that's basically the highest bar, OSPS Baseline 3, that any open source project could have, and it was a lot of work. Honestly, I think we started that work in April, and I've been churning at it. You guys, I've been boring the crap out of you whenever I'm like, okay, let's talk about CI security again. Here's attestations, and here's signing, and now we have SBOMs. But we do that now for all of our repos: Go, TypeScript, wasmcloud.com, and of course wasmCloud. Now there's a bit of a downside here, in that we needed to hit 80% code coverage for everything, and now our CI is a little bit slower. My bad. It's worth it, though, because it's been catching real regressions and real bugs. So the effort to get to that badge, I think, really made the project way better, and now we have excellent regression coverage. So I feel even braver to make even more sweeping changes. But yeah, 10:30 tomorrow. If anybody's here, come with me. I'm gonna go up on stage and get us a little physical award for wasmCloud. I ran into Brooks earlier, and I told him he better be there. I think we just quickly walk across the stage. We don't get to say our sandbox is better than your sandbox, unless they give me a mic. I'll sneak that in. But yeah, I'm pretty pumped. I'm not going to be a normal person until my 4 o'clock demo is over, so just a few more hours. And if you're waiting on me for reviews or anything, I'm sorry. I've been in KubeCon zone for a bit, but I'll come back out of it, I'll be back home on Friday, and I'll go back through my backlog then.
Bailey Hayes 16:30
Any questions? Anything else? That was everything I wanted to share, which I guess turns out was a lot. Normally we keep these KubeCon-from-the-floor calls pretty short, but I guess we've got so much to share. Oh yeah, Victor. Do you want to do that?
Victor Adossi 16:49
Yeah. One second. Let me share my screen here. The other screen. Okay, I think everyone should be able to see my screen. So one of the things we wanted to do was start making it easier for people to contribute with agents, and making it more likely that the agent puts up the right thing on the first go-around. There are lots of ways to do this, like the combination of skills, how you prompt, etc. And AGENTS.md is kind of a coarse way to do it, but the nice thing is that it works per repo, right? So I've put up a draft PR with a whole bunch of the stuff that I know I often end up adding to my prompts when I'm working on wasmCloud PRs, and sometimes when I'm working on JCO PRs as well.
This is not complete at all. I think there's a lot of stuff that I've added over time that hasn't made it in here. But I wanted to start fresh and just go through and think of things that often come up. And then at the bottom, since where AGENTS.md stops and skills should start is kind of a fuzzy line, for people who prefer using skills, there are a lot of skills, for example the ones we have in Cosmonic Labs that Bailey has put together, that are also worth considering. So those are actually at the bottom of AGENTS.md, and your agent should recommend those to you. And where your agent will install the skills is actually different, because people have different setups. You've got global skills that are available in .claude/skills, I think, and then a different one for OpenAI. But basically, with all the pieces of AGENTS.md, however long your AGENTS.md is, that's how much context you're going to lose. And a lot of times agents don't actually pay that much attention to the stuff at the beginning of the context after a certain amount of time. So this is better than nothing, but we'll see if we can improve it and get something that at least helps us make PRs a little bit better right out of the gate.
Victor Adossi 20:05
Oh yeah, thank you, Yordis. So the incremental disclosure thing, that's what we kind of do with the skills. It's a little bit easier to do in a skill than it is in the AGENTS.md that's at the top. I assume everyone can still see my screen. Yordis's comment is about moving from AGENTS.md to .agents/skills. If we have a .agents folder with skills in there, then it would make sense as the place to put it, and that's how a normal skill would be formatted. But I'd probably do that as a follow-on to this one. You can refer to a skill from either AGENTS.md or from somewhere else. But it's not clear it should actually go here versus in the place where we already have skills, because we have, for example, a wash skill. So there's a mix of where things should go. I think we should have more specific skills over there, right? So basically, what Yordis is suggesting is we should remove all the stuff that's at the top of here, and it would boil down to this section that we have at the bottom.
Victor Adossi 21:43
Just like, hey, you're doing Rust, use the Rust development skill. Hey, you're doing WIT interface design, use the WIT interface design skill. That's one approach. But I think we should still have some core set of directives, things that we want agents to do, in AGENTS.md, right? There are some things arguably worth paying that cost for, and I actually don't think that cost is that much. At this point, context windows are pretty large, but it's still something. Getting the core things we want agents that work on the codebase to always do, I think it's worth finding that middle ground between pushing everything out to "hey, your agent should use this skill for this thing, or look at this file," and something that is basically always in the context. Trying to read Yordis's comment: "There are studies showing that the file actually hurts." Yeah. I don't remember exactly, and this was actually a while ago, if I remember correctly. I think this was back when, rather than AGENTS.md, it was system prompts not necessarily helping as much. But yeah, there are some studies that suggest that AGENTS.md, or trying to steer an agent with a block of text at the beginning, doesn't help at all. I don't know that that's the case, since if that were the case, then skills wouldn't do anything either, whether you refer to them or not. So there's some level of: if it works, then it works. We have to try it to have it work. So feel free to try it ahead of time and propose changes as well. Thanks, Yordis, for adding some comments. Okay, but anyway, the PR is in the chat, and obviously it's on the repo. So I would love some more comments, or just things that you find yourself having to ask the agent to do. It'd be nice to just get them in one place, even if we decide to move all this out to other skills and pull them in more selectively. Oh, sorry, your hand is raised. Go ahead.
Yordis Prieto 25:11
Yeah, no, no. I was just saying it was not about the system prompt. It was about all these files, and even Anthropic and other companies are recommending to maybe delete them and see if you actually need them, right? Especially after Fable and Opus 5.5 and things like that. So what I'm saying is, ideally, put some metrics around whatever you're saying. For example, what we do is look at the commit history. Nowadays we don't like to squash commits anymore, so everything is in the history. Then you can go through the history of the commits that people are doing when trying to contribute, and analyze it and say, okay, I see that it's making the exact same mistake multiple times. Because otherwise, the good thing about moving what you're doing right here into skills is that the activation is a little bit cheaper, especially with tools that are now optimizing by searching for the skills and tools and then activating them based on the description or whatever. Because don't forget that this file is being pushed into every single context, even when you're not doing anything around it. So instead of everything at the root, just move it into different directories, or into skills, and progressively say, "Hey, point here."
Victor Adossi 26:30
Yeah, so the progressive pointing is good, and we have some of that. But they gave that sort of guidance before Fable and some of the newer models came out, and it's something you redo basically after every new batch of models comes out. These are things I've had to tell Fable, and more so Codex in particular, recent versions of Codex and Fable. So it's not that they just always get it right. Fable does a little bit better than Codex most of the time, actually. But I do hear what you're saying. For language-specific stuff, it's definitely a good idea to push it out, and we have that for Rust stuff. But there is some stuff that you generally want to always be there, right? Some of the stuff around Rust style, we can push off. But this is a Rust repo, so most of the time, if you're contributing, it's going to be Rust-related. Sometimes, oh, we do have Go code, for example. But stuff about how you make PRs, for example, shows up almost every time. Like having validation, right? Every PR will be like, "Here are the validation things," or "These are the things you do for validation." But that ends up being the same for every PR, and it doesn't add any meaningful content for anyone who's doing a review or anything like that, unless it's abnormal, and then it's usually something important. So that's a rule that's in here, for example. But yeah, this is just stuff that I found myself typing to make the PRs not terrible.
Bailey Hayes 28:48
It's still in draft. I'll give it a whirl. I have a lot of stuff that I've saved to my users folder, and I feel like I need to spread that around, because I'm using a lot more agents now, all simultaneously, largely because I want a lot of flavor now in PRs. So if I know I wrote it with Opus, then I want to run and test it with [another vendor's model], and vice versa. A lot of times it's easier to use the vendor agent for that. But I'm also having fun trying out new things like Pi and Hermes. So finding ways to make that vendor-neutral and work really well across lots of different agents is going to be important to me going forward. I myself have put up stupid PRs because my Codex doesn't know all the things that I have configured in my Claude, basically. So I'm trying to find ways to federate that all out. But thank you for putting it up, and I definitely appreciate the feedback. I feel like this stuff changes so fast. I have a hard time keeping up, but measuring what is working right now, today, is the best we can do. So let's do that.
Bailey Hayes 30:23
On the CONTRIBUTING.md front, I don't know if folks have noticed, I've been jamming in there a lot over the past couple weeks. Me and Victor, really, we've been putting a ton of stuff in there, and all of that really comes from OpenSSF and OSPS best practices and guidelines around security and all that kind of stuff. And I will say that I can definitely tell that it and our threat model are picked up by the LLMs every time, and I think that has really raised the bar when I'm hacking and doing silly things like, can I change out the allocator? Am I about to screw everything up? And it'll be like, you should consider this. So that's been a nice thing to have. So yeah, more changes like this are super welcome. We are definitely an AI-forward project at this point, and it's gotten to the point now where I think it's really starting to improve our quality. So thank you again, Victor, for sharing.
Bailey Hayes 31:41
Any other topics that folks want to go through? All right, then I think I'm gonna get back over there and say my sandbox is better than your sandbox. Appreciate everybody joining today, and we'll do a rundown next week on anything that we learned, what some of the hot topics were, and break out any changes we maybe want to make as a project. So thanks again, everybody, for joining.