Skip to main content
← Back

Building an ETL Pipeline from Wasm Components on wasmCloud

The June 11, 2025 wasmCloud community call opens with a fast-turnaround demo from Mike Nikles, who in seven days went from boxes on a whiteboard to a working drag-and-drop ETL pipeline builder running on wasmCloud. Each pipeline stage is a Wasm component — composed with wac, deployed as a wadm manifest, and wired together over NATS messaging. The discussion that follows digs into composing components statically vs. linking them dynamically with wRPC, the case for WASI P3 streams, and HTTP/2 support for agentic protocols. Brooks Townsend then kicks off the Q3 2025 roadmap planning and walks through a new consolidated NATS reference doc.

Key Takeaways​

  • A drag-and-drop ETL builder on wasmCloud: Mike Nikles demoed a UI where each node — an HTTP source, a customer-supplied processor, and a log sink — is a WebAssembly component dragged onto a canvas, connected into a pipeline, and deployed to a live wasmCloud host
  • From canvas to running app in one click: the builder generates a YAML config, uses wac to compose the in/out components, emits a wadm manifest with the right HTTP capability, pushes to a local OCI registry, and deploys — then visualizes the live pipeline back in the UI
  • wac composition vs. wRPC links: Mike composed components with wac to keep the component count low and the UI mapping clean; Colin Murphy and Brooks Townsend made the case that wRPC links keep components distributable and swappable on the fly, at the cost of network fallibility
  • WASI P3 streams are coming: Colin highlighted that Preview 3's unified stream type will map naturally onto an ETL pipeline's streaming data, replacing today's separate input/output stream interfaces — and offered a WIT example to follow
  • HTTP/2 is a frequent request: WASI-HTTP is HTTP/1.1 today; the group discussed bridging HTTP/2 ingress through a provider (or onto the NATS bus) to unlock agentic protocols, with full HTTP/2 expected to follow WASI P3
  • Leaning into NATS without locking users in: Taylor Thomas and the maintainers framed NATS as replacing several pieces of typical Kubernetes tooling, while ongoing host refactors add light abstractions so embedders aren't forced to back config with the NATS key-value store
  • A consolidated NATS reference doc landed: Brooks introduced a single docs page collecting every NATS subject, stream, and key-value bucket wasmCloud uses, plus production guidance on backing up the wadm manifests bucket
  • Q3 2025 roadmap planning is open: the quarterly roadmap discussion is the place to leave feedback; ideas floated include embedding the host in wash, a NATS-native messaging interface, and automating a "What's New" docs section

Chapters​

Meeting Notes​

Demo: A Drag-and-Drop ETL Pipeline Built from Wasm Components​

Returning a week after sketching the idea, Mike Nikles showed a working ETL pipeline builder running on wasmCloud. The UI presents a canvas where you drag in a source, a processor, and a sink and connect them into a flow — in the demo, an HTTP source fed data to a customer-supplied Wasm processor, which then wrote to a log sink. Each node is a WebAssembly component; customers simply author a component, upload it to the processor node, and deploy.

Clicking Deploy kicked off the interesting part. The builder reads its YAML config, uses wac to compose the in/out components into single units, generates a wadm manifest with the appropriate HTTP capability attached, and pushes the artifacts to a local OCI registry. wasmCloud then deploys the application, and the UI re-renders the now-live pipeline with data flowing through it. Mike noted the components communicate over NATS messaging so each can scale independently. He built this in about a week atop a single wasmCloud host running locally.

Composing with wac vs. Linking with wRPC​

Colin Murphy and Brooks Townsend used the demo to tease out a recurring design choice: compose components statically with wac, or link them dynamically with wRPC? Mike chose wac to reduce the number of components floating around — keeping a clean one-to-one mapping between the canvas nodes and what appears in wadm. Colin and Brooks noted that linking instead would let you swap or change a component on the fly, deploy a shared sink once and reuse it via links, and lean into wRPC's distributed nature. The trade-off: composed-together components can't "fail to go over the transport," whereas links add fallibility. Brooks flagged this as good documentation to write — when to choose one approach over the other.

Streams, Running on wasmCloud Itself, and the Edge​

Colin pointed out that WASI P3's unified stream type will map naturally onto a streaming ETL pipeline, and offered to share a WIT example using input and output streams. Brooks floated a "nerd-snipe" — running Mike's wit-to-wadm translation logic client-side as a Wasm component — and Mike noted the whole builder service could itself run on wasmCloud rather than as a standalone HTTP server. Looking ahead, Mike sketched extending the canvas with edge environments where dragging a component automatically deploys it to a target, a natural fit for wasmCloud's Kubernetes and multi-cloud distribution model.

Issue of the Week: Q3 2025 Roadmap Planning​

Brooks opened the Q3 2025 roadmap discussion as a call to action rather than a bug. The quarterly roadmap is where maintainers source feedback from people who aren't in Slack or on the call — an open space to scope and triage what the community wants next. Brooks said he would seed it with project-level ideas (e.g. making more mechanisms pluggable, embedding the wasmCloud host inside wash and adding a local-dev mode that doesn't require NATS JetStream) while distinguishing those from feedback informed by production users he works with at Cosmonic.

HTTP/2, Agentic Protocols, and the NATS Balance​

Massoud (ossfellow) raised HTTP/2 support, noting that the new AI/agentic protocols are largely built on HTTP/2 — and that WASI-HTTP is HTTP/1.1 today. The group discussed bridging: an HTTP/2 ingress server (or a custom provider) translating onto NATS messaging or HTTP/1.1 on the back end, with full HTTP/2 support expected to follow WASI P3. Taylor Thomas reframed the broader tension: wasmCloud aims to be flexible "at the right levels," leaning into NATS where it replaces several pieces of typical Kubernetes tooling, while host refactors add light abstractions so embedders aren't forced to back configuration with the NATS key-value store. The recurring principle: powerful defaults without locking users in.

Doc of the Week: A Consolidated NATS Reference​

Brooks introduced a new NATS reference section in the docs, answering the question maintainers hear most: what NATS streams, key-value buckets, and subjects do I need to set up to run wasmCloud in production? The page collects the subject structure for the wRPC protocol (e.g. allow-listing RPC with *.*.wrpc.>), the control interface, and the wadm API, plus the two key-value buckets wasmCloud uses, recommended operator signing-key structure, and the hub-and-spoke NATS topology. Brooks highlighted production guidance — back up the wadm manifests bucket (the source of truth for declarative app state), while wadm state itself is reconstituted from host heartbeats within ~30 seconds.

The discussion turned to documentation workflow: Massoud suggested a "What's New" section driven by a CI job over the changelog or doc tags, and Yordis Prieto asked to label pages explicitly as explanations, how-tos, or references (Diátaxis) so readers can tell at a glance what kind of page they've landed on. Brooks agreed and floated using documentation versions and release notes — possibly with LLM assistance — to surface new and stale docs as wasmCloud features land.

WebAssembly News and Updates​

This call is a snapshot of WebAssembly maturing from runtime to application platform. Mike Nikles's ETL builder shows real software being assembled out of WebAssembly components — composed with wac, packaged to OCI, and scheduled over NATS. The roadmap conversation tracks the ecosystem's near-term frontier: WASI Preview 3 bringing unified streams, futures, and async (and paving the way for HTTP/2), plus the standardization of capability interfaces like messaging. For ongoing updates, follow the wasmCloud blog and the WASI subgroup.

What is wasmCloud?​

wasmCloud is a CNCF project that lets you build applications using WebAssembly components and deploy them anywhere — cloud, edge, or Kubernetes clusters. It uses the WebAssembly component model to let you write business logic in any supported language (Rust, Go, Python, TypeScript, C#, and more) while the platform handles capabilities like HTTP, messaging, and key-value storage through a pluggable provider architecture. wasmCloud's host is built on Wasmtime and uses NATS as its lattice for distributed scheduling and communication. With built-in OpenTelemetry observability and Kubernetes integration, wasmCloud bridges WebAssembly's portable, sandboxed execution model and production cloud-native infrastructure.

Topic Deep Dive: Composing Wasm Components into Applications​

The heart of this meeting is what it looks like to build a real application out of WebAssembly components. Mike's ETL pipeline is, structurally, a graph of components — a source, processors, and sinks — and the demo surfaces the two ways wasmCloud lets you wire that graph together. Composition with wac fuses components into a single deployable unit ahead of time: fewer moving parts, no network hop between them, and a clean mapping from the visual canvas to the deployed wadm manifest. Dynamic linking with wRPC keeps components separate and swappable, deployed once and reused across pipelines via links — at the cost of an inter-component transport that can fail. The component model is what makes both possible: each node is a sandboxed unit with a typed interface, so a customer-supplied processor and a platform-provided sink can interoperate without either trusting the other's internals. As WASI Preview 3 lands its unified stream type, this composition story extends naturally to streaming data — exactly the shape an ETL pipeline wants.

Who Should Watch This​

This call is especially valuable for developers building data and integration pipelines who want to see a real drag-and-drop ETL workflow assembled from Wasm components (4:35), platform engineers weighing static wac composition against dynamic wRPC links for their own apps (11:27), and operators running wasmCloud in production who need the consolidated reference on NATS subjects, streams, and key-value buckets (45:34).

Up Next​

The next community calls feed directly into Q3 2025 roadmap planning, scheduled for the first call of the quarter in July. Watch for community feedback on HTTP/2 support, a possible NATS-native messaging interface, embedding the host in wash for local development, and documentation improvements like a "What's New" section and clearer Diátaxis labeling. Leave your ideas on the Q3 2025 roadmap discussion, and explore the new consolidated NATS reference.

Get Involved​

wasmCloud is a CNCF project and contributions are welcome. Join the community:

Full Transcript​

Read the complete transcript with speaker labels and timestamps:

Read the full transcript →