← index updated 2026-09-08 7 min 1,816 words Go programming firecracker networking linux

I Spent 29 Hours Debugging iptables to Boot VMs in 4 Seconds

Building a Firecracker MicroVM rootfs, wiring up TAP networking, debugging UFW's FORWARD chain, and writing a guest agent with raw vsock syscalls in Go.

The first time my host got a reply from the Firecracker guest over vsock, I sat there grinning. The VM had booted, its agent was listening, and I could send it a command. The reply came back in under a second.

I was about 30 hours into the project. Most of the previous 29 had gone into rootfs images and iptables rules.

Part 1 covered why I built this: running Claude Code with full tool permissions inside Firecracker MicroVMs. Getting that working meant building a root filesystem, configuring the host network and writing a guest agent to run commands and stream their output.

Building the rootfs

For this setup, I used a prebuilt Linux 6.1 LTS vmlinux kernel and an ext4 filesystem image as the root disk. Firecracker loads the kernel directly. The root filesystem took more work to prepare.

I built the image with Debian Bookworm and the tools I wanted Claude Code to have: Node.js 24, Python 3.11, Chromium for browser automation, git, curl and jq. I installed the Claude Code CLI globally via npm. The image came to about 4GB.

The guest agent is a Go binary that listens for commands from the host. It runs inside the rootfs as a systemd service:

sudo mount /opt/firecracker/rootfs/base-rootfs.ext4 /mnt
sudo cp bin/agent /mnt/usr/local/bin/agent
sudo chmod +x /mnt/usr/local/bin/agent

sudo tee /mnt/etc/systemd/system/agent.service <<'EOF'
[Unit]
Description=Orchestrator Guest Agent
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/agent
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target
EOF

sudo chroot /mnt systemctl enable agent.service
sudo umount /mnt

RestartSec=1 sets a one-second delay before systemd tries to restart the agent. The orchestrator polls vsock every 500ms while waiting for it to become ready.

You build this rootfs once, by hand. Every new VM gets a sparse copy of it.

VM lifecycle

internal/vm/manager.go runs these steps in order. If one fails, it cleans up the resources created so far and returns the error.

flowchart TD
    A["Copy rootfs (sparse)"] --> B["Mount & inject network config"]
    B --> C["Create TAP device"]
    C --> D["Add iptables rules"]
    D --> E["Setup jailer chroot"]
    E --> F["Write Firecracker config JSON"]
    F --> G["Launch via jailer --daemonize"]
    G --> H["Find PID, save metadata"]
    H --> I["VM ready — poll vsock"]

The sparse copy is the first thing that happens:

cmd := exec.Command("cp", "--sparse=always", BaseRootfs, vm.RootfsPath)

--sparse=always leaves holes in the copy where the source contains zero blocks. A 4GB image might use only 2GB of disk space. On my NVMe drive, the copy took under a second.

The manager mounts the copy and writes a systemd-networkd config with a static IP, /etc/resolv.conf for DNS, and /etc/hostname. It then unmounts the image and copies it into the jailer chroot.

That's two copies of the rootfs per VM. I could write the network config into the chroot copy and avoid the first one. The copy was taking less time than Firecracker took to boot, so I left that optimisation for later.

The jailer

Firecracker's jailer is a separate binary that creates a chroot, sets up minimal /dev entries (kvm, net/tun, urandom), and runs the Firecracker process inside it. The VM config is a JSON file:

vmConfig := map[string]interface{}{
    "boot-source": map[string]interface{}{
        "kernel_image_path": "/vmlinux",
        "boot_args":         "console=ttyS0 reboot=k panic=1 pci=off init=/sbin/init",
    },
    "drives": []map[string]interface{}{{
        "drive_id":       "rootfs",
        "path_on_host":   "/rootfs.ext4",
        "is_root_device": true,
        "is_read_only":   false,
    }},
    "machine-config": map[string]interface{}{
        "vcpu_count":  vm.VCPUs,
        "mem_size_mib": vm.RamMB,
    },
    "network-interfaces": []map[string]interface{}{{
        "iface_id":      "eth0",
        "guest_mac":     "06:00:AC:10:00:02",
        "host_dev_name": netCfg.TapDev,
    }},
    "vsock": map[string]interface{}{
        "guest_cid": vm.VsockCID,
        "uds_path":  "/vsock.sock",
    },
}

pci=off because Firecracker doesn't emulate PCI. Paths are relative to the jailer chroot. The vsock entry creates a Unix domain socket at /vsock.sock inside the chroot, which the host uses to talk to the guest.

Launch looks like this:

cmd := exec.Command(JailerBin,
    "--id", vm.JailID,
    "--exec-file", FCBin,
    "--uid", "0", "--gid", "0",
    "--cgroup-version", "2",
    "--daemonize",
    "--",
    "--config-file", "/vm-config.json",
)
cmd.Run()

After launch, the manager waits two seconds, finds the Firecracker PID with pgrep and saves it to a metadata file. If the orchestrator restarts, it reads these files to recover the VMs it was managing. The VM processes can keep running while the orchestrator is down.

Networking

The host's forwarding rules took most of the debugging time.

Each VM needs internet access for Claude Code to fetch packages, clone repos, and hit the Anthropic API. The approach: each VM gets a Linux TAP device on the host, a dedicated /24 subnet, and iptables rules for NAT.

IP allocation

Subnets are deterministic, derived from the VM name using FNV-1a hashing:

func NetSlot(name string) int {
    h := fnv.New32a()
    h.Write([]byte(name))
    return int(h.Sum32()%253) + 1
}

A VM assigned slot 61 gets subnet 172.16.61.0/24, guest IP 172.16.61.2 and TAP IP 172.16.61.1. This avoids running DHCP, but the hash can assign two VM names to the same slot. The 253 possible values still need collision handling.

TAP devices

A TAP device is a virtual ethernet interface. Firecracker attaches the guest's eth0 to it.

tap := &netlink.Tuntap{
    LinkAttrs: netlink.LinkAttrs{Name: cfg.TapDev},
    Mode:      netlink.TUNTAP_MODE_TAP,
}
netlink.LinkAdd(tap)
addr, _ := netlink.ParseAddr(cfg.TapIP + "/24")
link, _ := netlink.LinkByName(cfg.TapDev)
netlink.AddrAdd(link, addr)
netlink.LinkSetUp(link)

TAP names are fc-<vm-name>, truncated to Linux's 15-character interface-name limit.

The iptables rules

Three rules per VM:

// NAT — rewrite source IP when traffic exits the host
ipt.AppendUnique("nat", "POSTROUTING",
    "-s", cfg.Subnet, "-o", cfg.HostIface, "-j", "MASQUERADE")

// FORWARD — allow outbound from TAP
ipt.Insert("filter", "FORWARD", 1,
    "-i", cfg.TapDev, "-o", cfg.HostIface, "-j", "ACCEPT")

// FORWARD — allow established/related inbound
ipt.Insert("filter", "FORWARD", 1,
    "-i", cfg.HostIface, "-o", cfg.TapDev,
    "-m", "state", "--state", "RELATED,ESTABLISHED", "-j", "ACCEPT")

The position of the two Insert calls fixed the networking problem.

The UFW rule order

I originally used Append for the FORWARD rules. Traffic from the VM would leave the host fine (NAT worked), but return traffic got dropped. The VM could resolve DNS but couldn't complete TCP handshakes. I spent hours in tcpdump before checking where my rules sat in the FORWARD chain.

On this host, UFW's forwarding configuration dropped packets before they reached my appended ACCEPT rules. The rules were present, but their position meant they never got to allow the return traffic.

Inserting them at position 1 put them ahead of UFW's rules. The guest could then complete TCP connections.

The traffic path through a working VM:

Guest (172.16.61.2) → eth0 → TAP (fc-task-xxx) → FORWARD ACCEPT
→ NAT MASQUERADE (rewrite src to host IP) → host interface → internet
→ response → RELATED,ESTABLISHED → TAP → guest eth0

Each VM has its own TAP device and subnet. Separate subnets alone don't prevent VMs from reaching one another; host routing and firewall rules determine that. The Firecracker networking guide describes how the host connects guest interfaces to the network.

The guest agent

cmd/agent/main.go was 420 lines of Go at this point. The static binary starts on boot, listens on vsock port 9001 and handles five request types: ping, exec, write_files, read_file and signal.

When the orchestrator wants to run Claude Code, it sends an exec request with stream: true. The agent spawns the command, reads stdout and stderr line by line, and sends each line back as a framed event over the vsock connection. When the process exits, it sends an exit event with the exit code.

Claude Code can also start dev servers and file watchers that outlive the main command. These child processes inherit its stdout and stderr pipes. Waiting for those pipes to close can leave the agent stuck after the main command has exited, because a child still holds them open.

The fix has three parts:

// 1. Process group isolation
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}

// 2. Wait for the main process, not the pipes
<-waitDone

// 3. Kill the entire process group
pgid, _ := syscall.Getpgid(cmd.Process.Pid)
syscall.Kill(-pgid, syscall.SIGTERM)
time.Sleep(500 * time.Millisecond)
syscall.Kill(-pgid, syscall.SIGKILL)

Setpgid: true puts the command in its own process group. When the main process exits, the agent signals the group (-pgid addresses the process group). It sends SIGTERM, waits half a second, then sends SIGKILL.

The agent also gives the pipe-reading goroutines three seconds to drain. After that timeout, it sends the exit event so a stuck reader doesn't block the task.

The line-by-line reader uses a 256KB buffer for --output-format stream-json. Tool results can include file contents in a single line, so the reader needs room for those messages.

Credential injection

Before Claude Code runs, the orchestrator writes five things into the VM via vsock:

  • OAuth credentials from the host's ~/.claude/.credentials.json (mode 0600).
  • A settings file that allows all tools.
  • An environment script that sets CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=true.
  • Task metadata.
  • A marker file to create the output directory.

The orchestrator writes the prompt to a temporary file inside the VM and references that file in the command:

claudeArgs := fmt.Sprintf(
    "claude -p \"$(cat %s)\" --output-format stream-json --verbose",
    promptFile,
)
cmd := []string{"bash", "-c",
    "source /etc/profile.d/claude.sh && " + claudeArgs}

Destroying the VM deletes its rootfs, including the guest copy of the credentials. The host's credentials remain on the host.

Collecting results

After Claude Code finishes, the orchestrator searches for files it created:

// Anything in the output directory
vsock.Exec(jailID, []string{"find", outputDir, "-type", "f", "-not", "-name", ".keep"}, nil, "/root")

// Any new files under /root, created after the prompt was written
vsock.Exec(jailID, []string{"find", "/root", "-maxdepth", "2", "-type", "f",
    "-newer", "/tmp/claude-prompt.txt"}, nil, "/root")

Each file gets downloaded via vsock.ReadFile and saved to /opt/firecracker/results/<task-id>/. The runner also scans the accumulated output for Claude's total_cost_usd field to record what the task cost in API credits.

The orchestrator then kills the Firecracker process, removes the TAP device and deletes the VM's iptables rules. It deletes the jailer chroot and the VM state directory as well.

In this setup, the cycle from boot to teardown typically took 30-120 seconds, depending on the prompt. Boot took about four seconds and teardown about one. Most of the time went into Claude's task.

Part 3 covers the MCP server that lets Claude delegate tasks to itself, the streaming architecture, the web dashboard and the work needed before running this in production.