blog draft

This summer we rebuilt the lab. Storage was the reason: Longhorn writes were slow, and since all our VMs were disposable a clean install beat an in-place migration. The three nodes (compute-82, compute-83, and the GPU box inference-84) now run a fresh Harvester v1.8 cluster with Rook-Ceph inside it, Rancher came back as a vcluster addon, and the Pulumi stacks were re-applied on top.

Improve disk speed

Two workloads pushed the rebuild: Packer builds and Windows installs.

Packer builds

Packer builds used to mean babysitting ISO installs through VNC boot menus. Now they run from a Coder workspace whose Harvester identity is scoped to the work: it can create and delete build VMs in packer-dev, and add images to harvester-public but never overwrite them. Delete the workspace and the identity goes with it.

The whole point of the scoping is that agents build images without getting broad access to the cluster. Everything is kept inside a namespace, so a rogue agent can only do damage inside its own sandbox.

The runner images used to be built in Azure, and exporting both cost about $40 in data egress plus long compute builds. Now they’re built on-prem. The ubuntu-24.04 runner image builds in about 10 minutes with all upstream tests green (66 tests, 0 failures). Windows Server 2025 takes 37 minutes, but it’s a full unattended install. Both runners are published to harvester-public: ubuntu-24.04-runner-20260810 (128Gi) and windows-2025-runner-20260810 (164Gi).

Windows

Windows was the hardest part. The Debugging Windows on Harvester note has the full story; the short version is five separate faults, none of them visible in packer logs, kubectl events, or the serial console. The build would install Windows and then sit in “Waiting for WinRM…” until it timed out, which looks exactly like a network problem. The console screenshot is what actually diagnosed it, and two of the faults masked each other.

The five: the installer boot loop (fixed by shipping virtio drivers as a containerDisk instead of a PVC-backed CD), OOBE dying on deprecated Skip*OOB settings, the specialize pass never running because a RunSynchronousCommand path exceeded 259 chars (which invalidates the whole answer file), the specialize pass hanging on a bad flag to VMDP’s setup.exe, and no IP ever reported because qemu-ga failed on VSS/COM+ registration.

End result: a captured image with WinRM, OpenSSH, Cloudbase-Init, and the QEMU guest agent, 36m51s from packer build to image.

smoll-harvester

A big part of the rebuild was moving services off VMs and into k8s. On Harvester every VM costs at least 1GB of RAM and a full core, even when the service inside only needs a fraction of that. On smoll-harvester you can ask for 100m CPU and 256MB, and that’s what you pay. Small services that each used to get their own VM now share nodes, and you don’t lose memory to a guest OS and its system processes for every one of them. Coder, Gitea, mem0, DeerFlow, and n8n all run here, deployed with Fleet GitOps from the repo.

Zero trust

Public services now live behind Cloudflare tunnels on astral-labs.work, and every hostname is gated by Cloudflare Access with Vault OIDC as the only identity provider. Nothing public is reachable without logging in through Vault.

The pieces that don’t fit the browser flow are the interesting ones. coder ssh uses a cftoken.py header command that does the Access auth for non-browser clients. Vault’s JWT auth backend can’t send Access service-token headers, so k8s_api_proxy gets a path-scoped bypass for exactly /openid/v1/jwks, and the apiserver’s RBAC is scoped so anonymous auth unlocks just that one endpoint.

Agents are scoped too. A packer-builder workspace can create and delete VMs in packer-dev and add images in harvester-public, but it can’t modify or delete an existing golden image. Publishing is always additive.

Reduce maintenance

The old lab ran on hand-managed credentials: pasted kubeconfigs, certs placed by hand, service accounts created by clicking through Rancher. Every one of those was a calendar event I dreaded.

Expiring kubeconfigs

Rancher caps tokens at 90 days and its kubeconfigs at 30, and nothing reminded you until a pulumi up failed with a 401. Now every credential is derived: RancherUserToken mints a scoped token per service and rotates it 30 days before the cap. The kubeconfigs it feeds (CSI, cloud-provider, Coder’s VM provisioning) are minimal and one per service. When a token rotates, the workloads restart themselves via an annotation hash on the pod template, so nobody has to touch it. The hand-pasted ggl:harvester_kubeconfig config key is gone.

Setting TLS certs

TLS is handled by cert-manager with a Vault issuer on smoll-harvester. Workload certs are issued from Vault’s PKI and renew themselves. Trust the lab root CA once per machine and forget about it. Public hostnames terminate at the Cloudflare edge, VM services use Caddy.

Creating service accounts

Creating a service account used to be a Rancher clickathon, and the result was usually admin, because it was easier. Now a service module declares the rules it needs (ACCESS), and a shared context mints a dedicated Rancher user with matching Role/RoleBindings. One credential per service, no shared admin tokens. The whole derivation chain hangs off two roots that never leave the process that mints them.

Local AI

The three RTX 3090s in inference-84 now run the lab’s own AI stack.

Inference

vLLM + LiteLLM on a GPU VM. The active model is Muse-Glimmer-30B-FP8 at tensor-parallel 2 (~13GiB per card), with the vision tower on rank 0, which is why context is capped at 32768 instead of the model’s native 131072. It replaced Qwen3.6-27B-FP8, which couldn’t use all three cards anyway (its attention heads don’t divide by three). Running on 2 GPUs gave up some KV cache but improved token generation speed by about 30% under Qwen3.6, which is fine for a lab with fewer than ten users. Model configs live in code with the tuning that took a while to work out, so switching models is a one-line change plus a VM replace.

Embeddings

The third card runs a second vLLM container serving gte-Qwen2-1.5B-instruct as the embedder. The dimension is the constraint: mem0’s pgvector column is fixed at 1536 and reads no env var to change it, and this model’s hidden size is exactly 1536.

Memory: mem0

The last piece is mem0, long-term memory for the lab’s agents, running on smoll-harvester. The API and dashboard have their own local account system (upstream mem0 has no OIDC, verified against the tag we build), but the MCP bridge fixes that for anything acting on one person’s behalf: you log in with Vault in a browser and never handle a key.

claude mcp add --transport http --scope user mem0 \
  https://mem0-mcp.k8s.internal.galaxygridlabs.com/mcp

The bridge is its own authorization server. MCP clients self-register (RFC 7591), it serves /authorize and /token itself, delegates the login to Vault OIDC, and pins every tool call to the email on your token. That’s the authorization mem0 itself is missing: an API key is instance-wide, but through the bridge an agent can only reach your memories.

Other MCPs

mem0 isn’t the only MCP server in the lab:

  • Brave Search at brave.mcp.internal.galaxygridlabs.com gives agents web search without handing out API keys.
  • GitHub connects agents to the repos they’re working on.
  • Coder’s own lets Claude Desktop or Google Antigravity orchestrate Coder workspaces, so an agent can spin up a workspace as part of doing the work.

Ceph

Underneath it all is the storage the rebuild was about. Rook-Ceph runs hyperconverged inside the Harvester cluster: dedicated raw disks per node, a single replica-3 RBD pool with failure domain set to host, and ceph-block as the default StorageClass. VM images, root disks, and data disks all land on Ceph, and a guest PVC is a ceph-block volume hot-plugged into the VM through the CSI driver.

Replica-3 has a cost: Ceph acks a write only after all three replicas commit, so the slowest host paces every write. One box is a PERC H710, so its controller cache has to match the other nodes’ disks; in write-through it would bottleneck the whole pool. And Ceph is sensitive to clock skew, so NTP has to work from first boot.

The benchmarks aren’t all wins. Random writes are 4.3x worse than the old setup, and we need more benchmarking to understand the initial error. But Windows installs work and no longer time out, so even with worse benchmark numbers the migration was a success. The next step is moving the 10gig cards into x16 slots instead of x8 to drop the bifurcation bridge.

The trade we accepted: Harvester’s native VM backup is Longhorn-only, so Ceph-backed VMs have no backup button. Recovery is what the rebuild was: everything is IaC and the data is RBD snapshots. If the lab burns down, it’s a pulumi up away.