GGL Docs

Packer Builder

4 min read

Packer Builder Workspaces

How Coder workspaces get a safe, scoped Harvester identity for building Packer images, and how to use them.

Purpose

YOLO agents build Packer templates (ISO installs with VNC boot menus, image customization, export) in Harvester. They do this from a Coder kubernetes pod (on smoll-harvester) whose agent has a per-workspace Harvester identity scoped to packer-dev — able to create VMs, drive the console, and delete VMs only there. In harvester-public it can read base images and publish new ones, but nothing more: no update, patch, or delete, so an existing golden image can never be overwritten or replaced from a workspace.

Identity model

coder Deployment (smoll-harvester), runs as SA coder-workspace-provisioner
├── in-cluster: Role "coder-workspace-pod" in packer-builder ns (pod/PVC/secret)
├── /kubeconfig.yaml (harvesterKubeconfig): parent-cluster VM provisioning + packer-dev identity rules

packer-builder workspace (pod in packer-builder ns on smoll-harvester)
├── terraform (template) creates on parent harvester, via /kubeconfig.yaml:
│     SA coder-<ws> + 2 RoleBindings (packer-builder/packer-builder-read) + token Secret, in packer-dev
├── terraform (template) creates on smoll-harvester, in-cluster:
│     SA coder-<ws> (pod runs as it) + reader RoleBinding + Secret (packer kubeconfig) + PVC + Pod
├── /etc/packer-kubeconfig.yaml ← mounted from the Secret (packer-dev SA token + server/CA)
├── packer + kubectl + ggl_core_infra clone (persisted on the PVC at /persistent)
└── packer build -var kubeconfig=/etc/packer-kubeconfig.yaml ...
  • The Roles are owned by Pulumi (packer_builder_roles in src/2_app/services/harvester/scoped_credential.py for the parent cluster; coder-workspace-pod / coder-workspace-pod-reader in services/coder/coder.py for smoll-harvester). Templates only ever bind, never author rules.
  • The workspace pod’s own ServiceAccount can read exactly one thing — its packer-kubeconfig Secret — so the kubelet can mount it. All Harvester work goes through the packer-dev identity via packer/kubectl, never through the pod’s k8s SA.
  • Deleting the workspace runs terraform destroy → pod + PVC + Secret + packer-dev SA + bindings deleted. Every identity dies with the workspace.
  • A coder_script sweeps leftover build VMs (labeled app.coder.io/workspace=<ws>) on workspace stop.

RBAC

Parent harvester cluster

NamespaceRoleGrants
packer-devpacker-builderkubevirt VMs/VMI CRUD; start/stop/vnc subresources; PVCs; CDI DataVolumes + datavolumes/source create (cross-namespace clone authorization); VirtualMachineImages CRUD
harvester-publiccoder-packer-builderVirtualMachineImages get/list/watch/create (append-only — no update/patch/delete); DataVolumes get/list/watch/create/delete (staging clones for publishes); NetworkAttachmentDefinitions read
coder-vms + packer-devcoder-vm-provisioner (user)VM provisioning (coder-vms) + SA/RoleBinding/Secret manage in packer-dev (identity provisioning, via VM_ACCESS)

smoll-harvester cluster (where the workspace pod runs)

NamespaceRoleGranted toGrants
packer-buildercoder-workspace-podcoder-workspace-provisioner (coder server SA)pods/PVCs/secrets/configmaps/serviceaccounts + rolebindings CRUD (template provisioning)
packer-buildercoder-workspace-pod-readerper-workspace pod SA (bound by template)secrets get (mount packer kubeconfig)

Using it

  1. Create a workspace from the packer-builder template.

  2. cd ~/ggl_core_infra/src/3_onprem/vm_templates

  3. Build (always pass the three vars):

    packer build \
  -var kubeconfig=/etc/packer-kubeconfig.yaml \
  -var namespace=packer-dev \
  -var output_namespace=packer-dev \
  harvester-iso-ubuntu24-golden.pkr.hcl

  4. The finished image lands in packer-dev. To publish a golden image, pass -var output_namespace=harvester-public; the workspace identity can create images there but not modify or delete them, so publishing is always additive — replacing an existing golden image is still an admin job.

    Publishing across namespaces cannot use a direct export-from-volume: Harvester hardcodes the export DataVolume’s source PVC namespace to the image’s namespace, so the plugin first clones the finished build volume into harvester-public (a CDI cross-namespace clone DataVolume) and exports from that local clone. The staging clone is cleaned up automatically.

    The plugin names each image packer-<random hex> and puts the readable name in spec.displayName, so a “republish” is a new object alongside the old one, not a replacement. If a publish fails mid-export the plugin’s cleanup delete on the image gets a 403 and the half-finished image stays until an admin removes it (the staging clone is removed by the plugin).

The packer-builder subagent (subagents/packer-builder.md in the template) encapsulates this operating manual.

How to push / update the template

export CODER_HEADER_COMMAND=~/.local/bin/cftoken.py
coder template push --yes \
  --directory src/2_app/services/coder/coder_templates/packer-builder \
  packer-builder

Troubleshooting

  • 403 outside packer-dev — expected; that is the sandbox. The identity is namespace-scoped.
  • output_image_namespace hardcoded — the two golden .pkr.hcl files now take output_namespace (default packer-dev). Golden runs pass -var output_namespace=harvester-public.
  • 403 creating an image in harvester-public — check the verb. create is granted; a 403 on update/patch/delete there is the no-overwrite rule working as intended.
  • Staging clone rejected with “insufficient permissions in clone source namespace packer-dev” — the CDI cross-namespace clone authorization (datavolumes/source create in packer-dev) has not been applied yet; run pulumi up on src/2_app first.
  • Plugin behavior unchanged after a util/packer-harvester change — the plugin binary is baked into the packer-builder Docker image; rebuild it so workspaces install the new binary on pod start.
  • Pod fails to start — check the workspace build log; the bootstrap runs as the container entrypoint. Common: image pull (airgap), GGL CA fetch, or the packer plugin install.
  • Aux-CD upload (cd_files) 401/404 — the plugin’s steve-API upload may not accept an SA token through the rancher proxy; use http_directory / cloud_init_user_data (NoCloud) instead. The Ubuntu golden file already uses the NoCloud path.
  • Leftover VMs after a crash — kubectl -n packer-dev delete vm -l app.coder.io/workspace=<ws> or use the workspace’s stop-time sweep.

Graph View