The build works end to end as of 2026-08-09: 36m51s from packer build to a captured VirtualMachineImage, with WinRM, OpenSSH, Cloudbase-Init and the QEMU guest agent all provisioned in the guest.
What it took to get there, on the packer-harvester plugin (packer-harvester, test/build-windows-server-2025.sh). Five distinct faults, all solved:
- the installer boot loop — a PVC-backed driver ISO; use a
containerDisk - the unattended install dying at the end of OOBE — deprecated
Skip*OOBsettings - the specialize pass never running — a
RunSynchronousCommand<Path>over the 259-character cap, which invalidates the whole answer file - the specialize pass hanging forever — VMDP’s
setup.exegiven a bad flag, answering with a modal dialog - no IP ever reported —
qemu-ga -s installfailing on VSS/COM+ registration before it creates the service
Faults 2–5 all present as the same thing: Windows installs perfectly, and then the build sits in “Waiting for WinRM…” until it times out. Two of them additionally masked each other, so a bisect over one “cleared” it.
/IMAGE/INDEX 2 is now confirmed — long flagged as unverified and suspected more than once. The guest reports Windows Server 2025 Standard Evaluation via guestOSInfo, which is what index 2 is meant to select.
Written so the next attempt doesn’t re-run the same experiments. The recurring theme: every one of these is invisible to packer, kubectl and the serial console. Each presents as a build that installs Windows perfectly and then either shows a generic dialog on the VM console or sits in “Waiting for WinRM…” until it times out. Faults 2 and 3 also masked each other — with two independent causes, a bisect over one of them “cleared” it, which sent two rounds of investigation into the boot chain and the VM’s hardware while both real faults sat in the answer file.
If you are debugging a new Windows failure here, the single most useful habit is at the bottom: the console screenshot is the primary instrument. Neither of the OOBE dialogs that gated this work appears in packer’s logs, in kubectl events, or on the serial console — the build just sits in “Waiting for WinRM…” until it times out, which looks identical to a network problem.
Companion to Harvester Install Guide and Harvester Install Context.
Environment as tested
| Harvester | v1.8.1, KubeVirt v1.7.0 (virt-launcher:1.7.0-150700.3.21.1, SLES 15.7) |
| Nodes | compute-82 / compute-83 / inference-84, amd64 |
| Guest CPU | host-model → Broadwell-IBRS; sse4.2, popcnt, cx16, lahf_lm, 3dnowprefetch all present |
| ISO | windows_server_2025_en-us.iso (harvester-public/image-qvv7s), 8,152,356,864 B, Imported=True |
| Driver ISO (superseded) | virtio-win-0.1.285-1 as a PVC-backed CD-ROM caused the reset below; fixed by driver_container_disk instead (see “Root cause”) |
| VM shape | q35, disk sata, NIC e1000, 4 vCPU / 8 Gi |
The install ISO traces to the Microsoft Evaluation Center — see the commented-out win25iso entry in images.py pointing at go.microsoft.com/fwlink/?linkid=2293312. This matters for edition selection (below).
Start here: diff against a VM that works
harvester-public/winisotest was built through the Harvester UI from this same
ISO and boots its installer. Diff a failing build VM against it before forming
any hypothesis — it costs seconds and covers every field at once, where each
elimination below cost a 10-minute build cycle:
kubectl get vmi <build-vm> -n default -o json | jq -S '.spec' > a.json
kubectl get vmi winisotest -n harvester-public -o json | jq -S '.spec' > b.json
diff -u a.json b.jsonDoing this retired two of the hypotheses below outright. winisotest boots
with secureBoot: false and no SMM, so the OVMF secboot build is not required;
and with maxGuest: 31Gi, so the memory hotplug window is not the cause either.
What it had that the plugin’s build VM did not, before the fixes below:
winisotest | plugin (before) | |
|---|---|---|
| TPM | tpm: {} | absent |
| Boot order | CD-ROM 1, disk 2 | disk 1, CD-ROM 2 |
| Guest memory | 7936Mi, 256Mi under the pod limit (harvesterhci.io/reservedMemory) | 8Gi, equal to the limit |
| Input | usb tablet | absent |
| CPU | maxSockets: 1 | unset |
| VirtIO drivers | registry.suse.com/suse/vmdp/vmdp:2.5.5 containerDisk | virtio-win-0.1.285-1 ISO on a PVC-backed CD-ROM |
harvesterhci.io/os label | windows | absent |
The VirtIO drivers row turned out to be the fix for the boot loop. See below.
Know what this diff cannot show you. It compares the two VMs’ Kubernetes
specs, so it only surfaces differences in things both VMs have.
winisotest was installed by hand and has no answer file at all, so the
answer file — where the second bug actually lived — never appeared in any
diff, no matter how thorough. Driving the two specs to identical and still
failing is therefore not evidence that the fault is in the hardware; it is
evidence that the fault is in something the diff does not cover. That
misreading cost several build cycles here (see the eliminations table under
the OOBE section).
SOLVED — UEFI resets during WinPE kernel handoff
Windows Server 2025 requires UEFI. Under UEFI on this cluster, with the driver ISO attached as the third CD-ROM (fourth device overall, after the root disk), the installer could not start.
Symptom. WinPE’s Loading files... bar reaches 100%, the screen goes flat black for ~4 s, then the VM resets to the TianoCore splash. No bugcheck is ever drawn — there is no blue screen to read, because the reset happens before Windows’ display and EMS subsystems initialise. On the next POST nothing presses a key at the CD prompt, so it lands on No bootable option or device was found, which looks deceptively like a boot_command failure and is not one.
The reset was deterministic at t ≈ 264 s after the VMI reported Running, within ~2 s across every run.
Under BIOS the same ISO boots fine — Windows logo with spinner, then Setup’s blue UI. So the guest, ISO, disk bus and NIC are all sound; the fault was specific to the UEFI path.
Root cause: the driver ISO, not any hardware attribute
The fix is driver_container_disk = "registry.suse.com/suse/vmdp/vmdp:2.5.5"
instead of driver_iso_image_name = "virtio-win-0.1.285-1" — same VirtIO
drivers, delivered as a KubeVirt containerDisk (pulled and served as an
ephemeral CD-ROM, no VirtualMachineImage, no PVC) rather than a PVC-backed
ISO. Same slot, same boot order, same guest-visible effect. That one substitution
took the build VM from a deterministic reset at t≈264s to Setup’s language
screen, with every other variable held constant.
How this was found. A long bisect against winisotest (TPM, boot order,
reserved memory, CPU topology, machine type, os_type label, even the AHCI
device count) drove the two VMs’ QEMU command lines to identical except
for device count, and the reset still happened — which briefly looked like
the seed CD’s answer file was the cause instead (see “Ruled out” below). The
actual break came from stopping the attribute-by-attribute bisect and instead
building an exact topological replica of winisotest: same three devices
in the same slots, including how the third one is delivered. That is what
exposed containerDisk vs. ISO as the real variable — a difference the k8s
spec diff and even the QEMU command-line diff both under-weighted, because
both still just show “a CD-ROM in slot 3” either way. The mechanism (why an
ISO-backed CD-ROM in that slot specifically breaks WinPE’s kernel handoff
under UEFI, when the same bus/slot as a containerDisk does not) was not
root-caused further; the workaround is confirmed empirically and costs
nothing since Harvester’s own Windows template already does it this way.
Ruled out — do not re-test these
| Hypothesis | How it was tested | Result |
|---|---|---|
Display adapter (bochs-display under EFI vs VGA under BIOS) | Enabled the VideoConfig feature gate, set devices.video.type: vga, verified -device '{"driver":"VGA"...}' and zero bochs-display in the QEMU args | Not the cause — identical reset at t≈265 |
| Secure Boot / OVMF build | secureBoot: true + features.smm.enabled, verified OVMF_CODE.secboot.fd (→ ovmf-x86_64-smm-ms-code.bin, SMM + Microsoft keys) and smm=on | Not the cause — identical reset at t≈264. Confirmed again by winisotest, which boots with secureBoot: false |
Memory hotplug window (maxGuest 4x under LiveUpdate) | Pinned memory.guest == memory.maxGuest in the plugin | Not the cause — winisotest boots with maxGuest: 31Gi, 4x headroom, present |
| TPM absence | Added tpm: {} to match winisotest | Not the cause alone — reset persisted at t≈264 with TPM present, driver ISO still attached |
| Boot order (disk before CD-ROM) | Reordered to CD-ROM first, matching winisotest | Not the cause alone — reset persisted |
| Reserved memory (guest == pod limit vs. 256Mi under it) | Added the harvesterhci.io/reservedMemory split | Not the cause alone — reset persisted |
| AHCI device count (4 vs. 3) | Dropped the driver ISO entirely (no replacement), taking the count to 3 | Not the cause — reset persisted identically at t≈264 even with matching device count, which is what pointed (briefly, and wrongly) at the seed CD’s answer file instead |
| CPU features | 26100 kernel (shared with Win11 24H2) hard-requires SSE4.2 + POPCNT; both present on Broadwell-IBRS | Not the cause. The CloudStack thread blaming qemu64 does not apply — we are far richer than qemu64 |
| Truncated / corrupt ISO | 7.6 GiB, Imported=True, size == virtualSize; WinPE loads to 100% | Not the cause |
| Machine type | q35, same as the working coder template | Not the cause |
cidata answer-file seed CD | Removed entirely (separately from the driver-ISO fix) while the driver ISO was still absent | Not the cause of the t≈264s reset — with no driver ISO and no seed, the VM still needed the containerDisk fix to get past that point. (Whether the unattended install itself works is still open — see below.) |
Note the first two eliminations are only meaningful because the variable was isolated. An earlier BIOS test appeared to implicate the video adapter, but switching firmware also switches the adapter — two variables moved together, and the conclusion drawn from it was wrong. The same trap nearly repeated with the AHCI device count: dropping the driver ISO changed both the count and the driver-delivery mechanism at once, so the first “count matches, still resets” result briefly pointed at the wrong variable (the seed CD) until the container-disk substitution was tried on its own.
SOLVED — qemu-ga -s install dies on VSS/COM+ before it creates the service
The last blocker. With everything else working, the build reached a fully provisioned guest with a live WinRM listener and still stalled at “no IP reported”, because KubeVirt cannot see a Multus VM’s IP without the QEMU guest agent. The agent was never installed:
qemu-ga -s install exit 1
QEMU Guest Agent VSS Provider: errmsg Failed to pCatalog->GetCollection.
(Error: 8004e00f) COM+ was unable to talk to the
Microsoft Distributed Transaction Coordinator
-s install registers a VSS provider through COM+ before it registers the
service. MSDTC is not running on a fresh Server 2025 during specialize, so
that step fails and takes the whole install with it — leaving no service at
all. This is also the root cause one layer down of VMDP setup.exe’s “Failed
to open qemu-ga service”: the same registration, the same failure.
The fix is two-layered, in install-guest-agent.ps1:
Start-Service MSDTCfirst, which lets-s installsucceed properly and register the VSS provider too.- Fall back to
New-Serviceif no service exists afterwards. The VSS provider is only for filesystem-consistent snapshots — the build just needs the agent running to report an IP, so a failed VSS registration must never cost us the agent.
Use New-Service, not sc.exe: sc.exe wants its binPath= <value> pairs as
separate argv entries with = glued to the name, and the quoting around a path
containing spaces does not survive PowerShell’s argument handling — an
sc.exe create attempt here silently printed the usage text instead.
SOLVED — a modal dialog in the specialize pass hangs the build forever
install-guest-agent.ps1 ran VMDP’s setup.exe /qemu-ga /S. /S is an
NSIS-style silent switch that VMDP does not accept, so setup answered with a
modal “Setup Help” usage window — in a pass with no interactive user, on a
Start-Process -Wait. The RunSynchronousCommand never returned, specialize
stopped dead behind the dialog, and the build sat at “no IP reported” until
the watchdog gave up. Nothing appeared in the packer log, the serial console
or kubectl events; the only evidence was a console screenshot.
The disc’s own README.txt is the authority on its flags:
| Flag | Effect |
|---|---|
/qemu-ga | install just the guest agent (plus virtio_serial if needed) |
/lic_accepted | suppress the license dialog |
/no_reboot | suppress the “reboot now?” prompt |
So the working invocation is setup.exe /qemu-ga /lic_accepted /no_reboot.
Note that dropping /S alone would not have been enough — without
/lic_accepted the license prompt hangs it in exactly the same way.
Two habits follow, both now baked into the script:
- Never
-Waiton an installer in specialize without a timeout. It now usesWaitForExit(300000)and kills the process if it blocks, so a future unexpected dialog costs 5 minutes instead of the whole build. - Check an installer’s flags against its own docs, not against habit. You can read them on the build host in about a minute, no build cycle needed — the VMDP disc is a containerDisk and pulls straight from the registry:
TOKEN=$(curl -sS -G https://scc.suse.com/api/registry/authorize \
--data-urlencode "service=SUSE Linux Docker Registry" \
--data-urlencode "scope=repository:suse/vmdp/vmdp:pull" | jq -r .token)
curl -sSL -H "Authorization: Bearer $TOKEN" \
https://registry.suse.com/v2/suse/vmdp/vmdp/manifests/2.5.5 -o m.json
DIGEST=$(jq -r '.layers[0].digest' m.json)
curl -sSL -H "Authorization: Bearer $TOKEN" \
"https://registry.suse.com/v2/suse/vmdp/vmdp/blobs/$DIGEST" -o layer.tar.gz
tar -xzf layer.tar.gz # -> disk/VMDP-WIN-2.5.5.iso
isoinfo -J -f -i disk/VMDP-WIN-2.5.5.iso # layout
isoinfo -J -i disk/VMDP-WIN-2.5.5.iso -x /README.txt | tr -d '\r'The suse/vmdp/vmdp repo needs a bearer token (the realm is scc.suse.com,
not the registry), and the image is FROM scratch with no shell — so
kubectl run … --image=… cannot be exec’d into, and there is no package
manager to add tools to. Pull the layer instead.
SOLVED — RunSynchronousCommand <Path> is capped at 259 characters
Going over the cap does not fail that one command. It invalidates the
entire answer file, so the specialize pass never runs and Windows dies on
first boot with the same generic “The computer restarted unexpectedly or
encountered an unexpected error” dialog as the OOBE bug below. Straight from
C:\Windows\Panther\setuperr.log on a failed build’s disk:
[setup.exe] SMI data results dump: Source = Name: Microsoft-Windows-Deployment, ...
/settings/RunSynchronous/RunSynchronousCommand/[Order="1"]/Path
[setup.exe] SMI data results dump: Description = Value is invalid.
... identical for Order 2, 3 and 4 ...
IBS The provided unattend file is not valid; hrResult = 0x80220005
IBS Callback_Unattend_InitEngine:The provided unattend file
[C:\WINDOWS\Panther\unattend.xml] is not a valid unattended Setup answer file
The four commands were 413–420 characters each, having grown a
cidata-volume retry loop and a try/catch inlined into every command line.
The fix is one ~130 character command that locates the seed CD and launches
bootstrap.ps1,
which runs the four scripts itself with no length limit on the logic and a
transcript at C:\Windows\Temp\bootstrap.log.
check-autounattend.py
now enforces this (and the two traps below) before every build, from
build-windows-server-2025.sh. Each of the three costs a ~40 minute cycle to
find by running a build.
SOLVED — the unattended install died in oobeSystem, on two deprecated settings
<SkipUserOOB> and <SkipMachineOOB> in the oobeSystem pass of
autounattend.pkrtpl. Microsoft documents both as deprecated. With them
present the OS installed correctly every time and then died at the very end
of OOBE; removing just those two lines, changing nothing else, took the build
to a booted Windows lock screen.
Why it looked like several different bugs. The dialog depended on what else was in the answer file, which is what made this so hard to pin down:
| Answer file | Dialog |
|---|---|
| Full (windowsPE + specialize + oobeSystem) | “The computer restarted unexpectedly or encountered an unexpected error. Windows installation cannot proceed.” |
| Stripped to windowsPE + oobeSystem | ”Windows could not complete the installation. To install Windows on this computer, restart the installation.” |
Stripped, minus Skip*OOB | boots to the lock screen |
Neither message names the answer file, and both appear after a successful install (seen at 40% and 77% “Installing Windows Server”, then a clean reboot into the Windows Boot Manager on disk), so every earlier theory reached for the boot chain or the VM’s hardware instead.
Why the earlier bisect missed it. An earlier round varied the four
RunSynchronousCommand entries (4 → 2 → 1 → 0) and saw the identical failure
each time, concluding “not the answer file, must be the hardware”. That
bisect only ever varied the specialize pass. oobeSystem was constant
across all four runs — and across the “structurally different” Schneegans
reference file too, which carried the same deprecated settings. When a
bisect over one section of a config exonerates that whole config, check that
the sections you did not vary are actually held constant for a reason.
The eliminations that got there (2026-08-08)
Each of these was a full build cycle, and all of them are now known to have been chasing a symptom of the OOBE bug:
| Hypothesis | Test | Result |
|---|---|---|
Install CD-ROM on the SATA bus (the BdsDxe ... Time out in the serial log) | Added a cd_bus config knob to the plugin, set it to usb | Not the cause, and usb is unusable — OVMF took a #GP CPU exception inside UsbMassStorageDxe.dll with 0xAFAFAFAF poison in the registers, hanging on the TianoCore splash. That Boot0001 ... Time out line is cosmetic: OVMF falls through to the Windows Boot Manager on disk and boots fine |
Malformed answer file (a stray --> left inside <RunSynchronous> by earlier comment-out edits) | Removed it, verified well-formed | Not the cause — real bug, worth fixing, but the failure was identical without it |
disk_bus = virtio + VMDP pvvxblk injection | DriverPaths in 3 configurations, then drvload alone | Abandoned. DriverPaths always gave 0x80070103 - 0x40031 during disk apply; drvload alone installed but bugchecked INACCESSIBLE_BOOT_DEVICE (0x7B) on first boot, since it does not register the driver as boot-start in the installed image |
AHCI device count (4 vs winisotest’s 3) and Multus vs pod networking | Built an exact topological replica: dropped driver_container_disk, switched to the pod network, verified 3 devices via kubectl | Not the cause — identical failure with a byte-for-byte matching device list |
| Storage backend | kubectl diff of both PVCs | Not the cause — both ceph-block, volumeMode: Block, RWX |
features.acpi missing from the plugin’s spec | Compared VM vs VMI features across 9 cluster VMs | Not the cause — KubeVirt defaults acpi.enabled: true when the VM omits it |
Dead end: the pod network. Stay on Multus.
winisotest reports its IP with no QEMU guest agent running at all — its
status.interfaces entry carries infoSource: "domain" and the VMI has no
AgentConnected condition. That is a property of masquerade/pod networking,
where KubeVirt knows the pod IP directly. On Multus the agent is the only
source of that IP, which is why the build must carry the VMDP driver disk.
That makes the pod network look attractive — drop the driver disk, drop a
device, no agent needed. It does not work. The 10.52.0.0/16 pod network
is internal to the cluster and is not routable from the packer host. A build
on it runs to completion inside the guest — answer file valid, all four
provisioning scripts green, WinRM listener up — and then sits in “Waiting for
WinRM…” until it times out, because packer can never open 5985.
Do not use ping to check reachability here. ICMP is broken in the dev
container, and a pod IP appearing to answer ping is what made this look
routable in the first place. Ping proves nothing either way; check the port
that matters:
nc -zv -w 5 <vm-ip> 5985The corollary is that driver_container_disk is not optional on Multus. With
it empty, the guest’s own C:\Windows\Temp\bootstrap.log reads “No VMDP
CD-ROM found; guest agent will not be installed”, KubeVirt never learns the
IP, and the build stalls at “no IP reported” instead.
Still unverified
/IMAGE/INDEX 2 picks Standard + Desktop Experience by reasoning about WIM
layout, not observation. The install now completes, so this can finally be
confirmed from inside the guest with Get-CimInstance Win32_OperatingSystem.
Solved — and why
boot_command was too short, not too late
The CD-boot prompt was being missed because keys stopped arriving before it appeared, not because they arrived late. The giveaway: a failure screenshot showed Press any key to boot from CD or DVD...... with all six countdown dots rendered — the prompt displayed and timed out completely untouched.
Do not try to time the prompt. Cover the whole plausible POST window: boot_wait = "3s", then alternate <spacebar>/<enter> once a second for ~100 s (200 boot commands). Alternating covers both prompts — “press any key” takes either, and the Windows Boot Manager screen that follows needs <enter> specifically. Keys pressed before the prompt exists are harmless.
boot_wait is 3 s rather than ~0 because connecting to the VNC subresource the instant the VMI goes Running gets the connection torn down mid-sequence (write: broken pipe), losing every key sent before the reconnect.
Console capture steals the VNC console from boot_command
KubeVirt’s VNC subresource serves one client at a time. A screenshot loop running during the boot_command window kicks packer off; a no-sleep loop made packer burn all three VNC reconnect attempts and fail the build outright:
Failed to reconnect VNC: VNC handshake failed: read security types:
websocket: close 1006 (abnormal closure): unexpected EOF
Hold capture off until the key window plus margin has elapsed (~130 s). Packer closes its VNC connection after the boot commands and does not reopen it, so capture is free after that.
RunStrategy: Once makes the build VM un-restartable
RunStategy Once does not support manual restart requests
Now RerunOnFailure: it still starts on creation (unlike Manual, which waits for an explicit start request) and still stays down after the guest shuts itself down at the end of the build, which is what the image capture needs.
The driver CD cannot be found by elimination
When there were three CD-ROMs — the install ISO (cdrom-0), the cidata seed (cdrom-1) and virtio-win (cdrom-2) — “the only other CD besides cidata” picks the install ISO, finds no qemu-ga MSI, silently skips the guest agent, and the VM then never reports an IP — a 2-hour stall with no error. If still using driver_iso_image_name rather than driver_container_disk (see “Root cause” above), identify the driver CD by content (a vioserial folder at its root), not by position.
The guest agent matters because KubeVirt only reports a VM’s IP on this cluster’s Multus network once the agent is running, and the agent’s transport (VirtIO-serial) has no Windows inbox driver. driver_container_disk carries the same drivers without adding a CD-ROM to hunt through at all.
RunSynchronousCommand failures abort the whole specialize pass
Two landmines were sitting in autounattend.xml:
sc config WinRM start=auto— invalid,screquires a space afterstart=winrm quickconfig— exits non-zero whenever any network connection is still classifiedPublic, which it is that early in specialize
Either would have aborted specialize and left Setup in an unrecoverable error loop. Both are now in setup-winrm.ps1 on the seed CD, which uses Enable-PSRemoting -Force -SkipNetworkProfileCheck, tolerates its own errors, always exits 0, and leaves a transcript at C:\Windows\Temp\setup-winrm.log.
Edition selection: use /IMAGE/INDEX, not /IMAGE/NAME
The image name differs per media type for the same edition — volume-license calls it Windows Server 2025 SERVERSTANDARD, retail Windows Server 2025 Standard (Desktop Experience), and the evaluation media this ISO came from adds Evaluation on top. Index 2 is Standard + Desktop Experience on all three, since each edition is laid out Server Core first and full desktop second.
Unverified. Setup has never run far enough to read the answer file, so the index is reasoned-from-layout, not observed. It is the first thing to confirm once the VM boots.
Diagnostic techniques that worked
Periodic console capture, named by offset. util/bin/screenshot-harvester-console, one frame every ~2 s, filename = seconds since the VMI reported Running. Setup clears its own error dialogs by rebooting, so a capture only taken after a stall shows nothing useful. Compare frames cheaply by size or hash to find the transition, then only open those:
md5sum *.png | awk '{print substr($1,1,6), $2}' | uniq -c -w6
ls -la *.png | awk '{print $9, $5}' # loading screen ~5 KB, TianoCore ~13 KBFlags must precede the positional <vm-name> or Go’s flag package stops parsing and the tool silently writes to its default path — every frame overwrites one file.
The guest serial log. QEMU logs serial0 inside the virt-launcher pod. This is where OVMF’s BdsDxe boot decisions appear:
kubectl exec virt-launcher-<vm>-xxxxx -n default -c compute -- \
cat /var/run/kubevirt-private/<uid>/virt-serial0-log | tr -d '\000'BdsDxe: failed to load Boot0002 "UEFI QEMU HARDDISK QM00001 " ...: Not Found
BdsDxe: loading Boot0001 "UEFI QEMU DVD-ROM QM00003 " ...
BdsDxe: starting Boot0001 ...
BdsDxe: failed to start Boot0001 ...: Time out
That Windows writes nothing here before the reset is itself evidence — it dies before EMS initialises.
Read Windows’ own Setup logs off the stopped VM’s disk. This is the tool
that ended the guessing: C:\Windows\Panther\setuperr.log names the offending
answer-file element outright, where the console dialog is generic and the
packer log says nothing at all. Worth reaching for before the third
speculative rebuild, not after.
Stop the VM but keep the disk — the plugin’s PVCs outlive the VM, and only its
cleanup step (which a kill -9 skips) deletes them:
kubectl patch vm <vm> -n default --type merge -p '{"spec":{"runStrategy":"Halted"}}'Then mount the disk in a pod. Three things that look like they should work and do not:
quay.io/kubevirt/libguestfs-toolsships no appliance —guestfishfails with “cannot find any suitable libguestfs supermin”. It also has no package manager to fix that with.privileged: truealone is not enough. That image runs as non-root, so the container lands withCapEff: 0000000000000000andlosetupgetsPermission denied. It needsrunAsUser: 0as well.- The nodes have no NTFS driver — neither
ntfs3norntfsappears in/proc/filesystems, somount -t ntfs3fails regardless of privileges.
What works is Alpine plus FUSE ntfs-3g. Note --partscan does not create
loopNpN nodes without udev, so give losetup the partition offset by hand —
read the start sector from fdisk -l and multiply by 512 (the Windows OS
partition is the large “Microsoft basic data” one, partition 3 in our layout):
kubectl run winreader -n default --restart=Never --image=docker.io/library/alpine:3.20 \
--overrides='{"spec":{"containers":[{"name":"winreader","image":"docker.io/library/alpine:3.20",
"command":["sleep","3600"],"securityContext":{"privileged":true,"runAsUser":0},
"volumeDevices":[{"name":"d","devicePath":"/dev/disk"}]}],
"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"<vm>-disk-0"}}]}}'
kubectl exec winreader -n default -- sh -c '
apk add --no-cache ntfs-3g ntfs-3g-progs losetup >/dev/null
fdisk -l /dev/disk # read the start sector of partition 3
LO=$(/sbin/losetup --find --show --read-only \
--offset $((878592*512)) --sizelimit $((123336704*512)) /dev/disk)
mkdir -p /mnt/win && ntfs-3g -o ro "$LO" /mnt/win
tr -d "\000" < /mnt/win/Windows/Panther/setuperr.log | tail -40'The log is UTF-8 with a BOM despite looking UTF-16 at a glance — running it
through iconv -f UTF-16LE produces convincing CJK mojibake rather than an
error. tr -d "\000" is enough. setupact.log next to it has the full
timeline, and UnattendGC/setupact.log covers the specialize pass
specifically.
Diff the QEMU command line between a working and a failing config. Two ways to get it:
# From the virt-launcher log (split on line continuations, normalise UUIDs):
kubectl logs <pod> -n default | grep -o 'qemu-system-x86_64 .*-msg timestamp=on' \
| perl -pe 's/ \\\\-/\n-/g' | sed 's/\\"/"/g'
# Directly from /proc in the running compute container -- works any time the
# VM is up, no log-format guessing:
kubectl exec <virt-launcher-pod> -n <ns> -c compute -- bash -c \
'for d in /proc/[0-9]*; do if grep -qa qemu-system "$d/cmdline" 2>/dev/null; then tr "\0" "\n" < "$d/cmdline"; break; fi; done' | sortThis is the technique that generalises — it compares what the guest actually
gets rather than what the spec claims, and it is what surfaced the AHCI
device-count asymmetry. Know when it bottoms out, though: once two
command lines are identical apart from one device, matching that device’s
presence is not the same as matching how it’s delivered (containerDisk vs.
ISO-backed CD-ROM look identical as -device ide-cd,... on the QEMU command
line but differ in the backing -blockdev). When the diff stops finding
anything and the fault persists, stop refining the diff and instead build an
exact topological replica of the working VM — same devices, same slots,
same backing mechanism for each — rather than continuing to bisect individual
attributes. See “Root cause” above for how this played out.
Environment traps that cost time
pkill -f <pattern>kills the shell running it. The wrapper’s own command line contains the pattern, so pkill signals itself first (exit 144) and the real target survives — twice this left apacker buildrunning that was believed stopped. Resolve PIDs withpgrep -af, thenkillby number.- Killing the plugin process skips packer’s cleanup, orphaning the VM, its root-disk and install-ISO PVCs, the seed-CD PVC and
VirtualMachineImageif acidataseed was attached, and the driver-ISO PVC and image if usingdriver_iso_image_namerather thandriver_container_disk. Interrupt packer and wait (~3 min — its WinRM wait unwinds slowly) rather than killing the plugin, or clean up by hand afterwards:kubectl delete vm <name>first, then the PVCs, then theVirtualMachineImages — Harvester’s admission webhook rejects an image delete while its PVC still exists. - Two builds’ PVCs look alike; separate them by age before deleting anything.