GGL Docs

Harvester

3 min read

smoll-harvester

smoll-harvester is a nested RKE2 cluster provisioned on top of the parent Harvester cluster via Rancher’s node driver (smoll_harvester.py). It exists to run k8s workloads that authenticate to Vault using their own service account tokens.

VM networks

harvester-public-net and maldev are created by networks.py (create_all_networks()), alongside the images in images.py — before this they were a manual post-rebuild step, and a missing harvester-public-net NAD silently stalls smoll-harvester at “Waiting for viable init node” while the node driver’s pre-create check fails and CAPI loops create → delete.

Each network gets its own Clusternetwork plus a per-node Vlanconfig (NIC names differ per host, so each node is selected by kubernetes.io/hostname) and is attached untagged (vlan_id 0) — the Unifi port already carries the tag. The mgmt bond is never an uplink here; it carries Ceph replication and stays host-only. inference-84 has a single free wired NIC, which goes to harvester-public-net, so maldev covers compute-82/compute-83 only.

Cluster hardening changes

  • anonymous-auth=true added to the RKE2 kube-apiserver-arg. RKE2 disables anonymous auth by default (CIS 1.2.1); Vault’s k8s auth backend needs unauthenticated access to /.well-known/openid-configuration and /openid/v1/jwks for OIDC discovery.
  • oidc-reviewer ClusterRoleBinding scopes system:unauthenticated to just the system:service-account-issuer-discovery ClusterRole, so anonymous-auth only unlocks those two discovery endpoints and nothing else. Applied via a new smoll_harvester_proxy kubernetes provider (proxied straight at smoll-harvester itself, not the rancher local/management cluster).

JWT auth for workload identities (replaces k8s auth backend)

The original plan was Vault’s kubernetes auth backend (TokenReview against the apiserver). That failed testing — see vault-jwt-auth-over-kubernetes-auth — and has been superseded by Vault’s JWT auth backend, which validates smoll-harvester service account tokens directly against the cluster’s JWKS rather than calling back into the apiserver.

K8sApiProxy

New component: k8s_api_proxy.py. Deploys a Flatcar VM on the parent Harvester cluster (shares harvester-public-net with smoll-harvester nodes) running a Cloudflare Tunnel container in HTTP reverse-proxy mode (--url, not raw TCP passthrough) pointed at a smoll-harvester node’s apiserver (:6443).

Key design points:

  • HTTP mode means Cloudflare’s edge terminates TLS with its own publicly-trusted cert; the origin hop (cloudflared → node apiserver) uses --no-tls-verify since the node presents a self-signed, periodically-rotating rke2-server-ca cert. This means Vault’s JWT auth backend can fetch jwks_url over normal public trust — no jwks_ca_pem, so CA rotation no longer requires updating Vault config.
  • Two Cloudflare Access applications gate the hostname:
    • Hostname-wide app requiring a Cloudflare Access Service Token (for non-browser callers).
    • A path-scoped app for exactly /openid/v1/jwks with a bypass policy (no auth), since Vault’s JWT auth backend can’t send Access service-token headers. Access matches the most specific application, so this doesn’t widen access elsewhere. Defense in depth is provided by the apiserver’s own anonymous-auth RBAC scoping (see above).
  • node_ip is currently hardcoded (10.10.180.49, set via pulumi config set ggl:smoll_harvester_node_ip) pending a proper lookup — marked debug-only.

Vault backend

  • k8sjwt auth backend (active path) — pvault.jwt.AuthBackend with jwks_url pointed at K8sApiProxy.jwks_url. A k8sworkload role is bound to bound_audiences=["https://kubernetes.default.svc.cluster.local"] and hardcoded bound_subject="system:serviceaccount:default:default" (debug/PoC only — multi-user path mapping is a later problem), granting token_policies=["default", "k8sworkload"] with a 15-minute TTL. Left untouched during the DebugVM cleanup.
  • k8sworkload.hcl (k8sworkload.hcl) — policy hardcoded to workspaces/*/test@hul.to/* paths (list/read/create/update/delete on the KV v2 mount), mirroring the debug-only role binding above. Also left untouched.

Graph View

Backlinks