GGL Docs

Harvester

5 min read

smoll-harvester

smoll-harvester is a nested RKE2 cluster provisioned on top of the parent Harvester cluster via Rancher’s node driver (k8s.py). It exists to run k8s workloads that authenticate to Vault using their own service account tokens.

Scoped Rancher credentials

Rather than handing the admin rancher2:tokenKey to every consumer, scoped_credential.py provides reusable building blocks for minting dedicated low-privilege Rancher identities:

  • scoped_cluster_user(username, cluster_id, role_template_id, ...) — creates a Rancher user with login-only global access plus one cluster-scoped RBAC binding. Returns a ScopedClusterUser.
  • cluster_user_token(scoped_user, cluster_id, ...) — mints a non-expiring login token for that user via the custom RancherUserToken dynamic resource (native rancher2.Token can only mint tokens for the provider’s own identity).

scoped_harvester_kubeconfig composes them into the provisioner credential for the parent Harvester cluster (node driver), scoped to the smoll-harvester-vms project + read access to the shared image/network namespace. Feeds the harvester CloudCredential.

The two other wrappers that used to exist here — scoped_smoll_harvester_token (DebugVM’s kubeconfig) and scoped_vault_k8s_token (the dead kubernetes auth backend’s token_reviewer_jwt) — were removed along with their consumers; see DebugVM removal below.

Stale-token self-heal: RancherUserToken.diff() now probes the stored token against /v3/users?me=true and forces a replace if it 401/403s, since a dead token otherwise sits in state forever (none of api_url/username/password ever change to trigger a natural replace). See pulumi-up-401-stale-rancher-usertoken.

Kubeconfig Output plumbing: rancher_proxy_kubeconfig now resolves api_url/cluster_id inside .apply() rather than f-string-interpolating them directly, since either can be an unresolved pulumi.Output (e.g. ClusterV2.cluster_v1_id).

Cluster hardening changes

  • reserved_memory_size="-1" added to the harvester node driver config.
  • ignore_changes=["harvesterCredentialConfig"] on the harvester CloudCredential — Rancher ≥2.8.5 auto-renews the embedded token in place, so Pulumi no longer needs to push a fresh kubeconfig on every token rotation. This also sidesteps a preview hard-failure when the provisioner token is mid-replacement (stale-token self-heal above) and its value is unknown during preview.
  • anonymous-auth=true added to the RKE2 kube-apiserver-arg. RKE2 disables anonymous auth by default (CIS 1.2.1); Vault’s k8s auth backend needs unauthenticated access to /.well-known/openid-configuration and /openid/v1/jwks for OIDC discovery.
  • oidc-reviewer ClusterRoleBinding scopes system:unauthenticated to just the system:service-account-issuer-discovery ClusterRole, so anonymous-auth only unlocks those two discovery endpoints and nothing else. Applied via a new smoll_harvester_proxy kubernetes provider (proxied straight at smoll-harvester itself, not the rancher local/management cluster).

JWT auth for workload identities (replaces k8s auth backend)

The original plan was Vault’s kubernetes auth backend (TokenReview against the apiserver). That failed testing — see vault-jwt-auth-over-kubernetes-auth — and has been superseded by Vault’s JWT auth backend, which validates smoll-harvester service account tokens directly against the cluster’s JWKS rather than calling back into the apiserver.

K8sApiProxy

New component: k8s_api_proxy.py. Deploys a Flatcar VM on the parent Harvester cluster (shares harvester-public-net with smoll-harvester nodes) running a Cloudflare Tunnel container in HTTP reverse-proxy mode (--url, not raw TCP passthrough) pointed at a smoll-harvester node’s apiserver (:6443).

Key design points:

  • HTTP mode means Cloudflare’s edge terminates TLS with its own publicly-trusted cert; the origin hop (cloudflared → node apiserver) uses --no-tls-verify since the node presents a self-signed, periodically-rotating rke2-server-ca cert. This means Vault’s JWT auth backend can fetch jwks_url over normal public trust — no jwks_ca_pem, so CA rotation no longer requires updating Vault config.
  • Two Cloudflare Access applications gate the hostname:
    • Hostname-wide app requiring a Cloudflare Access Service Token (for non-browser callers).
    • A path-scoped app for exactly /openid/v1/jwks with a bypass policy (no auth), since Vault’s JWT auth backend can’t send Access service-token headers. Access matches the most specific application, so this doesn’t widen access elsewhere. Defense in depth is provided by the apiserver’s own anonymous-auth RBAC scoping (see above).
  • node_ip is currently hardcoded (10.10.180.49, set via pulumi config set ggl:smoll_harvester_node_ip) pending a proper lookup — marked debug-only.

Vault backends

  • kubernetes auth backend — removed. It was already dead (TokenReview failed testing) and depended entirely on DebugVM’s internal IP for kubernetes_host, so it was deleted along with DebugVM rather than kept as unreachable code. scoped_vault_k8s_token/VAULT_TOKEN_USER (its token_reviewer_jwt helper) were removed from scoped_credential.py too.
  • k8sjwt auth backend (active path) — pvault.jwt.AuthBackend with jwks_url pointed at K8sApiProxy.jwks_url. A k8sworkload role is bound to bound_audiences=["https://kubernetes.default.svc.cluster.local"] and hardcoded bound_subject="system:serviceaccount:default:default" (debug/PoC only — multi-user path mapping is a later problem), granting token_policies=["default", "k8sworkload"] with a 15-minute TTL. Left untouched during the DebugVM cleanup.
  • k8sworkload.hcl (k8sworkload.hcl) — policy hardcoded to workspaces/*/test@hul.to/* paths (list/read/create/update/delete on the KV v2 mount), mirroring the debug-only role binding above. Also left untouched.

DebugVM removed

services/debug_vm/ (the GCP f1-micro COS VM used to smoke-test K8sApiProxy’s public JWKS path) has been deleted entirely, along with everything that only existed to support it:

  • The DebugVM import/instantiation and debug_vm_ip export in __main__.py.
  • The dead kubernetes Vault auth backend block (see above) — its kubernetes_host was built from debug_vm.internal_ip, so it couldn’t survive without it.
  • scoped_smoll_harvester_token/DEBUG_TOKEN_USER and scoped_vault_k8s_token/VAULT_TOKEN_USER in scoped_credential.py.
  • Stale DebugVM mentions in K8sApiProxy’s docstring/comments.

The node_ip hardcoded-IP TODO in K8sApiProxy and the debug-only hardcoding in k8sjwt-k8sworkload/k8sworkload.hcl are unrelated to DebugVM and were intentionally left in place.

End-to-end result

Per commit ef4fbbe, an e2e test confirms k8s workloads on smoll-harvester can authenticate to Vault using their own service account JWT via the k8sjwt backend.

Cloudflare wiring

  • common/cloudflare/tunnels.py’s Tunnel component is now reused by K8sApiProxy (previously only used elsewhere).
  • setup_cloudflare_vault_auth in __main__.py — the ZeroTrustAccessIdentityProvider registering Vault as a Cloudflare Access OIDC identity provider is now live (previously commented-out TODO).
  • README documents the Cloudflare API token permissions needed: Tunnel edit, Access Apps/Policies edit, Access Identity Providers edit, Access Service Tokens edit, Zone DNS edit, Zone read.
  • CFTUNNEL_IMAGE bumped 2026.3.0 → 2026.7.2.

Graph View