Jello 2.0 vs 3.0 Architecture Review

Postmortem comparing Jello’s three implementations, written because the 3.0 rewrite has felt more painful than 2.0 despite being a second rewrite. See also Jello Architecture and Jello Re-Architecture.canvas for the original design notes.

Correcting the premise

“Jello 2.0” and “Jello 3.0” aren’t the first and second version — there have been three full rewrites of this service:

Gen	Commits	Span	Language/shape
1.0	5069422 → deleted aee6a5c	Aug 10 – Aug 27	TypeScript/Node, npm workspaces (Fastify + worker-thread conversation manager + React SPA)
2.0	e70639b “Jello headless rewrite (#93)” → 45869fe^	Aug 24 – Sep 7 (~2 weeks)	Go, single static binary, plain SQL, vanilla-JS frontend
3.0	45869fe “Jello3.0 harness (#210)” → HEAD	Sep 7 – now (~4 days)	Go, jello-server/jello-agent split over gRPC-over-stdio, Ent ORM, React/Vite frontend

1.0 was TypeScript and was deleted outright as “the pre-rewrite Node service” — dead weight once 2.0 had been running in production for a few days. 2.0 was already Go. So “3.0 vs 2.0” is a same-language rewrite: a working single-binary Go service replaced by a two-binary Go service with a different ORM, different transport, and a different frontend stack. That reframes the complaint — this isn’t “Go finally fixed what TS got wrong,” it’s “the second Go architecture is causing more pain than the first one did.”

All three generations’ code except 3.0 (current HEAD) had to be recovered via git show/git ls-tree on historical commits — 1.0 and 2.0 are both fully deleted from the working tree.

---

Jello 1.0 (brief) — TypeScript/Node, Aug 10–24

A Node 22 npm-workspaces monorepo: Fastify 5 API, a worker_thread-per-conversation manager that spawned the pi CLI and bridged its JSONL protocol, Drizzle ORM over Postgres, and a React 19 + Vite 6 + Tailwind 4 + Zustand SPA. Auth was OIDC via the lab’s Vault provider with opaque DB-backed session cookies; MCP OAuth 2.1 + Dynamic Client Registration was hand-rolled with AES-256-GCM token encryption. It ran for about two weeks, was declared “deprecated” mid-development once the Go rewrite (2.0) proved out, and was deleted wholesale in aee6a5c once nothing depended on it. docs/quartz/content/notes/Jello Architecture.md explicitly says to “port the approach, not the code” from its auth/session/crypto design — i.e. the ideas were sound, the implementation wasn’t worth carrying forward.

No deep bug-pattern analysis was done on 1.0 given its short life and full replacement; the useful signal is that its OIDC/session/crypto design held up well enough to be reused as a reference in later planning docs.

---

Jello 2.0 — the Go headless rewrite (e70639b → Sep 7)

Architecture

One static Go binary is the entire deployable. It embeds the built frontend via //go:embed, owns the Postgres connection, drives Coder workspace lifecycle through the real coder/coder/v2 SDK, and — since its Phase 2 — runs a second gRPC listener that Envoy’s ext_authz calls into for credential masking. The code’s own comment: “No other process runs in the pod.”

Request flow: browser → WebSocket (internal/ws) → conversation.Orchestrator → an in-memory per-conversation session → coder port-forward/SSH into the workspace → a per-harness CLI driver (internal/claudecli, internal/picli, internal/agycli) shelling out to the actual coding-agent CLI and translating its wire format into one shared harness.Event vocabulary.

The standout design piece is the credential-masking gateway: an Envoy egress proxy sits in front of LiteLLM/GitHub/MCP, agents dial per-conversation masked tokens, and Envoy’s ext_authz asks Jello for the real credential per request (resolved from a Postgres credential_grants table). No real secret ever reaches the workspace filesystem; rotation and revocation are just a DB row update, live on the next request with no agent restart. This matches the phased plan in Jello Architecture.md almost exactly.

API design

Plain net/http with Go 1.22+ method+path routing — no framework. Session middleware resolves cookies to a user; a same-origin CSRF guard backs SameSite=Lax. Login itself pivoted mid-stream: it started as generic OIDC against Vault (the originally planned approach) and was later replaced by “Sign in with Coder” (OAuth2+PKCE against Coder directly) — the router comments explicitly flag this as replacing “the Vault OIDC relying party this replaced.” Routes cover auth, conversations (CRUD/messages/attachments/shares), two separate OAuth-linking subsystems (internal/mcpauth for MCP servers, internal/serviceauth for LLM providers — see below), Web Push, the /mcp native-tool surface (its own bearer-JWT scheme, deliberately never accepting the session cookie), and the websocket.

DB schema

Plain SQL via pgx/v5, no ORM — the code’s own doc comment says so directly. Migrations are golang-migrate .sql files embedded and run idempotently at boot. The system grew to 26 sequential migrations (0001_init → 0026_agy_conversation_id), which is a good fossil record of how the design evolved reactively:

• 0001_init: just conversations + workspaces — no message-transcript table yet, because the transcript initially lived only in agentapi, outside Jello’s own DB.
• 0002_credential_proxy: oauth_credentials, credential_grants (with a CHECK (num_nonnulls(...) = 1) constraint), credential_grant_audit.
• 0003_identity_and_sharing: users, sessions, sharing.
• 0010_messages: a durable transcript table only appeared after agentapi was dropped in favor of directly-spawned CLI processes — i.e. the DB became the source of truth only once the architecture forced the issue, not by upfront design.
• 0014–0019: Coder OAuth2 client/user tokens, then MCP OAuth credentials twice (mcp_oauth_credentials, then a separate mcp_oauth_client) — schema churn that directly mirrors the duplicated auth subsystems below.
• 0020–0026: status simplification, attachment files, a “deleted” tombstone state, push subscriptions.

Technology stack

Go 1.26, coder/coder/v2 SDK, pgx/v5, golang-migrate, envoyproxy/go-control-plane (ext_authz gRPC), golang-jwt/v5, the official MCP Go SDK, webpush-go, embedded-postgres for tests (no real Postgres needed in CI). Frontend: hand-written vanilla JS, essentially one ~4,300-line file with vendored dompurify/highlight.js/marked.js — no framework, no virtual DOM, no build step. E2E: Playwright, including specs purpose-built to stress-test UI re-render races under simulated event floods (sidebar-provisioning-barrage.spec.js, rename-barrage.spec.js) — itself a signal of where the pain lived.

What worked well

• Per-harness driver isolation. claudecli/picli/agycli each implement the same harness.Event seam. Antigravity alone needed roughly a dozen follow-up fixes in its first couple months, but that churn stayed contained inside agycli and never leaked into the Claude or pi drivers.
• The credential-masking gateway. Shipped once in Phase 2 (0002_credential_proxy) and needed no further schema changes for the rest of 2.0’s life — the “a DB row is the revocation mechanism” design held up under real use.
• Write-ahead message persistence. After 05badae (#126) fixed a real incident — a conversation’s first prompt could be silently dropped if the websocket died mid-provisioning — no further “lost message” bugs appear in the remaining ~90 commits.
• Deployment simplicity. A single static binary meant zero deploy/packaging bug commits in this era, in explicit contrast to 1.0’s split-migration-container pattern.

Where bugs concentrated (119 commits, ~27 are fixes ≈ 23%)

Category	Approx. count	Representative evidence
Harness/CLI protocol churn	~15	65ff38a (silent pi crashes), b0f70eb (agy install 404), three separate agy model/flag corrections, 3f61d65 (LLM stalls/quota/oauth-expiry surfaced as generic errors)
Auth/credential-linking flip-flops	~10	95576d6 “agy links via real OAuth2 + PKCE (#179)” reverted by 45b3409 “switch agy account linking to a pasted-token flow (#183)” — because the OAuth2 approach violated Antigravity’s own TOS
Frontend re-render/state races (no diffing framework)	~12	sidebar churn from unconditional replaceChildren() on every event frame, model-picker races, in-flight renames clobbered by conversation updates
Workspace/conversation lifecycle	~8	viewing a conversation shouldn’t resume a stopped workspace; stopped→deleted tombstone rename; websocket-vs-fetch race dropping the first message
Mobile/PWA	~6	iOS file picker, mobile focus-zoom, diff-card overflow

Structural root causes, confirmed directly in code:

1. internal/conversation/orchestrator.go is 4,321 lines — session dial/reconnect, event-loop translation, per-harness config writing, and turn bookkeeping all in one file. The project’s own 3.0 planning doc names it as “the change magnet for most regressions.”
2. No frontend diffing. Vanilla-JS replaceChildren() on whole lists per event frame is why sidebar/rename bugs recurred and needed two separate structural fixes from two different event sources.
3. Two independently-built OAuth-linking subsystems — internal/mcpauth and internal/serviceauth, 2,646 combined lines (verified) — that converged on nearly the same provider/resolve/cookie/status shape because each was built per-use-case instead of off one shared token-grant core.
4. Third-party CLI fragility. Pinning exact harness-CLI versions still didn’t stop model lists, flags, and JSON envelopes from drifting, producing a steady trickle of small fixes.

Why it was scrapped for 3.0

Explicit and self-documented in src/2_app/services/jello/plans/3.0-rebuild.md (written alongside the Sep 2 planning commit 91f907e "What would a 3.0 look like?"), not inferred:

• “UI - Vanilla is causing too many issues and merge conflicts… A framework with real diffing/fine-grained reactivity gets that for free.”
• “2.0’s orchestrator ended up as one ~4,300-line file doing session dial/reconnect, event-loop translation, turn/status bookkeeping, and per-harness branching all together… keep those as separate components with narrow seams, not one growing switch statement.”
• “Don’t build two parallel auth subsystems… because they were built per use case instead of off a shared core.”
• The Antigravity OAuth-client reversal is cited directly as a lesson about vendor-TOS risk.
• A DB-tracked “has this session started” flag could drift from what the CLI actually had on disk, silently losing context on resume — flagged as a live, unresolved bug at the time of the rewrite decision.
• An innerHTML-from-markdown XSS surface was flagged as an open security-review item, never closed in 2.0.

There is no evidence of one catastrophic incident forcing the rewrite — it reads as an accumulated-complexity decision backed by specific, named, cited regressions.

---

Jello 3.0 — current architecture (45869fe → HEAD)

Architecture

3.0 splits the single 2.0 binary into two: jello-server (control plane — HTTP/SSE API, Postgres, Coder API calls) and jello-agent (runs inside the Coder workspace itself). The two talk gRPC tunneled over stdio, carried across a single multiplexed Coder SSH session (internal/agentdeploy, pkg/protocol/stdio) — jello-server execs jello-agent over SSH and speaks Protobuf-framed gRPC across that process’s stdin/stdout, rather than dialing a network port. This is a materially more exotic transport than 2.0’s REST-over-port-forward.

Persistence moved from hand-written SQL to Ent (pkg/auth/ent), an ORM whose schema doubles as the wire protocol source: the server↔agent gRPC contract (pkg/protocol/v1/agent.proto) is built entirely on Ent-generated protobuf messages — its own comment states “Zero handwritten message definitions exist in this file.” Ent schema entities: User, Session, Conversation, Turn, TurnEvent, ToolCall, Attachment, TokenGrant, OAuthClient, ProviderUsageCache, ConfigSync.

Harness support moved from three per-CLI packages (claudecli/picli/agycli) to a harnesses/{claude,pi,antigravity} package built against a shared pkg/harness.Harness interface, plus a pkg/catalog runtime-config system (config/catalog.yaml) for harness/model/provider metadata instead of hardcoded Go constants.

The Envoy credential-masking gateway is gone entirely — there is no envoy, ext_authz, or gateway package anywhere in the 3.0 tree. Provider OAuth tokens are now encrypted at rest in Postgres (TokenGrant, AES-256-GCM) but are written in plaintext onto the workspace disk over SSH (~/.claude/.credentials.json, an Antigravity OAuth token file) for the harness CLI to read directly. This is 3.0’s own security-review doc’s finding, not external speculation (plans/3.0-rebuild-security.md), which also flags a hardcoded fallback encryption key ("development-32-byte-secret-key-1234") that silently activates if JELLO_ENCRYPTION_KEY is unset, and a single-pass unsalted SHA-256 KDF. A “Zero-Trust Workspace Secrets Proxy” is listed as target hardening, not yet built.

Go module structure also changed sharply: 2.0 was one module; 3.0 is a go.work workspace of 17 separate go.mod modules (cmd/jello-server, cmd/jello-agent, harnesses/{claude,pi,antigravity}, internal/agent/{orchestrator,config,installer,staging}, internal/{agentdeploy,session,coderclient,testutil}, pkg/{catalog,auth,harness,protocol}).

API design

REST/SSE surface on jello-server, hand-written DTOs (ConversationDTO, TurnDTO, etc. in handlers_conversation.go) that manually mirror the Ent schema — duplicated a third time as hand-written TypeScript types on the frontend. A dedicated plan (plans/TODO-strictly-typed-TS-protos.md) exists to eliminate this by generating TypeScript directly from entpb.proto via protoc-gen-es, not yet executed. The /mcp native-tool surface persists from 2.0’s design intent, now backed by jellomcptoken.

DB schema

Same conceptual entities as 2.0 (conversations, turns/messages, users, sessions, token grants) but modeled through Ent rather than raw SQL, with schema changes now flowing through Ent’s codegen rather than hand-written migration files. Turn and TurnEvent carry deliberately-commented design decisions worth noting as good practice — e.g. Conversation.queue_paused and Turn.queue_seq/client_turn_id have inline comments explaining exactly why they’re persisted columns rather than derived state (avoiding a denormalized-status-drift bug class 2.0 also hit).

Technology stack

Go 1.26, Ent ORM + entproto/protobuf codegen, gRPC (google.golang.org/grpc), a 17-module go.work. Frontend switched from 2.0’s vanilla JS to React 19 + Vite 6 + Tailwind 3 + TypeScript, addressing the “vanilla is causing too many issues” complaint from the 3.0 planning doc directly. Testing keeps Playwright E2E and adds Go “wire replay” tests per harness that replay recorded CLI fixtures offline.

Where bugs concentrated so far (17 commits since merge, 14 are fixes — 82%)

This is the headline number: 2.0 ran at a ~23% fix-commit ratio over its two-week life; 3.0 has run at 82% over its first four days, including four consecutive “fix jello build” commits on day one:

Commit	Issue
c4c2eaa	GitHub MCP OAuth secret key name mismatch
dda23cd	Turn didn’t cancel when the client disconnected mid-stream
b199d41	Double-send crash — concurrent session bootstraps weren’t deduped
ed6b76f	Second, separate double-send crash — turn execution wasn’t queued per conversation
cea26e2	Queue state management overhaul
9d6e8b2	pi install source was wrong (@earendil-works/pi-coding-agent)
4e9819f, 71fa0c6	Mobile: iOS auto-zoom, single-tap navigation
0754be5, b3cf430	Antigravity usage tracker wasn’t wired to the real quota API; false “stall” detection
8a7b3ee	Live stream events blocking user-message hydration

The two double-send crashes are the clearest structural signal, and there’s a first-party root-cause doc for it (plans/queued-message-state-management.md) that is worth reading in full — its opening line: “There is no queue anywhere in jello 3.0.” Five rapid sends produced five concurrent fire-and-forget SSE streams racing one single-turn orchestrator, because:

• The client’s canSend never checked conversation status, and sessionMapRef/abortControllersRef were single slots silently overwritten per send.
• The server had no busy check — handleExecuteTurn would write a new Turn row and dispatch even if one was already running.
• The orchestrator rejected the second call, but destructively: the loser’s deferred cleanup stole the winner’s in-flight state, and the UI’s “activity” status was one unowned scalar written by six different call sites, five of them unconditionally — so whichever turn finished first flipped the whole conversation to idle regardless of what was still running.

This is a case of a feature that already existed in an earlier codebase (2.0’s TypeScript-era predecessor, per the doc’s own note that a message queue was written against “the old TypeScript 2.x stack” and simply never ported) not making it into either Go rewrite — 2.0 didn’t have it either, and 3.0 shipped without it a second time, discovering the gap only after real multi-message use.

Assessment

3.0’s core bets — React over vanilla JS, splitting the orchestrator god-object, unifying the two auth subsystems, schema-driven wire types — are each a direct, reasoned response to a specific, named 2.0 pain point, not scope creep. The frustration isn’t that the bets are wrong; it’s that 3.0 introduced new architectural surface area at the same time: a genuinely unusual transport (gRPC-over-stdio-over-SSH) instead of 2.0’s plain REST-over-port-forward, a schema-generates-wire-protocol coupling (Ent → entproto → gRPC) instead of hand-written DTOs, and a 17-module Go workspace instead of one module (with a documented ~7-minute cold build as a direct consequence). Each is defensible in isolation, but landing all of them simultaneously — plus dropping the credential-masking gateway that was 2.0’s one clearly-successful subsystem — means 3.0 is re-litigating problems 2.0 had already solved (credential isolation, basic message-queueing) while also debugging brand-new machinery (the stdio transport, the codegen pipeline) that 2.0 never had to worry about. That combination is a reasonable explanation for why 3.0 reads as more frustrating despite being the better-reasoned rewrite on paper.

Suggestions

1. Port the message queue immediately — it already exists conceptually (was built once for the TS stack, and 3.0 has a full root-cause doc and design already written in plans/queued-message-state-management.md); this is 3.0’s highest-leverage open bug.
2. Reinstate credential masking before wiring up more providers. Writing real OAuth tokens to workspace disk is a regression from 2.0’s best subsystem, and 3.0’s own security doc already scopes the fix (“Jello Secrets Proxy”).
3. Fix the encryption-key footgun now, not later — refuse to boot on the hardcoded dev key outside dev mode; this is a five-line change against a real production risk.
4. Treat the 17-module split and gRPC-over-stdio transport as the two highest-risk novel bets — they have no precedent in 2.0 to fall back on, so budget disproportionate testing/debugging time against them rather than assuming they’re “just infrastructure.”