The Kikapu Book

A complete, in-depth account of what Kikapu is, how every part of it works, why it was built that way, and what remains to be built.

Reading order: Part I gives you the idea and the vocabulary. Part II is a copy-paste guided path through a controlled development deployment. Part III explains the MVP machine that path uses. Parts IV and V explain trust and economy. Parts VI and VII cover operations, proof, and honest roadmap status.

This document is a rendered view. The authoritative contracts live in plan/architecture.md, plan/architecture-v2.md, plan/principles.md, and plan/design-decentralization-governance.md; where this book and those files disagree, the plan files win.

Table of contents

• Part I — The Idea
  • 1. What Kikapu is
  • 2. System topology
• Part II — Complete guided path
  • 3. First controlled deployment
• Part III — The Machine (MVP core)
  • 4. The domain model
  • 5. Identity and authentication
  • 6. Tenancy and workload isolation
  • 7. Scheduling
  • 8. The agent
  • 9. The tunnel
  • 10. Routing and TLS
  • 11. Telemetry
• Part IV — The Trust Plane
  • 12. Operating modes and local survival
  • 13. Signed state
  • 14. TUF and the supply chain
• Part V — The Economy
  • 15. Compute Units
  • 16. The ledger
  • 17. Usage receipts and disputes
• Part VI — Operations and Assurance
  • 18. Configuration and deployment
  • 19. Verification
  • 20. Process and governance
• Part VII — The Road Ahead
  • 21. Phase history
  • 22. Completed: Phases 20–24 and Phase 26 proof
  • 23. Deferred: Phase 25
  • 24. Open problems and launch debt
• Appendices
  • A. Constants and limits
  • B. Database migrations
  • C. Tunnel stream grammar
  • D. Glossary
  • E. File-path map

Part I — The Idea

1. What Kikapu is

1.1 The one-paragraph definition

Kikapu (Swahili: basket) is a self-hosted platform for running tenant workloads on community-donated Linux nodes. A central control plane (kikapu-server + PostgreSQL + Caddy) schedules containerized deployments; a daemon on each donated node (kikapu-agent) runs them with rootless Podman under per-tenant Linux users; and Caddy serves them to the public internet over HTTPS through persistent, agent-initiated WebSocket/yamux tunnels — the node never accepts an inbound connection. Every user is simultaneously an account and, through a one-to-one tenant, a deployment owner. The module path is github.com/isnakolah/kikapu; everything is Go (pinned toolchain 1.25.12), MIT-licensed, and developed in numbered, documented phases under plan/.

1.2 The problem it solves

Ordinary people have spare compute — a homelab box, an old laptop, a VPS with headroom — and other people need places to run small services. Connecting the two is hard for three reasons, and each one shaped Kikapu's architecture:

1. Donated machines are hostile territory in both directions. The platform cannot trust a donor's machine, and a donor cannot be expected to trust arbitrary tenant code. Kikapu's answer is layered tenant isolation (Chapter 6) combined with a brutally honest trust boundary: tenants are isolated from each other, but a malicious node owner with host root can read anything a workload does on their machine. Kikapu never pretends otherwise — the Acceptable Use Policy tells tenants plainly they "must not submit secrets or data they are unwilling to disclose to the selected node owner."
2. Donated machines sit behind NAT, firewalls, and dynamic IPs. You cannot route to them, port-forward to them, or give them DNS names. Kikapu's answer is the agent-initiated tunnel (Chapter 9): the node dials out; the center never dials in; nodes need no public ingress at all.
3. A volunteer network dies when its center dies. Central infrastructure will go down, get partitioned, or — worst case — be compromised. Kikapu's answer is the north-star invariant that governs the entire v2 wave of development:

The north-star invariant. Kikapu Cloud can disappear, the mesh can partition, and a coordinator can be compromised — yet every node should still know what it is running, who it trusts, which version is valid, whether it is safe to accept new work, and how to recover from a signed checkpoint.

This is not aspirational copy; it is proven by a named test. TestOutOfBandRecovery in test/e2e/zz_recovery_test.go removes the central plane entirely and asserts that the agent — using its signed desired-state cache and trust checkpoint — keeps its running deployment alive and correct with no control plane in existence.

1.3 What Kikapu is not (yet)

• Not a marketplace. Public marketplace scheduling is explicitly disabled pending legal review (Q-DOC3). Today Kikapu is private-deployment only: an operator runs a control plane for a community they administer.
• Not confidential compute. There is no claim that a workload is hidden from the host it runs on. The isolation goal, ratified as decision Q-DOC1, is precise: tenants are isolated from each other; node owners choose which workload classes they accept; no more is promised.
• Not a blockchain. The compute-unit economy (Part IV) is deliberately a non-transferable barter ledger in PostgreSQL, signed with Ed25519 hash chains. Decision Q-DOC11 rules that tokens, transferable CU, and on-chain settlement are unplanned and would require a separate design pass after Phase 25.
• Not finished. Phases 0–24 are implemented and validated at development level; production launch validation remains pending for Phase 21. Phase 25 self-publishing and governance is deferred by maintainer direction. Part VI catalogs open problems and launch debt.

1.4 The two horizons

Kikapu was built in two deliberate waves, and the codebase visibly contains both:

• The MVP (phases 0–13, plus hardening phases 14–19). A classical centrally-scheduled platform: one control plane, many agents, tunneled ingress, telemetry, a CLI, packaging, and an end-to-end test harness. This is Part II of this book. It is complete: all twenty architecture acceptance criteria exist as named tests.
• The v2 wave (phases 20–25). Governed by a formal design document (plan/design-decentralization-governance.md, imported from the maintainer's Kikapu Decentralization, Fallback & Governance v2), this wave converts the MVP into a decentralized, locally-survivable, federated compute fabric with a barter economy — "degrade locally before failing globally." Its threat model names seven adversaries (freeloader, Sybil, malicious host, malicious workload, colluding coordinator ring, reputation poisoner, key thief), and each phase of the wave is gated by a maintainer go/no-go checkpoint (G1–G5) and by recorded design rulings (Q-DOC1 through Q-DOC13). Parts III, IV, and VI cover this wave.

1.5 The security ratchet

One meta-rule governs every change and deserves front-page placement: implemented security is a floor, never a ceiling. When the v2 design document's stated non-goals appeared to conflict with the isolation the MVP had already shipped, the ruling (Q-DOC1, generalized into plan/principles.md and invariant-register entry I10) was that safeguards already in place may only be strengthened. Any proposed change that would weaken a shipped safeguard is cut. A concrete corollary: the rootless-Podman hardening flag set (Chapter 6) is declared never renegotiable by any phase — it is the only invariant in the whole conflict register with no renegotiation path at all.

1.6 Vocabulary you need for the rest of the book

2. System topology

2.1 The three parties and seven binaries

Three parties interact: the operator (runs the central plane), the node owner (donates a machine), and the tenant (deploys workloads). A fourth — the anonymous end user — just opens https://<app>.<tenant>.<root-domain> in a browser.

Seven binaries are built from cmd/:

2.2 The shape of the network

flowchart LR
  subgraph central [Central plane — only 80/443 exposed]
    caddy[Caddy] --- srv[kikapu-server]
    srv --- pg[(PostgreSQL)]
  end
  subgraph node [Donated node — no inbound ports at all]
    agent[kikapu-agent] --- pod[rootless Podman workloads<br/>bound to 127.0.0.1 only]
  end
  browser((Browser)) -- HTTPS --> caddy
  cli((kikapu CLI)) -- HTTPS API --> caddy
  agent -- outbound WebSocket+yamux tunnel --> srv

The single most important topological fact: the node initiates all connectivity. The agent registers outbound, heartbeats outbound, polls outbound, and opens the tunnel outbound. The control plane never dials a node address; the database schema does not even have columns for node public addresses or host ports (they were removed in Phase 1 and their absence is an acceptance criterion).

Why this direction? Three reasons compound. Practical: donated machines live behind NAT and dynamic IPs; outbound-only works everywhere without port forwarding, UPnP, or STUN. Security: a node with no listening ports has no remotely reachable attack surface — an attacker who scans a donor's home IP finds nothing that belongs to Kikapu. Policy: if user requests could name nodes or ports, tenants could probe or target specific donors; forbidding placement data in every request (a non-negotiable invariant, enforced at the API layer and in the CLI, which accepts no node/port flags at all) makes the donor pool anonymous from the tenant's point of view.

2.3 Life of a request (the 30-second version)

1. Provision. Operator runs admin create-user (user + tenant created atomically, with immutable Linux identity and a reserved subordinate-ID block) and mints keys. A node owner runs kikapu node init, which provisions a node record and writes the node key into /etc/kikapu/agent.yaml.
2. Register. The agent boots, loads or mints its Ed25519 node identity at <data-dir>/trust/node.key, and registers: version, runtime capabilities, offered classes, and its receipt public key (pinned by the server on first registration).
3. Tunnel. The agent dials GET /api/v1/agent/tunnel with its bearer node key, negotiates the kikapu.tunnel.v1 WebSocket subprotocol, and a yamux session is registered under its node ID. The routing manager hears "node connected" and reconstructs Caddy routes.
4. Deploy. A tenant runs kikapu deploy -f app.yaml. The server validates class + AUP, then — inside one SERIALIZABLE transaction — picks the least-loaded fitting node and inserts the deployment with its capacity reservation.
5. Converge. The agent's 5-second poll sees the new desired deployment. The reconciler ensures the tenant's Linux user exists, pulls the image, and starts a hardened rootless container publishing exactly one port on 127.0.0.1. It reports lifecycle states up the legal transition graph.
6. Route. Caddy answers https://app.tenant.example.org, asks the server's ACME ask endpoint whether the hostname is real, obtains a certificate on demand, and proxies to a central loopback listener, which opens a PROXY/1 stream over the node's tunnel; the agent dials 127.0.0.1:<current-port> and bytes flow. A traffic meter counts every proxied byte — evidence the economy will later demand.
7. Observe. Logs batch up LOG/1 streams with exactly-once DB semantics; live stats answer on STATS/1; Prometheus counters ship on METRICS/1.
8. Meter (economy on). A scheduler challenges random running deployments over RECEIPT/1; the agent measures a real interval and returns a node-signed receipt; after a 24-hour dispute window a finalizer converts accepted receipts into signed double-entry CU postings.

Each numbered step is a chapter of Part II–IV.

2.4 Code layout and the dependency discipline

All logic lives under internal/ (247 Go files) behind narrow packages; cmd/ binaries are thin shells. The layering rules are explicit and tested by review discipline:

• Handlers depend on services and repositories, never on a raw connection pool.
• The CLI and server never import each other — the only shared shapes are internal/manifest (the deployment manifest) and the DTO packages internal/api/request / internal/api/response.
• Domain types (internal/domain) carry no persistence or transport tags. They are pure Go: IDs, entities, state machines, sentinel errors.
• Changes go behind the narrow interfaces — Runtime, Stager, Runner, Manager, FileSystem, APIClient — and never bypass the validation chokepoints (tenant.Context.Validate, validateOperationIdentity, validateSpec).

Package groups:

Why this shape? The project charter (CONTRIBUTING-AI.md) is written for a development process in which AI agents make most changes. Narrow interfaces are the mechanism that makes that safe: every dangerous operation (spawning containers, editing /etc/subuid, opening tunnels) funnels through one small, heavily-tested seam, so a change that tries to route around a safety check is structurally visible in review. The technology choices follow the same conservatism: chi for routing (plain net/http handlers, middleware as a visible ordered stack), cobra for CLIs, pgx for PostgreSQL with sqlc-generated typed queries (SQL is written by hand, reviewed as SQL, and compiled to Go — no ORM magic between the reviewer and the query that enforces an invariant), and golang-migrate-style embedded SQL migrations that ship inside the binaries so a server can migrate itself at boot.

Part II — Complete guided path

3. First controlled deployment

This chapter is beginner path through one controlled Ubuntu 24.04-style central plane, one donated node, one tenant deployment, and its public HTTPS route. It proves development flow, not production launch readiness. Authoritative contracts remain plan/architecture.md, plan/architecture-v2.md, and plan/README.md.

3.1 Vocabulary, roles, and safety boundary

• Operator runs central plane, DNS, TLS, database backups, and key custody.
• Node owner donates Linux host and may drain or quarantine workloads.
• Tenant owns manifest and deployment lifecycle; scheduler chooses node, never tenant.
• Browser user reaches public HTTPS only.
• Root domain is operator-owned DNS suffix such as apps.example.net; control plane is control.<root-domain>, while workload URL is <app>.<tenant>.<root-domain>.

flowchart LR
  operator[Operator\nDNS, central plane, custody]
  tenant[Tenant\nmanifest and lifecycle]
  owner[Node owner\ndonated Ubuntu host]
  browser((Browser))
  subgraph trust1 [Operator trust boundary]
    central[PostgreSQL + kikapu-server + Caddy]
  end
  subgraph trust2 [Node-owner trust boundary]
    agent[kikapu-agent]
    podman[Rootless Podman\nper-tenant Linux user]
  end
  operator --> central
  tenant -- authenticated API --> central
  owner --> agent
  agent -- outbound authenticated tunnel --> central
  central --> browser
  agent --> podman

Tenant isolation is real; confidentiality from node-owner root is not. Node owner can inspect environment values, files, processes, and Podman state. Do not deploy secrets or data you would not hand that owner. Raw API keys are shown once; store them in a secret manager, never shell history, source files, logs, screenshots, or this book.

3.2 Prerequisites and inputs

Before command groups, prepare following. Brackets mean operator-provided input, never a default to invent.

DNS expectation: control.<root-domain> resolves to central host. For default on-demand TLS, each deployed application hostname must also resolve there. A wildcard *.<root-domain> does not cover <app>.<tenant>.<root-domain>; see operations TLS modes.

3.3 Start central plane and mint first tenant credential

On central host, set operator-owned values in deploy/.env. Generate password locally; never copy placeholder password into real installation.

cd /path/to/kikapu/deploy
cp compose.env.example .env
chmod 0600 .env
# Edit .env: set KIKAPU_POSTGRES_PASSWORD and KIKAPU_ROOT_DOMAIN.
# Pin KIKAPU_SERVER_IMAGE to approved release image or digest before production use.
docker compose config
docker compose up -d
docker compose ps

Expected: postgres, kikapu-server, and caddy become healthy; only TCP 80/443 and UDP 443 are published. docker compose config failure means missing or malformed environment input. Unhealthy server means inspect docker compose logs kikapu-server caddy postgres; do not expose server API or PostgreSQL directly to bypass Caddy.

After DNS propagates, set shell-only value [ROOT_DOMAIN] and verify liveness plus database readiness:

export ROOT_DOMAIN='[operator-owned root domain]'
curl --fail --silent --show-error "https://control.${ROOT_DOMAIN}/healthz"
curl --fail --silent --show-error "https://control.${ROOT_DOMAIN}/readyz"

Expected: both return success. healthz failure means HTTP process or DNS/TLS path unavailable; readyz failure means server cannot serve against PostgreSQL. Resolve before creating users.

Create initial user and tenant, then one laptop key. Command emits raw key exactly once; capture it directly in approved secret storage and do not paste it into documentation or command lines.

docker compose exec kikapu-server kikapu-server admin create-user --name alice
docker compose exec kikapu-server kikapu-server admin create-key --user alice --name laptop

Expected: user creation atomically creates one tenant with immutable Linux identity; key command returns one credential. Failure means keep raw credential out of retry output, fix named validation or server error, then retry only required command.

3.4 Bootstrap donated node and verify its tunnel

On node owner machine, install release artifacts only after staging bootstrap trust inputs from an already verified installation or maintainer-approved offline channel. Do not download 1.root.json or verifier from release being verified. CI-only kikapu-tuf init-keys artifacts are not production custody defaults.

First log in as node owner or operator account authorized to provision this node. Use masked prompt, KIKAPU_API_KEY, or stdin; no --api-key flag exists by design.

kikapu login --server "https://control.${ROOT_DOMAIN}"
sudo install -d -m 0700 /secure/kikapu
sudo install -m 0600 '[path-to-operator-provided-1.root.json]' /secure/kikapu/1.root.json
sudo install -m 0755 '[path-to-trusted-kikapu-tuf]' /secure/kikapu/kikapu-tuf

Expected: login validates key before persisting mode-0600 CLI configuration; install commands leave root artifact and verifier inaccessible to other users. Failure to secure either path is an installer stop, not permission to weaken file modes.

Create manifest and provision node configuration without starting service. classes is optional; default general remains sufficient for this path.

cat > node.yaml <<'YAML'
api_version: v1
name: donor-one
classes: [general]
resources:
  cpu: 4
  memory: 8Gi
  disk: 100Gi
YAML
sudo kikapu node init --no-start --file node.yaml

Expected: control plane records node and writes one node-scoped credential to /etc/kikapu/agent.yaml with mode 0600. Do not rerun against existing config expecting overwrite; inspect and repair intentional configuration instead.

Install approved release. Replace bracketed paths/version with release-custodian inputs.

sudo '[path-to-install-agent.sh]' \
  --version '[RELEASE_VERSION]' \
  --config /etc/kikapu/agent.yaml \
  --tuf-root /secure/kikapu/1.root.json \
  --tuf-client /secure/kikapu/kikapu-tuf
sudo systemctl status kikapu-agent --no-pager
journalctl -u kikapu-agent -n 80 --no-pager
kikapu node list
kikapu node inspect '[node-id-from-node-list]'

Expected: installer verifies TUF metadata and archive before install, enables agent, agent registers and opens outbound WebSocket/yamux tunnel, and inspect eventually reports node online. TUF, configuration, dependency, or signature errors fail before installation. Online without an active tunnel is not enough for public routing; node inspect is tunnel check. No inbound node port or node-public-address setup exists.

3.5 Deploy, browse, and operate workload

Tenant logs in with own stored key, then creates manifest. This is deliberately small public nginx example; image and environment remain visible to node-owner root. aup: v1.0 is required, and tenant cannot name node or host port.

kikapu login --server "https://control.${ROOT_DOMAIN}"
cat > kikapu.yaml <<'YAML'
api_version: v1
name: hello
image: docker.io/nginxinc/nginx-unprivileged:1.27-alpine
class: general
aup: v1.0
resources:
  cpu: 0.5
  memory: 256Mi
  disk: 1Gi
container_port: 8080
environment:
  APP_ENV: production
YAML
kikapu deploy --file kikapu.yaml
kikapu deploy inspect hello

Expected: scheduler picks fitting online node in one serializable transaction; agent creates tenant Linux identity and starts rootless hardened container bound only to node loopback. Identical re-apply is no-op. Admission failure usually means missing AUP/class or insufficient fitting capacity; never add placement data because interface rejects it.

URL shape is https://hello.<tenant-name>.${ROOT_DOMAIN}. Copy exact tenant/hostname shown by deployment inspection rather than guessing tenant name, then test it:

export DEPLOY_URL='[hostname reported for hello deployment]'
curl --fail --silent --show-error "$DEPLOY_URL"
kikapu logs --tail 200 hello
kikapu stats hello

Expected: Caddy terminates HTTPS, central tunneld proxies through authenticated node tunnel, agent dials current node-loopback port, and nginx response returns. Logs/stats prove tenant-scoped control paths. TLS failure normally means DNS or ACME ask eligibility; tunnel/route failure means inspect deployment and node before changing DNS.

Apply image/environment update through same manifest, then exercise lifecycle. Resource, name, class, and port fields are immutable after creation; change only image/environment here.

# Edit only image and/or environment in kikapu.yaml, then:
kikapu deploy update --file kikapu.yaml
kikapu restart hello
kikapu stop hello
kikapu deploy inspect hello
kikapu restart hello
kikapu remove hello

Expected: update converges in place; restart increments generation; stop preserves /data and withdraws public route; restart restores desired run; remove deletes container and deployment data and releases routing/capacity. Use kikapu logs --follow hello before removal when debugging.

3.6 Node-owner withdrawal controls

Drain is planned withdrawal: stop scheduling new work, then stop agent. Existing containers keep running until agent stops; public routes depend on active tunnel and therefore are withdrawn once agent stops.

kikapu node stop '[node-id]'

Quarantine is emergency local refusal for one deployment. It stops container on next reconciliation (normally five seconds), withdraws route, reports failed, and blocks later run convergence. removed intent still cleans up data.

sudo kikapu-agent --config /etc/kikapu/agent.yaml \
  --quarantine-deployment '[deployment-id]' \
  --quarantine-authority 'node-owner:incident-2026-07-17'
# Only same stable authority may reverse it:
sudo kikapu-agent --config /etc/kikapu/agent.yaml \
  --clear-quarantine '[deployment-id]' \
  --quarantine-authority 'node-owner:incident-2026-07-17'

Expected: drain removes node from new placement; quarantine fails closed if state file ownership, mode, symlink status, or content is unsafe. Do not edit quarantine JSON to bypass custody check.

3.7 Completed v2 journey: deliberate gates, not beginner defaults

flowchart LR
  local[Phase 20\nlocal survival and modes] --> trust[Phase 21\nsigned identity, TUF, checkpoint]
  trust --> receipts[Phase 22\nreceipts and CU ledger]
  receipts --> mesh[Phase 23\npinned coordinator failover]
  mesh --> reputation[Phase 24\nindependent validation and reputation]
  reputation --> settlement[Settlement and recovery\nreplay, checkpoint, rollback]

Advanced root custody, validator membership, production TUF keys, economy enabling, private coordinator setup, and reputation thresholds are operator gates. They are not copy-paste beginner defaults. Phase 25 remains deferred; launch needs real elapsed-time soaks, production custody, named custodians, managed-database recovery evidence, and legal approval before marketplace use.

Part III — The Machine (MVP core)

4. The domain model

4.1 Identifiers

Every entity ID is a lowercased ULID with a type prefix, minted by newID(prefix) in internal/domain/id.go: usr_, tnt_, key_, node_, dep_, evt_, and (economy) led_, rcpt_, chl_, dsp_, eco_.

Why ULIDs, and why prefixes? ULIDs are 128-bit like UUIDs but lexicographically sortable by creation time, which keeps B-tree indexes append-friendly and makes raw IDs useful in debugging ("this one is older"). The type prefix does two jobs: it makes logs self-describing, and it lets validators reject cross-type confusion cheaply — the tunnel's header parser, for example, enforces dep_ + 32 chars of [a-z0-9] before any handler runs, so a stream can never smuggle a node ID where a deployment ID belongs. IDs are also unguessable, which is why the known existence-leak problem P-008 (cross-tenant reads return 403 instead of 404) is graded negligible.

4.2 Sentinel errors

internal/domain/errors.go defines the error vocabulary the whole system speaks with errors.Is/%w: ErrNotFound, ErrUnauthenticated, ErrForbidden, ErrInsufficientResources, ErrTenantCapacity, ErrTemporarilyUnavailable, ErrInvalidArgument, ErrConflict, ErrInsufficientCredits ("insufficient compute units"), and three security-specific sentinels — ErrPrivilegedContainer, ErrDangerousMount, ErrHostNetworking — whose existence tells you where the runtime validator draws its red lines. Handlers map these to HTTP problem responses; repositories map PostgreSQL errors into them (repo/pg/common.go: mapErr). One vocabulary end-to-end means a serialization failure deep in a repository and a bad manifest at the API edge both surface predictably.

4.3 The entities

Node (domain/node.go): {ID, OwnerUserID, Name, Status, OfferedCPUCores float64, OfferedMemoryMB, OfferedDiskMB, Observed* (pointers — absent until first heartbeat), RuntimeCapabilities []string, OfferedClasses []string, AgentVersion, ReceiptPublicKey []byte, OperatingMode, LastHeartbeatAt, CreatedAt}. Capability strings are constants: rootless-container, disk_quota:hard, disk_quota:soft. NodeStatus is pending | online | offline | draining.

Tenant: 1:1 with user, created in the same transaction. Immutable linux_user matching kikapu_t_*, plus subuid_start/subgid_start — the base of its 65,536-ID subordinate block. Immutability is enforced in the database by the reject_tenant_identity_change trigger, not just in Go: even a buggy service cannot rename a tenant's Linux identity.

Deployment: name (unique among a tenant's active deployments), image, immutable class, exact aup_version, requested cpu/memory/disk, container_port, ephemeral node_local_port (the kernel-assigned loopback port on the node, re-read after every restart), desired_state ∈ {run, stopped, removed}, a 12-value status, restart_counter (the generation), report_sequence, and environment (plaintext env jsonb legacy column or env_ciphertext/env_key_version when envelope encryption is on). The public hostname <name>.<tenant>.<root-domain> is computed, never stored — one less thing to drift.

API key: scope user XOR node (a database CHECK), unique hash, revocation timestamps, and (Phase 21) at most one attached signed capability.

4.4 The deployment state machine

The twelve statuses and their legal edges live in one place — domain/status.go's allowedTransitions map — and both the server and the agent's reconciler consult the same table:

pending → scheduled → assigned → pulling_image → starting → running
running → updating   (image/env changed)        → starting
running → restarting (restart_counter++)        → starting
running → stopped | failed | removed
starting → failed
stopped → starting | removed
failed  → starting | removed
rescheduling → scheduled          (opt-in reschedule; fresh row)
removed → (terminal)

Self-transitions are always valid (idempotent re-reports). Any report outside these edges — or carrying a stale generation — is rejected with HTTP 409 without mutating state.

Why one shared table? Because there are two writers. The server mutates status when scheduling and sweeping; the agent reports status as containers move. If each side had its own idea of legality, an agent restart replaying old observations could drag a deployment backwards (e.g., re-report starting after the server marked it removed). With a single table plus two guards — the transition check and the generation guard (ReportStatus(nodeID, id, generation, sequence, status, port) only applies when generation == restart_counter and the sequence advances) — stale writers are structurally harmless. The integration test TestGenerationGuardRejectsEveryStaleGeneration enumerates this exhaustively.

4.5 The capacity-reservation matrix

ReservesCapacity(desired, status) answers the one question the scheduler cares about: does this deployment row currently hold resources on its node? Only desired = run combined with an active status reserves capacity; stopped, failed, and removed rows release it; rescheduling reserves on its freshly placed node (the stranded predecessor row was flipped to desired = removed, releasing the old node). This matrix is one of the four cross-phase contracts pinned in plan/architecture.md, and the SQL that computes free capacity is proven equivalent to the Go function by TestCapacitySQLMatchesDomainReservationMatrix — the same matrix, tested in both languages, because a disagreement between them is an oversubscription bug.

5. Identity and authentication

5.1 What a key is

A raw key is kk_<keyID>_<secret>. The server stores only bcrypt(secret) (cost 12) keyed by hash; the raw value is printed exactly once at mint. Scope is user or node, never both. Identity always derives from the key, never from the request body — this is a non-negotiable invariant; agent register and heartbeat payloads do not even have a node_id field to lie in.

Why bcrypt for machine keys? bcrypt's deliberate slowness is designed for low-entropy human passwords; Kikapu's secrets are high-entropy random strings where a fast keyed hash would be cryptographically sufficient. The question was litigated as problem P-007 and bcrypt was kept deliberately: it is the boring, well-audited choice, and its real cost — CPU burn per verification, which is a denial-of-service surface at cost 12 — is neutralized by the caching layer below. This is the security ratchet in action: replace a defense only when the replacement is strictly better, not merely faster.

5.2 The verification cache

Every request would otherwise cost a bcrypt verification, so internal/server/auth wraps the resolver in a short-TTL cache (KIKAPU_AUTH_CACHE_TTL, default 5 s, hard cap 60 s) whose entries are keyed by HMAC with a per-process random salt — the cache never stores raw secrets or even raw hashes, so a memory disclosure of the cache reveals nothing reusable. Two refinements matter:

• Negative caching (Phase 15): a repeated wrong secret costs one map lookup, not one bcrypt run. Without this, an attacker hammering a single invalid key would burn a bcrypt core per request.
• A bcrypt semaphore bounds concurrent verifications, so even cache-miss storms degrade gracefully instead of starving the scheduler.

The documented consequence, stated honestly in the operations guide: a revoked key remains valid for up to one cache TTL. Revocation is eventually consistent within 5 seconds by default, and the admin CLI path (which runs outside the server process and cannot purge its memory) documents exactly that window.

5.3 Rate limiting

Per-client-IP token buckets, tightest where it hurts attackers most: general API traffic at 20 req/s, burst 40; authentication failures at 1/s, burst 4. Client identity is the direct TCP peer unless it falls inside KIKAPU_TRUSTED_PROXIES, in which case the rightmost untrusted hop of X-Forwarded-For is used — the standard defense against header spoofing (never trust X-Forwarded-For from an arbitrary peer; only unwrap it across proxies you configured). 429 responses carry Retry-After: 1. The regression test name states the property: TestInvalidAuthFloodCannotDelayDifferentClient.

5.4 The middleware stack

The chi router applies a fixed, ordered stack — order is the security argument, so it is worth reading as a sentence: RequestID → Metadata → Logging → Recovery → BodyLimit (1 MiB) → RateLimit → Auth → Timeout. Bodies are capped before auth (a 32 MB body should never reach bcrypt); rate limiting sits before auth (unauthenticated floods are shed before they cost anything); timeouts wrap normal routes but exclude the streaming routes (tunnel upgrade, log follow), which manage their own deadlines — otherwise the 30-second HTTP write timeout would sever every tunnel.

5.5 Self-service key lifecycle

Phase 18 added user-scoped GET/POST/DELETE /api/v1/keys (metadata only; the raw key appears once in the create response) with an active-key cap (KIKAPU_MAX_ACTIVE_USER_KEYS, default 10, range 1–100) enforced by locking the owning user row before counting — two concurrent creates cannot race past the cap. Node keys remain admin-minted only. This endpoint is deliberately the seam where Phase 21 plugs in: when a trust root is configured, every key minted here is silently accompanied by a signed capability (Chapter 13), turning bare bearer tokens into revocable, signature-backed grants without changing the API surface.

6. Tenancy and workload isolation

This chapter is the heart of the security story. The design question: how do you run a stranger's container on a volunteer's machine such that (a) two strangers cannot touch each other, and (b) neither can touch the host more than the host owner agreed to? Kikapu's answer stacks four independent Linux mechanisms, so a failure of any single layer is not a compromise.

6.1 Layer 1 — a real Linux user per tenant

The Tenant Manager (internal/agent/tenant, contract in docs/tenant-manager.md) is the only component allowed to provision host identity, and it runs as root. Per tenant it creates: a system user named exactly kikapu_t_<suffix> with shell /usr/sbin/nologin, home /var/lib/kikapu/tenants/<tenant_id> mode 0700; one entry each in /etc/subuid and /etc/subgid; systemd lingering enabled plus a genuine wait for /run/user/<uid> (a readiness condition, not a sleep). Every filesystem inspection uses lstat and rejects symlinks, wrong owners, and wrong modes; subordinate-ID files are parsed in full before mutation (malformed, duplicate, or overlapping entries fail closed) and rewritten via temp-file + fsync + rename + directory fsync — the classic crash-safe file replacement, because a truncated /etc/subuid would break every tenant on the box.

Technology: why systemd linger matters. Rootless Podman needs a per-user systemd instance and a runtime directory (/run/user/<uid>) to manage cgroups and the user's D-Bus. Those normally exist only while a user is logged in. loginctl enable-linger makes them permanent for the tenant's never-logging-in system user. The agent passes rootless Podman exactly three environment variables — HOME, XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS — through sudo -u <tenant> env, scrubbing everything else.

6.2 Layer 2 — user namespaces and the subordinate-ID grid

Each tenant owns a globally-disjoint block of 65,536 subordinate UIDs/GIDs, allocated on a grid starting at base 2,000,000,000, with the allocator serialized and capped at 35,018 ranges (the arithmetic ceiling before 32-bit UID space runs out). Containers run with the fixed map 0:1:65536: container root is the tenant's first subordinate UID — a number that owns nothing on the host except that tenant's own container files.

Technology: what a user namespace buys you. Inside the container, the process believes it is UID 0 and can do UID-0 things to its own namespace (install packages, bind port 80 internally). On the host, the kernel translates every credential through the map, so "container root" is an unprivileged high UID. A container escape — a runtime bug that lets the process see the real filesystem — lands the attacker as UID ~2,000,000,000-something with no privileges and no reachable files belonging to any other tenant, because every other tenant's block is disjoint by construction. Why 65,536 per block? It is the conventional full range a container image expects (UIDs 0–65535 all mappable), so arbitrary images run unmodified. Why base 2e9? High enough to never collide with human users, system users, or typical corporate LDAP ranges on the donor machine.

Technology: why crun, not runc. The OCI runtime is pinned to crun because runc does not reliably support these high, fixed rootless mappings. This is a documented hard requirement checked by the installer.

6.3 Layer 3 — the hardening flag set

Every container is created through the typed internal/runtime interface — callers cannot supply usernames, Podman arguments, namespaces, devices, capabilities, sockets, or host paths, and the agent never composes a shell command (structured argv through an injected runner; injection-proof by construction). The flags themselves, enforced by internal/runtime/podman_test.go and never renegotiable:

• --cap-drop ALL and --security-opt no-new-privileges — no capabilities, and no re-gaining them via setuid binaries.
• --read-only root filesystem, with exactly one writable tmpfs: /tmp, 64 MiB, nosuid,nodev,noexec.
• --pids-limit 1024 — fork bombs die at a fence.
• CPU and memory cgroup limits from the immutable resource request.
• Networking: slirp4netns with exactly one ephemeral TCP port published on 127.0.0.1 (-p 127.0.0.1::<port>; the kernel picks the port, the agent re-reads it after every restart); class none gets --network none.
• Exactly one writable bind mount: /data, owned by the tenant's subordinate UID, staged by root under a traverse-only 0711 path (/run/kikapu/mounts).
• A forbidden-flag list as a test: --privileged, --pid=host, host namespaces, device mounts, runtime sockets, added capabilities, mounts outside the tenant root — a change that introduces any of these fails the suite.

Loopback-only publication is doing double duty. It is a security control (no LAN exposure on the donor's network — the donor's fridge cannot reach the workload) and the routing primitive: the only path to the workload is the tunnel, which is exactly what the architecture wants (Chapter 9).

6.4 Layer 4 — environment secrets, handled like secrets

Tenant environment variables are plaintext secrets to the node owner (stated plainly in the AUP) but must never leak anywhere else. Phase 16 rebuilt this path after finding that env passed as argv is world-readable via /proc/*/cmdline:

• Values go into a tenant-owned 0600 file inside a 0700 directory and reach Podman only as --env-file; the file is deleted immediately after podman create and orphans are reaped. The flag-set test requires --env-file and forbids --env.
• Caps: ≤128 entries, ≤64 KiB per value, ≤1 MiB aggregate; CR/LF/NUL rejected (env-file format cannot represent them safely).
• Diagnostics use safeCommandDiagnostic, which redacts before truncating — order matters; the reverse can slice a secret in half and leak the prefix.
• At rest in the central database, environments can be envelope-encrypted (Chapter 18): AES-256-GCM data keys wrapped by versioned 32-byte master keys, with dual-read migration so the feature can be enabled and rolled back losslessly.

6.5 Disk quotas

Two modes, advertised as node capabilities. Hard (disk_quota:hard): XFS project quotas — the agent assigns a durable project ID to the deployment's /data and sets bhard via xfs_quota; the kernel enforces the ceiling with zero race window. Soft (disk_quota:soft, the fallback): the agent walks /data each poll, warns on first overage, and stops the workload after 60 s of sustained overage. The plan records this honestly as a best-effort limitation, which is why hard quotas exist and why capability strings let the scheduler know which enforcement a node offers.

6.6 What is not defended, and the known cracks

Two open problems live in this layer and are tracked, not hidden. P-005: validateStagingInput Lstats a path and then mount --binds it — a TOCTOU window in which a symlink swap could redirect the bind. Graded low (workloads are namespaced; tenant accounts are nologin); the recorded fix direction is openat2(RESOLVE_NO_SYMLINKS) or post-mount verification. P-006: manifests may reference mutable image tags (:latest), so a registry re-push changes what runs — a supply-chain gap; the recorded fix is an opt-in server flag requiring digest references. And the boundary itself, restated because it is the most important honest sentence in the project: a malicious node owner with host root sees everything their node runs. The isolation machinery protects tenants from each other and the host from tenants — not tenants from the host.

7. Scheduling

7.1 The algorithm

Placement is least-loaded fit: among online nodes that (a) explicitly offer the deployment's class and (b) have free CPU, memory, and disk per the reservation matrix, pick the least loaded, and insert the deployment row — all inside one PostgreSQL SERIALIZABLE transaction, retried up to KIKAPU_SCHEDULER_RETRIES (default 5) times on serialization failure.

7.2 Why SERIALIZABLE instead of locks

The invariant is zero oversubscription (it is one of the six ratified SLOs, and its target is literally zero). Two concurrent applies must not both count the same free capacity and both fit. The classical fix is SELECT … FOR UPDATE on candidate nodes; Kikapu instead uses PostgreSQL's Serializable Snapshot Isolation and lets the database prove correctness.

Technology: how SSI works and why it fits here. Under SSI, PostgreSQL tracks read/write dependencies between concurrent transactions and aborts one (with SQLSTATE 40001) whenever a cycle would make the execution non-serializable. The programming model becomes: write the transaction as if it ran alone; catch the serialization failure; retry. Compared with explicit row locks this (a) cannot deadlock on lock ordering, (b) protects against phantom capacity — a new deployment row appearing between your read of the node's reservations and your insert, precisely the bug row locks on nodes alone would miss, and (c) keeps the transaction body identical to the single-writer logic, which matters in a codebase where reviewability is a stated security control. The cost — occasional retries under contention — is bounded and measured (placement p99 must stay under 2 s per the SLO gate). The repository layer centralizes the retry (isSerializationFailure in repo/pg/common.go), and the same machinery is reused by the CU ledger (Chapter 16), which has even stricter atomicity needs.

7.3 Declarative apply

kikapu deploy is an apply, not a create: an identical manifest is a no-op (noop), a changed image or environment converges in place (updated, driving running → updating → starting → running), and a change to any immutable field — name, resources, isolation, class, restart_policy, container_port — is a 409 whose error message names the differing field. Why declarative? Idempotence makes the CLI safe to re-run, CI-friendly, and race-free: two identical applies cannot create two deployments, and "what should be running" always has a single textual source of truth, which is also exactly the shape the agent's reconciler (Chapter 8) needs.

7.4 What requests can never say

No deployment request — and no CLI flag — can name a node, a host port, a node address, or an egress override. Placement constraints in the federated future (Phase 23) will be class/trust/region constraints only (invariant-register entry I4). The scheduler is the only entity that maps intent to machines; tenants express what, never where.

7.5 Node loss and rescheduling

The sweeper (every KIKAPU_SWEEP_INTERVAL = 5 s) marks nodes offline after KIKAPU_NODE_STALE_AFTER (20 s, validated ≥ 3× heartbeat interval) without contact. What happens to that node's deployments is opt-in per deployment (Phase 20.8): restart_policy: manual (default) leaves them stranded — visible, owned, recoverable by re-apply, but untouched; restart_policy: reschedule (allowed only for class stateless) supersedes the stranded row (its desired state flips to removed, releasing its reservation) and places a fresh row in status rescheduling on a new node. Because /data is node-local, the replacement starts with empty storage — which is precisely why the policy is restricted to the class that declared itself stateless, and why auto-reschedule was refused as a default (invariant I7: silent data loss is worse than visible downtime). If no node fits, the sweep retries forever; the intent is never dropped. The old node, if it comes back, finds its stale reports rejected by the generation guard.

8. The agent

8.1 Boot and wiring

kikapu-agent loads /etc/kikapu/agent.yaml — which must be a regular file, mode exactly 0600, with unknown YAML fields failing startup (a typo in a security-relevant key must not be silently ignored) — then agent.RunConfigured wires the machine. Order is enforced and meaningful:

1. Register first, everything else after. Registrar.Run blocks until the node has authenticated and its capabilities are recorded. Registration sends {AgentVersion, RuntimeCapabilities, OfferedClasses, ReceiptPublicKey} — the receipt key being the agent's Ed25519 node identity, loaded or minted at <DataDir>/trust/node.key (0600). Retries use full-jitter exponential backoff (1 s → 30 s). Auth failure is permanent (no point hammering with a bad key); transient failures trigger the signed-cache replay described below.
2. Then four concurrent loops under one cancellable context: Heartbeater (5 s; observed CPU/mem/disk + operating mode), Poller (5 s; fetches desired deployments), mode.Monitor (5 s; the safety-mode state machine of Chapter 12), and — via an OnRegistered hook — the tunnel Keeper plus the LogShipper (5 s cadence) and MetricsShipper (10 s).
3. Shutdown is bounded by ShutdownTimeout (default 2 s) and it is an error to exceed it: the systemd unit uses KillMode=process, so workloads survive agent restarts, and a fast, honest exit is part of the upgrade story.

Technology: why full jitter? When a control plane restarts, every agent notices at once. Plain exponential backoff synchronizes their retries into thundering herds; full jitter (sleep a uniform random duration in [0, min(cap, base·2^attempt)]) is the AWS-analyzed variant that spreads reconnection almost uniformly, minimizing both collision and time-to-recover. Kikapu uses it in the registrar and the tunnel keeper, and the e2e suite asserts recovery inside the 30-second SLO.

8.2 The reconciler

internal/agent/reconcile is the largest single idea on the node: converge local reality toward desired state, safely, repeatedly, and under partial failure. Handle([]DesiredDeployment) fans out one goroutine per deployment, each serialized by a per-deployment mutex (concurrent polls can never interleave operations on one deployment, while different deployments proceed in parallel). Each pass:

1. Identity and storage: Tenants.Ensure (idempotently provision the Linux user, subids, directories), then Runtime.CleanupEnvironment (reap orphaned env files).
2. Quarantine gate: if the deployment is quarantined — or the quarantine store cannot be verified — the workload is stopped, targets and telemetry are cleared, and failed is reported. Unreadable safety state fails closed.
3. Switch on desired state. For run: enforce disk quota (hard) or the 60-second-grace soft watcher; build the runtime spec through policy.Class (class → egress); inspect any existing container. Then the generation guard, agent-side: a local container from a newer generation than desired is a hard fail (the world has moved on); same generation converges in place (publish the proxy target and telemetry context, report running once); older generation is removed and rebuilt via createAndStart, which walks the legal lifecycle out loud — scheduled → assigned → pulling_image → (Pull) → starting → (create, start) → running with the freshly-read loopback port. For stop/remove: inspect, stop, clear maps, release quota, and — for remove — delete storage through a SecureStorageRemover that resolves the deployment root and refuses to delete anything outside it or through a symlink.
4. Outage-aware reporting. Status reports go through helpers that consult IsValidTransition first: when the agent replays cached state during a central outage, a record already running is rebuilt locally without fabricating illegal transitions at the server. fail reports only when desired ≠ removed and the edge is legal.

Why level-triggered reconciliation? The alternative — an imperative command channel ("start X now") — cannot survive missed messages, restarts, or partitions: a lost command is a permanently wrong node. A reconciler is level-triggered: it acts on observed difference between desired and actual, so every pass is a fresh chance to be right, crash recovery is free (restart and reconcile), and idempotence is structural. This is the same argument that shaped Kubernetes controllers, applied at single-node scale.

8.3 The signed desired-state cache

Every accepted desired-state snapshot is persisted to <DataDir>/state/desired.json, cryptographically signed — HMAC with a node-local key originally (Phase 20.3), the node's Ed25519 identity after Phase 21 (with a dual-accept window during rollout). On a transient control-plane failure, replayDesiredState feeds the cached snapshot through the same reconciler path exactly once, so containers keep converging with no control plane. The store rejects wrong ownership, wrong modes, symlinks, or bad signatures — a root-owned 0700 directory with 0600 regular files, verified before trust.

Why sign a file the node wrote itself? Because the threat is not the node lying to itself — it is anything else on the host (or a partial disk restore, or a confused deputy) substituting desired state. The signature binds the cache to the agent's own key, converting "some JSON found on disk" into "state this agent verifiably accepted from an authenticated control plane." It is the first brick of the north-star property: a node that can trust its own memory can survive alone.

8.4 The supervisor split (Phase 20.9)

Historically the agent runs as root — it must useradd, edit /etc/subuid, enable linger, stage bind mounts. Invariant-register entry I9 targets a future in which it does not. The mechanism shipped: kikapu-supervisor, a separate root process owning exactly six provisioning verbs — ensure, inspect, ensure_deployment, restore_private_root, quota_ensure, quota_release — behind a Unix socket (root-owned, 0660), while the agent runs unprivileged and calls across.

Technology: SO_PEERCRED. The socket is authenticated with the kernel's peer-credential facility: getsockopt(SO_PEERCRED) returns the actual uid/gid/pid of the connecting process as recorded by the kernel — unforgeable, unlike any application-level token, because the kernel itself vouches for who dialed. The supervisor accepts only a peer whose euid equals its --agent-uid flag. The verb list is a fixed allowlist; there is no "run this command" verb, so a compromised agent gains exactly six provisioning operations, not root. Dual-mode (in-process root remains the default) lasts one release; the remaining cutover work is tracked as problem P-015.

9. The tunnel

9.1 The design in one sentence

One outbound WebSocket per node, carrying one yamux session, over which either side opens cheap logical streams, each self-describing via a single bounded ASCII header line — this is the only transport between center and node, for proxying, logs, stats, metrics, and receipts alike.

9.2 Technology: why WebSocket, and why yamux on top

WebSocket (RFC 6455) solves the reachability problem: it starts life as an ordinary outbound HTTPS request, so it traverses NAT, corporate proxies, and firewalls exactly as well as a browser does; it upgrades in-place to a full-duplex byte stream; and it lets the server authenticate before upgrading — Kikapu verifies the bearer node key and its scope on the plain HTTP request and only then accepts the upgrade (TestTunnelAuthenticationAndScopeFailBeforeUpgrade proves an unauthenticated caller never reaches the WebSocket layer). The subprotocol is pinned to kikapu.tunnel.v1 (refusing to negotiate it is a policy violation close), messages are binary with compression disabled (the payloads are already-encrypted or incompressible proxied bytes; permessage-deflate would add CPU and the BREACH-class risk surface for nothing), and the library is the maintained github.com/coder/websocket v1.8.15.

yamux (github.com/hashicorp/yamux v0.1.2) solves the multiplexing problem: many logical conversations over that one physical connection. Opening a yamux stream is a few frames — no TCP handshake, no TLS handshake, no new NAT state — so proxying a burst of HTTP requests to a node costs almost nothing per request. yamux provides per-stream flow control (one slow log stream cannot stall a proxy stream), ordered delivery per stream, and keepalives (Kikapu: interval 10 s, timeout 30 s, so a dead link is discovered within the tunnel-recovery SLO). One quirk is handled explicitly: yamux 0.1.2 has no half-close, so when one copy direction of a proxied TCP connection finishes, the implementation closes that logical stream to release the opposite copier — the parent session and all other streams are untouched.

Why not alternatives? gRPC streams would drag in a heavy framework for what is byte proxying; raw TCP would not traverse middleboxes or allow auth-before-upgrade; HTTP/2 server push is the wrong shape and CONNECT-style tunnels lose the multiplexed cheap-stream property. This exact question was later re-ratified for federation: Q-DOC6 pins coordinator protocol v1 to "versioned HTTP + authenticated WebSocket/yamux, no gRPC."

9.3 The stream header grammar

Every stream begins with exactly one newline-terminated ASCII line, at most 256 bytes including the newline (KIKAPU_STREAM_HEADER_MAX_BYTES, tunable 64–4096):

PROXY/1 <deployment_id>\n
LOG/1 <deployment_id>\n
STATS/1 <deployment_id>\n
RECEIPT/1 <deployment_id>\n
METRICS/1\n

ReadHeader (in internal/tunnel/protocol.go) reads one byte at a time up to the cap — it cannot buffer unboundedly by construction — then validates: printable ASCII only, one or two fields, kind in the known set, and a well-formed dep_-prefixed ID exactly when the kind requires one. METRICS/1 must not carry an identity field (§10.3 explains why). A malformed header closes that stream only, increments a rejection counter, and leaves the session serving its other streams — a hostile or buggy peer cannot poison the shared connection. The /1 suffix on every kind is the versioning seam: a future incompatible framing becomes LOG/2 beside LOG/1, negotiated per-stream rather than per-connection.

9.4 The server side: Terminus and the registry

Terminus.ServeHTTP (already downstream of auth) validates the node ID, accepts the WebSocket with a 1 MiB read limit, wraps it in a yamux server session, and registers it. The Registry holds at most one session per node with carefully-ordered replacement semantics: a reconnecting node's new session is installed atomically as not-yet-ready, the prior session (and all its in-flight streams) is closed, then the replacement is marked ready. Cleanup callbacks carry the exact entry identity, so a delayed cleanup from the old session can never delete the new one. OpenStream waits on readiness, retries if the session it grabbed was superseded mid-open, and is cancellation-safe. An availability observer receives connected/disconnected node IDs — this is how the routing manager (Chapter 10) learns to build or withdraw routes without the tunnel package ever importing routing or Caddy code.

Why exactly one session per node? Two live sessions for one node would mean two possible paths for a PROXY/1 stream and split-brain routing. Making replacement atomic-and-destructive gives a simple universal truth: the newest authenticated connection is the node, and everything older is dead.

9.5 The data path

ServerProxy.Proxy(nodeID, deploymentID, downstream) re-authorizes on every request via the RepositoryProxyAuthorizer — deployment assigned to this node, status running, valid current port — then opens a stream, writes PROXY/1 <id>, and runs CopyBidirectionalObserved, which both shuttles bytes (honoring CloseWrite half-close toward TCP) and feeds the TrafficMeter (internal/tunnel/traffic.go): a bounded map (capacity 10,000 deployments) of centrally-observed proxied byte counts, explicitly documented as "receipt evidence, not billing truth" — Chapter 17 shows why that distinction is load-bearing. On the node, AgentProxy.Handle resolves the deployment in its local target registry and dials only 127.0.0.1:<node_local_port>, and ports are never cached inside a tunnel session — every stream re-reads the current port, because a container restart changes it and a stale port would hand traffic to whoever binds the old one next.

9.6 Counters

Connections, reconnects, streams, rejections, and proxied bytes are all atomic counters. Agent-side counters travel over METRICS/1; the server retains the latest snapshot for at most 10,000 node identities and exports everything on the separate Prometheus listener. Snapshots contain no credentials, headers, environments, or payloads; handshake failures expose only an HTTP status code.

10. Routing and TLS

10.1 The routability contract

A deployment gets a public route exactly when all four hold: status running; assigned node with a valid current loopback port; a DNS-safe computed hostname; and an active authenticated tunnel to that node. Remove any one — the container stops, the tunnel drops, the node drains — and the route is withdrawn. This four-way conjunction is what makes drain semantics honest (a draining node's workloads may still be running, but the moment its agent stops, the tunnel drops and the public route dies with it — never a stale route to a dead path) and is precisely how split-brain routing after a reschedule is avoided: only the active row's node has the tunnel.

10.2 tunneld: the loopback listener pool

For each routable deployment the routing manager binds a listener on 127.0.0.1 in the port range 49152–65535 (the IANA ephemeral range — nothing else legitimately claims it), taking the lowest available port; binding the socket is itself the allocation lock, so there is no separate port-registry to drift out of sync with reality. Caddy proxies https://<hostname> to that loopback port; accepting a connection there triggers ServerProxy.Proxy down the tunnel. Reconciliation is transactional in spirit: new listeners are started before one atomic replacement of Caddy's kikapu-dynamic subroute, and if Caddy rejects the config, the new listeners are cancelled and the prior route set stays — the system never passes through a state with routes pointing at nothing. Startup reconciliation is additive: routes some other tool configured in Caddy survive.

10.3 Technology: Caddy, its admin API, and on-demand TLS

Caddy was chosen because it solves the hardest edge problem — per-hostname public TLS — natively. Its admin API (a JSON config endpoint) lets the server programmatically replace exactly one named subroute without owning the rest of Caddy's config. In the deployment topology, that admin endpoint is a Unix socket in a shared private volume, mode 0660, with Caddy running as 0:65532 so the distroless server (GID 65532) can reach it and nothing else can — tightened from 0666 in Phase 14, a small change that closed "any process in any co-located container can reconfigure the edge."

On-demand TLS + the ask endpoint. Kikapu hostnames are <app>.<tenant>.<root> — two labels below the root, so a single wildcard certificate cannot cover them (a documented operational fact: *.example.org does not match app.tenant.example.org). Enumerating certificates per deployment in config would be slow and stateful. Instead Caddy's on-demand mode fetches a certificate at first TLS handshake for an unknown SNI — but unconstrained, that lets any stranger pointing DNS at your IP mint certificate issuance attempts. Caddy's guard is the ask endpoint: before issuance it calls GET /api/v1/acme/ask?domain=<hostname>, and Kikapu answers 200 only for control.<root-domain> or a hostname in its lock-guarded, in-memory allowed-hostname snapshot — rebuilt only after Caddy accepts a route reconciliation, with no database fallback whatsoever (Phase 17). The endpoint is deliberately unauthenticated (Caddy has no Kikapu key) and deliberately O(1): an unknown-SNI flood costs one map lookup per probe and can never amplify into PostgreSQL load. Cold start denies until the first reconciliation — fail closed. For fully private installs, an alternative template (caddy-internal.template.json) uses Caddy's internal CA; the e2e harness runs Pebble/internal-CA so the same issuance path is exercised without touching Let's Encrypt.

10.4 The control hostname

control.<root-domain> — the API itself — is always allowed by the ask endpoint and carries HSTS and security headers (Phase 14). Tenants use it for the CLI; Caddy terminates TLS for it like any other route.

11. Telemetry

11.1 Scope and the authorization spine

Telemetry means three things a tenant can see about their own deployment — lifecycle events, workload stdout/stderr, a live stats snapshot — plus operational metrics for the operator. Every ingest path shares one authorization spine: node identity comes from the authenticated tunnel session, and the server verifies the deployment is currently assigned to that node before accepting a LOG/1 batch or opening a STATS/1 probe. Neither a request body nor a stream header can speak for another node; the DeploymentNodeAuthorizer is the chokepoint.

11.2 Logs: at-least-once shipping, exactly-once storage

The agent tails container logs via Podman's k8s-file log driver (chosen over journald so log capture works identically for system-user tenants with no journal sessions, and survives journald config drift on donor machines) using --since from a persisted cursor. Framing, bottom-up:

• A line: valid UTF-8, capped 16 KiB, stdout/stderr distinguished.
• A batch: JSON, ≤256 lines, framed with a 4-byte big-endian length prefix, frame capped 64 KiB.
• A stream: LOG/1 <deployment_id>, batches acknowledged by sequence number.

Per deployment the agent keeps .log-cursor.json (0600, atomically replaced only after the server acks) holding the last acknowledged timestamp and next batch sequence — the agent-side cursor is the resumption source of truth. Delivery is therefore at-least-once; storage is made exactly-once by the database uniqueness key (deployment_id, batch_sequence, line_index) — a retried batch upserts into the same coordinates. This is the classic idempotent-consumer pattern: cheap, crash-proof under agent restarts, ack loss, and duplicate sends, with no distributed-transaction machinery. Structured server logs never record workload line content (tenant data does not belong in operator logs).

Tenant read paths: GET /api/v1/deployments/{id}/events?limit=, /logs?tail=&after=, /stats, with tail/limit bounded 1–1,000. Follow mode is a 1 Hz poll over a stable next_cursor — deliberately not a streaming HTTP response, keeping the API surface within uniform timeout middleware. Log rows older than KIKAPU_LOG_RETENTION (168 h) are pruned every KIKAPU_LOG_PRUNE_INTERVAL (1 h); events are never pruned (they are the audit trail).

11.3 Stats and metrics

STATS/1 is pull-based and honest: the server opens a stream with a 5-second deadline, the agent runs one podman stats --no-stream snapshot and returns CPU %, memory bytes, process count, and the UTC observation time. Any failure — stopped workload, missing tunnel, staleness, timeout, malformed frame — returns application/problem+json unavailable; cached stats are never substituted, because a stale number presented as live is worse than an error (and because Chapter 17 reuses this exact probe as anti-fraud evidence, where freshness is the entire point).

METRICS/1 carries the agent's counter snapshot every 10 s and is the one header with no identity field by design: the node's identity is taken from the authenticated tunnel peer, so a payload cannot relabel itself as another node — the field that could lie simply does not exist. A missing operating-mode value from an older agent is treated as healthy for one compatibility window; an unknown value fails the stream (forward-compatibility is a courtesy; garbage is not).

Prometheus export lives on a separate listener (KIKAPU_METRICS_LISTEN_ADDR, default 127.0.0.1:9090, never published by the compose stack): central tunnel counters, latest per-node agent counters, kikapu_agent_tunnel_metrics_last_received_timestamp_seconds for staleness alerting, and bounded one-hot mode gauges kikapu_server_operating_mode{mode} / kikapu_agent_operating_mode{node_id,mode}. Nodes expose no metrics port at all — their metrics ride the tunnel like everything else.

Part IV — The Trust Plane

12. Operating modes and local survival

12.1 The five modes

Phase 20.4 gave both the server and every agent an explicit safety-mode state machine with five values and a fixed precedence:

Frozen > Partitioned > Recovery > Degraded > Healthy

Ratified triggers (Q-DOC13): health is evaluated every 5 s; two consecutive failed cycles → Degraded; 20 s without authenticated contact → Partitioned; loss of authoritative database state, an integrity failure, or a persisted manual override → Frozen immediately; clearing the cause → Recovery; a successful reconciliation plus 30 s of continuous health → Healthy. The cardinal rule across every mode: existing workloads keep running. Modes gate new decisions (scheduling, finalizing money, accepting work), never the survival of what is already placed.

Manual freezes exist at both ends and both persist. Server: kikapu-server admin mode freeze --authority operator:key-id writes a row in operating_mode_overrides (schema constrains it to scope server, mode frozen, with a recorded authority). Node: kikapu-agent --mode-freeze --mode-authority … persists an owner-only 0600 mode-override.json, observed on the next 5-second cycle. A malformed, unreadable, symlinked, or wrongly-permissioned override store yields Frozen — when the safety switch itself is suspect, the safe interpretation is on.

12.2 Why modes, in CAP terms

The v2 design document reasons about partitions explicitly in CAP/PACELC vocabulary (design doc §12). Kikapu's control plane is a CP system — placement and money demand consistency, so during a partition the center refuses to guess. But a donated node losing sight of the center is an availability event locally: the correct behavior is "keep serving what you were told to serve, accept nothing new, remember everything." The mode machine is that sentence as code. Partitioned is not an error state — it is a planned operating condition with defined behavior, which is the deeper meaning of "degrade locally before failing globally." The economy consumes the same signal: Phase 22.5's finalizer freeze (Chapter 17) refuses to settle CU while the server is partitioned or frozen, because settling money on a partial view of evidence is how ledgers rot.

12.3 The drill

The whole property is proven end-to-end, not asserted. North-star drill part 1 (Phase 20.10) runs the chaos harness through partition/recovery sequences against the real mode machine, transition table, capacity matrix, and SLO evaluator across 14 simulated days (make test-chaos — deterministic, so CI can afford it). Part 2 is TestOutOfBandRecovery: the harness tears the entire central plane down; the agent — armed with its signed desired-state cache and trust checkpoint — restarts from nothing, replays state, and the deployment placed by scripts/e2e-demo.sh (which deliberately leaves one running as the drill's fixture) is still serving. Quarantine still works while partitioned (quarantine.json is node-local and authority-bound). Local survival is a tested behavior with a named test, in CI, on every change.

13. Signed state

Phase 21 replaces "trust whatever the database and bearer tokens say" with "trust what chains, by signature, to a key you have anchored." Everything below lives in internal/trust, internal/eventlog, and internal/trust/checkpoint, with persistence in migration 000008_trust_plane.

13.1 Technology: Ed25519 and domain separation

Every actor — node, tenant, publisher, coordinator, and the root/genesis authority — holds an Ed25519 keypair (trust.Identity). Ed25519 is the modern default signature scheme for good reasons: 32-byte public keys and 64-byte signatures (they fit in database columns and wire frames without ceremony), fast constant-time verification, deterministic signatures (no nonce-reuse catastrophe of the ECDSA kind — a real historical failure class), and a single safe parameter set with no negotiation to get wrong. It is in the Go standard library (crypto/ed25519), which matters in a project whose supply-chain story (Chapter 14) counts every dependency.

Two hygiene rules apply to every signed object in the codebase:

• Domain separation. Every signature covers "kikapu.trust.v1\0" + <per-object purpose> prepended to the canonical message — the receipt purpose is economy.receipt.v1, the event log's is eventlog.event, and so on. Consequence: a signature valid for one object type is bytes-level meaningless for any other, so a captured receipt signature can never be replayed as, say, a revocation. This kills cross-protocol replay as a class rather than case-by-case.
• Private keys never serialize. Identity has redacting String/LogValue implementations and no MarshalJSON; key files are written 0600; the never-log rule (invariant I5) covers them explicitly.

13.2 Anchors, attestations, capabilities

• A trust anchor is a root public key a party has decided to believe (server table trust_anchors, 32-byte key as primary key, role root or coordinator; on nodes, the anchor set ships with installation).
• An attestation binds a subject key to a role and name under an issuer's signature; AnchorSet.VerifyChain walks a chain leaf-first — each link's signature, expiry, and non-revocation — up to an anchor. This is a deliberately minimal PKI: no X.509, no ASN.1, no CRL distribution points; just typed JSON objects, canonical bytes, and Ed25519, small enough to audit in an afternoon.
• A capability is the trust plane's answer to API keys: a root-signed grant binding a Phase-18 key ID to a subject, role, and scope. It carries no secret — the bcrypt-verified secret remains the authentication factor; the capability adds authorization provenance (this key exists because the root said so) and revocability by signature (the root can kill it without a database round-trip being the source of truth). Capability.Verify(anchors, revoked, now) is the single gate: purpose string, expiry, signature, issuer anchored, and not revoked by key ID, subject, or issuer key.

Dual-accept, the migration trick. Setting KIKAPU_TRUST_KEY_PATH makes the server load-or-mint a root identity, anchor it, and mint a capability alongside every new API key (via the Phase-18 KeyService seam, WithCapabilityIssuer). The auth resolver then — on each cache miss, after bcrypt succeeds — rejects directly-revoked key IDs; accepts keys with no capability (the legacy population); and requires full verification for keys that have one. Fail-closed where it matters, additive where it must be: unsetting the env var restores pure API-key auth, and the window where both populations coexist is bounded to one release. This "run both, verify new, accept old, then ratchet" pattern is Kikapu's standard migration shape — the same move as HMAC→Ed25519 on the state cache and plaintext→ciphertext on environments. Invariant I3's endgame: identity from signatures chaining to a trusted key; API keys demoted to derived capabilities.

13.3 Revocation: union-merge, earliest wins

A revocation names one target — a subject, a public key, or a key ID — with a reason from a closed set (manual, quarantine, key_rotation, compromise). Revocation lists are issuer-signed, epoch-numbered, deterministically sorted, and append-only in storage (revocation_lists, unique (issuer_pk, epoch)). The set semantics are the important design decision: union-merge, monotonic, earliest-revocation-time-wins. Two partitioned halves of the network can each revoke things independently; on heal, both merge everything and converge to the same set. Revocation can only ever grow — no partition ordering, race, or replay can un-revoke a key. This is a CRDT (a grow-only set with a min-merge on timestamps), chosen precisely because revocation is the one security operation that must work during a partition, which is exactly when coordination is unavailable.

13.4 The event log and checkpoints

internal/eventlog is an append-only, hash-chained, signed log: each Event{Seq, PrevHash, Timestamp, Kind, Payload, SignerPK, Signature} hashes as SHA-256 over "kikapu.eventlog.v1\0" ∥ canonical-JSON, chains to its predecessor (genesis has a domain-separated constant hash), and is Ed25519-signed. VerifyFrom(startHash, events) re-derives the chain; TruncateBefore allows retention only up to a checkpoint that committed the head hash; NewFromHead restarts a chain from a checkpoint — the restore path. Nothing here is exotic: it is the certificate-transparency / git-history idea — tamper-evidence, not tamper-prevention — applied to operational events. An attacker with the signing key can append lies, but cannot rewrite history without breaking every downstream hash, and cannot make two observers believe different pasts once either has pinned a head. The CU ledger (Chapter 16) extends this exact chain, which is why the economy inherits its auditability for free.

A checkpoint is a signed snapshot of the slow-moving truth: root keys, accepted coordinators, revocations, valid release hashes, a reputation-state hash (Phase 24's slot, reserved now), and the event-log head. Checkpoints are what reset levels 3–4 restore from, and what a node consults when it must answer "whom do I trust?" with no network.

13.5 Reset levels: the shared rollback vocabulary

Incident response is pre-enumerated as six levels, so that during an actual incident nobody is inventing procedure (design doc §16; runbooks in docs/operations.md):

Levels 0, 1, 2, and 4 are drilled in CI on every run (make test-reset-drills); level 3 is explicitly deferred to Phase 24; level 5 is documented and manually drilled. The out-of-band channel is deliberately GitHub — infrastructure that is not Kikapu, so a total Kikapu compromise cannot poison its own recovery path; the manifest is genesis-signed, and the genesis public key arrived on the node via the installer, before any attacker was in the loop.

14. TUF and the supply chain

14.1 The problem plain checksums cannot solve

Phase 12's installer verified SHA-256 checksums — which authenticates integrity (the bytes you got are the bytes listed) but not authority (who says these are the right bytes?), and offers no defense against a compromised release page serving old-but-vulnerable versions with matching old checksums, or against frozen metadata hiding that updates exist. Software-update security is its own well-studied threat model, and Phase 21.2 adopted its standard answer wholesale rather than inventing one: TUF (The Update Framework), via the official go-tuf/v2 library.

14.2 Technology: the TUF model

TUF splits release authority across four signed metadata roles, each countering a specific attack:

• root — the trust anchor; certifies all other roles' keys. Kept offline; supports thresholds (Kikapu production policy: at least 2 of 3 independent root signers, init-root --root-threshold 2), so one stolen laptop signs nothing. Long expiry.
• targets — maps release filenames to hashes and sizes (Kikapu: 30-day expiry). This is what says "this archive is release v1.2.3."
• snapshot — pins the current version of every targets file (7-day expiry), preventing mix-and-match attacks that serve one consistent-looking but stale combination of metadata.
• timestamp — a short-lived (1-day) heartbeat re-signing the snapshot hash, preventing freeze attacks: a mirror that stops updating becomes visibly stale within a day.

Kikapu publishes consistent snapshots: version-prefixed metadata (N.targets.json, …) and hash-prefixed target files, so a client mid-download never sees a torn update. The verifier (kikapu-tuf verify, and internal/release inside the agent) additionally enforces rollback protection (version numbers never regress) and a ±5-minute clock-skew guard — beyond that skew the agent refuses to make signature-expiry decisions at all, because with a wrong clock, "expired" and "valid" are both guesses (the same guard the trust plane applies everywhere).

14.3 The bootstrap ceremony

Verification is only as trustworthy as its starting point, so install-agent.sh demands two out-of-band inputs, delivered by the operator and never fetched from the release being verified: the pinned 1.root.json and a trusted kikapu-tuf binary. The script's trusted_bootstrap_path check walks every path component of both inputs up to / asserting root ownership and no group/world write — closing the quiet local attack where an unprivileged user swaps a bootstrap file between the operator's check and the script's use. Then: verify the archive via full TUF metadata → extract → kikapu-agent --check-config → visudo -cf the sudoers drop-in → install with exact modes (config 0600, sudoers 0440, unit 0644) → validate the installed sudoers again → enable. Bad, expired, tampered, downgraded, symlinked, or unsupported anything fails before installation touches the system.

14.4 Provenance, SBOM, and the release pipeline

A tag push (v*) drives the whole chain: scripts/build-release.sh cross-compiles Linux/macOS × amd64/arm64 archives (with -trimpath -ldflags "-s -w" for reproducibility and size), signs TUF metadata, and emits checksums.txt; CI attaches an SPDX SBOM (the full dependency inventory, so a future CVE can be answered with a grep instead of an archaeology project) and Sigstore/in-toto provenance (a signed attestation binding the artifacts to the exact workflow, commit, and builder that produced them — the defense against "the release was built somewhere else than you think"); a multi-arch server image goes to ghcr.io/isnakolah/kikapu-server; and Trivy fails the build on HIGH/CRITICAL vulnerabilities (--exit-code 1 --ignore-unfixed). CI exercises the entire signing path on every push using ephemeral development keys — so the machinery is continuously proven — while the honest ledger in Part VI records that a production key-custody ceremony and a real signed tagged release remain launch-validation debt, not development debt.

14.5 Update awareness without self-update

Phase 28 adds signed kikapu-update.json as a normal TUF target. It names latest stable v0.<minor>.<patch> and exact component/OS/architecture asset plus SHA-256. TUF root rotation, metadata expiry, rollback protection, and target hash verification happen before Kikapu reads it. GitHub Releases, APT, Homebrew, WinGet, and moving year/quarter tags are delivery or discovery pointers, never release authority.

Run kikapu update check --output json in automation. Interactive human CLI sessions may offer an update at most once every 24 hours; CI, pipes, --no-input, JSON output, and KIKAPU_UPDATE_CHECK=off do not prompt. kikapu update apply delegates to package manager then exits; it never overwrites running executable. Agents use same signed target every 24 hours with jitter, retain workloads on every failure, report available version in heartbeat, and never self-update. Node owner chooses sudo apt update && sudo apt install kikapu-agent.

Part V — The Economy

15. Compute Units

15.1 What a CU is — and pointedly is not

A Compute Unit is a capacity entitlement: an accounting of how much compute a user has contributed to the pool versus consumed from it. It is stored as integer milli-CU (MilliPerCU = 1000), moves only between a user's account and a single system account, and cannot be transferred between users — there is no send, no trade, no market, and the schema cannot even represent a user-to-user posting. Decision Q-DOC11 makes the boundary explicit: no token, no blockchain, no transferable CU, no on-chain settlement; any of those would require a separate design pass after Phase 25.

Why barter, not currency? The moment CU becomes transferable it becomes money, and the design document's threat model immediately gets worse in every dimension: Sybil farming becomes profitable (mint fake work, sell the proceeds), regulatory surface appears, custody and theft matter, and the incentive to attack the metering pipeline scales with the exchange rate. Non-transferable capacity credit keeps the incentive story humble — the only thing farming CU buys you is the right to run more workloads on the same pool you cheated — while still solving the actual fairness problem of a donation economy: people who contribute should be able to consume, and freeloaders should hit a ceiling.

15.2 Technology: fixed-point arithmetic as a consensus primitive

Every economic quantity is int64 milli-CU, and the package comment of internal/economy/credit states the rule: "no float parse or rounding mode can enter settlement." This is not fastidiousness — it is determinism as a security property. Floating-point results can differ across architectures, compiler optimizations, and evaluation orders; a ledger whose entries depend on such results cannot be independently re-derived and verified, and Phase 24's reputation recompute (and any future federated settlement reconciliation) requires bit-identical re-derivation from the event log. Fixed-point integer math with explicit floor division gives every party the same answer forever. Prices use the same trick: basis points (10000 = 1.00), so "multiply by 0.83" becomes the exact integer operation amount × 8300 / 10000.

15.3 The Q-DOC5 formula set (ratified, executable)

All of this is code in internal/economy/credit/credit.go, not prose:

• Raw usage → milli-CU. RawMilliCU(usage) floors each term independently before adding: cpu_core_ms / 3600 + memory_MiB_ms / (1024·4·3600) + disk_MiB_ms / (1024·100·3600), overflow-checked. In human units: one CPU-core-hour ≈ 1 CU; 4 GiB-hours of memory ≈ 1 CU; 100 GiB-hours of disk ≈ 1 CU. Per-term flooring means rounding always favors the payer and is order-independent.
• Price. PriceBPS(consumption_30d, contribution_30d) = clamp(0.80, 1.20, consumption/contribution), i.e. heavy net consumers pay up to 20% more, heavy net contributors up to 20% less; zero contribution deliberately yields neutral 1.00 rather than an infinite price (a brand-new user is not punished for having no history). Recomputed daily per Q-DOC5. ApplyPrice floor-multiplies, and applies the same multiplier to spend and earn so a finalized non-self-dealing receipt stays balanced to the milli-CU.
• Starter allowance. One-time, non-transferable 10 CU per enrolled root lineage, with ratified spend caps of 1 CU/hour and 5 CU/day. A lineage verifies after 24 hours of non-self finalized receipt-backed service and at least one such receipt; finalization records eligibility and atomically issues the signed allowance exactly once.
• Decay. WeeklyEarnedDecay: earned balances at or below 100 CU never decay; above that, 1% per week (100 bps) of the balance decays. Purpose: prevent indefinite hoarding of claims against a pool whose donated capacity is inherently perishable — you cannot bank this year's spare laptop against 2030.
• Zero-earn self-dealing. PlanReceiptSettlement produces a spend for the tenant always, and an earn for the provider only when tenant ≠ provider. Running your own workload on your own node still costs CU but earns nothing — which converts the most obvious farming strategy (be your own customer in a loop) into a strictly losing move. The red-team exit criterion for Phase 22 is exactly this: synthetic non-consumed work must demonstrate earn ≈ 0.

16. The ledger

16.1 Shape: double-entry with one counterparty

internal/economy/ledger + internal/db/repo/pg/cu_ledger.go + migration 000009_cu_ledger. Three tables: cu_accounts (user accounts plus the single system account cuacct_system, distinguished by CHECK), cu_ledger_entries (append-only postings: positive amount, debit ≠ credit, unique (kind, reference_id)), and cu_ledger_state — one row, id = 1, holding the signed-chain head_hash and next_seq.

Every posting is classical double-entry: one debit, one credit, equal amounts, so the sum across all accounts is invariantly zero and any corruption is arithmetically visible. But where a bank has many counterparties, Kikapu has exactly one: the system account. A consumption_spend debits the tenant and credits system; a consumption_earn debits system and credits the provider; starter_allowance credits a user from system. Two user accounts can never appear in one posting — non-transferability enforced by schema shape, not policy code.

16.2 The single writer and the signed chain

Settle runs a SERIALIZABLE transaction (retried up to 5 times on serialization failure — the same machinery as the scheduler). Inside settleInTx: ensure both account rows exist; SELECT … FOR UPDATE the user account and the cu_ledger_state row; check funds (ErrInsufficientCredits — overdraw is rejected before any write, proven by TestCULedgerRejectsOverdrawWithoutPosting); call the injected sign(seq, prevHash) to mint an economy.settlement event chained onto the Phase-21 event log; verify the returned event's sequence and chain (eventlog.VerifyFrom) before trusting it; update both balances; insert the entry with event_seq, event_hash, and the raw signed bytes in event_document bytea; advance the chain head.

Three deliberate choices deserve their why:

• Locking the state row serializes all postings — a single-writer ledger by construction. Ledger throughput is bounded by receipt cadence (default: one challenge per running deployment per minute), so this costs nothing and buys a totally ordered, gap-free, hash-chained history — the property every audit and every future reconciliation leans on.
• bytea, not jsonb, for the signed document. PostgreSQL's jsonb normalizes: it reorders keys and collapses whitespace, which changes the bytes, which breaks the signature. Byte-preserving storage means the row is the verifiable object. (This was hit in development and recorded — the kind of lesson plan/implementation-log.md exists to keep.)
• The signature happens inside the transaction. Sign-then-commit or commit-then-sign both leave a window where the chain and the balances disagree after a crash. One transaction containing balance mutation, entry insert, event insert, and head advance means the ledger and its audit chain are never observable out of sync.

Balance(userID) reads consistently: a known user with no postings is 0; an unknown user is ErrNotFound — absence of history and absence of identity are different facts.

16.3 Audit events without money movement

Dispute lifecycle actions (Chapter 17) append root-signed economy.dispute.open / economy.dispute.resolve events into cu_audit_events (migration 000012) — sharing the same sequence/hash chain head as postings but moving no balances (the schema constrains their kinds to economy.dispute.%). The chain is therefore a complete narrative: every settlement and every intervention, in one verifiable order.

17. Usage receipts and disputes

17.1 The adversary, stated plainly

The economy's core question: how does the center know a node actually did the work it claims? The node is the adversary here — the freeloader/farming threat class from the design document. A node that could self-report usage would print CU. Kikapu's answer is a challenge-response receipt protocol requiring three independent evidence sources to agree, only one of which the node controls.

17.2 The protocol, step by step

Machinery: internal/economy/receipt, internal/telemetry/receipt_client.go / receipt_scheduler.go / receipt_finalizer.go, internal/db/repo/pg/usage_receipts.go, migrations 000010–000013.

1. Key pinning at registration. The agent's first registration delivers its Ed25519 receipt public key; the server pins it on the node row (32 bytes, exactly ed25519 size). All future receipts must verify against this key — a later attacker cannot substitute their own.
2. Scheduling. The ReceiptScheduler ticks every KIKAPU_RECEIPT_INTERVAL (1 min), shuffles the running-deployment list (PCG randomness — a node cannot predict when it will be measured), and dispatches up to KIKAPU_RECEIPT_CONCURRENCY (4) collections, with an active-set so one deployment is never double-challenged while a 55-second window is open.
3. Challenge. IssueChallenge verifies the deployment is running, assigned to this node, at the current generation, and that the node has a pinned key — then mints a random 32-byte nonce challenge with a ≤60 s lifetime, recording traffic_bytes_at_issue: the TrafficMeter's current count of centrally-proxied bytes for this deployment.
4. Measurement. Over a RECEIPT/1 stream the agent's AgentReceiptHandler checks the challenge is addressed to its node identity and deployment, then waits ~55 seconds — the receipt covers a real measured interval, not an instant — samples runtime stats, verifies the sample time falls inside the challenge window, and computes integral usage: CPUCoreMilliseconds, MemoryMiBMillis (floor of bytes × span / 2²⁰), DiskMiBMillis.
5. Signing. receipt.Sign binds everything: shape checks (interval > 0 and ≤ 60 s, UTC), a challenge answer SHA256("kikapu.receipt.challenge.v1\0" ∥ nonce ∥ proof), and an Ed25519 signature under purpose economy.receipt.v1. The nonce binding makes every receipt single-use and unforgeable in advance; the purpose string makes it meaningless outside this protocol.
6. Verification and the two independent checks. Submit runs SERIALIZABLE: locks the challenge and deployment rows (FOR UPDATE OF c, d), rejects consumed challenges (replay-proof — TestDisputeFreezesSettlement proves single-submit end-to-end), re-validates the deployment, verifies the signature against the pinned key — and then demands the two pieces of evidence the node does not control: (a) traffic: the TrafficMeter's byte count must have increased since issue — the center itself proxied new bytes to this workload during the window, so a workload nobody talked to cannot generate paid work; (b) the probe: the server independently opened a STATS/1 stream and observed positive memory/process figures inside the challenge window. Node-signed claim, center-observed traffic, center-observed liveness: three sources, one signer.
7. The dispute window. Accepted receipts rest for KIKAPU_DISPUTE_WINDOW (24 h). Either party to the receipt — the tenant or the node owner — may open exactly one dispute (POST /api/v1/receipts/{id}/disputes, party-authenticated) on an unfinalized receipt; opening appends a signed economy.dispute.open audit event atomically with the freeze row. An open or upheld dispute excludes the receipt from finalization; a rejected resolution releases it. Resolution is an operator action appending economy.dispute.resolve.
8. Finalization. The ReceiptFinalizer ticks, lists undisputed receipts past their window (bounded batches of 32), and calls Finalize: SERIALIZABLE, re-checks not-settled and no open dispute, plans the settlement via credit.PlanReceiptSettlement (currently at NeutralPriceBPS — the rolling 30-day price is not yet wired), and posts spend + optional earn through the same settleInTx as Chapter 16 — receipt finalization and ledger posting are one transaction, stamped back onto the receipt (settled_at, price_bps constrained 8000–12000, spend_entry_id, earn_entry_id, both unique). No receipt can settle twice.
9. Partition freeze (Phase 22.5 core). While the server's operating mode is partitioned or frozen, the finalizer settles nothing. Receipt evidence keeps accumulating append-only; money waits for a whole view of the world (TestReceiptFinalizerFreezesPartitionedAndFrozenModes).

17.3 Why this shape

Every alternative is cheaper and worse. Trust the node's self-report: printing money. Meter centrally only: the center cannot see CPU/memory on a machine it does not own. Continuous streaming metering: expensive, and a continuous stream is easier to fabricate than a random, nonce-bound, deadline-bound interval that must reconcile with traffic the center routed itself. The random-challenge design means a cheating node must fake CPU, memory, and inbound traffic through the center's own proxy, on the center's schedule, against a signature key pinned before the cheat began — while the dispute window gives the humans on both sides of every receipt a full day to object, and the neutral-price bootstrap keeps stakes low while the pipeline earns trust. The remaining honest gap: the make test-load receipt-loss budget (< 0.1%) is not yet met — Chapter 22 carries it.

Part VI — Operations and Assurance

18. Configuration and deployment

18.1 The central stack

The central plane is a health-ordered Docker Compose stack (deploy/docker-compose.yml): PostgreSQL 16.4 → kikapu-server → Caddy. The server image is distroless, non-root (UID/GID 65532) — no shell, no package manager, nothing to live off inside the container; migrations are embedded and run at boot (KIKAPU_MIGRATE_ON_BOOT=true). Two networks: edge (public; only Caddy's 80/443 are published) and control (internal; PostgreSQL, the API port, and the metrics port are never published). Caddy shares the server's network namespace so central loopback tunneld listeners are reachable by exactly one process in the world. The compose DSN uses sslmode=disable because PostgreSQL lives only on the private network — documented, with the instruction to move to verify-full the moment PG crosses an untrusted link.

Quick start: copy deploy/, cp compose.env.example .env, generate KIKAPU_POSTGRES_PASSWORD (openssl rand -base64 36), set KIKAPU_ROOT_DOMAIN, chmod 0600 .env, docker compose up -d, then check /healthz (process alive) and /readyz (PostgreSQL reachable — goes 503 when the DB is down, and there is a test for that). First identities come from docker compose exec kikapu-server kikapu-server admin create-user --name alice and admin create-key.

18.2 The configuration surface

Everything is environment variables, loaded and validated by internal/platform/config/server.go — bad values fail startup, not runtime. The full table (defaults in parentheses):

Note the pattern: every dangerous feature — env encryption, the trust plane, the economy — defaults off and is enabled by one variable, with a documented, lossless rollback (unset it). Features land dark, get validated, then get ratcheted on.

18.3 Envelope encryption for environments

With KIKAPU_ENV_ENCRYPTION_ENABLED=true, deployment environments at rest become AES-256-GCM ciphertexts under per-row data keys wrapped by versioned 32-byte master keys. Why envelope encryption? Rotating one master key means re-wrapping small data keys, not re-encrypting every row; and versioning makes rotation an online operation: add the new key to the keyring, switch KIKAPU_ENV_ACTIVE_KEY_VERSION, restart — the server dual-reads old versions and migrates legacy plaintext plus old-version rows in batches; after migration, drop the old key. Startup fails closed on malformed keys. Rollback (=false with the keyring retained) keeps decrypting existing rows while new writes return to plaintext — sacrificing at-rest encryption, never data. The threat this addresses is precisely the central database at rest (backups, disk theft, DB-level exposure) — not the node, where the AUP already concedes plaintext visibility to owner root.

18.4 The node, day 2

Install and upgrade both run install-agent.sh (Chapter 14's ceremony). Agent config knobs (docs/agent-configuration.md): api_key + control_plane_url required; data_dir (/var/lib/kikapu), supervisor_socket (opt-in unprivileged mode), offered_classes ([general]), local_state_cache_enabled (true), hard_disk_quota_enabled (false), log_tail_lines (200, ≤10,000), and the interval mirrors of the server's cadence table. The systemd unit keeps ReadWritePaths=/etc (documented honestly: useradd must rewrite passwd/shadow/subuid/subgid and their lock files — narrowing this is the supervisor's job) and KillMode=process so workloads outlive agent restarts. Backups: scripts/backup-central.sh captures the PG custom dump, Caddy volumes, deploy config, and checksums — deliberately excluding deploy/.env and the env master keys, which must be stored separately (a backup that contains its own decryption keys is not an encrypted backup). Upgrades: pin KIKAPU_SERVER_IMAGE to a digest, run scripts/upgrade-central.sh (forward migration runs once), re-run the installer on nodes; never use migration down for ordinary rollback. Node owners hold two unilateral controls that need no central permission: drain (kikapu node stop — immediate exclusion from scheduling, existing workloads keep running until the agent stops) and quarantine (--quarantine-deployment dep_… --quarantine-authority owner:key — stops the workload within one reconcile cycle, withdraws its route, and only the same authority can clear it).

19. Verification

19.1 The gate ladder

Kikapu's charter makes verification non-optional: every change names its gate and pastes evidence into the log. The ladder, cheapest to heaviest:

A charter rule with teeth: macOS and nested containers are not accepted substitutes for host claims. If a change touches rootless Podman behavior, the evidence must come from a real Linux host; "it passed on my Mac" is not evidence. (Problem P-001 exists because of this rule — macOS contributors get a Lima/Multipass/Vagrant VM story instead of an exemption.)

19.2 CI: five jobs

.github/workflows/ci.yml, all on ubuntu-24.04, Go 1.25.12:

1. packaging — shellcheck all scripts; compose config validation; build ephemeral TUF dev keys and run the entire release pipeline (build-release.sh, archive inventory, checksum verify); verify agent and supervisor units; build the server image and Trivy-scan it (HIGH/CRITICAL = fail).
2. unit-and-postgres — lint, sqlc drift, unit (race), integration against postgres:16.4, reset drills, mesh, and SLO contract gate.
3. linux-host — provision a real rootless-Podman host, make test-host as root.
4. e2e (10-min timeout) — the full journey; on failure captures journalctl -u kikapu-agent and compose state.
5. bash-demo — runs scripts/e2e-demo.sh, proving the documented user commands work, not just the Go APIs.

Tag pushes additionally drive the release workflow (cross-compiled archives, signed TUF metadata, SBOM, Sigstore provenance, ghcr images).

19.3 The six SLOs

Ratified at checkpoint G1 and enforced numerically by make slo: placement p99 < 2 s; convergence p99 < 60 s; tunnel recovery ≤ 30 s; offline detection ≤ 20 s; zero oversubscription; metric loss < 0.1%. The default input test/slo/ci-contract.json is explicitly a contract fixture — it proves the gate evaluates correctly, and is never to be presented as production evidence. The chaos harness emits real evidence JSONs and feeds them through the same gate.

19.4 What the test suites prove

• Acceptance (test/e2e/acceptance_test.go) — all twenty architecture criteria as named tests: HTTPS through the tunnel, logs and follow, stats, restart/update/stop-preserves-data/remove-cleans-up, runtime security and limits live (TestRuntimeSecurityAndLimits, TestTenantIsolation, TestWorkloadRunsAsTenant), policy rejections, egress classes, class opt-in, hard quota, key-revocation survival, control-plane-restart recovery, tunnel recovery, TestTrustPlaneDualAcceptEndToEnd, TestDisputeFreezesSettlement, and the bash demo.
• zz_recovery_test.go — TestOutOfBandRecovery, the north-star drill part 2, deliberately named zz to run last so it can destroy the harness's central plane.
• Integration/repo — migrations up-down-clean; capacity SQL ≡ domain matrix; CU ledger double-entry property + signed chain + overdraw rejection; starter allowance exactly-once; env ciphertext dual-read; dispute lifecycle; trust anchors/capabilities/revocation epochs.
• Integration/host — real rootless Podman lifecycle and security, tenant provisioning and isolation, immutable runtime identity, hard quotas, the supervisor's verb boundary.
• installer_test.go — builds a fake release + TUF root and proves the installer's verification actually rejects tampering.

The pattern worth naming: Kikapu prefers behavioral tests with property names (TestInvalidAuthFloodCannotDelayDifferentClient, TestCULedgerRejectsOverdrawWithoutPosting) over coverage percentages. The test list is the security-property inventory.

20. Process and governance

20.1 The maintenance loop

CONTRIBUTING-AI.md is a charter written for a project developed largely by AI agents, and its prime directive is document as you go — no change is complete until the plan reflects it. Every change follows: read plan/ → state intent → change behind the narrow interfaces → document (plan/implementation-log.md entry with validation evidence, plan/README.md status, plan/problems.md, docs/) → verify with the correct make gate → Conventional Commit. The source-of-truth hierarchy when documents disagree: principles.md → architecture.md (the four cross-phase contracts: capacity-reservation matrix, generation guard, configuration-defaults table, security boundary) → the active phase file + testing-strategy.md → AGENTS.md → docs/.

Why a charter this heavy? Because process is a security control here. The implementation log (187 KB and growing) makes every decision auditable; the phase files make scope tamper-evident; and the five non-negotiable invariants give any reviewer — human or machine — a checklist that no diff may cross: (1) never relax the workload hardening flag set; (2) never accept node placement, host ports, or node addresses in requests; (3) never log raw API keys or env values; (4) never compose Podman shell strings — structured argv only; (5) identity comes from the key, never the body. plan/problems.md is the honesty mechanism: known cracks (P-005 TOCTOU, P-006 mutable tags, …) are catalogued with impact grades and fix directions instead of being quietly forgotten.

20.2 v2 governance: G-checkpoints and Q-DOC rulings

The v2 wave (phases 20–25) is governed by plan/design-decentralization-governance.md (the design rationale) operationalized by plan/architecture-v2.md (the binding contracts). Two instruments:

• G1–G5 go/no-go checkpoints gate phase starts along the DAG …16 → [G1] → 20 → [G2] → 21 → [G3] → 22 → [G4] → 23 → 24 → [G5] → 25. Each demands evidence: G1 (recorded 2026-07-03) required closed retrospectives, the six ratified SLOs, the class taxonomy, and a versioned AUP; G2 required two green chaos/load weeks plus drill part 1; G3 required the recovery drill and TUF-signed releases; G4 will require ≥30 days of economy production data; G5 will require ≥30 days of reputation data and a re-passed out-of-band drill.
• Q-DOC rulings are recorded design decisions (all currently resolved): Q-DOC1 security ratchet; Q-DOC2 class taxonomy; Q-DOC3 AUP v1.0; Q-DOC4 trust roots and key custody; Q-DOC5 economy tunables; Q-DOC6 coordinator protocol; Q-DOC7 emergency governance (3-of-5 threshold role set in development; accountable named custodians required for production); Q-DOC8 reputation formula; Q-DOC9 licensing (MIT everywhere, no source-available runtime component); Q-DOC10 private-cluster minimums; Q-DOC11 no token/chain; Q-DOC12 control-plane durability (PostgreSQL managed off-mesh; node-local /data never the sole copy of control-plane state); Q-DOC13 mode triggers.

A crucial governance fact recorded 2026-07-04: the maintainer authorized an autonomous development runway in which deterministic simulated substitutes stand in for elapsed production time (G2 = 14 simulated days via the chaos harness; G4 = 30 simulated economy days; G5 = 30 simulated reputation days), each labeled simulated and each exercising the same numeric gates. Launch still requires the real thing — real elapsed-time soaks, production key ceremonies, named custodians, the off-mesh managed-DB drill. Development velocity was bought without discounting the launch bar, and the debt is itemized (Chapter 24).

20.3 The invariant-conflict register

plan/architecture-v2.md contains the only legal venue for renegotiating an MVP invariant: the register I1–I12. Each entry names the current invariant, the phase allowed to renegotiate it, and the replacement contract — e.g., I2: SERIALIZABLE central placement gives way to node-signed monotonic capacity grants with last-writer-wins reconciliation (Phase 23); I3: API-key identity gives way to Ed25519 roots (Phase 21, done); I9: root agent gives way to the supervisor split (Phase 20.9, dual-mode); I12: the central subordinate-ID allocator gives way to node-local disjoint blocks (Phase 23). And one entry exists to say no: the workload hardening flag set is never renegotiated by any phase. Decentralization may rearchitect who decides where code runs; nothing may ever weaken how it runs.

20.4 The plan-review gate

An optional, opt-in browser tool (.agent/plans/ + make plan-review, bound to 127.0.0.1) lets proposed plans be reviewed with blocking comments before promotion into plan/phase-NN. It augments the charter and weakens nothing; plan/ remains canonical, and the tool never edits plan/ or docs/ markdown itself.

Part VII — The Road Ahead

21. Phase history

What each completed phase actually delivered, condensed but complete. (Status source: plan/README.md; evidence: plan/implementation-log.md.)

The MVP wave (0–13) — complete

The hardening wave (14–19) — complete

From the 2026-07-02 security/usability review: 14 edge hardening (Caddy socket 0660, HSTS, documented /etc surface); 15 auth hardening (per-IP token buckets, negative cache, documented revocation window); 16 env-file secrets (off argv, 1 MiB caps, redact-before-truncate, 0600 agent config); 17 the O(1) ask cache with no DB fallback; 18 self-service key lifecycle with the capped, race-safe key API; 19 CLI ergonomics (name-or-ID addressing, field-naming immutability errors, completion).

Phase 20 — Bulletproofing & local survival — complete

Ten packages: Prometheus over METRICS/1; the make slo gate; the signed desired-state cache; the five-mode machine; workload classes + AUP admission + per-class egress + owner quarantine; XFS hard quotas; envelope-encrypted env; opt-in auto-reschedule; the supervisor privilege split; and the chaos/load/property harness with north-star drill part 1. G2 satisfied via the authorized 14-simulated-day substitute.

Phase 21 — Trust plane — complete (development; production launch validation pending)

Ed25519 identities everywhere; API keys backed by derived revocable capabilities with fail-closed dual-accept; TUF release metadata via go-tuf/v2 with multi-signer thresholds; SBOM + Sigstore/in-toto provenance; the TUF-verifying installer; union-merge revocations; the hash-chained signed event log; signed checkpoints and restore; the GitHub out-of-band recovery manifest; reset-level runbooks with CI drills for levels 0/1/2/4. G3 recorded 2026-07-15 as a development substitute; fresh-VM make e2e passed in 231 s the same day.

22. Completed: Phases 20–24 and Phase 26 proof

Phase 20 completed development survival controls: signed local desired state, operating modes, AUP and class admission, quotas with fallback, opt-in reschedule, quarantine, and chaos/load proof. Phase 21 completed development trust controls: identities, capabilities, revocation, TUF, signed checkpoints, and out-of-band recovery. Phase 22 completed opt-in CU receipts, dispute and append-only settlement behavior. Phase 23 completed private-only coordinator, identity-pinned failover, capacity-grant, observer, and peer-limit paths. Phase 24 completed development validator/reputation/settlement primitives, stateful-placement gating, checkpoint/replay direction, and reconciliation safeguards.

Phase 26 reconciled this completed development boundary on supported Linux: current generated code, release contents, cleanup-safe E2E, and aggregate gates. make verify-v2 is portable development proof; make verify-v2-linux additionally runs supported-Linux host tests and full E2E. These are development evidence, not production-duration launch evidence. Exact status and evidence remain in plan/README.md, Phase 20–24/26 files, and plan/implementation-log.md.

23. Deferred: Phase 25

Phase 25 — Self-publishing & governance — is deferred by maintainer direction while release work is skipped.

"Kikapu publishing Kikapu": a kikapu bootstrap seed; a system tenant class whose pinned placement is the one audited exception to I4; the control plane running on Kikapu itself (with authoritative PostgreSQL staying off-mesh per Q-DOC12); reproducible builder nodes proving bit-identical rebuilds; GitHub Actions deploying to the mesh with GitHub retained forever as the out-of-band recovery channel; progressive canary/attestation/ring rollouts with auto-rollback and N−1 compatibility; internal/governance threshold emergency keys whose forbidden verbs are structurally impossible (the emergency key cannot express "transfer CU" or "weaken hardening" — the verbs do not exist in its grammar); and a dormant voting mechanism, shipped disabled — activation deferred until real actors and real data exist to vote with. Reset level 5 is documented and manually drilled here.

Beyond 25 — explicitly unplanned

Tokens, transferable CU, on-chain settlement, and voting activation are not late phases; they are unplanned (Q-DOC11) and would require a new design pass with its own threat modeling.

24. Open problems and launch debt

Open problems (plan/problems.md)

Launch-validation debt (development complete ≠ launch ready)

The simulated-substitute authorization bought development speed, not a lower bar. Before production launch, all of the following remain owed: the real elapsed-time G2/G4/G5 soaks (two green chaos weeks; 30 economy days; 30 reputation days); the production TUF key-custody ceremony with independent root signers and a real signed tagged release; named accountable custodians for the Q-DOC7 emergency-governance roles (the development 3-of-5 set is placeholder); and the off-mesh managed-database HA/backup/restore drill (Q-DOC12). Every one is tracked in the plan, none is hidden in this book, and that is exactly how this project's honesty mechanism is supposed to work.

Appendices

Appendix A — Constants and limits

Appendix B — Database migrations

Appendix C — Tunnel stream grammar

stream        = header payload
header        = kind [ SP deployment-id ] LF        ; ≤256 bytes incl. LF, printable ASCII
kind          = "PROXY/1" | "LOG/1" | "STATS/1" | "RECEIPT/1" | "METRICS/1"
deployment-id = "dep_" 1*32( a-z / 0-9 )            ; required except METRICS/1 (forbidden)

Transport: WebSocket subprotocol kikapu.tunnel.v1 (binary, no compression, auth before upgrade) carrying one yamux session per node; either side opens streams; malformed headers close the stream, never the session.

Appendix D — Glossary

Ask endpoint — the public, O(1), DB-free hostname oracle Caddy consults before on-demand certificate issuance. Capability — root-signed grant backing an API key ID. Checkpoint — signed snapshot of roots/coordinators/revocations/releases/log-head. CU / milli-CU — non-transferable capacity entitlement; integer thousandths. Dual-accept — one-release window where legacy and signature-backed auth coexist. Generation — restart_counter; the staleness fence for status reports. Linger — systemd per-user instance persistence for never-logged-in tenants. North-star drill — the test pair proving survival without a central plane. Reservation matrix — the desired×status table deciding capacity occupancy. Security ratchet — shipped safeguards may only strengthen. Self-dealing — tenant == provider; spends but never earns. Stranded — deployment whose node went offline under restart_policy: manual. Sweeper — marks nodes offline after 20 s silence. TrafficMeter — central count of proxied bytes; receipt evidence, not billing truth. tunneld — the central loopback listener pool Caddy proxies into. Union-merge — revocation CRDT semantics; grow-only, earliest wins.

Appendix E — File-path map

This book reflects the repository as of 2026-07-17 (Phases 20–24 development-complete, Phase 25 deferred, Phase 26 Linux readiness reconciliation complete). Plan files remain source of truth; amend this book alongside them.