The Kikapu Book

Kikapu turns contributed machines into isolated application hosts. Tenant submits one manifest; Kikapu authenticates it, admits it against policy and capacity, schedules it, runs it as rootless Podman workload, and exposes it through outbound node tunnel. Same product can operate with hosted, self-hosted, or federated coordinators and can degrade locally during control-plane loss.

This Book is primary human explanation. Binding contracts live as Contract Issues in Kikapu Delivery; operational commands live in operations.md; contributor workflow lives in DEVELOPER.md; exact generated commands live in cli-reference.md.

Capability labels used here:

Implemented: code and repository-controlled tests exist.
Planned: accepted contract remains incomplete.
Launch proof pending: implementation exists, but external custody, elapsed soak, provider, supported-host, physical-machine, or production-drill proof remains open.

These labels describe evidence boundaries. Current acceptance and proof records are GitHub Issues, not repository checklists or historical implementation logs.

Words used in this Book

You do not need infrastructure experience to start. Six words describe most of Kikapu:

Workload: your application process, packaged as a Linux container image.
Manifest: a small YAML file saying which image to run, which port it listens on, and how much CPU, memory, and disk it needs. It describes intent; it never chooses a machine.
Tenant: one Kikapu account and its isolated workload namespace. One user has one tenant.
Node: an opted-in machine offering a bounded amount of Linux capacity.
Node owner: person or group controlling that machine. Host root remains more powerful than Kikapu and can inspect anything node runs.
Control plane: server, database, scheduler, and routing coordination that authenticate intent, choose fitting nodes, and distribute desired state.

One person can fill every role in a development setup:

Operator starts control plane and creates initial user credential.
Node owner opts in supported host with kikapu node init.
Tenant signs in and applies kikapu.yaml from CLI or console.

flowchart LR
    Operator[Operator starts control plane] --> Owner[Node owner donates bounded capacity]
    Owner --> Tenant[Tenant applies manifest]
    Tenant --> Scheduler[Scheduler chooses fitting node]
    Scheduler --> Workload[Rootless workload starts]
    Workload --> Proof[Status, logs, and public route are checked]

First successful local run proves repository-controlled parts work in that environment. It does not prove public availability, durable storage, protected release custody, provider setup, or protection from node-owner root.

1. Purpose and boundaries

Kikapu serves three roles:

tenant supplies signed intent and operates workload;
node owner offers bounded CPU, memory, disk, and accepted workload classes;
coordinator authenticates, schedules, distributes desired state, and reconciles signed truth.

Kikapu does not give tenants SSH, host shells, device access, host namespaces, runtime sockets, arbitrary mounts, user-selected nodes, host ports, or public node addresses. It isolates tenants from each other, but cannot protect tenant secrets from malicious node-owner root. Do not deploy secret you would not hand host owner. No confidential-compute claim exists.

Product principle: degrade locally before failing globally. Existing workloads survive coordinator loss. Admission, settlement, and updates freeze when safety cannot be proven. Signed checkpoints and independent recovery preserve route back from compromise.

Topology

flowchart LR
    Tenant[Tenant CLI or console] -->|HTTPS + credential| Server[kikapu-server]
    subgraph Control[Linux host → rootless Podman control-node]
      Server --> DB[(PostgreSQL)]
      Caddy[Caddy loopback proxy]
      Server --> Caddy
    end
    Owner[Node owner] -->|node init| Host[Linux node host]
    Agent[unprivileged kikapu-agent] -->|control + WebSocket/yamux| Server
    Agent -->|local authenticated socket| Supervisor[root kikapu-supervisor]
    Agent --> Podman[rootless Podman per tenant]
    Browser[Application client] --> Caddy
    Caddy --> Tunneld[loopback tunneld]
    Tunneld --> Server
    Server -->|PROXY/1| Agent
    Host --- Agent

The control node has identity, heartbeats, tunnels, and no tenant runtime: the scheduler excludes all coordinator nodes and the control-only agent rejects non-empty desired state. A separate kikapu-sentinel, outside the container, verifies pinned signed-event segments and alerts on healthz/readyz failures; it has no key, scheduler, recovery, host-control, or workload authority.

Native Linux is supported node baseline. Apple-silicon macOS hosts one Linux environment per node through Apple Container; workload runtime remains Linux/rootless Podman. Mac support requires logged- in owner and has no headless or sleep-survival claim.

2. Guided deployment

This path demonstrates controlled development deployment. Production needs launch checklist, approved release digest, real DNS/TLS, protected credentials, supported hosts, and explicit publish authorization.

Before copying commands, replace example domains, credentials, image digests, and passwords. Lines beginning with # are comments. Run control-plane commands in repository checkout. Run node commands on supported host donating capacity.

Start control node

Prerequisites: supported Linux rootless Podman, an immutable TUF-verified image, and checkout.

export KIKAPU_ROOT_DOMAIN=example.test
make control-init # development-only bootstrap custody
make control-status

make control-init is the explicit development-only first-start path; it resolves the exact digest-pinned image from signed TUF metadata and bootstraps the private control state. This is development control plane, not production proof.

Create user and key from trusted server host:

go run ./cmd/kikapu-server admin create-user --name alice
go run ./cmd/kikapu-server admin create-key --user alice

Raw kk_<keyID>_<secret> appears once. Store it in password manager; server stores bcrypt hash.

kikapu login --server https://control.example.test --api-key-file ./alice.key

Add node

On supported Ubuntu host, interactive path is:

kikapu node init

Wizard collects coordinator URL, masked owner credential, node name, offered resources, accepted classes, and explicit confirmation. --file, flags, stdin, environment, and --no-input support automation. Diagnose before mutating host:

kikapu node host doctor
kikapu node host inspect

Driver installs/verifies dependencies, registers node, provisions systemd units, and starts agent. Node initiates all network connections. See operations for exact host, Apple Container, service, drain, and recovery procedures.

Wait until node reports online before applying workload. Online means authenticated heartbeat and current offered capacity reached control plane. It does not mean host passed production soak.

Apply workload

api_version: v1
name: hello
image: ghcr.io/example/hello@sha256:replace-with-real-digest
aup: v1.0
class: stateless
restart_policy: reschedule
resources:
  cpu_millis: 250
  memory_mb: 128
  disk_mb: 256
container_port: 8080
environment:
  GREETING: hello

kikapu deploy --file kikapu.yaml
kikapu deployments list
kikapu logs hello --follow
kikapu stats hello

kikapu deploy is declarative apply. Identical input is no-op. Image/environment converge. Name, isolation, class, restart policy, and port stay immutable. Resource changes obey capacity and safe- state rules; disk grows only. Scheduler chooses node—manifest cannot.

Check journey in this order:

kikapu deployments list shows desired deployment and eventual running state.
kikapu logs hello --follow shows application output without environment values.
kikapu stats hello shows bounded live CPU, memory, and process observations.
Open computed workload hostname only after status is running and DNS/TLS route is configured.

If step fails, stop and read exact error. Do not work around placement, isolation, or credential checks. Operations guide owns recovery commands and host diagnostics.

Deployment flow

sequenceDiagram
    participant U as Tenant
    participant S as Server
    participant D as PostgreSQL
    participant A as Agent
    participant R as Rootless Podman
    U->>S: apply manifest + user credential
    S->>D: SERIALIZABLE admit + place + reserve
    A->>S: authenticated desired-state poll
    S-->>A: tenant-scoped desired state
    A->>R: pull/create/start with fixed hardening
    A->>S: generation + sequenced status and port
    S-->>U: running hostname

Stop preserves /data; remove deletes workload data after container removal. Node owner can drain/resume/retire capacity without learning tenant or workload details from owner API.

3. Current design

Domain and API

Core entities are user, tenant, API key, node, deployment, event, log entry, trust identity, capability, receipt, settlement entry, coordinator, and reputation record. ULIDs identify stored entities. All HTTP routes live under /api/v1; errors use application/problem+json.

User has exactly one tenant. User credential resolves user and tenant scope; node credential resolves one node. Console session resolves user scope only. Global operator requests additionally need live local role and fresh step-up where defined. Payloads never select authenticated identity.

Scheduling

Creation and placement occur in one SERIALIZABLE transaction. Scheduler locks eligible online capacity rows in deterministic order, filters by class/trust/region constraints, calculates free CPU, memory, and disk, and chooses highest normalized remaining-resource score; node ID breaks ties. Reservation is inserted before commit, so concurrent placement cannot oversubscribe.

Manual deployment remains assigned when node goes offline. restart_policy: reschedule is allowed only for reschedulable stateless class. Sweeper marks old row removed and creates fresh replacement atomically on fitting online node. /data is node-local and never follows replacement. Generation and report sequence reject stale node updates; old node removes superseded container on reconnect.

Identity and isolation

flowchart TB
    Credential[resolved credential] --> Scope[user, tenant, node, or operator scope]
    Scope --> Desired[tenant-scoped desired state]
    Desired --> Agent[unprivileged agent]
    Agent -->|fixed peer-credential verbs| Supervisor[root supervisor]
    Agent -->|fixed sudo Podman switch| TenantUser[kikapu_t tenant user]
    TenantUser --> Namespace[rootless user namespace]
    Namespace --> Container[cap-drop ALL; no-new-privileges; read-only root]
    Container --> Data[/tenant/deployment/data only]

Each node allocates immutable disjoint 65,536-ID subordinate blocks for tenants it hosts. Tenant root is 0700; container UID 0 maps into tenant subordinate range. Runtime always applies CPU, memory, PID 1024, capability drop, no-new-privileges, read-only root, 64 MiB /tmp, loopback publication, and only deployment /data mount. Privileged mode, host namespaces, devices, sockets, extra capabilities, and foreign resolved mounts fail closed.

Environment values are bounded, reject CR/LF/NUL, encrypt at rest when key configured, travel in desired state and signed local cache, and reach Podman through short-lived env file—not command argv. Node-owner root can still read them.

Agent and host boundary

Agent registers, heartbeats, polls desired state, reconciles per deployment, ships logs/metrics, and maintains tunnel. Signed local desired-state cache lets restart preserve known work during coordinator outage. All loops cancel through context and use deterministic intervals.

Root supervisor has no network listener. Peer-credential-authenticated local socket exposes fixed provision, quota, symlink-safe bind-stage/release, and tenant env-file verbs. Agent sudo authority is only fixed Podman executable as provisioned tenant. Supervisor or node-owner-root compromise remains full host compromise.

Tunnel, routing, and telemetry

Agent opens authenticated WebSocket to coordinator and runs yamux inside it. Streams start with bounded line: PROXY/1 <deployment_id>, LOG/1 <deployment_id>, STATS/1 <deployment_id>, or METRICS/1. Unknown, oversized, unauthorized, stale, and unavailable streams fail independently.

flowchart LR
    Client -->|HTTPS app.tenant.domain| Caddy
    Caddy -->|central loopback port| Tunneld
    Tunneld -->|PROXY/1 deployment| Terminus[server tunnel terminus]
    Terminus -->|WebSocket/yamux| Agent
    Agent -->|127.0.0.1 ephemeral port| Workload

Agent reports running only after local port accepts TCP. Proxy resolves current port at stream time. Caddy route appears only for running deployment with active tunnel. On-demand TLS ask endpoint authorizes only current routable hostnames; alternative per-tenant DNS-01 wildcard is supported.

Logs ship at least once with stable sequence and store idempotently. Stats remain live observations. Prometheus metrics travel through tunnel; nodes expose no metrics port. Authorization follows same tenant/node spine as desired state.

4. Resilience and operating modes

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Degraded: repeated failed health
    Degraded --> Partitioned: 20s authenticated silence
    Healthy --> Frozen: operator or integrity failure
    Degraded --> Frozen: authority loss
    Partitioned --> Recovery: contact restored
    Frozen --> Recovery: setting authority clears
    Recovery --> Healthy: reconcile + 30s healthy
    Recovery --> Degraded: health fails

Healthy admits and settles. Degraded trusts known actors only and queues settlement. Partitioned operates locally/regionally and freezes CU. Frozen admits no new work or updates but preserves running workloads. Recovery reconciles signed truth before gradual return. Higher safety mode always wins; manual freeze can be cleared only by same authority.

Recovery vocabulary:

Pause scheduling.
Quarantine actor.
Rotate keys.
Roll reputation to checkpoint and recompute.
Roll network truth to signed checkpoint.
Threshold-approved genesis reset.

Levels 0–4 have repository drills. Genesis procedure is documented; production drill is launch proof pending.

5. Trust, recovery, economy, and federation

Signed trust and releases

Ed25519 roots identify tenants, nodes, publishers, and coordinators. API keys become revocable capabilities. Domain-separated signatures prevent payload reuse. Revocations union on partition heal and never disappear. Signed append-only event chains provide recomputable audit truth.

TUF authorizes release versions, expiry, and rollback. SBOM and provenance bind artifacts to source and builders. Nodes discover updates but verify before action. Independent signed recovery manifest remains usable when Kikapu is unavailable.

CU economy

flowchart LR
    Work[consumed workload interval] --> Receipt[node-signed receipt]
    Receipt --> Challenge[randomized challenge]
    Challenge --> Verify[server + validator verification]
    Verify --> Ledger[strong local CU ledger]
    Ledger --> Summary[signed settlement summary]
    Summary --> Federation[coordinator reconciliation]

CU rewards verified consumed work, never reservation or presence. It is bounded, identity-gated, non-transferable, and not token. Partition freezes spend; reconciliation replays ordered signed events and rejects overdraw. Reputation is actor- and class-specific, uses adversarial probes and independent validators, and recomputes from signed events after rollback.

Scoped federation

Coordinator is replaceable scheduler/state distributor, not owner of network or workload data path. Private clusters select their own authority. Nodes maintain bounded local, regional, trust, random, and coordinator peer rings. Mutually authenticated channels carry only signed resource summaries, classes, health observations, checkpoint hashes, and coordinator lists—never logs, secrets, customer metadata, or workload internals.

Cross-coordinator placement uses node-signed monotonic capacity grants. Local scope still schedules transactionally. Heal uses signed-timestamp last-writer-wins with node-owner intent above coordinator intent. Credits remain strongly consistent and frozen during partition; desired state may continue locally; revocations always union.

Trust and economy topology

flowchart TB
    Roots[tenant, node, publisher, coordinator roots] --> Caps[revocable capabilities]
    Roots --> Events[signed event chains]
    Events --> Checkpoint[signed checkpoints]
    Events --> Reputation[per-class reputation]
    Receipts[verified usage receipts] --> Ledger[CU ledger]
    Ledger --> Settlement[signed summaries]
    Checkpoint --> Recovery[independent recovery]
    Reputation --> Admission[class/trust filters]
    Settlement --> Coordinators[scoped federation]

Planned release governance

Self-publishing remains planned. Design requires trusted reproducible builders, TUF, SBOM, provenance, canaries, validator attestations, automatic rollback, and independent recovery. Emergency threshold authority may freeze, revoke, rotate, quarantine, or rollback. It cannot read secrets, rewrite workloads, seize private nodes, redirect value, or bypass owner policy. Voting ships disabled and may govern trust/protocol transitions only—never routine placement.

6. Design changes

Kikapu evolved capabilities inside one product:

root network agent → unprivileged agent plus narrow root supervisor;
centralized API-key identity → signed roots plus derived capabilities;
manual node-loss recovery → opt-in stateless rescheduling with empty replacement storage;
single coordinator → scoped federation with local sovereignty and signed capacity grants;
central tenant subordinate IDs → node-local disjoint allocation.

Rejected designs: blockchain as live DB, all-to-all global gossip, automatic stateful relocation, silent emergency administration, transferable CU governance weight, and self-publishing without independent rollback.

7. Verification and current limits

Gate ladder:

make docs-check
make verify
make verify-fabric
# Supported Linux host only:
make verify-fabric-linux

make verify covers docs, lint, unit, and PostgreSQL integration. make verify-fabric adds build, sqlc freshness, CLI docs, management parity, reset drills, SLO, chaos, load, and mesh tests. Linux gate adds host integration and full E2E. Site, Playwright, packaging, release-content, physical Mac, provider custody, production publication, elapsed soaks, and drills retain separate evidence.

Implemented repository behavior includes control plane, rootless runtime, tunnel/routing, telemetry, local survival, trust, CU economy, federation, reputation, consoles, unified Linux/macOS host drivers, and signed distribution path. Planned work includes complete self-publishing, progressive rollout, constrained emergency governance, and disabled-by-default voting integration. Launch proof pending includes production credentials/custody, live OAuth/TLS, fresh remote Linux and multi-provider validation, physical Apple-silicon lifecycle, elapsed CU/reputation soaks, production recovery drills, and authorized publication. Current defects and external gates live as native Issues under Kikapu Delivery.

8. Reference map

Installation, host lifecycle, TLS, backup, trust, economy, troubleshooting: operations.md
Repository layout, workflow, gates, docs ownership: DEVELOPER.md
Exact commands and flags: cli-reference.md
Binding architecture and acceptance criteria: Kikapu Delivery
Acceptable use: acceptable-use-policy-v1.md