The mesh — nodes, meshauthority and its authentication

Every sklet node holds a WireGuard address handed out by meshauthority; a signed request under an admitted key is the only way in. This is the exact contract, as the code enforces it.

What the mesh is

Every sklet node carries a WireGuard interface, wg0, with one address in 10.90.0.0/24. A service running in a space on node A is reachable from a space on node B at <service>.<space>.internal, which resolves to A's mesh address. The front uses the same path to proxy a public host to the origin that runs it (ADR-021). Nothing else crosses the mesh: image pulls, telemetry and package downloads leave through the node's own egress.

One process, meshauthority, is the registry: who holds which WireGuard public key, which mesh address, which dialable endpoint, and which (space, service) pairs. It runs on the front, listens on :4300 over TLS, and is the control plane only. Once peers are configured, traffic is peer-to-peer WireGuard and keeps flowing while the authority is down.

Design records: ADR-006 (the mesh), ADR-008 (cross-host relay), ADR-019 (private origins behind the front), ADR-021 (front routes), ADR-024 (admission, TLS, contested routes).

Address plan

A node's WireGuard endpoint is <private ip>:51820. The listen port is fixed in the binary (listenPort in wg_linux.go) and the security group opens it between nodes only.

A node's identity

sklet agent creates its mesh identity on first start and never changes it (mesh.EnsureIdentity). It lives in ~/.config/sklp/mesh/, 0600, outside the cache so sklp clean cannot remove it:

Two keys on purpose: WireGuard's is a key-agreement key (X25519) and must not double as a signature key.

Wiping the directory makes a new node: new ID, new keys, new mesh address, and the old ID stays in the registry as a silent peer until someone removes it (see Rebuilding a node).

The registry

meshauthority keeps the peers in memory and persists them to one JSON file (MESH_DATA_FILE, atomic write + rename), on a volume of the front. Every node re-registers its full state every 5 seconds, so the file is a cache of live reality; the only durable value in it is the address allocation.

A peer record:

retired keeps, for every removed ID, its signing key and its old mesh address: a removed ID can only come back under the same key, and gets its address back if it is still free.

Endpoints

All /v1/peers routes require Authorization: Bearer <MESH_TOKEN> (constant-time compare, 401 otherwise). GET /health needs nothing.

Request signing

Every signed request carries three headers:

The payload is four lines joined by \n:

<METHOD>
<path>            e.g. /v1/peers/register
<timestamp>
<hex sha256(body)>   body empty => sha256 of the empty string

Method and path pin the signature to one operation; the timestamp is accepted within 5 minutes of the authority's clock (MaxClockSkew); the body hash covers the published services, so they cannot be rewritten on the way. The payload is built identically on both sides (mesh.SigningPayload in the authority, signingPayload in the client), duplicated rather than imported because the two are separate Go modules; tests on both sides pin it byte for byte.

Order of checks on a write: bearer token, then signature (401), then admission and identity (403).

Answers a node can get

Admission and revocation

MESH_ADMITTED_KEYS is the comma-separated list of admitted signing public keys (hex, case-insensitive). It is the trust list of the mesh:

• a registration under any other key is refused with 403;
• a peer already registered whose key leaves the list disappears from GET /v1/peers and from every node's wg0 on their next refresh, and can no longer heartbeat, register or delete anyone;
• an empty list keeps the mesh open to any holder of the token and prints WARNING: MESH_ADMITTED_KEYS is empty at startup. Never run production that way.

The list lives in .sklp/meshauthority/deploy.yaml, so changing it is a redeploy of the authority (one rolling restart, the data path is unaffected). The keys are public; only the token and the TLS key are secrets.

Identity is trust on first use on top of admission: the first registration of an ID binds its signing key, and every later request for that ID must be signed by it. Holding the token is therefore never enough to take over a node, and holding an admitted key is never enough to take over another node's ID.

The token and TLS

MESH_TOKEN is one shared secret proving membership of the mesh, nothing more; identity comes from the signing key. Every node reads it from SKLP_MESH_TOKEN (rendered into ~/.config/sklp/node.env, 0600) and removes it from its environment once read, so space workers never inherit it.

The authority refuses to start without MESH_TLS_CERT (MESH_ALLOW_PLAINTEXT=1 exists for local tests only). The certificate is self-signed, generated once with MESH_TLS_HOSTS as SANs, stored on the data volume next to the registry, and reused across restarts so the pin stays valid. At startup the authority prints certificate fingerprint (pin this on every peer): <sha256 hex>.

Every node pins that fingerprint: sklet host configure refuses to render a unit unless the authority URL is https:// and the fingerprint is a full SHA-256 (mesh.CheckAuthority). The client verifies the presented leaf against the pin and nothing else; there is no CA.

Data plane

On every start the agent joins before any space is launched (mesh.Join):

1. EnsureIdentity loads or creates the files above.
2. Register with the pinned client. On success: wg0 is created with the returned address /24, its full peer set is replaced (ReplacePeers), each peer gets AllowedIPs = <mesh ip>/32 and its endpoint, and the list is written to peers.json.
3. On failure: if peers.json exists, wg0 comes up from it and the agent logs meshauthority unreachable, brought wg0 up from the last known peer list; the node serves traffic in degraded mode and keeps retrying. A node that never joined stays out until the authority answers.

Then the network controller's reconciler runs every 5 seconds: it calls Register with the node's current spaces and the (space, service, port) triples it serves, receives the peer list, replaces wg0's peers, and refreshes its peer cache. There is no separate heartbeat loop; a full register is the heartbeat.

The peer cache (netctrl.PeerCache):

• ignores peers not seen for 30 seconds (peerStaleAfter);
• maps <space>.<service> to the mesh address of the one peer publishing it;
• when two live peers publish the same pair it routes nothing for that pair and logs netctrl: <space>.<service> is published by peers A and B; not routing it until one withdraws (ADR-024: contested routes go dark, the front answers 503);
• maps a peer ID to its endpoint host, which the front's relay uses.

Cross-node resolution: a container asks its space DNS for <service>.<space>.internal; a name the node does not serve itself is answered with the remote node's mesh address. The controller binds its HTTP proxy and TCP listeners on the node's mesh address as well as on the space side, so the remote container connects to <mesh ip>:80 (HTTP) or the declared TCP port over wg0, and the local controller hands the stream to the container. The controller accepts any mesh peer on those listeners: the mesh proves the peer is an admitted node, not which space the request comes from (ADR-024 consequence, ADR-023 must not trust an identity header for having arrived over wg0).

The control API relay

Private origins have no public address (ADR-019). Their control API (:6464, TLS with a per-node self-signed certificate and a per-node bearer token under ~/.config/sklp/control/) is bound on the private interface only (privateOriginListen), and they advertise themselves as https://<node-id>.<relay domain>:6464 (advertise_via in node.yaml).

The front runs --node-relay-domain <relay domain> and a wildcard DNS record *.<relay domain> points at its public address. Its control listener peeks the TLS ClientHello of every connection: an SNI of the form <node-id>.<relay domain> is spliced, bytes untouched, to that peer's control API on its private address, looked up in the peer cache by node ID. Any other SNI is served by the front's own control API.

Consequences:

• the origin's certificate pin and token stay end to end; the front cannot read or inject anything;
• a node unknown to the mesh, stale for 30 seconds, or not admitted is not relayed: the front closes the connection;
• everything the operator or the core does to a private node goes through this path: sklp deploy apply|ls|logs|exec|restart, git reconcile, image rolls, restore drills, sklp deploy mesh, and Data connections of type space (POST /v1/spaces/<space>/dial/<service>, a raw TCP stream to the service's host-side port);
• this traffic uses the private network, not wg0.

Operations

Read the mesh

sklp deploy mesh ls               # via the current context's node

Columns: ID, mesh IP, endpoint, last seen, number of services, spaces, sign key. The node relays GET /v1/peers with its own token; the operator never needs MESH_TOKEN.

On a node: journalctl --user -u sklet-agent | grep -E 'mesh|netctrl'. On the front, the authority's log names every refused registration: refused <id>: signing key <key> is not in MESH_ADMITTED_KEYS.

Add a node

1. Provision it on the private network and configure node.yaml:

   mesh:
     authority_url: https://10.73.0.10:4300
     cert_fingerprint: <sha256 printed by meshauthority>
     endpoint: 10.73.0.60:51820
   advertise_via: nodes.front.<zone>

   and SKLP_MESH_TOKEN=<token> in node.env, then sklet host configure.

2. The agent starts, generates its identity and is refused with 403. Read its signing key in the authority's log or in ~/.config/sklp/mesh/sign.key on the node (public key = last 64 hex characters).

3. Append the key to MESH_ADMITTED_KEYS in .sklp/meshauthority/deploy.yaml and redeploy the authority from a clean main (sklp deploy context use mesh-front && sklp deploy apply meshauthority).

4. Within 5 seconds the node registers, gets a mesh address, and appears in sklp deploy mesh ls with its sign key.

Revoke a node

Remove its key from MESH_ADMITTED_KEYS and redeploy the authority. The node vanishes from the peer list; every other node drops it from wg0 at its next refresh (5 seconds); its routes stop resolving; the front no longer relays its control API. Its ID stays bound to that key, so nobody else can register it.

Rebuild a node

A rebuilt VM has a new identity. The old ID stays in the registry with last_seen_at frozen. Once it has been silent for 2 minutes:

sklp deploy mesh rm <old id>

The node relays a signed DELETE; the old address returns to the pool (reserved first for a return of the same key). Do not remove a live peer: the authority refuses it, and it would re-register at once anyway.

Then admit the new key exactly as for a new node. If the volume was kept (~/.config/sklp/mesh/ intact), nothing changes: same ID, same key, same address.

Rotate the token

Change MESH_TOKEN in the skalpai project secrets (the authority reads it from its stack config), redeploy the authority, then update node.env on every node and restart each agent. Between the two, nodes fail to register and keep serving from peers.json; the front goes last, because a reconfigured node drops peers it cannot see and the front is the one everyone must still reach.

Rotate the TLS certificate

Delete tls.crt/tls.key on the authority's volume and restart it: a new certificate is generated and its fingerprint printed. Until each node's cert_fingerprint is updated (sklet host configure re-renders the unit and restarts the agent), that node cannot register and runs from peers.json. Same order as above: origins first, the front last.

Move the authority to TLS from plaintext

Documented in ADR-024 and done on 2026-09-29: authority on TLS with the ready: tcp probe (its HTTP probe cannot speak TLS), then every origin (https:// URL and fingerprint), then the front, then the binaries.

Threat model in one table

The residual risk, accepted in ADR-024, is the compromised admitted node publishing a space whose real node is down. Closing it means declaring which node may publish which space; that is not built.

Environment and flags reference

meshauthority:

sklet agent (rendered by sklet host configure from node.yaml):