sklpdocs

Front and origin nodes over the WireGuard mesh

One public front terminates TLS and proxies over wg0 to origin nodes that expose no port at all. This is the exact order that brought the first pair live, including the two traps met on the way.

The model — one public front, private origins

Two kinds of sklet nodes share one private network and one WireGuard mesh (ADR-006):

NodePublic portsHoldsRuns
front80 / 443, control API, the origins' control API relayDNS records, ACME state, the mesh peer registrymeshauthority + the gateway
originnone: no Floating IP at all (PUBLIC_IP=false)the application and its datathe workload spaces

The front terminates TLS and proxies every public host to a service on an origin over wg0, using routes of the form <space>/<service>. It stores no application data and no application secret, so a compromised or saturated front is destroyed and rebuilt, and the origin was never reachable from the internet in the first place.

meshauthority lives on the front because an authority that hands out mesh addresses must not depend on a machine it addresses. It is the control plane only: once peers are configured, WireGuard carries traffic peer-to-peer.

The front is also the only public address of the whole mesh (ADR-019). An origin has no Floating IP: it leaves through the shared network's router for image pulls and telemetry, and is reached, for deploys, image watches, logs and exec, through the front's control API relay. The front peeks the TLS server name of every connection on its control port; <node-id>.nodes.front.<zone> is spliced, still encrypted, to that peer's control API on its private address. TLS is never terminated there, so the origin's pinned certificate and token stay end to end.

This page is the exact path that put the first pair live on OVH on 2026-09-11: mesh-front-v1 (front) and sklet-auth-v1 (origin for the identity provider), driven from infra/instance/.

0. The network comes first

The private network is its own stack, mesh-net-v1: network, subnet and the router whose external gateway SNATs every Floating-IP-less node's egress. No node owns it, so the front stays disposable even with prod attached:

cd infra/instance
pnpm exec cdktf deploy mesh-net-v1 --auto-approve   # env/mesh-net-v1.env: PRIVATE_CIDR=10.73.0.0/24

Every node below joins it with SHARED_NETWORK_NAME=mesh-net-v1-net and SHARED_SUBNET_NAME=10.73.0.0/24.

1. Declare the two stacks

Each machine is a SklpHostStack id in infra/instance/main.ts, opted in by the presence of env/<id>.env. The committed .env.example files document every variable; the real .env is gitignored.

Front (env/mesh-front-v1.env):

PUBLIC_HTTP=true
SHARED_NETWORK_NAME=mesh-net-v1-net
SHARED_SUBNET_NAME=10.73.0.0/24
PRIVATE_CIDR=10.73.0.0/24
PRIVATE_IP=10.73.0.10
FLAVOR_NAME=d2-2
DATA_VOLUME_GB=10              # ACME state + meshauthority's peer registry
DNS_SUBDOMAIN=front
MESH_AUTHORITY_URL=https://10.73.0.10:4300  # itself; pin its certificate fingerprint on every node (see The mesh)
MESH_TOKEN=<openssl rand -hex 32>
MESH_PEER_CIDR=10.73.0.0/24
CORE_CIDR=<skalpai core egress ip>/32
NODE_RELAY=true                # *.nodes.front.<zone> -> this Floating IP, agent relays

Origin (env/sklet-auth-v1.env): joins the front's network instead of creating one, same token, no public HTTP.

PUBLIC_HTTP=false
PUBLIC_IP=false                # no Floating IP; subnet-only security group
ADVERTISE_VIA=nodes.front.<zone>   # the front's node_relay_domain output
SHARED_NETWORK_NAME=mesh-net-v1-net
SHARED_SUBNET_NAME=10.73.0.0/24
PRIVATE_CIDR=10.73.0.0/24
PRIVATE_IP=10.73.0.20
FLAVOR_NAME=d2-2
DATA_VOLUME_GB=20
DNS_SUBDOMAIN=auth             # private name only; app domains live on the front
MESH_AUTHORITY_URL=https://10.73.0.10:4300
MESH_TOKEN=<same as the front>
MESH_PEER_CIDR=10.73.0.0/24

The mesh only routes between peers on the same OpenStack network. Two stacks each on their own dedicated network cannot reach one another over their private IP at all, whatever the security group says.

2. Provision the front first

cd infra/instance
pnpm exec cdktf synth
pnpm exec cdktf deploy mesh-front-v1 --auto-approve

cloud-init installs sklet from the published dist, applies cap_net_bind_service,cap_net_admin as root, and starts sklet-agent.service in the user's lingered session. Wait for the control API, then register a deploy context without ever printing the token:

until curl -sk -m 5 https://<front-ip>:6464/ -o /dev/null; do sleep 15; done
sklp deploy login \
  --server https://<front-ip>:6464 \
  --token "$(ssh -i ssh-keys/mesh-front-v1-key architect@<front-ip> cat ~/.config/sklp/control/token)" \
  --ca-fingerprint <fingerprint printed in the agent journal> \
  --context mesh-front --space meshauthority

At this point the agent journal shows mesh join failed (continuing without it): expected, the authority is not running yet.

3. Deploy meshauthority on the front

The manifest is .sklp/meshauthority/deploy.yaml. Its two variables have no project_id, so the host environment resolves them:

MESH_TOKEN=<token> SCW_SECRET_KEY=<registry key> \
  sklp deploy -context mesh-front -space meshauthority meshauthority
ssh architect@<front-ip> 'curl -sfk https://10.73.0.10:4300/health'

The manifest binds ports: ["0.0.0.0:4300:4300"]. The bind address is honoured since #1198; before that every host port silently bound loopback and peers got connection refused. The security group already limits 4300 to MESH_PEER_CIDR.

4. Bring the front into its own mesh

The agent joins the mesh at boot, before any space is launched. On the node that hosts meshauthority that first join can therefore never succeed. Since #1200 the reconciler persists the peer list on every successful registration (~/.config/sklp/mesh/peers.json), and the next restart brings wg0 up from that cache. So, once /health answers:

ssh architect@<front-ip> 'systemctl --user restart sklet-agent.service; sleep 20; ip -br addr show wg0'
# wg0  UNKNOWN  10.90.0.2/24

If the node runs a sklet older than #1200, seed the cache by hand from the registry (GET /v1/peers with the bearer token) before the restart.

A reconfigured ports: mapping does not rebind a host port the agent already opened; a change of bind address needs an agent restart (issue 1fe1bd38).

Boot order matters. The agent joins the mesh once, at startup. A node that boots while meshauthority is not yet answering, the front itself and any origin provisioned before this step, logs mesh join failed (continuing without it); its reconciler then registers on its own cadence and writes peers.json, but wg0 only comes up at the next agent restart. Deploy the front and meshauthority first, then the origins, and restart the agent once on every node that booted too early:

systemctl --user restart sklet-agent.service && sleep 20 && ip -br addr show wg0

An empty wg0 on a fresh origin is this, not a network fault (issue 911201b4).

A dual-homed node misses its first join for a different reason. When the mesh arrives on a second NIC — a node whose primary NIC is on another network, like the cluster's — meshauthority answers fine, but the interface itself is not routable yet: OpenStack attaches it after the instance is up and netplan applies it around the moment the agent starts. The join leaves too early and fails with no route to host, which reads like a firewall problem and is not one. Same remedy, one restart, and the same rule applies — the agent joins once, so nothing recovers on its own:

systemctl --user restart sklet-agent.service && sleep 20 && ip -br addr show wg0

Expect this on every dual-homed node you provision, not occasionally: it has reproduced on every one so far (issue b50be31e). The node looks healthy in the meantime — it logs node ready and serves its control API — while the front resolves it and then times out relaying to it.

5. Provision the origin

The origin has no public address, so every command reaches it through the front: SSH jumps over it, the control API goes through its relay.

pnpm exec cdktf deploy sklet-auth-v1 --auto-approve
alias ossh='ssh -i ssh-keys/sklet-auth-v1-key -J architect@<front-ip> architect@10.73.0.20'
until ossh 'curl -sk -m 5 https://127.0.0.1:6464/ -o /dev/null'; do sleep 15; done
ossh 'ip -br addr show wg0; cat ~/.config/sklp/mesh/node-id; curl -s ifconfig.me'
# wg0  UNKNOWN  10.90.0.3/24   — joined at first boot, the authority was up
# <node-id>
# <the router's SNAT address>  — egress works without a Floating IP
sklp deploy login --server https://<node-id>.nodes.front.<zone>:6464 \
  --token "$(ossh cat ~/.config/sklp/control/token)" \
  --ca-fingerprint <fingerprint> --context sklet-auth --space <app space>

The login URL is the one the agent wrote itself into ~/.config/sklp/node-config.yaml from --advertise-via. Register the node in the Skalpai app with that same server_url: core, image watches and the CI spawner dial it and land on the origin through the front, certificate pinned.

If the last line printed nothing, the router does not SNAT ports without a Floating IP on this cloud; check openstack router show <front>-router for enable_snat.

Check the tunnel both ways: ping 10.90.0.3 from the front, ping 10.90.0.2 from the origin.

6. Route a public host to the origin

Routes are declared in the front's own manifest. A deploy file must declare at least one service, so on the front they live next to meshauthority:

gateway:
  email: <acme email>
 
spaces:
  meshauthority:
    gateway:
      routes:
        - service: urbangate-production/web      # "<space>/<service>" = mesh target
          host: id.urbangate.dev
          path: /
          kind: web
          security_headers: true
        - service: urbangate-production/hydra
          host: id.urbangate.dev
          path: /oauth2
          kind: auth
          max_body: 64Ki

The front resolves urbangate-production to the origin's mesh IP through the peer registry and sends Host: <service>.<space>.internal upstream. Point the DNS record at the front's Floating IP, then re-apply the manifest to the mesh-front context: traefik obtains the certificate and the route goes live.

7. Deploy the application on the origin

The origin is an ordinary sklet node: sklp deploy -context sklet-auth <stack> by hand, or the git path (push → core → node) once the repository's deploy-root.yaml names the node and the Skalpai project holds its secrets. Register the node in the app with its server_url, token and ca_fingerprint, exactly as for any other node.

Swapping sklet on a live node

The published installer is the update path; a manual swap drops the file capabilities, and without them cross-space DNS and wg0 silently die:

curl -fsSL https://api.skalpai.dev/install/sklet.sh | sh
sudo setcap 'cap_net_bind_service,cap_net_admin=+ep' ~/.local/bin/sklet
systemctl --user restart sklet-agent.service

Recap

  1. Front env, cdktf deploy the front, sklp deploy login.
  2. Deploy meshauthority (host env resolves its variables), check /health on the private IP.
  3. Restart the front's agent once: wg0 comes up from the peer cache.
  4. Origin env on the front's network, cdktf deploy, it joins at first boot.
  5. Public routes as <space>/<service> in the front's manifest, DNS to the front, re-apply.
  6. Deploy the workload on the origin by hand or through git reconcile.

Moving a public node behind the front

A node that already serves traffic on its own Floating IP (prod) joins the model in place, without recreating the VM. Each step is reversible until the last one; public traffic never stops.

  1. Second NIC on the mesh network: MESH_NETWORK_NAME=mesh-net-v1-net, MESH_PORT_IP=10.73.0.5, MESH_PEER_CIDR=10.73.0.0/24, then cdktf deploy. The port is attached hot. On an already-booted node write the netplan file by hand (DHCP, use-routes: false); a fresh node gets it from cloud-init.
  2. Mesh flags in the agent unit (--mesh-authority-url, --mesh-endpoint <mesh ip>:51820, SKLP_MESH_TOKEN) and one agent restart: wg0 comes up, the peer announces its spaces and services.
  3. Routes on the front: declare every public host as <space>/<service> in the front's manifest, apply, and compare each host with curl --resolve <host>:443:<front ip> against the node itself before touching DNS.
  4. --advertise-via nodes.front.<zone> on the node, restart, then PATCH /api/stacks/:id/gateway/nodes/:nodeId with the relay server_url and the node's fingerprint. No delete: routes and deployment records stay.
  5. DNS of every zone to the front's IP, then one front agent restart: traefik only retries a failed ACME order on restart.
  6. Resync image watches: an empty push on each repository the node serves, so the watches take the relay address. Watches created by hand with no repository stay on the old address.
  7. PUBLIC_IP=false + PUBLIC_HTTP=false, cdktf deploy: the Floating IP is detached (kept in the account when it was FLOATING_IP), 80/443 close, SSH goes through the front. Egress continues through the node's network router, check it with curl ifconfig.me from the node.