sklp on a VM — agent, control API, remote deploy

Run the whole lifecycle on one machine. A control-plane agent boots empty, opens a TLS control API, and waits for a remote deploy — no cluster, no Kubernetes, no SSH for day-to-day. This is the exact, end-to-end path, from binary to live HTTPS.

The model — one node, kubelet-style

A sklp VM runs a single control-plane agent: an empty node that boots with zero services, opens a TLS control API on :6464, and waits for a remote deploy to hand it a stack. It inverts the Kubernetes model — the kubelet boots first and empty; the workload arrives later, over the wire. No cluster, no Kubernetes, no SSH needed for day-to-day deploys once the agent is up.

Everything below is the exact, end-to-end path: provision the binary, run the agent, register a control-plane context, seal your secrets, and apply a stack from your workstation.

1. Get the binary onto the VM

The agent is the sklp binary. It must match the version your workstation deploys with, because the agent is what runs your services (and the native gateway) inside the space.

The Linux CLI is a fully static build (CGO_ENABLED=0), so it runs on any Linux with no glibc match required. Two ways to install it:

# a) public installer (curl | sh) — pulls the published static binary
curl -fsSL https://api.skalpai.dev/install/sklp.sh | sh
#    installs to ~/.local/bin/sklp
 
# b) copy a specific build yourself (matching uncommitted/local changes)
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o sklp ./apps/sklp/cmd/sklp
scp sklp architect@<vm-ip>:/tmp/sklp-new
ssh architect@<vm-ip> 'install -m 0755 /tmp/sklp-new ~/.local/bin/sklp'

Version match matters. The published installer only serves what the last publish-cli run pushed to the dist bucket. If you deploy from a branch that isn't published yet, copy the binary yourself (option b) — otherwise the agent runs an older sklp than your workstation and silently ignores newer config fields.

2. Run the agent (systemd)

The agent runs as a user systemd service (rootless — no root, no daemon socket). A minimal unit:

# ~/.config/systemd/user/sklp-agent.service
[Unit]
Description=sklp control-plane node (empty agent, control API)
 
[Service]
ExecStart=%h/.local/bin/sklp deploy agent --listen :6464 --gateway-mode singleton
Environment=SKLP_GATEWAY_ACME_EMAIL=you@example.com
Restart=always
RestartSec=5
 
[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now sklp-agent.service
loginctl enable-linger $USER   # keep the service running after logout

On first start the agent generates its own identity and prints it:

sklet: control API listening on https://0.0.0.0:6464
  token file:  /home/architect/.config/sklp/control/token
  fingerprint: <generated-sha256-fingerprint>
 
Remote control ready. On your workstation run:
  sklp deploy login --server https://<vm-ip>:6464 --token "$(ssh <vm-ip> cat /home/architect/.config/sklp/control/token)" --ca-fingerprint <fp>

The token is never printed — only its path on the VM. Logs (systemd journal, any collector shipping them off-host) are read by more people than a terminal session, so the token stays in a 0600 file instead. The line the agent prints is ready to paste as-is: $(ssh <vm-ip> cat <path>) is a shell command substitution, not sklp doing anything special. When you run the pasted line on your workstation:

1. the shell evaluates the $(...) part first — it opens an SSH connection to the VM, runs cat <tokenPath> there, and captures its stdout;
2. the shell substitutes that captured token, verbatim, in place of $(...);
3. only then does sklp deploy login run, with --token <the-real-token> as a plain argument — sklp itself never touches SSH.

This needs ssh <vm-ip> to already work passwordless from your workstation (a key already trusted by the VM). Sanity-check that piece on its own before pasting the full line:

ssh <vm-ip> cat /home/architect/.config/sklp/control/token   # should print the token, nothing else

The fingerprint, unlike the token, is not a secret — it is printed in the clear and safe to copy from the log.

The identity is per-agent. The token and CA fingerprint are tied to this agent instance. Destroy and recreate the VM (or restart the agent from a clean state) and it mints new ones — the old context stops connecting with a fingerprint mismatch. That is the anti-MITM pin doing its job, not a bug: re-run deploy login with the new values.

Ship the node's logs

The agent ships its own logs and its spaces' logs to Skalpai when SKLP_LOG_INGEST_KEY holds an ingest-source key (sklp_ing_…, created in the app). Without it the agent runs fine and simply ships nothing — the only symptom is silence, so it is worth checking rather than assuming:

systemctl --user show sklet-agent -p Environment | tr ' ' '\n' | grep SKLP_LOG_INGEST_KEY

Add it in a drop-in rather than by editing the unit, so a redeploy that rewrites the unit does not drop the key:

mkdir -p ~/.config/systemd/user/sklet-agent.service.d
printf '[Service]\nEnvironment=SKLP_LOG_INGEST_KEY=%s\n' "$KEY" \
  > ~/.config/systemd/user/sklet-agent.service.d/ingest-key.conf
chmod 600 ~/.config/systemd/user/sklet-agent.service.d/ingest-key.conf
systemctl --user daemon-reload
systemctl --user restart sklet-agent

A user unit reads its drop-ins from ~/.config/systemd/user/<unit>.d/ as well as from /etc/systemd/user/<unit>.d/, and the home directory wins. So the key goes in without root — and it keeps winning afterwards, including over a value a later provisioning run writes under /etc. Put the key in both places or neither; the same key in both is the one combination that cannot surprise you.

The value belongs to the machine, not to the repo: provisioning passes it as LOG_INGEST_KEY and the unit reads it as SKLP_LOG_INGEST_KEY. A key that is missing rather than wrong is the common case, and it is invisible from the outside — the agent reports healthy either way.

Exporting fails loudly, at least:

traces export: failed to send to https://api.skalpai.dev/v1/traces:
  401 Unauthorized (body: {"error":"missing x-api-key header"})

That line, repeating in journalctl --user -u sklet-agent, means the key never reached the process. It stops the moment the drop-in is in place.

3. Register the context (kubeconfig-style)

Contexts are how one workstation targets many VMs — like kubectl contexts. Save one from the values the agent printed:

sklp deploy login \
  --server https://<vm-ip>:6464 \
  --token <token> \
  --ca-fingerprint <fingerprint> \
  --space production \
  --context prod-lungor
 
sklp deploy context use prod-lungor
sklp deploy ls        # → "space production has no services"  (agent reachable)

sklp deploy ls answering — even with "no services" — confirms the workstation is authenticated to the agent over the pinned TLS channel. No SSH, no tunnel: the control API is reachable directly on :6464.

4. Seal your secrets

Deploy secrets (DB creds, registry keys, ACME DNS creds) live in a committable, Pulumi-style config: .sklp/<stack>/config.yaml. Values are AES-256-GCM, encrypted under SKLP_SECRETS_PASSPHRASE — the file is useless without the passphrase, so it is safe to commit.

Bulk-import a dotenv in one shot:

export SKLP_SECRETS_PASSPHRASE='choose-a-strong-passphrase'
sklp config import --secret --stack prod .env
sklp config ls --stack prod        # keys shown; values as <secret>

At deploy time sklp resolves ${VAR} references in deploy.yaml from this store (and from env_files). For a remote apply, the client reads the values on your workstation and ships them to the agent over the TLS control API — a detached agent has no access to your .env, so the values travel encrypted in-flight, never written to the agent's disk in the clear.

5. Apply the stack

From the stack repo, with the context selected and the passphrase exported:

export SKLP_SECRETS_PASSPHRASE='...'
sklp deploy context use prod-lungor
sklp deploy prod            # applies .sklp/prod/deploy.yaml to the remote space

space "production": created (7 change(s))
  - added service postgres
  - added service nats
  - added service core
  - added service web
  - added service migrate
  - added service restore-db
  - updated gateway routes

Watch it converge:

sklp deploy get production
# SERVICE   INSTANCE     STATUS
# postgres  postgres-0   ready
# nats      nats-0       ready
# core      core-0       ready
# web       web-0        ready
# migrate   migrate-0    done

The space name in deploy.yaml must match the context's space (here production). A mismatch fails with "space X is not declared" — the agent targets the space its context names, and the stack must declare that space.

6. Expose it — the gateway

A space publishes services through the gateway, configured per space under spaces.<name>.gateway.routes. The front door binds :9080 (HTTP, routed by Host) and :9443 (HTTPS, routed by SNI); a host-level NAT redirect sends :80/:443 there (sklp gateway redirect).

spaces:
  production:
    gateway:
      routes:
        - service: web
          host: lungor.fr

Two front-door modes exist, but the mode is a property of the node, not of any stack's deploy.yaml:

The agent chooses at boot with --gateway-mode singleton or --gateway-mode per-space (the flag is mandatory — there is no default the agent picks silently). A deploy.yaml cannot declare or override it: the mode governs the shared front door for every stack on the node, so letting one stack's manifest set it would let whichever stack deploys first decide the node's front door for every other stack after it. deploy.yaml only declares routes and (for per-space) the ACME contact/provider — never the mode itself.

ExecStart=%h/.local/bin/sklp deploy agent --listen :6464 --gateway-mode singleton

7. Automatic HTTPS — native ACME (per-space)

In per-space mode the in-worker L7 obtains real Let's Encrypt certificates via the dns-01 challenge — the only challenge that works behind the SNI-passthrough mux, and the only one that issues wildcards. The node must already be running with --gateway-mode per-space (see above); deploy.yaml then only configures the ACME contact/provider, not the mode:

gateway:
  email: you@example.com
  acme:
    enabled: true
    dns_provider: ovh          # provider-agnostic; ovh ships first
    propagation_seconds: 120   # DNSSEC zones propagate slower

The DNS API credentials (e.g. OVH_APPLICATION_KEY / OVH_APPLICATION_SECRET / OVH_CONSUMER_KEY) are read from the sealed config like any other secret. The managed domains are derived from the route hosts — no domain list to maintain. Certs and the ACME account persist on disk (the acme.json equivalent), so a restart never re-hits the CA. Dev/bare hosts (*.test, localhost) fall back to a self-signed cert automatically.

8. Rate limiting (per-space)

Any route can be rate-limited per client IP with a token bucket:

      routes:
        - service: web
          host: lungor.fr
          rate: 50     # sustained requests/sec per client IP
          burst: 100   # bucket size (max instantaneous); required when rate > 0

Over-limit requests get 429 Too Many Requests + Retry-After. Buckets are per-worker (per space); idle client buckets are evicted so a source-IP scan can't grow memory unbounded. RemoteAddr is the true peer — the mux fd-passes the client connection, so no upstream proxy rewrites the source IP.

Recap

1. Binary on the VM (installer or scp a matching build).
2. Agent as a rootless systemd user service on :6464.
3. Context registered with the agent's token + CA fingerprint.
4. Secrets sealed with sklp config import --secret.
5. Apply sklp deploy <stack> from your workstation — no SSH.
6. Gateway routes expose services; the node runs in singleton (traefik) or per-space (native) mode, set once via sklp deploy agent --gateway-mode — never from a stack's deploy.yaml.
7. HTTPS via native dns-01 ACME (per-space) or traefik (singleton).
8. Rate limiting per route, per client IP (per-space).