sklp backup — stateful services to S3

Deploy the DB tier of a stack with backups to S3 using only existing deploy primitives — no new sklp code, no new YAML field. A worked, end-to-end-tested example.

Two backup levels

Level	Postgres	NATS / files
Snapshot (periodic dump)	`pg_dump` → S3 — official image	`nats account backup` → S3
PITR (recover to the second)	continuous WAL → S3 — needs a custom image	not possible (no WAL)

backup: {command, schedule} is the universal floor — the supervisor fires the command on a cron schedule inside the live service. PITR reuses the CloudNativePG backup mechanism outside k8s: the barman-cloud-* CLIs (not the operator, which needs the k8s API). The WAL stream is what makes recovery "to the second".

The hard constraint — PITR requires a custom image

archive_command is an in-process Postgres hook: Postgres itself runs barman-cloud-wal-archive on every closed WAL segment, so barman must live in the postgres container — no separate space service can stream the WAL.

Installing barman at boot (apt-get install in a command wrapper) fails in a space: the network namespace blackholes private ranges and external DNS resolves the Debian mirrors IPv6-only with no v6 route, so apt cannot fetch:

W: Failed to fetch http://deb.debian.org/... Temporary failure resolving 'deb.debian.org'
sh: barman-cloud-wal-archive: not found

So for PITR, bake the image (the build runs on the host, where the network is fine):

# Dockerfile.pg-barman
FROM postgres:17-bookworm
RUN apt-get update \
 && apt-get install -y --no-install-recommends barman-cli-cloud python3-boto3 \
 && rm -rf /var/lib/apt/lists/*

sklp image push -dockerfile Dockerfile.pg-barman -context .sklp/backup \
  -image rg.fr-par.scw.cloud/skaplai/data/postgres-barman -tag 17

Snapshot-only Postgres and NATS use 100% official images — apk add aws-cli works inside the space; only Debian's apt is unreachable.

deploy.yaml — the data tier

One deploy holds the DB services a stack needs; it declines into spaces. App services (core/web) drop in as siblings and reach the DBs by space DNS (postgres:5432, nats:4222).

version: 1
kind: deploy
name: data
 
services:
  # Postgres: PITR with the prebuilt barman image.
  postgres:
    image: rg.fr-par.scw.cloud/skaplai/data/postgres-barman:17
    expose: "5432"
    volumes: ["pgdata:/var/lib/postgresql/data"]
    env:
      POSTGRES_PASSWORD: ${PGPASSWORD}
      POSTGRES_DB: app
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT_URL: ${AWS_ENDPOINT_URL}
      BACKUP_BUCKET: ${BACKUP_BUCKET}
    # Continuous WAL archiving = the PITR backbone (in-process hook).
    command:
      - postgres
      - -c
      - "archive_mode=on"
      - -c
      - "wal_level=replica"
      - -c
      - "archive_command=barman-cloud-wal-archive --cloud-provider aws-s3 s3://${BACKUP_BUCKET}/${SPACE}/postgres postgres %p"
    ready:
      log: "database system is ready to accept connections"
      timeout: 120s
    # Nightly base backup + 14-day retention prune. No `set -e` (a pg WARNING on
    # stderr would trip it); guard each step with `|| exit 1`.
    backup:
      schedule: "0 2 * * *"
      command:
        - sh
        - -c
        - >
          barman-cloud-backup -z --cloud-provider aws-s3
          "s3://${BACKUP_BUCKET}/${SPACE}/postgres" postgres || exit 1;
          barman-cloud-backup-delete --cloud-provider aws-s3
          --retention-policy "RECOVERY WINDOW OF 14 DAYS"
          "s3://${BACKUP_BUCKET}/${SPACE}/postgres" postgres || exit 1;
          echo "postgres base backup + prune done"
 
  # NATS: periodic snapshot (no WAL, file-storage streams only).
  nats:
    image: nats:2.10-alpine
    command: ["-js", "-sd", "/data"]
    expose: "4222"
    volumes: ["nats-data:/data"]
    ready:
      log: "Server is ready"
      timeout: 30s
 
  # Sidecar holding the nats + aws CLIs; carries the cron snapshot.
  nats-backup:
    image: natsio/nats-box:0.14.5
    depends_on: [nats]
    command: ["sh", "-c", "apk add --no-cache aws-cli >/dev/null 2>&1 || true; sleep infinity"]
    env:
      NATS_URL: "nats://nats:4222"
      BACKUP_BUCKET: ${BACKUP_BUCKET}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT_URL: ${AWS_ENDPOINT_URL}
    # `nats account backup` exits non-zero on an empty account → `|| true`.
    backup:
      schedule: "*/30 * * * *"
      command:
        - sh
        - -c
        - >
          ts=$(date +%Y%m%dT%H%M%S);
          rm -rf /tmp/snap && mkdir -p /tmp/snap;
          nats account backup /tmp/snap -s "$NATS_URL" || true;
          tar czf "/tmp/nats-$ts.tgz" -C /tmp/snap . || exit 1;
          aws s3 cp "/tmp/nats-$ts.tgz" "s3://${BACKUP_BUCKET}/${SPACE}/nats/nats-$ts.tgz"
          --endpoint-url "${AWS_ENDPOINT_URL}" || exit 1;
          echo "nats snapshot nats-$ts.tgz pushed"
 
spaces:
  production: {}

Snapshot-only Postgres (no custom image)

If you don't need PITR, drop barman — official image + pg_dump:

  postgres:
    image: postgres:17-alpine
    expose: "5432"
    volumes: ["pgdata:/var/lib/postgresql/data"]
    env: { POSTGRES_PASSWORD: ${PGPASSWORD}, POSTGRES_DB: app, AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}, AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}, AWS_ENDPOINT_URL: ${AWS_ENDPOINT_URL} }
    ready: { log: "database system is ready to accept connections", timeout: 120s }
    backup:
      schedule: "0 2 * * *"
      command:
        - sh
        - -c
        - >
          apk add --no-cache aws-cli >/dev/null 2>&1 || true;
          ts=$(date +%Y%m%dT%H%M%S);
          pg_dump -U postgres -Fc app > /tmp/d.dump || exit 1;
          aws s3 cp /tmp/d.dump "s3://${BACKUP_BUCKET}/${SPACE}/postgres/app-$ts.dump"
          --endpoint-url "${AWS_ENDPOINT_URL}" || exit 1;
          rm -f /tmp/d.dump; echo "postgres dump app-$ts.dump pushed"

`${...}` vs `$shell` — load-time expansion

sklp expands only ${VAR} (braces) at load time, over the whole file (comments included), from the process env:

${BACKUP_BUCKET}, ${AWS_*}, ${PGPASSWORD} → resolved by sklp at deploy (must be set in the env). ${SPACE} is kept verbatim and filled per-space.
shell/container vars ($ts, $NATS_URL) use bare $ — a ${braced} there fails the manifest parse.

Env required at sklp deploy time: PGPASSWORD, BACKUP_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL=https://s3.fr-par.scw.cloud.

Backup command rules

The supervisor execs the command via runc exec with no stdin/TTY and discards stdout/stderr. Two failure modes to avoid:

No streaming pipe under set -e. pg_dump | aws s3 cp - + a pg WARNING on stderr → SIGPIPE → non-zero exit. Dump to a file, then upload.
Guard expected non-zero. nats account backup exits 1 on an empty account; wrap it || true so an empty DB isn't a backup failure.

Changing a backup: command requires a full supervisor restart (sklp deploy delete + redeploy), not just a reconcile — the scheduler reads the commands at startup.

Restore — a one-shot job

Stop the target service (free its volume), deploy the job once, then start the service.

jobs:
  restore-postgres:                       # PITR restore
    image: rg.fr-par.scw.cloud/skaplai/data/postgres-barman:17
    volumes: ["pgdata:/var/lib/postgresql/data"]
    env:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT_URL: ${AWS_ENDPOINT_URL}
      BACKUP_BUCKET: ${BACKUP_BUCKET}
      TARGET_TIME: "2026-06-11 14:32:07"  # the instant to recover to
    command:
      - sh
      - -c
      - >
        rm -rf /var/lib/postgresql/data/*;
        id=$(barman-cloud-backup-list --cloud-provider aws-s3
             "s3://$BACKUP_BUCKET/${SPACE}/postgres" postgres | tail -1 | awk '{print $1}');
        barman-cloud-restore --cloud-provider aws-s3
          "s3://$BACKUP_BUCKET/${SPACE}/postgres" postgres "$id" /var/lib/postgresql/data;
        touch /var/lib/postgresql/data/recovery.signal;
        {
          echo "restore_command='barman-cloud-wal-restore --cloud-provider aws-s3 s3://$BACKUP_BUCKET/${SPACE}/postgres postgres %f %p'";
          echo "recovery_target_time='$TARGET_TIME'";
          echo "recovery_target_action='promote'";
        } >> /var/lib/postgresql/data/postgresql.auto.conf;
        echo "restore staged; start postgres to replay WAL up to $TARGET_TIME"
 
  restore-nats:                           # snapshot restore
    image: natsio/nats-box:0.14.5
    depends_on: [nats]
    env:
      NATS_URL: "nats://nats:4222"
      BACKUP_BUCKET: ${BACKUP_BUCKET}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT_URL: ${AWS_ENDPOINT_URL}
    command:
      - sh
      - -c
      - >
        apk add --no-cache aws-cli >/dev/null 2>&1 || true;
        key=$(aws s3 ls "s3://${BACKUP_BUCKET}/${SPACE}/nats/" --endpoint-url "${AWS_ENDPOINT_URL}"
              | sort | tail -1 | awk '{print $4}');
        aws s3 cp "s3://${BACKUP_BUCKET}/${SPACE}/nats/$key" /tmp/snap.tgz --endpoint-url "${AWS_ENDPOINT_URL}";
        rm -rf /tmp/snap && mkdir -p /tmp/snap && tar xzf /tmp/snap.tgz -C /tmp/snap;
        nats account restore /tmp/snap -s "$NATS_URL";
        echo "nats restore done"

Verified end-to-end

Tested with MinIO running as a space service (minio:9000) and snapshot schedules at */1. The scheduler logged backup "postgres" ok / backup "nats-backup" ok every minute, and the objects landed in the bucket:

postgres/app-20260612T082352.dump   1.5 KiB
postgres/app-20260612T082450.dump   1.5 KiB   ← one per minute, automatic
nats/nats-20260612T082349.tgz       86 B
nats/nats-20260612T082449.tgz       87 B

List the bucket from inside the space (MinIO is only reachable on the space net):

sklp deploy exec --space <space> nats-backup -- \
  sh -c 'aws --endpoint-url "$AWS_ENDPOINT_URL" s3 ls s3://<bucket>/ --recursive --human-readable'

Isolation boundary (honest)

All services share the space's network bridge (one network namespace, e.g. 10.88.0.0/24) and the same rootless user namespace. The space blackholes private ranges (10/8, 172.16/12, 192.168/16). DBs are reachable by any service in the space, an escape lands on the same unprivileged host UID, and the blast radius includes the other services' volumes. Fine for a trusted single-tenant stack; untrusted workloads need a stronger boundary (separate space, or a gVisor/Kata-class runtime) — out of sklp scope today.