CI pipelines

The model

A pipeline is a YAML file under .sklp/pipelines/. sklp run <name> loads .sklp/pipelines/<name>.yaml, diffs the branch to find the impacted scopes, and runs only the steps those scopes select, in parallel along a DAG, each one cached by content.

Concept	Where	What it decides
`steps:`	pipeline	the units of work, ordered by `depends_on:`
`scopes:`	pipeline	named glob lists, matched against the diff
`tools:`	pipeline + step	pinned toolchain versions a step opts into
`services:`	pipeline + step	sidecars (Postgres, NATS…) a step needs live
`space:`	pipeline	inside a Space container (default) or on the host
`on:`	pipeline	which branches may select and run it

CI is never GitHub Actions. There is no .github/workflows/ driving sklp: remote runs are dispatched by the Skalpai app, local runs by sklp run.

Pipeline file format

# yaml-language-server: $schema=../../apps/sklp/pipeline/pipeline.schema.json
version: 1
kind: pipeline
name: ci
 
on:
  branches: ['**', '!main']
 
tools:
  go: "1.26.6"
  pnpm: "11.3.0"
 
secrets:
  - GITHUB_TOKEN
 
scopes:
  core:
    - "apps/core/**"
    - "go.work"
  web:
    - "apps/web/**"
    - "pnpm-lock.yaml"
 
services:
  timescaledb-test:
    image: timescale/timescaledb:2.17.2-pg16
    ready:
      tcp: localhost:5432
      timeout: 60s
 
steps:
  - name: vet
    scope: [core]
    tools: [go]
    run: go vet ./...
 
  - name: test
    scope: [core]
    tools: [go]
    depends_on: [vet]
    services: [timescaledb-test]
    run: go test -race -count=1 ./...

Top-level fields

Field	Meaning
`version`	always `1`
`kind`	`pipeline`; anything else is rejected by `sklp run`
`name`	pipeline name
`space`	`false` runs on the host instead of a Space container
`extends`	sibling file(s) whose `scopes`, `tools`, `env`, `secrets` are merged in
`on`	`branches:` globs (last match wins, `!` excludes) and `clean_tree:`
`tools`	`name: version` pins
`env` / `env_files`	plain env, and gitignored dotenv files (`.env.local`)
`secrets`	names resolved before the run; missing → fail fast
`registries`	push credentials used on a CI node (see Publish)
`cache`	persisted directories keyed by content
`scopes`	`name: [globs]`
`services` / `jobs`	sidecars and one-shot jobs, same shape as `sklp dev`
`tag`	publish-tag strategy, exported as `$SKLP_PUBLISH_TAG` / `{{ publish_tag }}`
`steps`	the work

Step fields

Field	Meaning
`name`	unique; what `depends_on` references
`run`	shell snippet run via `sh -c` (required for a shell step)
`kind`	empty = shell step; `publish` = build + scan + push an image
`scope`	scopes that select the step; none = always runs
`depends_on`	step names that must finish first
`tools`	which pinned tools are put on `PATH`
`services`	sidecars that must be Ready before `run`
`secrets`	per-step opt-in to pipeline secrets (both levels are required)
`env`, `cwd`, `if`	step env, working dir, condition
`cache`	`false` opts the step out of the content cache
`inputs` / `outputs`	extra cache-key inputs and produced paths
`network`	`none` unshares the network namespace
`filesystem`	default `confined` (Landlock: no `$HOME`); `open` lifts it

An unknown key is parsed and ignored, never reported: a typo enforces nothing. Use the schema header for editor validation.

Space vs host

By default a pipeline runs in a Space: a rootless container whose rootfs is built from .sklp/space.yaml, the same recipe the dev stack uses. There is no separate runner image to maintain.

space: false runs the steps directly on the host. Publish pipelines need it: a rootless container cannot create the containers an image build runs.

Running a pipeline

sklp run                     # list the pipelines in .sklp/pipelines/
sklp run ci                  # run .sklp/pipelines/ci.yaml
sklp run -file path/x.yaml   # a pipeline outside .sklp/pipelines/

See sklp run. The runner also accepts:

Flag	Effect
`-resolve-only`	print the plan (impacted scopes + changed paths), run nothing
`-base-ref <ref>`	base of the impact diff (default `HEAD~1`)
`-force`	every scoped step is in scope, whatever the diff
`-only a,b`	only steps scoped to `a` or `b`; wins over `-force`
`-no-pull`	skip the `git pull --ff-only` before the diff
`-no-cache`	bypass the content cache (run everything, write nothing)
`-no-remote-cache`	ignore `SKLP_REMOTE_CACHE_URL` for this run
`-jobs N`	max parallel steps (default: the CPU budget)
`-fail-fast`	cancel in-flight steps on the first failure
`-resume`	skip steps that passed in the previous run
`-quiet`, `-tail-lines N`, `-no-hotkeys`	live output controls
`-env-file <path>`	env file for the Space run (default `<repo>/.env`)
`-runner-image <ref>`	override the Space image (`$SKLP_CI_RUNNER_IMAGE`)

Impacted-service scoping

The runner diffs HEAD against the base ref and matches every changed file against each scope's globs. A step runs when any changed file matches any glob of any of its scopes.

sklp run ci -resolve-only            # what would run
sklp run ci -base-ref origin/main    # diff against main, not HEAD~1
sklp run ci -only web                # just the web steps
sklp run ci -force -no-cache         # a real full rebuild

No usable diff (git error, missing base) means run everything.
A step with no scope: always runs, and -only never selects it.
An unknown scope in -only or in a step is a hard error.
-force and -only do not bypass the cache; add -no-cache.

CI on a pull request

git push feat/x  →  sklp flow end opens the PR
GitHub webhook   →  core
core             →  CI node (sklp-run-api) runs the selected pipelines

sklp flow end pushes the branch and opens the PR; the PR triggers CI on the node, so CI is not run locally first. Pipelines are selected by their on: branches against the diff. To check a change before it leaves the machine:

sklp flow end --local

When no CI node is registered for the project, the PR reports the node as unreachable. That is expected, not an incident: sklp flow end --local is then the validation that counts.

Follow a remote run with sklp ci status and sklp ci watch:

sklp ci status --branch feat/x   # current status, no tail
sklp ci watch <run-id>           # replay + tail logs until done

Both take --project (defaults to SKLP_PROJECT) and --json.

Running a CI node

sklp run --listen turns a Linux VM into a CI node: it serves the run API (default :6464, TLS + Bearer token) and executes pipelines for the refs it is sent. --install makes it a systemd --user service.

sklp run --listen --advertise 10.0.0.4 --work-root /var/cache/sklp --concurrency 2 --install

The pipeline YAML comes from the branch under test, so a node refuses any pipeline that sets space: false unless it was started with --allow-host-steps. A host pipeline runs as the node user with the run API's environment, its token file and every run's workdir in reach; only enable it on a node that runs trusted refs, never one that builds pull requests from forks.

The CI rootfs and caches

The CI Space's rootfs is built on the fly from .sklp/space.yaml. Inside it:

Mount	Holds
`/workspace`	the repo checkout
`/cache-shared`	`GOMODCACHE` / `GOPATH`, content-addressed, shared
`/cache`	`GOCACHE`, `HOME`, per-task state

The tool cache grows with every checkout. sklp cache gc evicts least-recently-used buckets, skipping those held by a live Space:

sklp cache status
sklp cache gc --dry-run
sklp cache gc --max-size 12G --runner-max-size 8G

Never delete anything under ~/.cache/sklp by hand; use sklp clean.

To build or inspect the runner image from the recipe, sklp runner build:

sklp runner build --print        # show the generated Dockerfile
sklp runner build --tag my-runner

Repository pipelines

Pipeline	What it does
`sklp run ci-skalpai`	lint + test of core, web, www (every branch but main)
`sklp run generate`	regenerate the OpenAPI spec and the TypeScript client
`sklp run regen-schemas`	regenerate `pipeline.schema.json`; fails on drift
`sklp run publish`	build and push impacted images (see Publish)

regen-schemas declares on: { never: true }: no push selects it, it runs by name only.

Git hooks

Local gates are git hooks, not a hosted CI. Activate the versioned hook once per clone:

git config core.hooksPath .githooks

.githooks/pre-commit runs sklp run regen-schemas whenever apps/sklp/pipeline/schema*.go, apps/sklp/pipeline/spec/ or apps/sklp/cmd/genschema/ is staged. On drift, stage the regenerated JSON and commit again.