Skip to content

How we ship

What stands between a merge and production at Sailfork, what rolls back on its own, and what we don't have yet.

By Sailfork engineering, 3 minutes to read

A sailboat on open water beside a buoy, with a lighthouse on a headland under a bright sun
On this page

Sailfork is weeks old and every merge to main deploys. That is a lot of change in a short time, so this post describes the machinery that limits the damage, and says plainly where it stops. Everything below is read from the scripts in the repository, not from memory.

Before a pull request#

Checks run on the author’s machine, not in CI. scripts/check.sh finds the crates a branch touched, adds every workspace crate that depends on them, and runs:

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test for those crates
  • for client/, web/ and edge/: format check, lint, type check and a production build
  • the generated API catalog drift check (scripts/ops-drift.sh), when the API surface changed
  • a timing of the CLI’s start-up, when the change reaches the CLI

We removed the pull request CI check on 2026-11-12. Running it took 17 minutes of every server deploy and its 7.5 GiB peak on an 8 GiB runner was the likeliest cause of a run ending without an error. That makes the pre-merge gate a discipline, not an enforced rule, and it’s the weakest part of this list. Bringing a cheaper enforced check back is open work.

What a deploy does#

A merge starts one deploy run. The server goes first, then the site, so the site never runs ahead of the API it calls. Each step is skipped when nothing it is built from changed.

  1. Build the image in CI from the merged commit.
  2. Start the image before releasing it. scripts/check-image.sh runs the built image and fails the run if it doesn’t come up.
  3. Release to Fly. The site uses a blue-green deploy: new machines come up beside the old ones and take traffic only when their health checks pass. The server’s primary has one machine on a volume, which can’t be blue-green, so the release clones it first. The clone serves while the primary restarts and is removed afterward. A single-machine rolling deploy once cost us 16 seconds of failed requests, which is why the clone exists.
  4. Wait for the live commit. The run polls until the site reports the new commit, then asks every machine, one at a time, to confirm it serves it within 90 seconds.
  5. Probe. From before the release until 30 seconds after it, the script requests the home page and the version endpoint every second. A server release also probes the git host. One failed request makes the release a failed deploy, whatever else went right.

Only one deploy runs at a time. A run that has started is never cancelled, because a half-finished blue-green leaves cordoned machines behind (incident 2026-11-13).

Rollback#

When fly deploy fails, or the live app never reports the new commit, or a machine doesn’t serve it, scripts/fly-release.sh redeploys the image the machines ran before and still fails the run. The log says rolled back <app> to <image>, or NOT rolled back when there was no single earlier image to return to, in which case a person does it by hand. If the new image is already serving and only the probe saw failures, we don’t roll back: the new image is the one answering, and the run is recorded as an incident.

A blue-green deploy that hangs past 600 seconds is stopped and finished by the script, which keeps at least one machine serving at every step. That exists because flyctl once hung with the green machines healthy and still cordoned (run 1020).

Each image is tagged with the commit and its build time, so a forced redeploy of the same commit never overwrites the image a rollback returns to.

Migrations#

Database migrations are additive: new columns and tables, and the old ones stay. A rollback of the site or the server therefore never meets a schema it can’t read, and rolling back doesn’t undo a migration.

Staged rollout where we have it#

The sandbox host daemon, drydockd, rolls out in rings: internal organizations first, then a 5% canary, then everyone, with a 24-hour soak between rings. A bad release stops at the first ring.

The server and the site don’t have a canary. They have a probe, health checks and an automatic rollback. A bad release reaches all users until the probe or the health checks catch it.

What we don’t have yet#

  • Enforced CI on pull requests.
  • Published test counts or coverage. We won’t quote numbers we haven’t measured for this post.
  • A canary stage for the server and the site.
  • A third party who has reproduced our failover and git-history checks. The harness results we’ve published are ours.

Incidents#

When the outside checks see a part of Sailfork fail, the status page opens an incident and keeps its updates there. We write up the timeline, cause and what changed for each one.

Changes you depend on#

Every change goes in the changelog. The CLI is at 0.x, so expect commands and flags to change between releases; sailfork upgrade --rollback puts back the previous binary.

Loading Sailfork