Upgrade Softwarden

Status: written November 3, 2026 (M4 E26-3, US-M4-03, D67, D68). Not yet run on a pilot VM; nothing is deployed. Record each upgrade in the log at the end. Rules: a fresh backup before every upgrade; pinned image tags; forward-only migrations. Rollback means restoring the pre-upgrade backup, never running migrations backwards.

Step 2 takes a worker backup set with backup_now. Older releases use pg_dump directly.

Upgrading to Phase 9 (worker backups). The first start generates backup_key in the secrets volume and adds the softwarden-backups volume; the System page then warns until an Admin confirms the key is copied off the host (backup runbook). That volume is on the database’s disk: each set needs about the database’s compressed size plus the attachment files, and a run refuses if it would leave less than the larger of 2 GB and 10 % of the disk free (SOFTWARDEN_BACKUP_MIN_FREE_MB). Check df -h before the first night, or bind-mount the backups to another disk. From Phase 9 on, step 2 can be docker compose exec worker python manage.py backup_now --inline.

Before you start (5 min)

• Read the release notes for anything that needs a setting changed.
• Pick a quiet time. Sign-in is unavailable for a minute or two while the app restarts and migrates.
• Have the off-host backup location and the .env copy at hand.

Steps

1. Write the manifest (1 min)

What must survive the upgrade: every record review, every completed renewal cycle and the audit log so far (§13.5), each as a count and a checksum; and every record, contract and membership by its id, with its organization (and a record’s kind, a membership’s user; not its role, which may change). Those are compared row by row: one added after the manifest is fine (the app keeps running until step 4), one missing or now in another organization fails the check. So a record, contract or membership someone deletes between this step and the upgrade also shows as missing: write the manifest right before the backup in step 2, and tell people not to delete anything until the upgrade is done. The manifest also records the latest applied migration of each Softwarden app ("migrations": {"api": "0001_initial", "catalog": "0019_split_apps", "core": "0001_initial", ...}).

cd ~/softwarden
docker compose exec app python manage.py upgrade_check --write-manifest /tmp/before.json
docker compose cp app:/tmp/before.json ./upgrade-$(date +%F)-before.json

2. Back up (5 min)

From Phase 9 on, take an encrypted worker backup set (the backup runbook restores it):

docker compose exec worker python manage.py backup_now --inline
cp .env softwarden-$(date +%F).env

On an older release, without backup_now, dump the database directly:

docker compose exec -T db pg_dump -U softwarden -Fc softwarden > softwarden-$(date +%F)-pre-upgrade.dump

Encrypt both files and copy them off the host (D67) before going on. Without them there is no way back.

3. Pin and fetch the new release (2 min)

Change one line in .env, then fetch that image before step 3a, so the pre-check runs the new code.

With the release images published to GHCR:

# .env
SOFTWARDEN_IMAGE=ghcr.io/softwarden/softwarden:<new tag>

docker compose pull --ignore-buildable

From source, check out the release and build it under a local tag:

git fetch --tags && git checkout <new tag>
# .env: SOFTWARDEN_IMAGE=softwarden:<new tag>
docker compose build

Never use latest: the tag in .env is the record of what is running.

Pinned images. The other images are pinned in the release itself, so the same release always runs the same database and proxy: compose.yaml names postgres:17.11-trixie and caddy:2.11.9, and both Softwarden images build on python:3.13.18-slim-bookworm (compose.dev.yaml’s Mailpit is v1.21.8). Each is named by its tag followed by the digest the tag resolved to (postgres:17.11-trixie@sha256:…), so even a re-pushed tag can’t change what runs; Docker uses the digest and the tag is there for people to read. A test refuses a compose image on a floating tag such as postgres:17 or without a digest, and ops/ci/check-pins.sh does the same for the Dockerfiles and the workflows’ actions. Dependabot proposes new tags and digests as pull requests. Two copies it doesn’t update: the services: postgres image in .github/workflows/ci.yml (twice) must equal compose.yaml’s db image, and CHECK_IMAGE in scripts/setup.sh and $CheckImage in scripts/setup.ps1 must equal the Dockerfile’s FROM python: reference. Tests fail until they match, so copy the new reference into them on the Dependabot branch. The PostgreSQL image stays on Debian trixie, the base postgres:17 resolved to when earlier installs created their databases: changing the C library under an existing data directory can change text collation and invalidate indexes. A new release may move these pins forward. A PostgreSQL minor (17.11 to 17.12) needs no action. A move to another Debian base or major version is called out in the release notes with its own steps.

3a. Pre-check the data (1 min)

Run the new image’s pre-check before starting it. run replaces the image’s command, so nothing migrates:

docker compose run --rm --no-deps app python manage.py upgrade_check --precheck

It ends with “Ready to migrate.” or lists rows that would stop a pending migration, and exits 1. If it lists rows, fix them as in Cross-organization references before step 4. (Images from before Phase 1 don’t have --precheck; skip this step when upgrading to one of those.)

4. Start it (3 min)

docker compose stop app worker           # nothing serves the old code while the schema changes
docker compose up -d --wait              # migrate runs and exits 0, then app, worker and proxy start
docker compose ps -a                     # migrate "Exited (0)", app "healthy", worker "healthy"
curl -fsS https://<PUBLIC_HOSTNAME>/readyz   # {"status": "ok", ... "migrations": "ok"}; 503 for up to a minute until the worker ticks (add -k with Caddy's local CA)

The one-shot migrate service runs the new migrations; the app starts only after it exits 0 (docker compose logs migrate if it doesn’t). The old app and worker are stopped first because up -d would otherwise keep them running on the old code while the schema changes under them. Migrations only go forward; a test refuses any change to a migration already shipped.

Upgrading to the first Phase 1 image adds the init (generated secrets) and proxy (Caddy, HTTPS on 80/443) services, and the app moves from http://127.0.0.1:8080 to https://<PUBLIC_HOSTNAME>. .env must hold the existing POSTGRES_PASSWORD (init stops otherwise) and keep SOFTWARDEN_SECRET_KEY and SOFTWARDEN_SECRETS_KEY; afterwards run docker compose exec app python manage.py rotate_credential_keys once and back up the new credential key (bootstrap_admin prints it once). First-run setup stays closed (core migration 0006).

5. Check it (2 min)

docker compose cp ./upgrade-$(date +%F)-before.json app:/tmp/before.json
docker compose exec app python manage.py upgrade_check --manifest /tmp/before.json

Every line should start with ✓ (OK on a console that can’t show the check mark, such as a Windows console on cp1252; a failure is ✗ or FAIL), ending with “All upgrade checks passed.” It checks, as the first Admin, through the real web stack: the sign-in page, Needs attention, a record with its reviews, a renewal with completion evidence, the worker, and that the reviews, renewal completions and audit events in the manifest are unchanged and every record, contract and membership in it is still there in its organization (a manifest written before 1.0 has no such rows; those three lines then say “not in this manifest”). “Database migrated” lists the latest migration of every app. “Migrations at or past the manifest” confirms no app is behind what the manifest recorded; an app behind it means the database is older than the manifest (the wrong backup was restored) and the check fails. A manifest written before the catalog split holds one "migration" value, the latest catalog migration, and is still accepted: it is compared as the catalog app’s.

Then run docker compose exec app python manage.py doctor. It should show no FAIL, and the worker, schedules and backup should be OK. Sign in yourself and open Needs attention.

Roll back

If step 4 or 5 fails and the fix isn’t obvious, put the previous SOFTWARDEN_IMAGE back in .env and restore the step 2 backup. A worker set is restored with ops/restore.sh --yes <set>.dump.enc (backup runbook, “Restore”), then docker compose up -d. A pg_dump file from an older release is restored like this:

docker compose down                                    # keeps the volume
# put the previous SOFTWARDEN_IMAGE back in .env
docker compose up -d db
docker compose exec -T db dropdb -U softwarden softwarden
docker compose exec -T db createdb -U softwarden softwarden
docker compose exec -T db pg_restore -U softwarden -d softwarden < softwarden-<date>-pre-upgrade.dump
docker compose up -d
docker compose exec app python manage.py upgrade_check --manifest /tmp/before.json

Work done between the backup and the rollback is lost; tell the IT staff what to re-enter.

Long-running migrations

Phase 2 migration catalog 0031_search_index (global search) adds a stored generated search_vector column to 11 tables (catalog_resource, catalog_application, catalog_component, catalog_aiagent, catalog_integration, catalog_vendor, catalog_tag, catalog_aiprovider, catalog_aimodel, catalog_person, catalog_attachment). Adding a stored generated column rewrites each table under an ACCESS EXCLUSIVE lock, and the migration then builds its GIN and trigram indexes without CONCURRENTLY. While it runs, every read and write of those tables waits, so the app is effectively unavailable. At the pilot’s size, and at seed_scale’s 1,000 or 5,000 resources, it takes seconds. A large install should apply it in a maintenance window and let step 4 finish before announcing the upgrade done. It runs in one transaction, so an interrupted run changes nothing.

Cross-organization references

Phase 1 migrations catalog 0021_composite_keys and imports 0002_importrow_tenancy add composite foreign keys (D3): each reference must point at a row of the same organization. The application has always checked this, so a normal install has no such rows. A database edited by hand or by a script may have some, and then the migration stops with an IntegrityError naming the constraint. Each migration runs in a single transaction, so nothing has changed, but the app won’t start until the rows are fixed. upgrade_check --precheck (step 3a) runs these same queries while the migrations are pending; plain upgrade_check also runs them as long as either migration is unapplied.

Run a query with docker compose exec -T db psql -U softwarden softwarden -c "<query>". Each returns the ids of the offending rows:

The organization columns that catalog 0020 and imports 0002 add are copied from the parent row (the Resource, the ImportRun), and the run and resource keys of those rows are built from the same value, so those constraints can’t fail.

To fix a row, point it at the right organization’s row, or clear the reference where the field allows it (owners, team, department, vendor, created_by_import and the three ImportRow resources are nullable). A RecordReview pointing at another organization’s record or Person was written wrongly. Keep the backup from step 2, and record what you changed in the log below. Then rerun step 3a.

Upgrade rehearsal from the previous release (every CI run)

CI’s migrate-from-previous job proves on every push that a backup of the previous release restores and migrates under the new image. The script is ops/ci/restore-previous.sh, and it runs the same way on a developer machine (Linux, or Git Bash with Docker Desktop on Windows):

bash ops/ci/restore-previous.sh              # the previous release: newest v* tag before HEAD, else 9ca7f2e
bash ops/ci/restore-previous.sh --ref v1.0.0 # a particular release

1. It checks the previous release out with git worktree into a temporary folder and builds its image, and builds the current checkout’s image.
2. It starts the previous release as its own Compose project (sw-restore-prev-<id>, its own volumes; the default project is never touched), runs seed_demo, and backs it up with that release’s ops/backup.sh (database, manifest, attachment files, checksums). That is the script the operator of that release has. Under Git Bash it runs with the current ops/common.sh, because earlier releases refuse every key file there: NTFS keeps no Unix file modes.
3. It starts the current image as a second project (sw-restore-cur-<id>), with the previous installation’s credential keys, as a real restore would have, and runs ops/restore.sh --rehearse --migrate against the set, then a real ops/restore.sh --yes. restore.sh runs the migrate service on the restored database before it starts the app again, then checks the manifest and the attachment files.
4. It runs migrate, migrate --check, upgrade_check --manifest (the manifest the previous release wrote) and doctor. Any failure stops the run with a non-zero exit.
5. It removes both projects with their volumes, matched by project name, then the worktree, the temporary folder and its two image tags. It prints a summary of each step.

A previous release that still publishes its database (before PRE_RELEASE_REVIEW F-16) publishes it on 127.0.0.1:55434 (RESTORE_PREV_DB_PORT); this commit’s stack publishes none (RESTORE_CUR_DB_PORT, 55435, is only set for an older checkout). Caddy isn’t started. --keep-images keeps the two image tags for a second run.

Cutting a release

The owner’s steps; nothing is published until the tag is pushed. X.Y.Z below is the version (1.0.0 for the first release). For 1.0.0, step 1 is already done in the repository (the ## [1.0.0] - 2026-11-08 section and version 1.0.0 everywhere); start at step 2, and if you tag on a later day, change the date in that heading first. The full first-release list, from creating the repository to the fresh-VM run, is RELEASE_CHECKLIST_1.0.0.md.

1. Prepare the release commit on main:

   • in CHANGELOG.md, rename ## [Unreleased] to ## [X.Y.Z] - <date> and add a new empty ## [Unreleased] above it;
   • set version to X.Y.Z in apps/api/pyproject.toml (/healthz/ops and the OpenAPI schema read it from there), mcp/pyproject.toml and apps/web/package.json, __version__ in mcp/softwarden_mcp/__init__.py, and appVersion in deploy/helm/softwarden/Chart.yaml (bump the chart’s own version too when the chart changed);
   • refresh the lock files’ own entries: (cd apps/api && uv lock), (cd mcp && uv lock), (cd apps/web && npm install --package-lock-only);
   • run the gate (apps/api/tests/test_upgrade_ci.py fails when any of these versions differ from each other or from the newest CHANGELOG release, compared as PEP 440 versions so 1.1.0rc1 in uv.lock matches 1.1.0-rc.1; INSTALL.md must pin the newest stable release, so leave it alone for a pre-release), commit and push.

2. Wait for CI to be green on that commit, including migrate-from-previous and audit: gh run list --branch main --limit 1, or the Actions tab.

3. Tag and push the tag: git tag -a vX.Y.Z -m "Softwarden X.Y.Z" then git push origin vX.Y.Z. Check the notes first with bash ops/ci/changelog-section.sh X.Y.Z; the workflow runs the same script.

4. Watch the release workflow and approve it: gh run watch (or the Actions tab, workflow “Release”). .github/workflows/release.yml runs, in order:

   • Preflight (ops/ci/release-preflight.sh): the tag is a strict vX.Y.Z or vX.Y.Z-pre (no leading zeros); the tagged commit is on main; apps/api/pyproject.toml and Chart.yaml’s appVersion say the same version; the CHANGELOG heading has a real date no later than tomorrow; neither image has this version in GHCR and there is no GitHub Release for it. A re-run for a version already published fails here: publish a new patch version instead.
   • CI: the whole of ci.yml on the tagged commit.
   • Approve: the run waits on the release environment; approve it in the run’s page (Review deployments).
   • Images: each image is built without the Actions cache, pushed by digest only (no tag yet), with an SBOM and a signed provenance attestation.
   • Smoke: both digests start as the Compose stack, and /readyz and doctor must pass.
   • Publish: the digests get their tags, X.Y.Z, X.Y while it is the highest stable release of that line, and latest only for the highest stable release (a pre-release such as v1.1.0-rc.1 gets only its own tag), and the GitHub Release is created with the notes and both digests.

   A failure before Publish leaves only untagged digests in GHCR: nothing an installation pulls has moved. Fix the cause, delete the tag (git push --delete origin vX.Y.Z, then git tag -d vX.Y.Z), and tag again; the untagged digests can be deleted in the package settings.

   A failure in Publish is different: once X.Y.Z is on an image, that version is published and a re-run refuses it. Publish tags X.Y.Z on both images first, then X.Y and latest on both, then creates the Release, so finish the remaining steps by hand from the run’s log and never retag a digest. The digests are in the run’s digest-softwarden and digest-softwarden-mcp artifacts (and in the “Tag the digests” log):

   • a missing version tag: docker buildx imagetools create --tag ghcr.io/softwarden/<image>:X.Y.Z ghcr.io/softwarden/<image>@<digest>;
   • a missing moving tag, only those the log’s Tags: line lists: the same command with --tag ghcr.io/softwarden/<image>:X.Y and/or :latest;
   • a missing Release: gh release create vX.Y.Z --verify-tag --title "Softwarden X.Y.Z" --notes-file notes.md, with bash ops/ci/changelog-section.sh X.Y.Z > notes.md plus an “Images” list naming both tags and their digests, as the workflow writes it (add --latest=false if latest wasn’t in the Tags: line, --prerelease for a pre-release).

   Check each tag with docker buildx imagetools inspect ghcr.io/softwarden/<image>:X.Y.Z (its digest must be the attested one) and gh attestation verify. GHCR names are the repository owner’s, lowercased. A pre-release tag needs a CHANGELOG section of its own, headed with its full version (## [1.1.0-rc.1] - <date>); the workflow refuses a tag without one, before anything is published.

5. The first time only: in the GitHub package settings (your profile or organization › Packages › softwarden and softwarden-mcp › Package settings), link both packages to the repository and make them public (or grant the readers access). Check with docker pull ghcr.io/softwarden/softwarden:X.Y.Z from a machine that isn’t signed in to ghcr.io, and gh attestation verify oci://ghcr.io/softwarden/softwarden:X.Y.Z --owner Softwarden.

6. Pin the new tag in your own .env and Helm values (docs/INSTALL.md, and “Upgrading” above).

From the first push to main after the tag, migrate-from-previous rehearses the upgrade from v1.0.0 instead of 9ca7f2e. (CI runs on pushes to main and on pull requests, and the release workflow runs it again on the tagged commit; on the first release both rehearse from 9ca7f2e.)

Restore drill (quarterly, and before real data)

D67 asks for a restore to a fresh Compose stack before real data is loaded, then every quarter, on the version currently running:

1. On a spare VM (or a second project directory with its own volume), follow the install guide up to step 4 with the same SOFTWARDEN_IMAGE and a copy of .env.
2. Restore the latest nightly backup as in Roll back.
3. Run upgrade_check --manifest with the manifest written just before that backup.
4. Record the date, the backup used, the time taken and the result below. A pass means every check is ✓.

Log