Runbook: Restore a secrets/env file from the encrypted backup

Issue: #1249 — the prior secrets backup was unrecoverable (operator-laptop rclone routine silently died ~2 months + the age private key was unavailable). This runbook covers the server-resident, monitored replacement.

What the backup is

Producer: scripts/backup-secrets.sh, driven by systemd/secrets-backup.timer (daily 02:30 ET) on the compute server.
Contents: a tar.gz of every /opt/sequency/config/*.env, streamed through age and encrypted to a public recipient. Plaintext never touches disk.
Location: s3://sequencydata/secrets-backup/<UTC-timestamp>.tar.age (DO Spaces, endpoint nyc3.digitaloceanspaces.com), objects named YYYYMMDD-HHMMSSZ.tar.age.
Freshness monitor: scripts/check-backups.sh (check 4) fails if the newest object is > 25h old or cannot be listed.

Decryption requires the age private identity, which is escrowed OUT of this repo and off the servers (key manager / offline). Without it the bundles are unrecoverable — that was the root failure in #1249. Confirm the identity is escrowed and test-restored before relying on this backup.

Restore procedure

Run from the compute server (has the DO creds) unless noted. Substitute the service/env-file you are restoring; signal-delivery.env is used as the example.

1. Fetch the bundle

# On compute — reuse the DO creds the backup script uses.
set -a; . /opt/sequency/config/corporate-actions-daemon.env; set +a
export RCLONE_CONFIG_SECRETS_TYPE=s3
export RCLONE_CONFIG_SECRETS_PROVIDER=DigitalOcean
export RCLONE_CONFIG_SECRETS_ACCESS_KEY_ID="$DO_SPACES_KEY"
export RCLONE_CONFIG_SECRETS_SECRET_ACCESS_KEY="$DO_SPACES_SECRET"
export RCLONE_CONFIG_SECRETS_ENDPOINT="${DO_SPACES_ENDPOINT#https://}"

# List available bundles, newest last.
rclone lsf secrets:sequencydata/secrets-backup/ | sort

# Download the one you want to a private scratch dir.
WORK="$(mktemp -d)"; chmod 700 "$WORK"
rclone copyto "secrets:sequencydata/secrets-backup/20260706-073000Z.tar.age" \
  "$WORK/bundle.tar.age"

2. Decrypt with the escrowed identity

Retrieve the age private identity from escrow (key manager / offline). Do not write it to a shared or persistent location — use the scratch dir and delete it after.

# Place the identity at $WORK/age-identity.txt (mode 600), from escrow.
install -m 600 /dev/null "$WORK/age-identity.txt"
# ... paste/copy the escrowed identity into it ...

age -d -i "$WORK/age-identity.txt" -o "$WORK/bundle.tar.gz" "$WORK/bundle.tar.age"

3. Extract the target env file

# Inspect what's in the bundle (all *.env are at the archive root).
tar -tzf "$WORK/bundle.tar.gz"

# Extract just the one you need into the scratch dir.
tar -xzf "$WORK/bundle.tar.gz" -C "$WORK" signal-delivery.env

4. Install with correct ownership + mode, then restart

/opt/sequency/config/*.env are mode 600. signal-delivery.env is owned root:root; the app-managed ones (corporate-actions-daemon.env, agent-framework.env) are sequency:sequency. Match whatever the service expects — install sets owner/group/mode atomically:

sudo install -o root -g root -m 600 \
  "$WORK/signal-delivery.env" /opt/sequency/config/signal-delivery.env

sudo systemctl restart signal-delivery
curl -sf http://localhost:8105/health && echo " signal-delivery OK"

5. Clean up the scratch dir (destroys the plaintext + identity copy)

rm -rf "$WORK"

Deriving `SUPABASE_JWKS_URL` if it is missing

If a restored (or hand-rebuilt) env file has SUPABASE_URL but not SUPABASE_JWKS_URL, derive it — the API (api/app/core/auth.py) and signal-delivery both use Supabase’s GoTrue JWKS endpoint:

SUPABASE_JWKS_URL="${SUPABASE_URL%/}/auth/v1/.well-known/jwks.json"

Example: SUPABASE_URL=https://abc123.supabase.co → SUPABASE_JWKS_URL=https://abc123.supabase.co/auth/v1/.well-known/jwks.json.

First-time setup / enabling the backup

Server-side units deploy automatically (deploy-worker.yml Stage 5 copies systemd/*.{service,timer} to /etc/systemd/system/). To activate:

# On compute, one-time:
sudo apt-get install -y rclone age
sudo systemctl daemon-reload
sudo systemctl enable --now secrets-backup.timer
systemctl list-timers | grep secrets-backup      # confirm scheduled

# Run once manually and confirm success:
sudo systemctl start secrets-backup.service
journalctl -u secrets-backup.service -n 30 --no-pager

# Confirm the freshness monitor sees it (run on compute):
scripts/check-backups.sh          # expect: [OK] secrets-backup-fresh (...)

/opt/sequency/config/secrets-backup.env must exist with the public recipient:

SECRETS_BACKUP_AGE_RECIPIENT=age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# optional overrides:
# SECRETS_BACKUP_BUCKET=sequencydata
# SECRETS_BACKUP_PREFIX=secrets-backup
# SECRETS_BACKUP_RETENTION_DAYS=90

Operator-only steps this PR does NOT (and cannot) do

This PR is code/diagnosis only. It intentionally does not handle, generate, or place any secret or key. The following are required, one-time, by the operator before the backup is trustworthy:

Generate an age keypair: age-keygen -o age-identity.txt — the file’s # public key: line is the recipient; the body is the private identity.
Escrow the private identity in a key manager (or offline/paper), OUTSIDE this repo and OFF both prod servers. Losing it = unrecoverable bundles again.
Set the public recipient on the server: create /opt/sequency/config/secrets-backup.env with SECRETS_BACKUP_AGE_RECIPIENT=age1... (mode 600).
Confirm DO Spaces creds are present on compute (DO_SPACES_KEY/DO_SPACES_SECRET in corporate-actions-daemon.env).
Install tooling on compute: apt-get install -y rclone age.
Enable the timer: systemctl enable --now secrets-backup.timer.
Test end-to-end BEFORE trusting it: run secrets-backup.service once, then perform a full decrypt+extract with the escrowed identity (steps 1-3 above) and diff against a live env file. Only then is the backup real.

Start

Operating

API

API reference

Runbooks

Secrets restore

Runbook: Restore a secrets/env file from the encrypted backup

What the backup is

Restore procedure

1. Fetch the bundle

2. Decrypt with the escrowed identity

3. Extract the target env file

4. Install with correct ownership + mode, then restart

5. Clean up the scratch dir (destroys the plaintext + identity copy)

Deriving `SUPABASE_JWKS_URL` if it is missing

First-time setup / enabling the backup

Operator-only steps this PR does NOT (and cannot) do

​Runbook: Restore a secrets/env file from the encrypted backup

​What the backup is

​Restore procedure

​1. Fetch the bundle

​2. Decrypt with the escrowed identity

​3. Extract the target env file

​4. Install with correct ownership + mode, then restart

​5. Clean up the scratch dir (destroys the plaintext + identity copy)

​Deriving SUPABASE_JWKS_URL if it is missing

​First-time setup / enabling the backup

​Operator-only steps this PR does NOT (and cannot) do

Runbook: Restore a secrets/env file from the encrypted backup

What the backup is

Restore procedure

1. Fetch the bundle

2. Decrypt with the escrowed identity

3. Extract the target env file

4. Install with correct ownership + mode, then restart

5. Clean up the scratch dir (destroys the plaintext + identity copy)

Deriving `SUPABASE_JWKS_URL` if it is missing

First-time setup / enabling the backup

Operator-only steps this PR does NOT (and cannot) do