> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sequency.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets restore

# Runbook: Restore a secrets/env file from the encrypted backup

**Issue:** [#1249](https://github.com/sequencyhq/sequency-data/issues/1249) — the
prior secrets backup was unrecoverable (operator-laptop rclone routine silently
died \~2 months + the age private key was unavailable). This runbook covers the
server-resident, monitored replacement.

## What the backup is

* **Producer:** `scripts/backup-secrets.sh`, driven by `systemd/secrets-backup.timer`
  (daily 02:30 ET) on the **compute** server.
* **Contents:** a `tar.gz` of every `/opt/sequency/config/*.env`, streamed
  through `age` and encrypted to a **public** recipient. Plaintext never touches
  disk.
* **Location:** `s3://sequencydata/secrets-backup/<UTC-timestamp>.tar.age`
  (DO Spaces, endpoint `nyc3.digitaloceanspaces.com`), objects named
  `YYYYMMDD-HHMMSSZ.tar.age`.
* **Freshness monitor:** `scripts/check-backups.sh` (check 4) fails if the newest
  object is > 25h old or cannot be listed.

> Decryption requires the **age private identity**, which is escrowed OUT of this
> repo and off the servers (key manager / offline). Without it the bundles are
> unrecoverable — that was the root failure in #1249. Confirm the identity is
> escrowed and test-restored before relying on this backup.

***

## Restore procedure

Run from the **compute** server (has the DO creds) unless noted. Substitute the
service/env-file you are restoring; `signal-delivery.env` is used as the example.

### 1. Fetch the bundle

```bash theme={null}
# On compute — reuse the DO creds the backup script uses.
set -a; . /opt/sequency/config/corporate-actions-daemon.env; set +a
export RCLONE_CONFIG_SECRETS_TYPE=s3
export RCLONE_CONFIG_SECRETS_PROVIDER=DigitalOcean
export RCLONE_CONFIG_SECRETS_ACCESS_KEY_ID="$DO_SPACES_KEY"
export RCLONE_CONFIG_SECRETS_SECRET_ACCESS_KEY="$DO_SPACES_SECRET"
export RCLONE_CONFIG_SECRETS_ENDPOINT="${DO_SPACES_ENDPOINT#https://}"

# List available bundles, newest last.
rclone lsf secrets:sequencydata/secrets-backup/ | sort

# Download the one you want to a private scratch dir.
WORK="$(mktemp -d)"; chmod 700 "$WORK"
rclone copyto "secrets:sequencydata/secrets-backup/20260706-073000Z.tar.age" \
  "$WORK/bundle.tar.age"
```

### 2. Decrypt with the escrowed identity

Retrieve the age **private identity** from escrow (key manager / offline).
**Do not** write it to a shared or persistent location — use the scratch dir and
delete it after.

```bash theme={null}
# Place the identity at $WORK/age-identity.txt (mode 600), from escrow.
install -m 600 /dev/null "$WORK/age-identity.txt"
# ... paste/copy the escrowed identity into it ...

age -d -i "$WORK/age-identity.txt" -o "$WORK/bundle.tar.gz" "$WORK/bundle.tar.age"
```

### 3. Extract the target env file

```bash theme={null}
# Inspect what's in the bundle (all *.env are at the archive root).
tar -tzf "$WORK/bundle.tar.gz"

# Extract just the one you need into the scratch dir.
tar -xzf "$WORK/bundle.tar.gz" -C "$WORK" signal-delivery.env
```

### 4. Install with correct ownership + mode, then restart

`/opt/sequency/config/*.env` are mode `600`. `signal-delivery.env` is owned
`root:root`; the app-managed ones (`corporate-actions-daemon.env`,
`agent-framework.env`) are `sequency:sequency`. Match whatever the service
expects — `install` sets owner/group/mode atomically:

```bash theme={null}
sudo install -o root -g root -m 600 \
  "$WORK/signal-delivery.env" /opt/sequency/config/signal-delivery.env

sudo systemctl restart signal-delivery
curl -sf http://localhost:8105/health && echo " signal-delivery OK"
```

### 5. Clean up the scratch dir (destroys the plaintext + identity copy)

```bash theme={null}
rm -rf "$WORK"
```

***

## Deriving `SUPABASE_JWKS_URL` if it is missing

If a restored (or hand-rebuilt) env file has `SUPABASE_URL` but not
`SUPABASE_JWKS_URL`, derive it — the API (`api/app/core/auth.py`) and
signal-delivery both use Supabase's GoTrue JWKS endpoint:

```
SUPABASE_JWKS_URL="${SUPABASE_URL%/}/auth/v1/.well-known/jwks.json"
```

Example: `SUPABASE_URL=https://abc123.supabase.co`
→ `SUPABASE_JWKS_URL=https://abc123.supabase.co/auth/v1/.well-known/jwks.json`.

***

## First-time setup / enabling the backup

Server-side units deploy automatically (deploy-worker.yml Stage 5 copies
`systemd/*.{service,timer}` to `/etc/systemd/system/`). To activate:

```bash theme={null}
# On compute, one-time:
sudo apt-get install -y rclone age
sudo systemctl daemon-reload
sudo systemctl enable --now secrets-backup.timer
systemctl list-timers | grep secrets-backup      # confirm scheduled

# Run once manually and confirm success:
sudo systemctl start secrets-backup.service
journalctl -u secrets-backup.service -n 30 --no-pager

# Confirm the freshness monitor sees it (run on compute):
scripts/check-backups.sh          # expect: [OK] secrets-backup-fresh (...)
```

`/opt/sequency/config/secrets-backup.env` must exist with the **public**
recipient:

```
SECRETS_BACKUP_AGE_RECIPIENT=age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# optional overrides:
# SECRETS_BACKUP_BUCKET=sequencydata
# SECRETS_BACKUP_PREFIX=secrets-backup
# SECRETS_BACKUP_RETENTION_DAYS=90
```

***

## Operator-only steps this PR does NOT (and cannot) do

This PR is code/diagnosis only. It intentionally does **not** handle, generate,
or place any secret or key. The following are required, one-time, by the
operator before the backup is trustworthy:

1. **Generate an age keypair:** `age-keygen -o age-identity.txt` — the file's
   `# public key:` line is the recipient; the body is the private identity.
2. **Escrow the private identity** in a key manager (or offline/paper), OUTSIDE
   this repo and OFF both prod servers. Losing it = unrecoverable bundles again.
3. **Set the public recipient on the server:** create
   `/opt/sequency/config/secrets-backup.env` with
   `SECRETS_BACKUP_AGE_RECIPIENT=age1...` (mode 600).
4. **Confirm DO Spaces creds** are present on compute
   (`DO_SPACES_KEY`/`DO_SPACES_SECRET` in `corporate-actions-daemon.env`).
5. **Install tooling on compute:** `apt-get install -y rclone age`.
6. **Enable the timer:** `systemctl enable --now secrets-backup.timer`.
7. **Test end-to-end** BEFORE trusting it: run `secrets-backup.service` once,
   then perform a full decrypt+extract with the escrowed identity (steps 1-3
   above) and diff against a live env file. Only then is the backup real.