> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sequency.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Production alerting

# Production Alerting

Origin: data#1925, data#1926, data#1927 (2026-08-17/18). All three incidents
were **silent**. Nothing paged; each was found by hand.

This runbook covers what alerting exists, how to activate delivery, what each
rule maps onto, and — explicitly — what is still not covered.

***

## 1. The stack, as it actually is

| Component         | State                                                                                                                          |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Prometheus server | **Running** on the compute host at `:9090`. Its config is **NOT in this repo** — it is managed on the host.                    |
| Prometheus rules  | In this repo: `monitoring/prometheus-rules/*.yml`. Loaded via the host's `rule_files:`.                                        |
| **Alertmanager**  | **NOT INSTALLED.** No receiver, route, or config exists on any host or in any repo.                                            |
| Grafana           | Running at `:3000`; dashboards in `monitoring/grafana-dashboards/`. Not actively used.                                         |
| node\_exporter    | Running at `:9100`.                                                                                                            |
| redis\_exporter   | Expected on the data server at `:9121`. **No evidence in this repo** — `alert-service-health.sh` reports if it is unreachable. |
| Alert delivery    | Slack incoming webhook (`SLACK_WEBHOOK_URL`) + GitHub Actions webhook (`PROBE_ALERT_WEBHOOK`).                                 |

> **The single most important fact:** because no Alertmanager is installed,
> writing a Prometheus rule delivers nothing on its own. Before this work, all
> 26 existing rules could fire and reach no human. `alert-dispatch.sh` is the
> bridge that makes them real. It is a **stopgap for Alertmanager**, not a
> replacement — no grouping, inhibition, silencing, or escalation.

### How an alert reaches a human today

```
Prometheus rule fires
   -> Prometheus /api/v1/alerts
      -> alert-dispatch.timer (every 2 min) -> alert-dispatch.sh
         -> alert-notify.sh -> $SLACK_WEBHOOK_URL -> Slack

systemd unit fails
   -> OnFailure=sequency-alert-notify@<unit>.service -> alert-notify.sh -> Slack

Service failed / Redis loading / Prometheus down  (Prometheus-independent)
   -> alert-service-health.timer (every 5 min) -> alert-service-health.sh
      -> alert-notify.sh -> Slack

Deploy fails on main
   -> deploy-failure-alert.yml (workflow_run) -> $PROBE_ALERT_WEBHOOK
```

***

## 2. Operator activation steps

Nothing below contains a secret. Both webhook values are bearer credentials and
must never be committed.

### Step 1 — Slack webhook on the compute host (REQUIRED)

Without this, every host-side alert is a journal line and nothing more.

```bash theme={null}
ssh sequency@compute.sequencyhq.com
grep -q '^SLACK_WEBHOOK_URL=' /opt/sequency/.env && echo "already set" || echo "NOT SET"

# If not set, append it (file is mode 600, owned by sequency):
sudo sh -c 'printf "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ\n" >> /opt/sequency/.env'
```

Verify delivery end-to-end:

```bash theme={null}
set -a; . /opt/sequency/.env; set +a
/opt/sequency/scripts/alert-notify.sh "Sequency alerting test — please ignore."
```

A message in Slack means the path works. `SLACK_WEBHOOK_URL is NOT set` or
`Slack webhook POST FAILED` in the output means it does not.

### Step 2 — GitHub webhook for deploy failures (REQUIRED)

```
Repo -> Settings -> Secrets and variables -> Actions -> New repository secret
Name:  PROBE_ALERT_WEBHOOK
Value: <Slack incoming webhook URL>
```

This secret is already used by `synthetic-auth-probe.yml`,
`clickhouse-storage-canary.yml` and `pattern-corpus-canary.yml`. If it is
already set, deploy-failure alerting works with no further action.

Prove it without breaking anything:

```bash theme={null}
gh workflow run deploy-failure-alert.yml -R sequencyhq/sequency-data -f drill=true
```

This posts a clearly-labelled `:test_tube: DRILL` message.

### Step 3 — Reload Prometheus so the new rules load (REQUIRED)

New rules in `monitoring/prometheus-rules/alerts.yml` are inert until
Prometheus re-reads them. **The rule files are not copied to the host by CI**
— confirm where the host reads them from before reloading.

```bash theme={null}
ssh sequency@compute.sequencyhq.com
grep -A5 rule_files /etc/prometheus/prometheus.yml     # confirm the configured path

# Sync the repo's rules to that path if it is not already the checkout, then:
promtool check rules /etc/prometheus/rules/alerts.yml   # MUST pass before reload
sudo systemctl reload prometheus                        # or: curl -XPOST localhost:9090/-/reload
```

Confirm the rules are loaded:

```bash theme={null}
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name' | sort | grep -E 'RedisLoading|BrokerOptions|CriticalServiceTargetMissing'
```

### Step 4 — Verify the timers are armed (after the next deploy)

```bash theme={null}
ssh sequency@compute.sequencyhq.com
systemctl list-timers 'alert-*' --no-pager
sudo systemctl status alert-dispatch.service alert-service-health.service --no-pager
```

Both timers are enabled and started automatically by the deploy. If they are
missing, `sudo systemctl enable --now alert-dispatch.timer alert-service-health.timer`.

### Step 5 (recommended, not done here) — install Alertmanager

`alert-dispatch.sh` has no silencing. During a long incident it will notify
once per alert and once on resolution, but there is no way to mute a known
issue short of stopping the timer. Installing Alertmanager and pointing
Prometheus at it is the durable fix; `alert-dispatch.timer` should be stopped
at that point to avoid duplicate notifications.

***

## 3. Per-rule signal mapping

Every rule below maps to a signal that is exported **today**. `scripts/check-alert-metrics.py`
(wired into `data-ci.yml`) fails CI if any alert references a metric nothing
emits — a dead alert reads as coverage it does not provide (#1673).

### Added by this change

| Alert                               | Signal                                                 | Emitted by                                    | `for:` | Severity |
| ----------------------------------- | ------------------------------------------------------ | --------------------------------------------- | ------ | -------- |
| `RedisLoadingDataset`               | `redis_loading_dataset == 1`                           | redis\_exporter                               | 10m    | critical |
| `RedisRestartLoop`                  | `redis_uptime_in_seconds < 900`                        | redis\_exporter                               | 15m    | critical |
| `CriticalServiceTargetMissing`      | `absent(up{job=...})` ×4                               | Prometheus scrape                             | 10m    | critical |
| `BrokerOptionsChainAuthFailures`    | `rate(te_options_chain_auth_failures_total[5m]) > 0.1` | `services/trade-executor/metrics.go`          | 5m     | critical |
| `BrokerOptionsChainAuthBreakerOpen` | `te_options_chain_auth_breaker_open == 1`              | `services/trade-executor/metrics.go`          | 5m     | critical |
| `ContractSelectionDisabled`         | `te_contract_selection_disabled == 1`                  | `services/trade-executor/metrics.go`          | 10m    | critical |
| `PatternDetectorCycleErrors`        | `pd_cycle_errors > 0`                                  | `services/pattern-detector/scheduler.go:1122` | 30m    | warning  |

Threshold note for the broker rules: data#1926 logged 118,594 401s in 32h
(\~1.03/s). The 0.1/s floor is two orders of magnitude below the observed rate
and far above the zero a healthy credential produces.

### Non-Prometheus checks

| Condition                               | Mechanism                                                     | Cadence    |
| --------------------------------------- | ------------------------------------------------------------- | ---------- |
| Any of the 6 services not active/failed | `alert-service-health.sh` → `systemctl is-active`             | 5m         |
| Redis TCP unreachable                   | `alert-service-health.sh` → `/dev/tcp` probe (no credential)  | 5m         |
| Redis loading                           | `alert-service-health.sh` → scrapes `redis_exporter` directly | 5m         |
| redis\_exporter unreachable             | `alert-service-health.sh`                                     | 5m         |
| Prometheus down                         | `alert-service-health.sh` → `/-/healthy`                      | 5m         |
| Watchdog cannot recover a service       | `service-watchdog.service` `OnFailure=`                       | 2m         |
| Alert dispatcher itself broken          | `alert-dispatch.service` `OnFailure=`                         | 2m         |
| Deploy failed on main                   | `deploy-failure-alert.yml`                                    | per deploy |

***

## 4. Recovery: services stuck `failed` (data#1925)

The canonical case. A data-server power loss left Redis loading an 18.4GB RDB
for \~25 minutes; every Go service exited on Redis `LOADING`, exhausted
`StartLimitBurst=10`, and stayed `failed` **permanently**.

```bash theme={null}
ssh sequency@compute.sequencyhq.com
systemctl --failed

# 1. Is Redis still loading? If so, WAIT — restarting services now just burns
#    the start limit again, and restarting REDIS restarts the load from zero.
curl -s http://10.0.0.2:9121/metrics | grep '^redis_loading_dataset'

# 2. Once Redis is serving, recover each failed unit. Services do NOT
#    self-heal once StartLimitBurst is exhausted.
for svc in market-signal-engine pattern-detector trade-executor backfill-api sequency-api daily-daemon; do
  sudo systemctl reset-failed "$svc" 2>/dev/null || true
  sudo systemctl start "$svc"
done

# 3. Verify
systemctl --failed
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8081/health
```

**Do not restart Redis to "fix" a slow load.** That is how the 2026-08-17
outage stretched to \~50 minutes: systemd's start timeout was shorter than the
RDB load time, so systemd kept killing the load mid-flight.

***

## 5. GAPS — not covered today

Recorded honestly rather than papered over with rules that cannot fire.

### 5.1 data#1927 (missing-relation query errors) is NOT specifically covered

pattern-detector queried two non-existent tables 27,362 times in 32h. There is
**no metric** for it, and **no log-based alerting exists at all** — no Loki, no
promtail, no Vector, no log shipper of any kind. The journal is the only
record.

`PatternDetectorCycleErrors` (`pd_cycle_errors > 0`) is real coverage for
cycle-level failure but is **not** a missing-relation detector; those queries
are not known to increment it.

To close this properly, one of:

1. **A metric at the query site (preferred, \~1 day).** A
   `pd_query_errors_total{table,code}` counter incremented where ClickHouse/
   FalkorDB errors are handled, then a rule on
   `rate(pd_query_errors_total{code="UNKNOWN_TABLE"}[10m]) > 0`. Cheapest,
   most precise, and passes `check-alert-metrics.py` by construction.
2. **A schema-conformance CI check.** Assert every table named in Go query
   strings exists in `clickhouse/*.sql`. Catches the class before deploy, but
   not at runtime.
3. **Log-based alerting (largest).** Deploy Loki + promtail on compute, add a
   Grafana/Loki ruler. Also unlocks the whole "repeated error in the journal"
   class, including data#1926's 401 log lines.

### 5.2 Other broker call sites have no auth metrics

Only the options-chain path has `te_options_chain_auth_*`. Other Alpaca call
sites (order placement, positions, account, market data) have **no auth-failure
metrics**, so a retired credential on any of those paths is still silent. The
`te_options_chain_auth_*` pattern should be generalised to a shared
`broker_api_errors_total{endpoint,status}` counter. Not invented here — the
metric does not exist yet, and a rule on it would be exactly the dead alert
`check-alert-metrics.py` exists to prevent.

### 5.3 backfill-api and daily-daemon have no Prometheus coverage

`backfill-api` exposes **no `/metrics` endpoint at all**; `daily-daemon`'s
scrape job label is not established anywhere in this repo. Both are covered by
`alert-service-health.sh` (systemd state) but by **no Prometheus rule**. Adding
a `/metrics` handler to backfill-api would close half of this.

### 5.4 The Prometheus config is not in version control

There is no `prometheus.yml` in this or any repo. Scrape targets, job labels
and `rule_files:` are host-managed and unreviewable. A target silently dropped
from the scrape config is invisible to code review —
`CriticalServiceTargetMissing` is a partial mitigation for the four services
whose job labels are known from `slo.yml`, but the durable fix is to bring the
Prometheus config into the repo and deploy it via CI.

### 5.5 No silencing, grouping, or escalation

`alert-dispatch.sh` notifies once per alert and once on resolution. There is no
way to mute a known-ongoing issue except stopping the timer, and no escalation
if Slack is not being read. Alertmanager is the fix (§2 Step 5).

### 5.6 Delivery is single-channel and unmonitored

Everything lands in one Slack webhook. If the webhook is revoked or the channel
is archived, `alert-notify.sh` logs the failure at `user.err` — but nothing
alerts on *that*, and it cannot: the alerting path is the thing that is broken.
A periodic heartbeat to Slack (visible absence rather than an alert) would
close it.

## 6. Host-side exporter state that CI cannot deploy (2026-09-11, data#2589)

Two exporter changes were made by hand on the data server. Neither unit is in
this repo, so nothing re-applies them: a host rebuild loses both, and the loss is
silent. This section exists so the next person looking for them finds them.

### 6.1 postgres-exporter now authenticates over the Unix socket

`pg_up` had been `0` since the 2026-08-17 PSU outage, so `PostgresDown` fired
continuously for over three weeks while PostgreSQL itself served the API
normally. The exporter was running and reachable; its stored DSN password had
gone stale and every scrape logged `password authentication failed for user
"sequency"`.

`/etc/default/postgres-exporter` (root-owned, mode 600) now holds:

```
DATA_SOURCE_NAME=postgresql:///sequency?host=/var/run/postgresql&sslmode=disable
```

The exporter runs as `User=sequency` and `pg_hba.conf` carries `local all all
peer`, so a socket connection needs no credential at all. This removes a stored
secret rather than rotating one. The previous file is kept at
`/etc/default/postgres-exporter.bak-2026-09-11`.

Further improvement, not done: the exporter connects as the application role. A
dedicated role with `pg_monitor` would be least-privilege, at the cost of a role
to create and a password to store — which is what the socket change just
eliminated, so it is a genuine trade rather than a clear win.

### 6.2 node-exporter now reports systemd unit state

Until this change nothing alerted when a backup job failed. The four backup units
on the data host carry no `OnFailure=` handler, no live rule covered failed
systemd units, and `alert-service-health.sh` watches six long-running services on
the *other* host. A silent backup failure on a single-host data tier that already
lost power once this year is the gap with the most to lose.

`/etc/systemd/system/node-exporter.service` (root-owned) now runs:

```
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100 \
  --collector.systemd \
  --collector.systemd.unit-include="(pgbackrest|clickhouse-backup)-(incr|full)\.service|(postgresql|clickhouse-server|redis-server|node-exporter|postgres-exporter|redis-exporter)\.service"
```

The include list is deliberate: the default `.+` would export every unit on the
box. Ten units produce 50 series, one per unit per state. The previous file is at
`/etc/systemd/system/node-exporter.service.bak-2026-09-11`.

`BackupJobFailed` and `DataHostUnitFailed` in `monitoring/prometheus-rules/alerts.yml`
read these series. **Adding a unit to either alert's regex requires adding it to
the exporter's include list too**, or the series will not exist and the rule will
match nothing. `BackupJobFailed` carries an `absent()` guard for exactly that
mistake; `DataHostUnitFailed` does not, because those services have independent
coverage through their own scrape targets.

### 6.3 What is still not covered

A backup that never *runs* is as bad as one that fails, and neither rule catches
it: a oneshot that is never triggered simply stays `inactive`. pgBackRest exposes
no last-successful-backup metric here, and `scripts/pgbackrest-check.sh` is not on
a timer. Backup *recency* therefore remains unmonitored. `pg_stat_archiver` does
now flow, so continuous WAL archiving failure is covered even though backup
recency is not.

### 6.4 The API metrics scrape presents the internal secret

`/metrics` had returned 401 to Prometheus since 2026-06-07, so no API metric
existed and `ServiceDown{job="sequency-api"}` fired for three months (data#2589).

It was tempting to exempt `/metrics` from the auth middleware. **That would have
been an exposure**: `/metrics` is reachable from the internet through nginx, and
only the 401 was protecting it, so an exemption would have published every
internal gauge and endpoint label. Verified at the time:

```
https://api.sequency.sh/metrics     -> 401
https://worker.sequency.sh/metrics  -> 401
```

Prometheus here is 3.3.0, which supports `http_headers` in a scrape config, so the
**scraper** presents the secret instead and the endpoint's auth posture is
unchanged. `/opt/prometheus/prometheus.yml`, `sequency-api` job:

```yaml theme={null}
    http_headers:
      X-Internal-Secret:
        files: [ /opt/prometheus/api-scrape-secret ]
```

The secret lives in `/opt/prometheus/api-scrape-secret` (mode 600, owned by the
Prometheus user) rather than inline, because `prometheus.yml` is mode 644. Neither
file is in this repo — `prometheus.yml` is host-managed and the secret must not be
committed — so a host rebuild loses this and the API target silently goes down
again. `promtool check config` was run before the reload; do the same on any edit,
because an invalid config makes the reload a no-op and leaves stale rules loaded.

The previous config is at `/opt/prometheus/prometheus.yml.bak-2026-09-11`.

### 6.5 Two exit-protection alerts are suspended, for different reasons

Suspended in `alerts.yml` on 2026-09-14 by operator decision (data#2593):

* `PositionsWithoutExitProtection` — paged on a legitimate state. Exit protection
  is not mandatory for discretionary trading (data#2094). Restore when data#2094
  Gap 2 derives `exit_intent` from the strategy that placed the order.
  `ExitProtectionBrokerStateUnknown` was suspended in the first version of that
  change and has been **kept**, for two reasons. CI pins it:
  `test_unknown_broker_state_metric_has_multiprocess_safe_paging_rule` asserts its
  exact expression, `severity: critical` and `for: 2m`, so removing it fails the
  build — a deliberate guard. And its cause is not the discretionary question: it
  means broker-resident protection could not be **read**, which is a real failure.
  Today that is the rejected broker credential, the same one that has 9 of 11
  positions auth-suppressed, so its blocker is the BYOK credential substrate in
  data#2037 rather than data#2094.

`ExitProtectionReconcilerStale` remains active and is the signal that matters
while those two are off: it fires if the sweep stops running or its gauge
disappears.

Note for anyone tempted to downgrade rather than suspend a noisy alert:
`alert-dispatch.sh` reads `severity` only to label the Slack message and does not
filter on it. A warning notifies exactly as loudly as a critical.

### 6.6 clickhouse-sink is now a scrape target

The sink exports `sink_consumer_lag`, `sink_dead_letters_total`,
`sink_retry_queue_size` and `sink_rows_written_total` on `:8110`, and **nothing
scraped it** until 2026-09-14 (data#2613). Added to `/opt/prometheus/prometheus.yml`:

```yaml theme={null}
  - job_name: clickhouse-sink
    static_configs:
      - targets: ["localhost:8110"]
    metrics_path: /metrics
```

This matters more than an ordinary target because data#2456 cut the Redis stream
caps, and for scores, signals, indicators and levels the stream is the only route
to durable storage. The cap is therefore how long the sink may stay broken before
data is lost silently. `clickhouse-sink` was added to
`CriticalServiceTargetMissing` for the same reason: `prometheus.yml` is
host-managed and not in this repo, so a rebuild drops this target, and losing it
would put the deadline back out of sight.

`ServiceDown` already covers the sink now that it is scraped — that rule has no
job filter.

Two alerting choices worth recording so nobody "fixes" them:

* **No rule on rows-written.** Measured 2026-09-15: the four history tables take
  over a million rows each per session, then legitimately go silent from the
  session boundary to the next open, with Sunday at zero. A `rate()==0` rule
  would page every night and every weekend.
* **No rule on the LEVEL of `sink_consumer_lag`.** It sits at a static \~1.93M
  pending backlog (data#2613), so any absolute threshold either fires forever or
  is meaningless. `SinkConsumerLagGrowing` watches the hourly delta instead;
  measured drift is \~500/hour against a 50,000 threshold, and a stalled sink would
  grow at roughly 425k/hour. The metric is recomputed from `XPENDING` each cycle
  and was verified stable across a sink restart, so the `offset 1h` comparison is
  restart-safe.