Production Alerting

Origin: data#1925, data#1926, data#1927 (2026-08-17/18). All three incidents were silent. Nothing paged; each was found by hand. This runbook covers what alerting exists, how to activate delivery, what each rule maps onto, and — explicitly — what is still not covered.

1. The stack, as it actually is

Component	State
Prometheus server	Running on the compute host at `:9090`. Its config is NOT in this repo — it is managed on the host.
Prometheus rules	In this repo: `monitoring/prometheus-rules/*.yml`. Loaded via the host’s `rule_files:`.
Alertmanager	NOT INSTALLED. No receiver, route, or config exists on any host or in any repo.
Grafana	Running at `:3000`; dashboards in `monitoring/grafana-dashboards/`. Not actively used.
node_exporter	Running at `:9100`.
redis_exporter	Expected on the data server at `:9121`. No evidence in this repo — `alert-service-health.sh` reports if it is unreachable.
Alert delivery	Slack incoming webhook (`SLACK_WEBHOOK_URL`) + GitHub Actions webhook (`PROBE_ALERT_WEBHOOK`).

The single most important fact: because no Alertmanager is installed, writing a Prometheus rule delivers nothing on its own. Before this work, all 26 existing rules could fire and reach no human. alert-dispatch.sh is the bridge that makes them real. It is a stopgap for Alertmanager, not a replacement — no grouping, inhibition, silencing, or escalation.

How an alert reaches a human today

Prometheus rule fires
   -> Prometheus /api/v1/alerts
      -> alert-dispatch.timer (every 2 min) -> alert-dispatch.sh
         -> alert-notify.sh -> $SLACK_WEBHOOK_URL -> Slack

systemd unit fails
   -> OnFailure=sequency-alert-notify@<unit>.service -> alert-notify.sh -> Slack

Service failed / Redis loading / Prometheus down  (Prometheus-independent)
   -> alert-service-health.timer (every 5 min) -> alert-service-health.sh
      -> alert-notify.sh -> Slack

Deploy fails on main
   -> deploy-failure-alert.yml (workflow_run) -> $PROBE_ALERT_WEBHOOK

2. Operator activation steps

Nothing below contains a secret. Both webhook values are bearer credentials and must never be committed.

Step 1 — Slack webhook on the compute host (REQUIRED)

Without this, every host-side alert is a journal line and nothing more.

ssh [email protected]
grep -q '^SLACK_WEBHOOK_URL=' /opt/sequency/.env && echo "already set" || echo "NOT SET"

# If not set, append it (file is mode 600, owned by sequency):
sudo sh -c 'printf "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ\n" >> /opt/sequency/.env'

Verify delivery end-to-end:

set -a; . /opt/sequency/.env; set +a
/opt/sequency/scripts/alert-notify.sh "Sequency alerting test — please ignore."

A message in Slack means the path works. SLACK_WEBHOOK_URL is NOT set or Slack webhook POST FAILED in the output means it does not.

Step 2 — GitHub webhook for deploy failures (REQUIRED)

Repo -> Settings -> Secrets and variables -> Actions -> New repository secret
Name:  PROBE_ALERT_WEBHOOK
Value: <Slack incoming webhook URL>

This secret is already used by synthetic-auth-probe.yml, clickhouse-storage-canary.yml and pattern-corpus-canary.yml. If it is already set, deploy-failure alerting works with no further action. Prove it without breaking anything:

gh workflow run deploy-failure-alert.yml -R sequencyhq/sequency-data -f drill=true

This posts a clearly-labelled :test_tube: DRILL message.

Step 3 — Reload Prometheus so the new rules load (REQUIRED)

New rules in monitoring/prometheus-rules/alerts.yml are inert until Prometheus re-reads them. The rule files are not copied to the host by CI — confirm where the host reads them from before reloading.

ssh [email protected]
grep -A5 rule_files /etc/prometheus/prometheus.yml     # confirm the configured path

# Sync the repo's rules to that path if it is not already the checkout, then:
promtool check rules /etc/prometheus/rules/alerts.yml   # MUST pass before reload
sudo systemctl reload prometheus                        # or: curl -XPOST localhost:9090/-/reload

Confirm the rules are loaded:

curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name' | sort | grep -E 'RedisLoading|BrokerOptions|CriticalServiceTargetMissing'

Step 4 — Verify the timers are armed (after the next deploy)

ssh [email protected]
systemctl list-timers 'alert-*' --no-pager
sudo systemctl status alert-dispatch.service alert-service-health.service --no-pager

Both timers are enabled and started automatically by the deploy. If they are missing, sudo systemctl enable --now alert-dispatch.timer alert-service-health.timer.

Step 5 (recommended, not done here) — install Alertmanager

alert-dispatch.sh has no silencing. During a long incident it will notify once per alert and once on resolution, but there is no way to mute a known issue short of stopping the timer. Installing Alertmanager and pointing Prometheus at it is the durable fix; alert-dispatch.timer should be stopped at that point to avoid duplicate notifications.

3. Per-rule signal mapping

Every rule below maps to a signal that is exported today. scripts/check-alert-metrics.py (wired into data-ci.yml) fails CI if any alert references a metric nothing emits — a dead alert reads as coverage it does not provide (#1673).

Added by this change

Alert	Signal	Emitted by	`for:`	Severity
`RedisLoadingDataset`	`redis_loading_dataset == 1`	redis_exporter	10m	critical
`RedisRestartLoop`	`redis_uptime_in_seconds < 900`	redis_exporter	15m	critical
`CriticalServiceTargetMissing`	`absent(up{job=...})` ×4	Prometheus scrape	10m	critical
`BrokerOptionsChainAuthFailures`	`rate(te_options_chain_auth_failures_total[5m]) > 0.1`	`services/trade-executor/metrics.go`	5m	critical
`BrokerOptionsChainAuthBreakerOpen`	`te_options_chain_auth_breaker_open == 1`	`services/trade-executor/metrics.go`	5m	critical
`ContractSelectionDisabled`	`te_contract_selection_disabled == 1`	`services/trade-executor/metrics.go`	10m	critical
`PatternDetectorCycleErrors`	`pd_cycle_errors > 0`	`services/pattern-detector/scheduler.go:1122`	30m	warning

Threshold note for the broker rules: data#1926 logged 118,594 401s in 32h (~1.03/s). The 0.1/s floor is two orders of magnitude below the observed rate and far above the zero a healthy credential produces.

Non-Prometheus checks

Condition	Mechanism	Cadence
Any of the 6 services not active/failed	`alert-service-health.sh` → `systemctl is-active`	5m
Redis TCP unreachable	`alert-service-health.sh` → `/dev/tcp` probe (no credential)	5m
Redis loading	`alert-service-health.sh` → scrapes `redis_exporter` directly	5m
redis_exporter unreachable	`alert-service-health.sh`	5m
Prometheus down	`alert-service-health.sh` → `/-/healthy`	5m
Watchdog cannot recover a service	`service-watchdog.service` `OnFailure=`	2m
Alert dispatcher itself broken	`alert-dispatch.service` `OnFailure=`	2m
Deploy failed on main	`deploy-failure-alert.yml`	per deploy

4. Recovery: services stuck `failed` (data#1925)

The canonical case. A data-server power loss left Redis loading an 18.4GB RDB for ~25 minutes; every Go service exited on Redis LOADING, exhausted StartLimitBurst=10, and stayed failed permanently.

ssh [email protected]
systemctl --failed

# 1. Is Redis still loading? If so, WAIT — restarting services now just burns
#    the start limit again, and restarting REDIS restarts the load from zero.
curl -s http://10.0.0.2:9121/metrics | grep '^redis_loading_dataset'

# 2. Once Redis is serving, recover each failed unit. Services do NOT
#    self-heal once StartLimitBurst is exhausted.
for svc in market-signal-engine pattern-detector trade-executor backfill-api sequency-api daily-daemon; do
  sudo systemctl reset-failed "$svc" 2>/dev/null || true
  sudo systemctl start "$svc"
done

# 3. Verify
systemctl --failed
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8081/health

Do not restart Redis to “fix” a slow load. That is how the 2026-08-17 outage stretched to ~50 minutes: systemd’s start timeout was shorter than the RDB load time, so systemd kept killing the load mid-flight.

5. GAPS — not covered today

Recorded honestly rather than papered over with rules that cannot fire.

5.1 data#1927 (missing-relation query errors) is NOT specifically covered

pattern-detector queried two non-existent tables 27,362 times in 32h. There is no metric for it, and no log-based alerting exists at all — no Loki, no promtail, no Vector, no log shipper of any kind. The journal is the only record. PatternDetectorCycleErrors (pd_cycle_errors > 0) is real coverage for cycle-level failure but is not a missing-relation detector; those queries are not known to increment it. To close this properly, one of:

A metric at the query site (preferred, ~1 day). A pd_query_errors_total{table,code} counter incremented where ClickHouse/ FalkorDB errors are handled, then a rule on rate(pd_query_errors_total{code="UNKNOWN_TABLE"}[10m]) > 0. Cheapest, most precise, and passes check-alert-metrics.py by construction.
A schema-conformance CI check. Assert every table named in Go query strings exists in clickhouse/*.sql. Catches the class before deploy, but not at runtime.
Log-based alerting (largest). Deploy Loki + promtail on compute, add a Grafana/Loki ruler. Also unlocks the whole “repeated error in the journal” class, including data#1926’s 401 log lines.

5.2 Other broker call sites have no auth metrics

Only the options-chain path has te_options_chain_auth_*. Other Alpaca call sites (order placement, positions, account, market data) have no auth-failure metrics, so a retired credential on any of those paths is still silent. The te_options_chain_auth_* pattern should be generalised to a shared broker_api_errors_total{endpoint,status} counter. Not invented here — the metric does not exist yet, and a rule on it would be exactly the dead alert check-alert-metrics.py exists to prevent.

5.3 backfill-api and daily-daemon have no Prometheus coverage

backfill-api exposes no /metrics endpoint at all; daily-daemon’s scrape job label is not established anywhere in this repo. Both are covered by alert-service-health.sh (systemd state) but by no Prometheus rule. Adding a /metrics handler to backfill-api would close half of this.

5.4 The Prometheus config is not in version control

There is no prometheus.yml in this or any repo. Scrape targets, job labels and rule_files: are host-managed and unreviewable. A target silently dropped from the scrape config is invisible to code review — CriticalServiceTargetMissing is a partial mitigation for the four services whose job labels are known from slo.yml, but the durable fix is to bring the Prometheus config into the repo and deploy it via CI.

5.5 No silencing, grouping, or escalation

alert-dispatch.sh notifies once per alert and once on resolution. There is no way to mute a known-ongoing issue except stopping the timer, and no escalation if Slack is not being read. Alertmanager is the fix (§2 Step 5).

5.6 Delivery is single-channel and unmonitored

Everything lands in one Slack webhook. If the webhook is revoked or the channel is archived, alert-notify.sh logs the failure at user.err — but nothing alerts on that, and it cannot: the alerting path is the thing that is broken. A periodic heartbeat to Slack (visible absence rather than an alert) would close it.

6. Host-side exporter state that CI cannot deploy (2026-09-11, data#2589)

Two exporter changes were made by hand on the data server. Neither unit is in this repo, so nothing re-applies them: a host rebuild loses both, and the loss is silent. This section exists so the next person looking for them finds them.

6.1 postgres-exporter now authenticates over the Unix socket

pg_up had been 0 since the 2026-08-17 PSU outage, so PostgresDown fired continuously for over three weeks while PostgreSQL itself served the API normally. The exporter was running and reachable; its stored DSN password had gone stale and every scrape logged password authentication failed for user "sequency". /etc/default/postgres-exporter (root-owned, mode 600) now holds:

DATA_SOURCE_NAME=postgresql:///sequency?host=/var/run/postgresql&sslmode=disable

The exporter runs as User=sequency and pg_hba.conf carries local all all peer, so a socket connection needs no credential at all. This removes a stored secret rather than rotating one. The previous file is kept at /etc/default/postgres-exporter.bak-2026-09-11. Further improvement, not done: the exporter connects as the application role. A dedicated role with pg_monitor would be least-privilege, at the cost of a role to create and a password to store — which is what the socket change just eliminated, so it is a genuine trade rather than a clear win.

6.2 node-exporter now reports systemd unit state

Until this change nothing alerted when a backup job failed. The four backup units on the data host carry no OnFailure= handler, no live rule covered failed systemd units, and alert-service-health.sh watches six long-running services on the other host. A silent backup failure on a single-host data tier that already lost power once this year is the gap with the most to lose. /etc/systemd/system/node-exporter.service (root-owned) now runs:

ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100 \
  --collector.systemd \
  --collector.systemd.unit-include="(pgbackrest|clickhouse-backup)-(incr|full)\.service|(postgresql|clickhouse-server|redis-server|node-exporter|postgres-exporter|redis-exporter)\.service"

The include list is deliberate: the default .+ would export every unit on the box. Ten units produce 50 series, one per unit per state. The previous file is at /etc/systemd/system/node-exporter.service.bak-2026-09-11. BackupJobFailed and DataHostUnitFailed in monitoring/prometheus-rules/alerts.yml read these series. Adding a unit to either alert’s regex requires adding it to the exporter’s include list too, or the series will not exist and the rule will match nothing. BackupJobFailed carries an absent() guard for exactly that mistake; DataHostUnitFailed does not, because those services have independent coverage through their own scrape targets.

6.3 What is still not covered

A backup that never runs is as bad as one that fails, and neither rule catches it: a oneshot that is never triggered simply stays inactive. pgBackRest exposes no last-successful-backup metric here, and scripts/pgbackrest-check.sh is not on a timer. Backup recency therefore remains unmonitored. pg_stat_archiver does now flow, so continuous WAL archiving failure is covered even though backup recency is not.

6.4 The API metrics scrape presents the internal secret

/metrics had returned 401 to Prometheus since 2026-06-07, so no API metric existed and ServiceDown{job="sequency-api"} fired for three months (data#2589). It was tempting to exempt /metrics from the auth middleware. That would have been an exposure: /metrics is reachable from the internet through nginx, and only the 401 was protecting it, so an exemption would have published every internal gauge and endpoint label. Verified at the time:

https://api.sequency.sh/metrics     -> 401
https://worker.sequency.sh/metrics  -> 401

Prometheus here is 3.3.0, which supports http_headers in a scrape config, so the scraper presents the secret instead and the endpoint’s auth posture is unchanged. /opt/prometheus/prometheus.yml, sequency-api job:

    http_headers:
      X-Internal-Secret:
        files: [ /opt/prometheus/api-scrape-secret ]

The secret lives in /opt/prometheus/api-scrape-secret (mode 600, owned by the Prometheus user) rather than inline, because prometheus.yml is mode 644. Neither file is in this repo — prometheus.yml is host-managed and the secret must not be committed — so a host rebuild loses this and the API target silently goes down again. promtool check config was run before the reload; do the same on any edit, because an invalid config makes the reload a no-op and leaves stale rules loaded. The previous config is at /opt/prometheus/prometheus.yml.bak-2026-09-11.

6.5 Two exit-protection alerts are suspended, for different reasons

Suspended in alerts.yml on 2026-09-14 by operator decision (data#2593):

PositionsWithoutExitProtection — paged on a legitimate state. Exit protection is not mandatory for discretionary trading (data#2094). Restore when data#2094 Gap 2 derives exit_intent from the strategy that placed the order. ExitProtectionBrokerStateUnknown was suspended in the first version of that change and has been kept, for two reasons. CI pins it: test_unknown_broker_state_metric_has_multiprocess_safe_paging_rule asserts its exact expression, severity: critical and for: 2m, so removing it fails the build — a deliberate guard. And its cause is not the discretionary question: it means broker-resident protection could not be read, which is a real failure. Today that is the rejected broker credential, the same one that has 9 of 11 positions auth-suppressed, so its blocker is the BYOK credential substrate in data#2037 rather than data#2094.

ExitProtectionReconcilerStale remains active and is the signal that matters while those two are off: it fires if the sweep stops running or its gauge disappears. Note for anyone tempted to downgrade rather than suspend a noisy alert: alert-dispatch.sh reads severity only to label the Slack message and does not filter on it. A warning notifies exactly as loudly as a critical.

6.6 clickhouse-sink is now a scrape target

The sink exports sink_consumer_lag, sink_dead_letters_total, sink_retry_queue_size and sink_rows_written_total on :8110, and nothing scraped it until 2026-09-14 (data#2613). Added to /opt/prometheus/prometheus.yml:

  - job_name: clickhouse-sink
    static_configs:
      - targets: ["localhost:8110"]
    metrics_path: /metrics

This matters more than an ordinary target because data#2456 cut the Redis stream caps, and for scores, signals, indicators and levels the stream is the only route to durable storage. The cap is therefore how long the sink may stay broken before data is lost silently. clickhouse-sink was added to CriticalServiceTargetMissing for the same reason: prometheus.yml is host-managed and not in this repo, so a rebuild drops this target, and losing it would put the deadline back out of sight. ServiceDown already covers the sink now that it is scraped — that rule has no job filter. Two alerting choices worth recording so nobody “fixes” them:

No rule on rows-written. Measured 2026-09-15: the four history tables take over a million rows each per session, then legitimately go silent from the session boundary to the next open, with Sunday at zero. A rate()==0 rule would page every night and every weekend.
No rule on the LEVEL of sink_consumer_lag. It sits at a static ~1.93M pending backlog (data#2613), so any absolute threshold either fires forever or is meaningless. SinkConsumerLagGrowing watches the hourly delta instead; measured drift is ~500/hour against a 50,000 threshold, and a stalled sink would grow at roughly 425k/hour. The metric is recomputed from XPENDING each cycle and was verified stable across a sink restart, so the offset 1h comparison is restart-safe.

Start

Operating

API

API reference

Runbooks

Production alerting

Production Alerting

1. The stack, as it actually is

How an alert reaches a human today

2. Operator activation steps

Step 1 — Slack webhook on the compute host (REQUIRED)

Step 2 — GitHub webhook for deploy failures (REQUIRED)

Step 3 — Reload Prometheus so the new rules load (REQUIRED)

Step 4 — Verify the timers are armed (after the next deploy)

Step 5 (recommended, not done here) — install Alertmanager

3. Per-rule signal mapping

Added by this change

Non-Prometheus checks

4. Recovery: services stuck `failed` (data#1925)

5. GAPS — not covered today

5.1 data#1927 (missing-relation query errors) is NOT specifically covered

5.2 Other broker call sites have no auth metrics

5.3 backfill-api and daily-daemon have no Prometheus coverage

5.4 The Prometheus config is not in version control

5.5 No silencing, grouping, or escalation

5.6 Delivery is single-channel and unmonitored

6. Host-side exporter state that CI cannot deploy (2026-09-11, data#2589)

6.1 postgres-exporter now authenticates over the Unix socket

6.2 node-exporter now reports systemd unit state

6.3 What is still not covered

6.4 The API metrics scrape presents the internal secret

6.5 Two exit-protection alerts are suspended, for different reasons

6.6 clickhouse-sink is now a scrape target

​Production Alerting

​1. The stack, as it actually is

​How an alert reaches a human today

​2. Operator activation steps

​Step 1 — Slack webhook on the compute host (REQUIRED)

​Step 2 — GitHub webhook for deploy failures (REQUIRED)

​Step 3 — Reload Prometheus so the new rules load (REQUIRED)

​Step 4 — Verify the timers are armed (after the next deploy)

​Step 5 (recommended, not done here) — install Alertmanager

​3. Per-rule signal mapping

​Added by this change

​Non-Prometheus checks

​4. Recovery: services stuck failed (data#1925)

​5. GAPS — not covered today

​5.1 data#1927 (missing-relation query errors) is NOT specifically covered

​5.2 Other broker call sites have no auth metrics

​5.3 backfill-api and daily-daemon have no Prometheus coverage

​5.4 The Prometheus config is not in version control

​5.5 No silencing, grouping, or escalation

​5.6 Delivery is single-channel and unmonitored

​6. Host-side exporter state that CI cannot deploy (2026-09-11, data#2589)

​6.1 postgres-exporter now authenticates over the Unix socket

​6.2 node-exporter now reports systemd unit state

​6.3 What is still not covered

​6.4 The API metrics scrape presents the internal secret

​6.5 Two exit-protection alerts are suspended, for different reasons

​6.6 clickhouse-sink is now a scrape target

Production Alerting

1. The stack, as it actually is

How an alert reaches a human today

2. Operator activation steps

Step 1 — Slack webhook on the compute host (REQUIRED)

Step 2 — GitHub webhook for deploy failures (REQUIRED)

Step 3 — Reload Prometheus so the new rules load (REQUIRED)

Step 4 — Verify the timers are armed (after the next deploy)

Step 5 (recommended, not done here) — install Alertmanager

3. Per-rule signal mapping

Added by this change

Non-Prometheus checks

4. Recovery: services stuck `failed` (data#1925)

5. GAPS — not covered today

5.1 data#1927 (missing-relation query errors) is NOT specifically covered

5.2 Other broker call sites have no auth metrics

5.3 backfill-api and daily-daemon have no Prometheus coverage

5.4 The Prometheus config is not in version control

5.5 No silencing, grouping, or escalation

5.6 Delivery is single-channel and unmonitored

6. Host-side exporter state that CI cannot deploy (2026-09-11, data#2589)

6.1 postgres-exporter now authenticates over the Unix socket

6.2 node-exporter now reports systemd unit state

6.3 What is still not covered

6.4 The API metrics scrape presents the internal secret

6.5 Two exit-protection alerts are suspended, for different reasons

6.6 clickhouse-sink is now a scrape target