Skip to main content

Runbook: confluence_version mislabel checks

The debate_recommendation_outcomes.confluence_version column carries the scoring lineage of every row. This runbook lists operator queries for day-to-day monitoring + post-deploy verification of #486 / #488 / #489 zombie-producer cleanups.

Tag values

Smoke checks (investigative — NOT correctness gates)

These surface high-signal candidates worth investigating. Each pattern can occur legitimately on a complete-v5 row (0.0 is a valid v5 score). Monitor for spikes vs. baseline rather than treating any individual row as proof of corruption.

Querying debate_recommendation_outcomes by lineage

The canonical query patterns for new research and operator scripts. Pick the filter that matches your cohort intent — the choice is load-bearing for forward-return validity.

Per-pass tag reference

(The reconstructed-with-source-prefix tags carry the originating live emission’s generation through Pass 3 backfill — emitted by the filler when it can infer the source from score_history provenance, falling back to 'reconstructed_unknown' otherwise.) The standard cohort for forward-return calibration, win-rate analysis, and any aggregation feeding downstream agents:
Pinning the lower bound to the #497 deploy date excludes pre-instrumentation empty-string rows even if they coincidentally satisfy other predicates (belt-and-braces against schema-restore edge cases — see Backup / restore pin below).

Post-migration filter (excludes only pre-#497 untagged rows)

When the cohort intent is “everything tagged” (legacy comparisons, producer-distribution analysis, zombie-cleanup verification):
This includes legacy and reconstructed rows. Useful for the cutover regression signal (see below). NOT valid for forward-return calibration without further filtering.

Forbidden mixes

These pooling patterns produce statistically invalid cohorts. They will look like normal SQL — there is no schema-level guard.
If a research question requires both live and reconstructed rows, run two separate queries and report metrics side-by-side rather than pooling.

Smoke C — binary correctness gate (CI)

.github/workflows/smoke-c-confluence-arithmetic.yml runs nightly at 06:00 UTC (02:00 ET) and fails the build on any non-zero count of v5 rows where abs(net − (bull − bear)) > 0.0001 over the trailing 24h. This is the binary correctness gate: the spec calls the pattern structurally impossible for a correctly-tagged v5 row, so any non-zero count is proof of #497-class drift. Investigate immediately:
  1. Cross-reference the failure window against debate_outcome_write_error_total{error_type="column_drift"}. Any non-zero column_drift rate during the same window points at a writer-side OUTCOME_COLUMNS desync.
  2. If column_drift is zero, the upstream scoring producer is computing net inconsistently with bull − bear. Inspect recent commits to scoring writers and confluence_v5 parsers.
  3. The CI workflow opens / updates a P0-Critical tracking issue automatically.
Smoke C is structurally distinct from A and B (investigative-only); treat A/B baseline drift as a research signal, treat Smoke C as a page-on-call gate.

Aggregation rule

Any algo running avg/sum/...(*_v5) over date ranges spanning pre/post deploy MUST filter WHERE confluence_version = 'v5' for v5-pure analysis. The orchestrator-boundary completeness derivation makes this a strict predicate: rows where one v5 side was None get 'v5_partial' and are excluded by definition. For diagnostic v5-coverage queries that include partial rows, use:

Reconstructed-row exclusion rule

Forward-return calibration jobs MUST exclude every reconstructed-lineage tag — bare legacy 'reconstructed', 'reconstructed_unknown', and every '<source>_reconstructed' variant — to avoid look-ahead bias. Pass 3 rebuilds score context post-hoc from score_history, so any forward-return measured on those rows leaks future state.
The 'reconstructed' (no prefix) literal is the legacy Pass 3 output, deprecated in #497 Item 4. New rows emit '<source>_reconstructed' (e.g. 'v5_reconstructed', 'v4.3.1_reconstructed') when score_history carries a known source generation, falling back to 'reconstructed_unknown'. The bare 'reconstructed' literal MUST NOT appear on any row written after the Item 4 deploy — its presence on a recent row is a regression signal.

Cutover regression signal (post-#486 / #488 / #489)

Tag distribution by week — count('v4.3.1') and count('v4.2') on new rows should approach zero post-cleanup. Express as a ratio against daily volume, not absolute zero (a single stuck producer would pass an absolute check on aggregate):
Persistent non-zero ratio is a zombie-producer regression.

Backup / restore pin

Restoring a clickhouse-backup snapshot taken before migration 032 deployed:
  1. Before resuming writes, run scripts/apply-db-migrations.sh against the restored DB.
  2. Otherwise: writer’s INSERT fails because OUTCOME_COLUMNS includes confluence_version against a column-less restored table.
  3. schema_migrations table is part of the restore — runner sees the migration as not-applied and re-applies it.

v6-launch checklist (future)

For any future scoring generation:
  1. Update parser at stock_parser.py:681-701 to detect and tag the new generation.
  2. Widen Literal["v5","v5_partial","v4.3.1","v4.2"] on MarketDataContext.confluence_version to include the new value. Deploy API.
  3. Only then enable the new generation’s producer path. The Literal will crash debate orchestration if a parser emits a tag the model doesn’t accept — this is the desired failure mode (deliberate cutover beats silent shipping), but step 2 must precede step 3 by at least one full deploy cycle.

Rollback shapes

See spec §Deploy → Rollback for safe rollback shapes. Summary:
  1. Safe default: revert Python only; keep Go preservation patch deployed.
  2. Acceptable with cost: revert both; lose confluence_version on rows in horizon backlog touched during the rollback window.
  3. Rejected: Go-only revert while keeping new Python emission — reverted Go filler wipes fresh tags within an hour.

On-call paging — debate_outcome_write_error_total

The Prometheus counter debate_outcome_write_error_total (labels: error_type) is incremented by api/app/services/debate_outcome_writer.py on every failed insert attempt — including retries — and the final exception re-raises into the asyncio default handler. Page on-call when:
error_type="column_drift" is the highest-priority signal: it indicates OUTCOME_COLUMNS and the writer’s row tuple have desynced — exactly the #497 failure mode this runbook exists to detect. error_type="ch_insert_error" over a sustained 5-min window points to ClickHouse degradation — the nightly Pass 3 filler is the safety net but operators should still be paged so the time-series gap is bounded. The counter is exposed at https://api.sequencyhq.com/metrics (the same FastAPI /metrics endpoint that already serves request latency / count metrics). Prometheus on the compute server scrapes this in the existing job; no new scrape config is required.
  • Spec: docs/superpowers/specs/2026-04-25-debate-outcomes-confluence-version-design.md
  • Issues: #486, #488, #489, #491, #496, #497