Runbook: confluence_version mislabel checks
The debate_recommendation_outcomes.confluence_version column carries the
scoring lineage of every row. This runbook lists operator queries for
day-to-day monitoring + post-deploy verification of #486 / #488 / #489
zombie-producer cleanups.
Tag values
Smoke checks (investigative — NOT correctness gates)
These surface high-signal candidates worth investigating. Each pattern can occur legitimately on a complete-v5 row (0.0 is a valid v5 score). Monitor
for spikes vs. baseline rather than treating any individual row as proof
of corruption.
Querying debate_recommendation_outcomes by lineage
The canonical query patterns for new research and operator scripts. Pick
the filter that matches your cohort intent — the choice is load-bearing
for forward-return validity.
Per-pass tag reference
(The reconstructed-with-source-prefix tags carry the originating live
emission’s generation through Pass 3 backfill — emitted by the filler
when it can infer the source from
score_history provenance, falling
back to 'reconstructed_unknown' otherwise.)
Canonical v5-pure filter (RECOMMENDED default)
The standard cohort for forward-return calibration, win-rate analysis, and any aggregation feeding downstream agents:Post-migration filter (excludes only pre-#497 untagged rows)
When the cohort intent is “everything tagged” (legacy comparisons, producer-distribution analysis, zombie-cleanup verification):Forbidden mixes
These pooling patterns produce statistically invalid cohorts. They will look like normal SQL — there is no schema-level guard.Smoke C — binary correctness gate (CI)
.github/workflows/smoke-c-confluence-arithmetic.yml runs nightly at
06:00 UTC (02:00 ET) and fails the build on any non-zero count of v5
rows where abs(net − (bull − bear)) > 0.0001 over the trailing 24h.
This is the binary correctness gate: the spec calls the pattern
structurally impossible for a correctly-tagged v5 row, so any non-zero
count is proof of #497-class drift. Investigate immediately:
- Cross-reference the failure window against
debate_outcome_write_error_total{error_type="column_drift"}. Any non-zerocolumn_driftrate during the same window points at a writer-sideOUTCOME_COLUMNSdesync. - If
column_driftis zero, the upstream scoring producer is computingnetinconsistently withbull − bear. Inspect recent commits to scoring writers andconfluence_v5parsers. - The CI workflow opens / updates a P0-Critical tracking issue automatically.
Aggregation rule
Any algo runningavg/sum/...(*_v5) over date ranges spanning pre/post
deploy MUST filter WHERE confluence_version = 'v5' for v5-pure analysis.
The orchestrator-boundary completeness derivation makes this a strict
predicate: rows where one v5 side was None get 'v5_partial' and are
excluded by definition.
For diagnostic v5-coverage queries that include partial rows, use:
Reconstructed-row exclusion rule
Forward-return calibration jobs MUST exclude every reconstructed-lineage tag — bare legacy'reconstructed', 'reconstructed_unknown', and
every '<source>_reconstructed' variant — to avoid look-ahead bias.
Pass 3 rebuilds score context post-hoc from score_history, so any
forward-return measured on those rows leaks future state.
'reconstructed' (no prefix) literal is the legacy Pass 3
output, deprecated in #497 Item 4. New rows emit
'<source>_reconstructed' (e.g. 'v5_reconstructed',
'v4.3.1_reconstructed') when score_history carries a known source
generation, falling back to 'reconstructed_unknown'. The bare
'reconstructed' literal MUST NOT appear on any row written after the
Item 4 deploy — its presence on a recent row is a regression signal.
Cutover regression signal (post-#486 / #488 / #489)
Tag distribution by week —count('v4.3.1') and count('v4.2') on new
rows should approach zero post-cleanup. Express as a ratio against
daily volume, not absolute zero (a single stuck producer would pass an
absolute check on aggregate):
Backup / restore pin
Restoring aclickhouse-backup snapshot taken before migration 032
deployed:
- Before resuming writes, run
scripts/apply-db-migrations.shagainst the restored DB. - Otherwise: writer’s
INSERTfails becauseOUTCOME_COLUMNSincludesconfluence_versionagainst a column-less restored table. schema_migrationstable is part of the restore — runner sees the migration as not-applied and re-applies it.
v6-launch checklist (future)
For any future scoring generation:- Update parser at
stock_parser.py:681-701to detect and tag the new generation. - Widen
Literal["v5","v5_partial","v4.3.1","v4.2"]onMarketDataContext.confluence_versionto include the new value. Deploy API. - Only then enable the new generation’s producer path. The Literal will crash debate orchestration if a parser emits a tag the model doesn’t accept — this is the desired failure mode (deliberate cutover beats silent shipping), but step 2 must precede step 3 by at least one full deploy cycle.
Rollback shapes
See spec §Deploy → Rollback for safe rollback shapes. Summary:- Safe default: revert Python only; keep Go preservation patch deployed.
- Acceptable with cost: revert both; lose
confluence_versionon rows in horizon backlog touched during the rollback window. - Rejected: Go-only revert while keeping new Python emission — reverted Go filler wipes fresh tags within an hour.
On-call paging — debate_outcome_write_error_total
The Prometheus counter debate_outcome_write_error_total (labels:
error_type ∈ ) is incremented by api/app/services/debate_outcome_writer.py
on every failed insert attempt — including retries — and the final
exception re-raises into the asyncio default handler.
Page on-call when:
error_type="column_drift" is the highest-priority signal: it indicates
OUTCOME_COLUMNS and the writer’s row tuple have desynced — exactly the
#497 failure mode this runbook exists to detect.
error_type="ch_insert_error" over a sustained 5-min window points to
ClickHouse degradation — the nightly Pass 3 filler is the safety net but
operators should still be paged so the time-series gap is bounded.
The counter is exposed at https://api.sequencyhq.com/metrics (the same
FastAPI /metrics endpoint that already serves request latency / count
metrics). Prometheus on the compute server scrapes this in the existing
job; no new scrape config is required.
Cross-links
- Spec:
docs/superpowers/specs/2026-04-25-debate-outcomes-confluence-version-design.md - Issues: #486, #488, #489, #491, #496, #497