> ## Documentation Index > Fetch the complete documentation index at: https://docs.sequency.sh/llms.txt > Use this file to discover all available pages before exploring further. # Confluence version mislabel check # Runbook: `confluence_version` mislabel checks The `debate_recommendation_outcomes.confluence_version` column carries the scoring lineage of every row. This runbook lists operator queries for day-to-day monitoring + post-deploy verification of #486 / #488 / #489 zombie-producer cleanups. ## Tag values | Tag | Meaning | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `'v5'` | Complete v5 — both bull and bear sourced from `confluence_v5` | | `'v5_partial'` | One v5 side populated, other None (zero-coerced downstream) | | `'v4.3.1'`, `'v4.2'` | Legacy generation; should approach zero post-cleanup | | `'v5_reconstructed'`, `'v5_partial_reconstructed'`, `'v4.3.1_reconstructed'`, `'v4.2_reconstructed'` | Pass 3 backfill with **lineage-preserved** source — reconstructed from a `score_history` row whose own `confluence_version` was the prefix. Look-ahead-bias risk; exclude from forward-return calibration. (#497 Item 4) | | `'reconstructed_unknown'` | Pass 3 backfill where `score_history` carries no `confluence_version` (pre-#491) or unknown value. Same look-ahead-bias risk; same exclusion rule. | | `'reconstructed'` | **Legacy literal — should NEVER appear on rows written after #497 Item 4 deploy.** Was the bare tag emitted by the pre-Item-4 Pass 3 path; survives only on rows backfilled before that deploy. Treat as `*_reconstructed` for cohort selection. | | `''` | Pre-instrumentation legacy row; lineage unknown | ## Smoke checks (investigative — NOT correctness gates) These surface high-signal candidates worth investigating. Each pattern can occur legitimately on a complete-v5 row (`0.0` is a valid v5 score). Monitor for **spikes vs. baseline** rather than treating any individual row as proof of corruption. ```sql theme={null} -- A. All-three-zero — historical fingerprint of #496 mislabel. -- Investigate spike vs. weekly baseline. SELECT count() FROM sequency.debate_recommendation_outcomes WHERE confluence_version = 'v5' AND confluence_bull_v5 = 0 AND confluence_bear_v5 = 0 AND confluence_net_v5 = 0 AND debate_timestamp > now() - INTERVAL 1 DAY; -- B. Asymmetric-zero — one v5 side zero while the other is non-zero. -- Useful for diagnosing partial-population leakage. SELECT count() FROM sequency.debate_recommendation_outcomes WHERE confluence_version = 'v5' AND ((confluence_bull_v5 = 0) != (confluence_bear_v5 = 0)) AND debate_timestamp > now() - INTERVAL 1 DAY; -- C. Arithmetic-inconsistency (HARD GATE — investigate any non-zero count). -- Net non-zero with both bull and bear at zero is structurally impossible -- when bull/bear are sourced and net = bull − bear. Indicates a writer -- or scoring bug. SELECT count() FROM sequency.debate_recommendation_outcomes WHERE confluence_version = 'v5' AND abs(confluence_net_v5) > 0 AND abs(confluence_bull_v5) + abs(confluence_bear_v5) = 0 AND debate_timestamp > now() - INTERVAL 1 DAY; ``` ## Querying `debate_recommendation_outcomes` by lineage The canonical query patterns for new research and operator scripts. Pick the filter that matches your cohort intent — the choice is load-bearing for forward-return validity. ### Per-pass tag reference | Tag | Source | Use case | | ------------------------------------------------------------------------- | ------------------------------------ | --------------------------------------------- | | `'v5'` | Live emission, both v5 sides present | Primary research cohort | | `'v5_partial'` | Live emission, one v5 side fallback | Diagnose parser gaps; do NOT pool with `'v5'` | | `'v4.3.1'`, `'v4.2'` | Live emission, full fallback | Legacy-comparison only | | `'v5_reconstructed'`, `'v4.3.1_reconstructed'`, `'reconstructed_unknown'` | Pass 3 backfill | Diagnostic; look-ahead risk | | `''` (empty) | Pre-#497 rows | Excluded from research | (The reconstructed-with-source-prefix tags carry the originating live emission's generation through Pass 3 backfill — emitted by the filler when it can infer the source from `score_history` provenance, falling back to `'reconstructed_unknown'` otherwise.) ### Canonical v5-pure filter (RECOMMENDED default) The standard cohort for forward-return calibration, win-rate analysis, and any aggregation feeding downstream agents: ```sql theme={null} SELECT * FROM sequency.debate_recommendation_outcomes WHERE confluence_version = 'v5' AND debate_timestamp >= toDateTime('2026-04-26 00:00:00'); -- post-#497 only ``` Pinning the lower bound to the #497 deploy date excludes pre-instrumentation empty-string rows even if they coincidentally satisfy other predicates (belt-and-braces against schema-restore edge cases — see Backup / restore pin below). ### Post-migration filter (excludes only pre-#497 untagged rows) When the cohort intent is "everything tagged" (legacy comparisons, producer-distribution analysis, zombie-cleanup verification): ```sql theme={null} SELECT * FROM sequency.debate_recommendation_outcomes WHERE confluence_version != ''; ``` This includes legacy and reconstructed rows. Useful for the cutover regression signal (see below). NOT valid for forward-return calibration without further filtering. ### Forbidden mixes These pooling patterns produce statistically invalid cohorts. They will look like normal SQL — there is no schema-level guard. ```sql theme={null} -- INVALID: reconstructed rows have look-ahead bias from the score_history -- rebuild, so pooling them with live-emission v5 inflates apparent -- accuracy. Forward-return calibration on this cohort is unreliable. WHERE confluence_version IN ('v5', 'reconstructed') -- INVALID for the same reason regardless of source-prefixed reconstructed -- variants: WHERE confluence_version IN ('v5', 'v5_reconstructed') -- INVALID: pools two distinct generations. Use only when the analysis -- is explicitly cross-generation comparison (e.g. v5 vs v4.3.1 -- per-symbol accuracy) and even then, GROUP BY confluence_version. WHERE confluence_version IN ('v5', 'v4.3.1') ``` If a research question requires both live and reconstructed rows, run two separate queries and report metrics side-by-side rather than pooling. ## Smoke C — binary correctness gate (CI) `.github/workflows/smoke-c-confluence-arithmetic.yml` runs nightly at 06:00 UTC (02:00 ET) and fails the build on any non-zero count of v5 rows where `abs(net − (bull − bear)) > 0.0001` over the trailing 24h. This is the **binary correctness gate**: the spec calls the pattern structurally impossible for a correctly-tagged v5 row, so any non-zero count is proof of #497-class drift. Investigate immediately: 1. Cross-reference the failure window against `debate_outcome_write_error_total{error_type="column_drift"}`. Any non-zero `column_drift` rate during the same window points at a writer-side `OUTCOME_COLUMNS` desync. 2. If `column_drift` is zero, the upstream scoring producer is computing `net` inconsistently with `bull − bear`. Inspect recent commits to scoring writers and `confluence_v5` parsers. 3. The CI workflow opens / updates a P0-Critical tracking issue automatically. Smoke C is structurally distinct from A and B (investigative-only); treat A/B baseline drift as a research signal, treat Smoke C as a page-on-call gate. ## Aggregation rule Any algo running `avg/sum/...(*_v5)` over date ranges spanning pre/post deploy MUST filter `WHERE confluence_version = 'v5'` for v5-pure analysis. The orchestrator-boundary completeness derivation makes this a strict predicate: rows where one v5 side was None get `'v5_partial'` and are excluded by definition. For diagnostic v5-coverage queries that include partial rows, use: ```sql theme={null} WHERE confluence_version IN ('v5', 'v5_partial') ``` ## Reconstructed-row exclusion rule Forward-return calibration jobs MUST exclude every reconstructed-lineage tag — bare legacy `'reconstructed'`, `'reconstructed_unknown'`, and every `'_reconstructed'` variant — to avoid look-ahead bias. Pass 3 rebuilds score context post-hoc from `score_history`, so any forward-return measured on those rows leaks future state. ```sql theme={null} -- Correct: SQL-level pattern match catches every reconstructed shape, -- including future source-prefixes added under #491. WHERE confluence_version NOT LIKE '%reconstructed%' ``` The `'reconstructed'` (no prefix) literal is the **legacy** Pass 3 output, deprecated in #497 Item 4. New rows emit `'_reconstructed'` (e.g. `'v5_reconstructed'`, `'v4.3.1_reconstructed'`) when `score_history` carries a known source generation, falling back to `'reconstructed_unknown'`. The bare `'reconstructed'` literal MUST NOT appear on any row written after the Item 4 deploy — its presence on a recent row is a regression signal. ## Cutover regression signal (post-#486 / #488 / #489) Tag distribution by week — `count('v4.3.1')` and `count('v4.2')` on new rows should approach zero post-cleanup. Express as a **ratio** against daily volume, not absolute zero (a single stuck producer would pass an absolute check on aggregate): ```sql theme={null} SELECT toStartOfWeek(debate_timestamp) AS week, countIf(confluence_version = 'v4.3.1') / count() AS v43_ratio, countIf(confluence_version = 'v4.2') / count() AS v4_ratio FROM sequency.debate_recommendation_outcomes WHERE debate_timestamp > now() - INTERVAL 30 DAY GROUP BY week ORDER BY week DESC; ``` Persistent non-zero ratio is a zombie-producer regression. ## Backup / restore pin Restoring a `clickhouse-backup` snapshot taken before migration 032 deployed: 1. **Before resuming writes**, run `scripts/apply-db-migrations.sh` against the restored DB. 2. Otherwise: writer's `INSERT` fails because `OUTCOME_COLUMNS` includes `confluence_version` against a column-less restored table. 3. `schema_migrations` table is part of the restore — runner sees the migration as not-applied and re-applies it. ## v6-launch checklist (future) For any future scoring generation: 1. Update parser at `stock_parser.py:681-701` to detect and tag the new generation. 2. Widen `Literal["v5","v5_partial","v4.3.1","v4.2"]` on `MarketDataContext.confluence_version` to include the new value. Deploy API. 3. Only then enable the new generation's producer path. The Literal will crash debate orchestration if a parser emits a tag the model doesn't accept — this is the desired failure mode (deliberate cutover beats silent shipping), but step 2 must precede step 3 by at least one full deploy cycle. ## Rollback shapes See spec §Deploy → Rollback for safe rollback shapes. Summary: 1. **Safe default:** revert Python only; keep Go preservation patch deployed. 2. **Acceptable with cost:** revert both; lose `confluence_version` on rows in horizon backlog touched during the rollback window. 3. **Rejected:** Go-only revert while keeping new Python emission — reverted Go filler wipes fresh tags within an hour. ## On-call paging — `debate_outcome_write_error_total` The Prometheus counter `debate_outcome_write_error_total` (labels: `error_type` ∈ {`column_drift`, `length_mismatch`, `ch_insert_error`, `unknown`}) is incremented by `api/app/services/debate_outcome_writer.py` on every failed insert attempt — including retries — and the final exception re-raises into the asyncio default handler. Page on-call when: ```promql theme={null} sum by (error_type) (rate(debate_outcome_write_error_total[5m])) > 0 ``` `error_type="column_drift"` is the highest-priority signal: it indicates `OUTCOME_COLUMNS` and the writer's row tuple have desynced — exactly the \#497 failure mode this runbook exists to detect. `error_type="ch_insert_error"` over a sustained 5-min window points to ClickHouse degradation — the nightly Pass 3 filler is the safety net but operators should still be paged so the time-series gap is bounded. The counter is exposed at `https://api.sequencyhq.com/metrics` (the same FastAPI `/metrics` endpoint that already serves request latency / count metrics). Prometheus on the compute server scrapes this in the existing job; no new scrape config is required. ## Cross-links * Spec: `docs/superpowers/specs/2026-04-25-debate-outcomes-confluence-version-design.md` * Issues: #486, #488, #489, #491, #496, #497