​

Runbook — lineage_warnings Triage (#527)

​

What this is

debate_recommendation_outcomes.lineage_warnings is an Array(LowCardinality(String)) column populated at write time from a frozen vocabulary (4 active contamination + 2 informational Pass 3 + 2 reserved-deferred tags). Empty array means no detected contamination. A live-emission v5 row (confluence_version ∈ {'v5', 'v5_partial'}) with contamination warnings (net_sign_mismatch, zero_triple, magnitude_outlier_v5, conviction_inconsistent) is the silent-leak detection cohort — what should never happen if the v5 pipeline is healthy. Smoke D gates exclusively on this cohort. Pass 3 reconstructed rows (v5_reconstructed, v5_partial_reconstructed) carry only informational tags (reconstructed, optionally v5_source_unknown) — Pass 3 does NOT call derive_lineage_warnings, so these rows are invisible to Smoke D by construction. Surface them in cohort filtering for calibration jobs (filter out reconstructed cohort to avoid forward-looking-bias contamination of v5 calibration). Activating contamination surveillance on Pass 3 rebuilds is a Phase 1.5 follow-up.

​

Vocabulary

​

Active contamination warnings (Phase 1)

​

Informational tags (Pass 3 reconstruction; NOT contamination)

​

Reserved (deferred to Phase 1.5)

These tags are in LINEAGE_WARNING_VOCAB but never emitted from Phase 1 derivation. Activation requires data outside the orchestrator hot path (prev-day v43 lookup, cross-table tier observation). Tracked in the Phase 1.5 follow-up issue.

​

Response when smoke-d fires

1. Query the offending rows (matches the Smoke D gate exactly — live-emission v5 family + hasAny on contamination tags):
   SELECT symbol, debate_timestamp, confluence_version, lineage_warnings,
          confluence_bull_v5, confluence_bear_v5, confluence_net_v5
   FROM sequency.debate_recommendation_outcomes
   WHERE debate_timestamp > now() - INTERVAL 1 DAY
     AND confluence_version IN ('v5', 'v5_partial')
     AND hasAny(lineage_warnings, ['net_sign_mismatch', 'zero_triple', 'magnitude_outlier_v5', 'conviction_inconsistent'])
   ORDER BY debate_timestamp DESC
   LIMIT 100;
   
2. Bucket by warning tag (most common first).
3. If a single tag dominates (>80% of rows), follow that tag’s “First-look” location above.
4. If multiple tags co-occur on the same rows, the producer is upstream of all of them — likely the parser. Check stock_parser.py recently-shipped changes (git log -- api/app/services/stock/).
5. File a tracking P0 referencing the smoke-d issue. Roll back the most recent debate_outcome_writer or stock_parser deploy if no fix in <2h.

​

Adding new vocab

1. Add tag to LINEAGE_WARNING_VOCAB in api/app/services/debate/lineage_warnings.py.
2. Add detection logic in derive_lineage_warnings.
3. Add a row to the table above + a “First-look” location.
4. Cardinality ceiling 32 — if you’re approaching it, audit existing tags first.