> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sequency.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Data agents

# sequency-data — Agent Instructions

## Shared delivery policy

All AI agents working in this repository must read and follow
`docs/agent-delivery-discipline.md`. It is the vendor-neutral source for active
packet scope, vertical delivery, token discipline, verification economy, and the
five delivery states. Repository-specific rules below add to that policy.

## Governed delegated implementation

Delegated Sequency implementation, inventory, architecture, security, and diff
review must use the discoverable `.agents/skills/sol-luna-promotion` skill and
its one-use task packet. Sol retains orchestration, exact-diff review, promotion,
deployment, and acceptance ownership; the skill does not replace repository
tests, CI, or security review.

GitHub: [sequencyhq/sequency-data](https://github.com/sequencyhq/sequency-data)

Parent product context: `../CLAUDE.md`, `../AGENTS.md`, operator `../THREATMODEL.md`.
This file is the **repository-local** guide for backend, data plane, agents, and deploy.

## Mission

Own market data ingestion, scoring, patterns, options, debates, backtests, lifecycle/scenario
pipelines, and the FastAPI graph API that powers sequency-app and sequency-charts consumers.

## Layout

| Path                             | Contents                                                                  |
| -------------------------------- | ------------------------------------------------------------------------- |
| `api/`                           | Python FastAPI (`app/`), workers, tests — primary HTTP surface :8081      |
| `services/`                      | Go services + `signal-delivery` (Elixir Phoenix)                          |
| `services/market-signal-engine/` | MSE :8100 — realtime indicators/scores; **sole options writer** (OPRA WS) |
| `services/pattern-detector/`     | Patterns + structural confluence :8090                                    |
| `services/trade-executor/`       | Autonomous / paper / live execution :8095                                 |
| `services/daily-daemon/`         | Daily VP, pivots, fundamentals, news :8091                                |
| `services/backfill-api/`         | Historical backfill :3001                                                 |
| `clickhouse/`                    | Forward-only CH DDL (`NNN_name.sql`)                                      |
| `migrations/`                    | Forward-only Postgres DDL                                                 |
| `agents/`                        | Python agent fleet + YAML configs                                         |
| `catalog/`                       | Feature / contract catalog (linted in CI)                                 |
| `systemd/`                       | Unit + timer definitions for prod                                         |
| `scripts/`                       | Deploy helpers, `apply-db-migrations.sh`, lint tools                      |
| `docs/`                          | Superpowers specs, runbooks, feature flags                                |

## DSOM contract & geometric-detection invariants (2026-08-05)

Standing rules from this week's deterministic geometric-detection work — read
before touching pattern/setup/signal/proposal/execution surfaces:

1. **DSOM is the governing contract.** Read
   `docs/superpowers/specs/2026-08-04-decision-support-object-model.md`
   before touching any of the five objects it defines: Observation → Setup →
   Signal → Proposal → Execution.
2. **OA-13**: Setup identity is the `lifecycle_id` dual-key; `event_id` is
   the confirmation fact and is NULL on prospects. **OA-14**: the word
   "revision" is banned — three named refs replace it (see DSOM). **OA-15**:
   "Evidence Envelope" is the semantic name for the existing Decision Lens wire object;
   `chart_contracts.Observation` remains the documented legacy wire name (no breaking rename).
3. **No LLM-derived price, ever.** Chart geometry/levels come from the
   corpus-backed `/pattern-events/symbol/{symbol}/insights` endpoint
   (`api/app/services/pattern_insight_projection.py`); this is a view, never
   a detector. The legacy app-side `analyze` proxy is deleted — do not
   reintroduce a path where an LLM produces a drawn/actionable price
   (security-review-verified invariant).
4. **Pattern-catalogue work is governed by epic #1654** (production-readiness
   spec `docs/superpowers/specs/2026-08-07-geometric-detection-production-readiness.md`).
   Class A construction is AUTHORIZED in tranches under the ratified
   classification pass (`2026-08-06-class-a-adj2-classification-pass.md`) +
   OQ-PDE-002 discipline. The `_shape` substrate EXISTS (#1634 shipped
   2026-08-08: migration `090_shape_facts.sql` + `pkg/patterns/shapes.go`)
   and TRANCHE 1 HAS BEGUN: `doji` is the first registered shape emitter
   (`pkg/patterns/detect_shapes.go`), so the substrate is no longer inert.
   **Registering an emitter still persists nothing in prod** — the nightly
   stays gated on the operator's `/opt/sequency/shape-facts-nightly-enabled`
   marker, created only after a supervised 5y backfill + dual-build gate.
   OQ-PDE-002 v0.3 is EFFECTIVE for tranche 1: its §8.4 declares the cases
   for the 8 constructible geometry-only shapes, §8.5 the one-detector-per-PR
   mechanics (each shape lands with its own OQ cases, its A-SH-09 band
   declared from an INDEPENDENT reference implementation before the emitter
   is written, and its §5 execution rows). Contextual shapes are gated on
   AC-CTX (#1588); `kicking` on AC-VOL stage two. Classes B–G need their own
   ratified classification passes first.
5. **Screener filters fail closed.** Build requests with registry filter
   ids; an unsupported filter leaf returns `422 unsupported_filter`, it is
   never silently dropped.

(The lint/deploy-verification rule this work also established is folded into
**Verification commands** below, not repeated here.)

## Data ownership (do not invent schemas)

* **ClickHouse** — OHLCV, scores, signals, indicators, options, lifecycle/outcomes
* **FalkorDB** (`trading_knowledge`) — graph market context (1-hop store; not a full KG replacement)
* **PostgreSQL** — users, debates, configs, risk profiles, relational state
* **Redis Streams** — realtime bus (same Redis as FalkorDB module)
* **Qdrant** — vectors for analogs / historical setups

Column/type mismatches between writers and readers are production incidents. Prefer verifying
CH column types and Go scanner types against live schema files.

## Deploy & migrations

1. Work on a **clean worktree** if `data/` root is dirty/stale (common).
2. Commit + push to `main` (or PR → merge with gates).
3. CI: tests → cross-compile Go → package → upload compute → migrations → restart → health.
4. **Never** SSH-apply migrations or SCP binaries.
5. Migration runner: `scripts/apply-db-migrations.sh` (CH + PG, `schema_migrations` tracking).
6. Post-deploy:\
   `ssh sequency@compute.sequencyhq.com "curl -s -o /dev/null -w '%{http_code}' http://localhost:8081/health"`

Workflow: `.github/workflows/deploy-worker.yml`.\
Env on server: `/opt/sequency/.env` (mode 600). **systemd `EnvironmentFile=` overrides
`Environment=`**; deploy Stage 2 overlays *preserved server* `.env` over package env — server wins.

## Verification commands (confirm in CI if scripts change)

```bash theme={null}
# Python (from repo root) — ALWAYS via scripts/lint-api.sh, never a bare
# `uvx ruff` / venv ruff: it is the single pinned-version source (ruff
# 0.16.0) that local, the `api-lint` required PR check, and the deploy
# pipeline all run. On 2026-08-04 three different ruff versions being in
# play (deploy's pinned uvx, the api venv's 0.14.10, a developer's local)
# broke main even though PR checks were green — PR-green did not imply
# deploy-green. After merging to `main`, verify the **deploy** workflow run,
# not just the PR's checks.
scripts/lint-api.sh
venv/bin/python -m pytest path/to/tests -v --tb=short   # from api/

# Go (per service / module)
go test -v -race ./...

# Elixir signal-delivery
cd services/signal-delivery && MIX_ENV=test mix test

# Catalog / contract when those schemas change
# (see scripts/catalog_lint, scripts/contract_lint)
```

**Ruff in scratch trees:** always use `api/pyproject.toml` (length 100). Default ruff 88 fails CI.

## Auth & internal secrets

* User routes: Supabase JWT; `AUTH_ENFORCEMENT_MODE=strict` in prod.
* Service-to-service: `X-Internal-Secret` / `INTERNAL_SECRET_<SERVICE>` — hard 403, no fallback.
* Per-service secrets do not fall back to a global secret.
* Read `../THREATMODEL.md` before new trust boundaries or `/security-review`.

## Options & market data landmines

* MSE owns OPRA WebSocket options quotes; OI from Trading API (daily).
* Do not reintroduce Alpaca REST polling for options steady-state.
* Microstructure / analysis should read CH (MSE-populated), not call Alpaca from random jobs.
* `market_data_daily` includes after-hours; date column is often `day` not `bucket`.

## Python preferences

* Prefer **Polars + numpy**; **no pandas** in new code (numpy stats ≠ pandas).
* IndicatorBank stays scalar-Python where established.
* asyncpg: JSONB may arrive as strings; pool bound to creating loop; careful with TIMESTAMPTZ vs naive dt.

## ClickHouse landmines

* Never alias an aggregate to a column name a sibling aggregate reads → `ILLEGAL_AGGREGATION`
  (data#1472 / #1345).
* User settings belong in `users.d/`, not `config.d/` (wrong placement → crash loop).
* CH FINAL + `capture_source='live'` semantics matter for lifecycle parity monitors.

## FalkorDB landmines

* Never `RETURN {map}` — use aliased returns.
* openCypher dialect quirks; Go type coercion on query results is a recurring bug class.

## Scoring ownership

* **MSE** = realtime scores/signals writer
* **pattern-detector** = structural / pattern side
* Registry / ontology: see app docs `documentation/ontology/SCORING_REGISTRY.md` when touching scores
* Confluence v5 field names are easy to typo (`entry_confluence_net_v5` vs `entry_net_confluence_v5`) —
  grep before inventing

## Git / transport

* Local `data/` is often **dirty and behind remote**. Verify with `gh`, not only `git status`.
* Prefer worktrees: `~/projects/sequency/.worktrees/data-<topic>`
* Helpers for stuck transport may exist as `.git/gh_push*.py` — do not force-push `main`
* Org ruleset: `security-review-gate` required on `main`

## Active backend programs (re-check memory)

* **Track P** lifecycle/scenarios — wire to prod on merge; monitors not data-accrual gates
* Intraday alpha Phase G shipped; RF2 calendar seal
* Footprint order-flow inert behind `FOOTPRINT_SYMBOLS`
* Backtestable confluence / strategy-run look-ahead issues — treat as capital-safety

## Spec discipline

When editing specs under `docs/`:

1. Verify every named symbol against current code.
2. Read full surrounding blocks before delete/replace guidance.
3. Echo-sweep the doc after every correction.
4. Spec example responses may be slimmer than shipped structs — project full fields when building UI/API.

## Issues

`gh issue create -R sequencyhq/sequency-data`\
Cross-cutting work defaults to this repo.