@@ -1,14 +1,12 @@# finance — Project Plan and Resume Doc`finance` is a self-hosted, real-timeish market-watching web app for stocks,ETFs, indexes, and commodities: live charts, key stats, fundamentals, SECfilings, and at-a-glance health reads. A single Rust + axum binary backed bySQLite, with a Vite frontend. Deploys at `finance.bythewood.me`, published onGitHub as `finance`.`finance` is a self-hosted, demand-driven market watcher for stocks, ETFs,indexes, and commodities: live charts, key stats, fundamentals, and SEC filings.A single Rust + axum binary backed by SQLite, with a Vite frontend. Deploys at`finance.bythewood.me`, published on GitHub as `finance`.It is for *watching* the market only. No portfolio, no holdings, no money orcost-basis tracking, no accounts, no auth. Single operator. **Not investmentadvice** — every synthesized read carries that disclaimer.cost-basis tracking, no accounts, no auth. **Not investment advice.**---
@@ -34,213 +32,214 @@ commit + auto-deploy (`git push server master`) and a clean breakpoint.## Status_Last updated: 2026-05-30 (Phase 7 committed + deployed at `645b351`; roadmap complete)_**Major refactor in progress (the "distill + ETF-first" rewrite).** This planwas fully rewritten 2026-05-30 from a sprawling 3,700-line resume doc into thisfocused roadmap. The decisions driving it are in the Decisions log under2026-05-30; the short version:- **Short-horizon prediction is being dropped.** Next-day / next-week "picks" are a coin-flip gamble (the backtest's own ~50% win rates prove it) and they drove most of the live-data demand that tripped the rate limits. Day/Week go; Month/Quarter are reframed as a non-advice *quality leaderboard*.- **Data is going Yahoo-only.** Stooq is being removed entirely (its bulk download is CAPTCHA-gated and unscriptable; its per-symbol API has an undocumented daily-hit cap that kept blocking us). Yahoo serves both deep history and daily updates. See **Data-source policy**.- **ETFs become true first-class citizens** with their own identity and an ETF "quality" read, clearly separated from stocks.- **Everything gets distilled** into a fast-scannable, dual-first (mobile + desktop) design while keeping the futuristic-clean "Paper Ledger" look.**Current work:** Phases 1–7 are **done, committed, and deployed** (Phase 7 islive at `645b351`). With Phase 7 the roadmap is **complete** — remaining itemsare all in Backlog / parked. Prod verified: home + `/health` return 200 and servethe new footer + health verdict markup.Phase 7 outcome (health distillation + footer + live breadth):- **Systems verdict on `/health`.** A one-line plain read distilling the whole page — `● All systems normal · both data sources healthy · N jobs on schedule · last fetch 4m ago` — sits above the Endpoints/Jobs/Log detail. Built client-side in `health.js` (`renderVerdict`) from the same snapshot that drives the detail, so it stays live and never disagrees with it. Tone is the worst thing on the page: a tripped breaker or errored job → bad (red); a recovering breaker or stale job → warn (amber, pulsing dot); else ok (green). A mid-fetch job is normal and doesn't darken the tone (the existing "Fetching now —" banner names it). Verified on dev: reads green "All systems normal · both data sources healthy · 10 jobs on schedule · fetching now" at desktop + mobile.- **Two-tier footer (Paper Ledger).** Replaced the single-line footer with the house pattern: a `footer__grid` (About + data attribution / Pages / Elsewhere, `// LABEL` mono headers) over a slim `.footer-bar` (`© {{ now.year }} Isaac Bythewood · Some rights reserved` + a GitHub icon link to `github.com/overshard/finance`). Modeled on `repos` but in Paper Ledger tokens (warm-paper wash, hairline rules, ink text). The stale-Stooq attribution (fixed early in Phase 4) folds into the About column as "Market data via Yahoo Finance · fundamentals & filings via SEC EDGAR · prices are delayed · not investment advice." The `.footer-bar` now carries the mobile bottom-nav clearance (it's the last element before the fixed nav); the desktop block trims it back. Verified: 3-up at desktop, About-spans-full + 2-up links at mobile, copyright bar clears the bottom nav.- **Live market summary (the Phase-5 deferral).** The hero verdict + headline figures + breadth were page-load snapshots; now a new `src/summary.rs` owns the verdict vocabulary (`market_verdict` + `vix_tone`, moved out of `home.rs` so the page render and the live push share one wording) and a cheap `market_summary(pool, session)` (two aggregate reads: breadth over the curated stocks + the lead index / VIX, resolving last-vs-prev exactly as the home queries do). A new `StreamEvent::Summary` carries it; the scheduler publishes it after an intraday sweep that touched a **pulse ticker** (the broad index or ^VIX — the curated stocks behind breadth only move at the daily close, so a sweep that hit only a viewed stock page is skipped), after `daily_close`, and on a session flip (the lead index swaps cash↔future). `base/stream.js` re-broadcasts it as a `finance:summary` window event; a new `home.js` patches the hero verdict/detail/figures + breadth counts/bar in place — no second EventSource. **The live push couldn't be exercised on dev** (closed-market weekend, no quotes stream — same caveat as Phase 6's live tick); it runs first in prod during market hours. The patcher itself was verified structurally by dispatching a synthetic `finance:summary`: the verdict, all three figures, the breadth counts, and the proportion-bar widths all updated correctly.- **Verified on dev:** `cargo build` + `vite build` clean; home + health render at desktop (1280) and mobile (390) with no console errors; the new footer and health verdict read correctly; the synthetic-summary patch updates every hero + breadth node (screenshots reviewed, then deleted).Phase 6 outcome (symbol-page distillation + live intraday on chart):- **1D / 1W intraday range buttons.** Two new ranges at the front of the chart's range bar serve today's / the week's 15-minute bars from `intraday_bars` on a minute axis (the rest stay daily). `history_api` branches to `intraday_history` for them: bars are stored as epoch-**ms** so each `ts` is ÷1000 to the UNIX *seconds* lightweight-charts wants; a new `BarTime` untagged enum lets the `candles` field carry either a `YYYY-MM-DD` string (daily) or a number (intraday). 1D resolves the most recent NY trading day present (so a weekend shows Friday); 1W is a rolling 7 days (well inside the 14-day retention). The daily-only overlays (SMA/EMA/RSI/benchmark) come back empty and their toggles hide on intraday — only Volume stays. A dashed **prev-close** reference line (the latest daily close) anchors the session move.- **Live, without a second connection.** `stream.js` now re-broadcasts each SSE quote as a `finance:quote` window event; `chart.js` listens and, while an intraday range is shown, ticks the trailing bar's close/high/low in place. A 60s background re-fetch (local DB only, no Yahoo call) folds in newly-stored 15m bars so new bars appear without a click. (The live tick itself couldn't be exercised on dev — today is a closed-market weekend so no quotes stream; the wiring is structurally verified, like Phase 4's NAV fetch. It runs first in prod during market hours.)- **Mobile above-the-fold reorder.** Per the user's call: the full interactive chart is **promoted** above the fold (no separate mini-chart), and the top-right health/quality **donut is desktop-only** — on mobile a flattened full-width **verdict line** sits right under the price (`● Verdict · NN% · ↑ trajectory`, coloured by overall grade; the trajectory clause shows for a stock, not an ETF). So the phone order is price/change → verdict → chart.- **Fixed a latent bug in passing:** `.ind-btn { display: inline-flex }` was overriding the `[hidden]` attribute, so the benchmark toggle never actually hid on benchmark-less symbols (and wouldn't hide the daily-only toggles on intraday). Added an `.ind-btn[hidden] { display: none }` re-assert.- **Verified on dev:** `cargo build` + `vite build` clean; `/api/.../history? range=1D|1W` returns `intraday:true` with numeric times + empty overlays + prev_close, while `range=1Y` is unchanged (`intraday:false`, SMAs populated). Screenshots reviewed (then deleted) at mobile (390) + desktop (1280): stock + ETF; 1D shows intraday candles, the prev-close line, a "over 16 hours" span caption, and only the Volume toggle; switching back to a daily range restores the toggles; the mobile verdict line reads correctly (stock with trajectory, ETF without) with the donut hidden, and desktop shows the donut with the verdict line hidden. No new console errors (only the benign restart/stream chunked-encoding noise).Phase 5 outcome (dashboard redesign → "how is the market doing TODAY"):- **Hero verdict.** A two-line plain read at the top blending the broad index move, breadth, and the VIX risk tone into a lead + clause (e.g. "Higher, but narrow." / "Markets higher with narrow participation."), over a compact index-chip strip, with the headline figures (S&P %, % green, VIX tone) and a non-advice note. `build_hero` + `market_verdict` + `vix_tone` in `src/routes/home.rs`, fed by the already-loaded index/commodity cards + breadth (zero extra queries). **The verdict's direction tracks the broad index's sign** so it never contradicts the "S&P +x%" shown beside it; breadth only sets direction when no index price exists. A near-flat index reads "mixed" even when breadth skews. (Found + fixed during review: a +0.12% S&P with weak breadth had read "Broadly lower" — a direct contradiction; the deadband was tightened from ±0.15% to ±0.05% and the breadth fallback narrowed.)- **Breadth band.** Advancers / decliners counts + % green over a single up/flat/down proportion bar. `breadth()` reuses the `load_stocks` scan (no extra query); a stock without a computable change is excluded so a missing quote never reads as "flat".- **ETF band (the Phase-4 deferral, now built).** Five curated quality cards (`VOO`, `VTI`, `QQQ`, `BND`, `GLD`) — intraday sparkline + day move + the Phase-4 quality verdict pill — over a compact gainers/losers strip across the whole curated ETF set, each pill dotted by quality grade. `load_etfs` rolls every seeded ETF (price + Yahoo metadata + SEC AUM + top-10 concentration) into the `etf_quality` read, reusing the same NAV-freshness gate as the symbol page.- **Band order:** Hero · Indexes · Breadth · ETFs · Stock movers · Industries · Risk & commodities · Quality leaderboard. The existing Industries (sector up/down) panel was **kept as its own band** (user's call), not folded into breadth.- **Verified on dev:** `cargo build` + `vite build` clean; home renders all bands at desktop (1280) and mobile (390) with no console errors; hero reads consistently with the figures beside it; breadth counts reconcile (197 adv / 306 dec = 39% green); ETF cards show quality pills and the movers strip ranks correctly (screenshots reviewed, then deleted). Because today is a weekend the index cards correctly resolve to futures (ES=F, …) and breadth dates to the prior close.- **Known limitation (Phase 7 polish candidate):** the hero verdict and breadth are page-load snapshots — the sparkline cards still stream live, but the verdict sentence + breadth counts do not recompute intraday. Live breadth would need a server-pushed breadth event on the stream hub.Phase 3 outcome (drop short-horizon prediction → quality leaderboard):- **The whole picker is gone.** Removed the four horizon rankers (`pick_day/week/month/quarter` + `PickInput`), `src/picks.rs`, the `/backtest` route + page + JS, the scheduler's once-a-day snapshot job, and the backtest-only `models::latest_annual_inputs_as_of` / `FILING_LAG_DAYS`. Migration `0013_drop_picks.sql` drops the `picks` table and sweeps its stale `data_status` / `fetch_log` / `meta` rows so `/health` doesn't show a frozen "picks" job.- **Three overlapping home panels merged into one Quality leaderboard.** Top picks, Strongest & weakest, and Stock health all collapsed into a single non-advice "Healthiest / Most concerning" surface driven by the existing `compute::health_read` composite (fundamentals + trajectory + leadership stability). The old strongest/weakest panel's trailing-year return is folded into each leaderboard row as a quiet price anchor; the non-advice disclaimer rides on the section.- **`compute::standing` kept** — the movers panel still uses it for each row's strength badge, and the symbol + search pages use it too. Only the home *strongest/weakest panel* was removed, not the standing read itself.- **Verified:** `cargo build` + `vite build` clean (no `backtest` entry); migration applied on boot (picks table dropped, zero stale picks rows, `data_status` job list no longer lists picks); `/backtest` + `/api/backtest` → 404; `/api/health` → 200 with no picks job; home renders the leaderboard with trailing returns and none of the removed panels (screenshot reviewed).Phase 2 outcome (universe curation, deployed in `b016b25`):- **Stocks reconciled to the current S&P 500.** Fetched the live Wikipedia constituent list (503) and diffed against ours: an exact match, zero changes needed — the stock list was already current.- **ETF roster curated to iShares + Vanguard (43 ETFs).** Dropped the other-issuer thematic/sector funds (SMH, ARKK, SCHD, XLK/XLF/XLE/XLV); kept the SPY/QQQ/DIA/GLD/SLV proxies; added the most-common Vanguard + iShares funds incl. the core holdings IAU, IBIT, VBIL. Issuer/category tags come from the existing Yahoo `fund_metadata` (no new schema) per the user's choice.- **Seed now reconciles on every boot.** `seed::sync_universe` (upsert + prune) runs each boot via `run_boot_seed`, so a CSV edit (symbols added or dropped) takes effect on deploy without a manual re-seed. Pruning is `is_seeded = 1 AND ticker NOT IN (csv)`, cascading cleanly; user-added symbols (`is_seeded = 0`) are never touched. This also swept up 7 stale rows an older non-pruning seed had left behind (MSTR, NET, RIVN, SHOP, SNOW, SOFI, MRVL — popular names from a pre-S&P-narrowing universe; none are current S&P 500 members). **If the user wants those back, that's a separate "popular non-S&P watchlist" decision (see backlog).**- **Fixed a Phase-1 data-quality bug: Yahoo `range=max` downsampling.** Yahoo silently returns weekly/monthly bars for `interval=1d&range=max` on symbols with long histories (all the futures, ^RUT/^VIX, and every freshly-backfilled multi-year ETF were monthly; some were 1-bar). Two-part fix: the provider refetches a bounded `range=10y` window (which Yahoo serves at true daily granularity) when it detects a downsampled response; and `run_history` self-heals stored coarse/single-bar series (recent-density test) by replacing them with a clean daily re-fetch — so **prod self-heals on deploy**. Verified: every seeded symbol now holds genuine daily bars (median gap 1 day), no symbol is 1-bar, ^SPX's deep daily history (back to 1789) is preserved, and the density test flags zero false positives across all 562 symbols.**Dev server:** kept running in the background via `make` during sessions sothe user can review progress live._Last updated: 2026-06-04 (Phases A-D all built + verified on dev; roadmapcomplete; everything uncommitted, holding per the user's call to batch thecommit/deploy)._**Major refocus in progress (the "demand-only" rewrite).** The previous roadmap(the "distill + ETF-first" rewrite, Phases 1-7, all deployed at `645b351`) shippeda broad always-on dashboard. The user has now steered a focus shift: the appcould not get enough live data to show what they actually wanted, and the timedbackground sweeps were spending API budget around the clock for symbols nobody waslooking at. The new principle:> **Nothing is fetched unless a human is actively looking at it.** With nobody on> the site, the app makes zero outbound calls.The new shape (decided 2026-06-03, see the Decisions log for the full Q&A):- **Drop every timed network sweep.** Remove the scheduler's `daily_close`, `sec`, `dividends`, `fund_metadata`, `fund_nav`, `earnings_calendar`, `asset_profile`, and the periodic `history` jobs. Keep only the local `prune` (no network) and the demand-driven intraday poll (already gated by the viewer-interest registry). Symbol data, including daily history, is fetched **on page load when stale** and on an explicit manual refresh, never on a timer.- **Symbol pages become pull-on-demand.** Landing on a symbol (or hitting its manual refresh button) pulls the latest data with a **clear loading bar** over every piece being fetched and a **clear data-age indicator** on everything. Fast Yahoo data (quote / intraday / daily history) refreshes live on load; slow, rate-limited SEC data (fundamentals, filings, holdings, NAV) is pulled only when missing or stale. Manual refresh re-pulls everything.- **Dashboard is fully reworked into a TradingView-style view:** a big real-time **normalized %-vs-SPX overlay** graph for the day, very clear about market hours (pre / regular / after-hours / closed), over a **session-scoped editable watchlist** of stocks/ETFs that refresh every ~5 minutes while the dashboard is open, with data-age indicators throughout. SPX is the fixed baseline. Starters: VTI, VXUS, BND, IAU, IBIT. Polls only while someone has the dashboard open. Also keeps three always-on market reads (user's call): overall market volume, ^VIX (risk tone), and SMA 50/200 as general-interest overlays.- **Remove the Industries page entirely** (not useful without real-time sector data).- **Search stays** as the way to jump directly to any symbol for its on-demand deep data. The full universe CSV remains a *searchable catalog* (metadata only; each symbol's data is fetched only when it is first viewed).**Current work:** the demand-only roadmap (Phases A-D) is **complete and verifiedon dev**. Everything is uncommitted — the user chose to hold and batch onecommit + deploy. The old surface is still live in prod until that deploy.Phase D outcome (cohesion + polish):- **Removed the now-dead `StreamEvent::Summary` machinery** (the old dashboard's live breadth/verdict push, unused since Phase C). Deleted `src/summary.rs` (`market_summary` / `market_verdict` / breadth) and the `summary` module; the one survivor, `vix_tone`, moved into `compute.rs`. Dropped the `Summary` stream variant, the scheduler's session-flip + intraday publishes, `sse_summary`, and the `finance:summary` re-broadcast in `stream.js`.- **Trimmed `/health` to the demand-only reality.** `job_meta` / `job_rank` now carry only the surviving `intraday` (and `prune`) jobs instead of arms for the eight removed sweeps; the page lede was rewritten to "fetches market data on demand … the only timed job is the live intraday poll." Fixed a "1 jobs" pluralization in the systems verdict.- **Data-age on the dashboard reads.** `market_reads` now carries an `asof` (freshest quote across ^SPX/^VIX/SPY); the reads strip shows "Prices as of {time}", kept live by `hero.js` — matching the symbol page's per-section ages.- **Verified on dev:** `cargo build` + `vite build` clean, zero warnings; `/api/health` lists only the `intraday` job with the on-demand guards (sec/yahoo) healthy and the Yahoo budget reflecting on-demand calls; the dashboard chart + reads + "Prices as of …" caption render with zero console errors and no Summary references left in the running app; `/health` renders the new copy.Phase D follow-up (dashboard freshness + graph legibility, from user feedback afterthe close): the dashboard felt stale — "prices as of 7:40pm" while it was 8:25pm —because it only updated via the market-hours intraday poll (which skips non-futureswhen the market is closed, since stocks/ETFs don't trade then) and had **no on-openrefresh** like the symbol pages. Fixes:- **On-open refresh.** New `GET /api/dashboard/refresh` → `scheduler::refresh_quotes` pulls fresh quotes for the watchlist + baseline (^SPX/^VIX/SPY) once when the page opens, regardless of session, skipping anything quoted in the last ~5min (so a reload doesn't re-hit Yahoo). `hero.js` calls it on init, then redraws — so opening the dashboard always shows current data and a current "as of" time. Verified: the caption jumped 7:40pm → 8:32pm on open (`refreshed: 7`) with the market closed.- **Closed-state clarity.** The banner sub now reads "Prices update during market hours" when closed (was a static "all times ET"), so a frozen price reads as expected, not broken.- **Graph legibility.** Each line now carries a `title` (its ticker / "S&P 500") that lightweight-charts labels at the line's last value on the price axis, and the palette was swapped for eight well-separated hues — so each line is identifiable by name + colour, not colour alone. Verified at the close: IAU / BND / S&P 500 / VTI / VXUS / IBIT each labelled on the axis, zero console errors.Phase C outcome (dashboard rework: session watchlist + %-vs-S&P-500 hero graph):- **Session-scoped watchlist.** New `watchlist` table (migration `0015`) keyed on an opaque `fin_sid` cookie (no accounts; minted via SQLite `randomblob`, no new crate). `src/watchlist.rs` owns the session resolve/seed/list/add/remove; a first visit seeds the starters (VTI, VXUS, BND, IAU, IBIT) and sets the cookie, an existing cookie's list is used as-is (even empty — a user who clears it isn't re-seeded). `routes/watchlist.rs` exposes `POST /api/watchlist` (add) + `/remove`. Add reuses a new `ensure_symbol` (refactored out of `add_symbol`) so a brand-new ticker is validated + backfilled into the universe first.- **Normalized %-vs-S&P-500 hero graph.** `GET /api/dashboard` returns the day's series (S&P baseline + each watchlist symbol, each as % change from the session's first bar) plus the market reads. `home/scripts/hero.js` draws them on one lightweight-charts axis (ink baseline + a muted non-semantic palette), with a legend, re-fetched every 60s and on tab focus. Symbols with no intraday bars are cleanly dropped from the graph (their card still shows "no intraday data").- **Market reads kept (user's call):** S&P level/move, VIX + tone (reuses `summary::vix_tone`), market **volume** (proxied off SPY — cash indexes carry no share volume on Yahoo — today vs its 65-day average → Heavy/Normal/Light), and the S&P's **50/200-day** stance (`compute::sma`). SMA 50/200 stay on symbol charts; the open sub-decision (a daily SPX chart on the dashboard) was resolved as **not now** — the dashboard carries the 50/200 read as a one-line trend stat instead.- **Market-hours banner** (Regular / Pre-market / After hours / Market closed) with a coloured session dot, kept live by hero.js.- **Watchlist cards** reuse the `spark-card` markup (so the base stream client live-ticks price + sparkline) with a hover remove button; an add box sits in the header. Add/remove reload the page so the cards, the stream tickers, and the graph re-sync.- **~5-minute poll throttle** added to `run_intraday`: a viewed symbol quoted within the last ~4m45s is skipped, so a dashboard left open polls each symbol about once every five minutes (light on budget, plenty for delayed data). The baseline reads (^SPX/^VIX/SPY) carry `data-ticker` so they stay polled while the dashboard is open.- **Stripped the old dashboard** entirely: home.rs's breadth / movers / ETF band / quality leaderboard / hero verdict code and structs, the old `home.js` (+ its `finance:summary` patcher), and the dead `compute::trailing_return`.- **Verified on dev (curl + Playwright):** migration applies on boot; first visit sets `fin_sid` and seeds the 5 starters; `/api/dashboard` returns the baseline + watchlist series (with real intraday point counts) and all four reads (S&P +0.22%, VIX 16.06 Steady, volume 54.4M Light, "Above its 50- and 200-day"); add (TSLA) and remove (via both API and the UI buttons) work and persist across reloads; the hero overlay draws all five lines as % from open with the S&P baseline; the banner reads "Market closed"; clean at desktop (1280, 4-up reads) and mobile (390); zero console errors; `cargo build` + `vite build` clean with zero warnings.- **Known limitations (Phase D / backlog):** (1) because symbols are polled on the ~5-min cadence and were last touched at different times in dev, the graph's series can span slightly different day-windows, showing a small x-axis gap; with all dashboard symbols polled together during live hours they align. (2) The dead `StreamEvent::Summary` machinery was **removed in Phase D** (`vix_tone` moved to `compute.rs`). (3) Pre/regular/after-hours **shading** on the graph itself was not built — the banner carries the session; shading is a backlog polish candidate.Phase B outcome (on-demand symbol data + loading bar + data age + manual refresh):- **New SSE refresh pipeline.** `scheduler::refresh_plan` decides which steps a symbol's refresh runs (the two price steps — live quote, daily history — always; the slow SEC / metadata steps only when their `*_synced_at` is stale, or on `force`). `scheduler::refresh_step` runs each, reusing the existing guarded `backfill_*` helpers. A new SSE route `GET /api/symbols/{ticker}/refresh?force=` (`routes::symbols::refresh_stream`) streams a `plan` event, a `step` event before and after each step, and a final `done {reload}`.- **Step set:** stock → Live quote · Daily history · Fundamentals/filings/leadership (SEC) · Earnings · Sector & industry · Dividends. ETF → Live quote · Daily history · Holdings & filings (SEC) · Fund details & NAV · Distributions. Index/future → Live quote · Daily history. History uses an **incremental** since-last-bar fetch (deep `range=max` only when no history exists). The quote step publishes to the hub so an open page patches its price live; the ETF NAV step re-adds a small `store_fund_nav` and uses the (now un-`allow(dead_code)`-ed) `yahoo.fund_nav`.- **Frontend (`symbol/scripts/refresh.js` + markup + `symbol.scss`):** a header Refresh button + status text + a thin progress bar. On load the page auto-runs a refresh (`force=0`); the button runs `force=1` (everything). The bar fills per step and names the current one. On `done`: if a **deep** (server-rendered) section was refreshed it reloads to show the new data + age; otherwise the live price was already patched and it just settles. A one-shot `sessionStorage` skip flag set before our own reload prevents that reload from re-triggering the auto-refresh — **no reload loop**.- **Data age everywhere (already present, now honest):** every section heading already showed "synced … ago" off its `*_synced_at`; the reload refreshes them. The stale "lands on the next sweep" empty-state copy was rewritten to "pulled on demand — hit Refresh".- **Staleness windows:** SEC fundamentals/filings 7d, leadership 30d, Yahoo metadata (earnings/profile/dividends/fund metadata) 7d. `force` ignores them.- **Verified on dev (Playwright + curl):** the SSE emits plan → per-step running/ok → done; AAPL `force=0` ran quote+history+earnings+profile (the stale ones) and left fundamentals/dividends (4d) and leadership (11d, inside its 30d window) untouched, each showing its true age. A fresh symbol (NVDA) did a single reload then settled ("Updated just now", no loop, zero console errors). The manual Refresh button on VTI ran the full ETF step set with the bar reading "Updating: Distributions", reloaded once, and settled. (Live in-market price tick couldn't be exercised — after-hours during the check — but the quote step stores + publishes exactly as the intraday job does.)Phase A outcome (strip background sweeps + remove Industries):- **Scheduler gutted to demand-only.** Removed `run_history`, `run_sec`, `run_dividends`, `run_fund_metadata`, `run_fund_nav`, `run_earnings_calendar`, `run_asset_profile`, `run_daily_close_if_due` (and the `is_due` / `schedule_next` due-check helpers + their constants, ~1340 lines). The loop now only broadcasts market-session changes (+ a local-DB summary on a session flip), runs the demand-driven `run_intraday` (viewed symbols only), and the local `run_prune`. The per-symbol on-demand pull (`backfill_symbol` + its `store_*` / `backfill_*` helpers) is **kept** — it is the on-demand path Phases B/C build on.- **Boot seed is metadata-only.** `run_boot_seed` now just calls `seed::sync_universe` (local upsert + prune, no network); the history backfill is gone from boot. `make seed` (the `seed` subcommand) keeps `seed::run` for an explicit, user-invoked bulk backfill.- **Removed jobs swept from `/health`.** `register_endpoints` now also `DELETE FROM data_status WHERE job NOT IN ('intraday','prune')` so a prod DB's old job rows don't linger as stale.- **Industries removed entirely:** deleted `src/routes/industries.rs`, the `industries` frontend entry + its Vite input, both `industries_*` templates, the home "Today's industries" band (`IndustryRow` / `industry_panels` + context vars + the now-dead `StockRow.sector` / `asset_profile_synced_at` fields), the `compute::industry_returns` / `seasonality` block + its tests, and every `/industries` nav/footer link. The symbol-page sector/industry tags are now plain `<span>`s (no link). The sector/industry *symbol data* (Yahoo assetProfile) is kept.- **Retained-for-next-phase (marked `#[allow(dead_code)]`):** `yahoo.fund_nav` (Phase B on-demand NAV), `market::{et_date,is_et_weekday,after_close}` (Phase C market-hours), `db::get_meta`.- **Verified on dev:** `cargo build` + `vite build` clean, zero warnings; boot does only local work (universe sync 562 + prune), and an 8s idle window makes **zero** outbound calls; `/` → 200 with no "industr" remnants, `/industries` → 404, `/api/health` lists only `intraday`. (Live intraday couldn't be exercised — no open market / viewer during the check; the path is unchanged from before.)---
@@ -252,58 +251,64 @@ monospace ledger figures, restrained serif headings. Tokens are CSS customproperties in `base.scss :root`.**Color is semantic and sparing.** Green / amber / red mean good / ok / bad(price moves, health reads, data-health states) — never decoration. Chart(price moves, data-age states, data-health states), never decoration. Chartindicator lines are the one deliberate exception (a muted non-semantic palette).**Scannability is the bar.** The user must be able to:- Land on the **dashboard** and tell *how the market is doing TODAY* in one glance — a one-line plain verdict + the index strip + market breadth.- Land on a **stock** and immediately read its health, trajectory, and key figures without hunting.- Land on an **ETF** and immediately read what it holds, what it costs, and how it's trending — with a clear visual separation from stocks.**Data age is always visible and honest.** Because data is now fetched on demandrather than on a timer, every figure on the site carries a clear, plain age read("live", "2m ago", "stale, refreshing", "as of Fri close"). A stale figure isnever shown as if fresh. A refresh in flight shows a loading bar, not a spinnerguessing-game.**Market hours are explicit.** The dashboard makes the current session(pre-market / regular / after-hours / closed) unmistakable, and the day graphdelineates those periods rather than drawing one undifferentiated line.**Scannability is the bar.** Land on the dashboard and read how your watchlist isdoing against the market today in one glance. Land on a symbol and read itsprice, trend, and key figures (with their ages) without hunting.**Dual-first, not mobile-first-only.** Desktop is information-dense and should*use* its space; mobile distills to the key signals (clear hierarchy, nothingimportant below a second scroll). Neither is an afterthought.*use* its space; mobile distills to the key signals. Neither is an afterthought.**Polish last.** Features land first; one focused UI polish pass closes eachvisual phase rather than nibbling polish mid-build.**Polish last.** Features land first; one focused UI polish pass closes the workrather than nibbling polish mid-build.---## Data-source policy (the important reference)All data is **free, no account, no API key.** The user considers *never hittinga rate limit* critical: every outbound call goes through a persistentAll data is **free, no account, no API key.** The user considers *never hitting arate limit* critical: every outbound call goes through a persistent`EndpointGuard` (DB-backed reactive circuit breaker + hard per-hour budget +request pacing; survives restarts; see `src/guard.rs`).request pacing; survives restarts; see `src/guard.rs`). The demand-only refocus*reduces* outbound traffic sharply, since idle time now means zero calls.**Yahoo Finance is the only price source (as of 2026-05-30; Stooq removed).****Yahoo Finance is the only price source.**- **Deep daily history:** `v8/finance/chart?interval=1d&range=max` returns a symbol's entire daily OHLCV in one call. Used once per symbol to backfill `daily_prices`.- **Daily updates:** the once-a-day `daily_close` job already touches every symbol; it appends that day's bar to `daily_prices` (no extra requests).- **Intraday + live quotes:** `v8/finance/chart?interval=15m&range=1d`. symbol's entire daily OHLCV in one call. Fetched on demand when a viewed symbol's stored history is stale or absent (no more periodic sweep).- **Intraday + live quotes:** `v8/finance/chart?interval=15m&range=1d`. Polled only for symbols a browser is currently viewing.- **ETF / fundamentals metadata:** `v10/finance/quoteSummary` (crumb-gated; the provider does the `fc.yahoo.com` primer + `getcrumb` dance, caches the crumb, rotates on 401/403). Modules: `fundProfile`, `calendarEvents`, `assetProfile`.- Budget: 1000 req/hr on the `yahoo` guard. 429/503/401/403 surface as the typed `RateLimited` the guard trips on. Fetched on demand for a viewed symbol when stale.- Budget: 1000 req/hr on the `yahoo` guard.**SEC EDGAR** (no key, contact email in User-Agent; 600/hr guard): stockfundamentals (`companyfacts`), filings (`submissions`), leadership (Form 3/4/5),ETF holdings/AUM (N-PORT, `company_tickers_mf.json`). Indexes don't file.**Freshness tiers (deliberate, to stay on-budget):**- **Live intraday (SSE-polled):** ONLY the dashboard's headline indexes and the single symbol whose page is currently open. Demand-driven via the viewer- interest registry in `src/stream.rs` — nothing is polled when nobody's watching.- **Daily close:** the entire rest of the universe. One snapshot per trading day.- A viewed fund during live market hours shows today's real-time intraday on its chart (Phase 6).ETF holdings/AUM (N-PORT, `company_tickers_mf.json`). Fetched on demand for aviewed symbol when its SEC data is missing or stale. Indexes don't file.**Freshness model (demand-only):**- **Live intraday (SSE-polled):** ONLY the symbols a browser is currently showing, via the viewer-interest registry in `src/stream.rs`. A dashboard watchlist symbol refreshes ~every 5 minutes while watched; an open symbol page refreshes faster. Nothing is polled when nobody's watching.- **History + deep data:** fetched on page load when stale, and on manual refresh. There is no background sweep keeping idle symbols warm; a symbol nobody visits simply holds whatever it last had, and is refreshed the next time it is viewed.---
@@ -315,29 +320,30 @@ migrations in `migrations/` applied on boot.- **`src/providers/`** — one trait per concern: `QuoteProvider` / `HistoryProvider` (Yahoo), `FundamentalsProvider` (SEC). `http.rs` builds the shared reqwest clients. (Stooq provider removed in Phase 1.) shared reqwest clients.- **`src/guard.rs`** — the persistent per-endpoint `EndpointGuard` (see policy).- **`src/scheduler.rs`** — one long-lived 60s-tick tokio task running market-hours-aware jobs (history backfill, daily_close, demand-driven intraday, SEC sweep, dividends, fund_metadata, earnings, asset_profile, prune). Each writes `data_status` + `fetch_log` and pings the stream hub.- **`src/scheduler.rs`** — the 60s-tick tokio task. **Shrinking in Phase A** to just: broadcast market-session changes, run the demand-driven intraday poll (viewed symbols only), and run the local prune. All timed network sweeps are being removed.- **`src/stream.rs`** — `tokio::broadcast` hub + per-ticker viewer-interest registry; `/stream` SSE forwards quote / market / health events. registry; `/stream` SSE forwards quote / market / health events. This registry is the heart of the demand-only model and stays central.- **`src/market.rs`** — US session clock (Closed/Pre/Regular/Post) via `chrono-tz`. No holiday calendar (deliberate).- **`src/compute.rs`** — pure numeric code: indicators (sma/ema/rsi), graded fundamental ratios, health read, range-meter positions, sparkline SVG. fundamental ratios, range-meter positions, sparkline SVG.- **Templates** — minijinja in `templates/` with a Jinja2-faithful HTML formatter (so `/` isn't escaped in URLs). **Frontend** — Vite from `frontend/static_src/` (entries: base, home, symbol, health, search), built with bun, served hashed at `/static/`. formatter. **Frontend** — Vite from `frontend/static_src/` (entries: base, home, symbol, health, search; the `industries` entry is being removed), built with bun, served hashed at `/static/`.**Key tables:** `symbols` (universe + denormalized latest price/snapshot +per-source `*_synced_at` staleness columns), `daily_prices` (permanent deepOHLCV), `intraday_bars` (15m, pruned 14d), `quotes`, `fundamentals` (long/narrowSEC facts), `filings`, `dividends`, `fund_profiles` + `fund_holdings` (ETFN-PORT), `fund_metadata` (ETF Yahoo data), `leadership`, `picks` (beingreworked), `endpoint_guard`, `data_status`, `fetch_log`.per-source `*_synced_at` staleness columns), `daily_prices` (deep OHLCV),`intraday_bars` (15m, pruned 14d), `quotes`, `fundamentals`, `filings`,`dividends`, `fund_profiles` + `fund_holdings`, `fund_metadata`, `leadership`,`endpoint_guard`, `data_status`, `fetch_log`. **New in Phase C:** a sessionwatchlist table (sid -> tickers).`kind` values: `stock`, `etf`, `index`, `future` (commodities/futures).
@@ -346,400 +352,210 @@ reworked), `endpoint_guard`, `data_status`, `fetch_log`.## RoadmapPhases are ordered but reorderable; each is a context-clearing breakpoint thatends verified + committed + deployed. Pain-point mapping to the user's brief:data/guardrails → P1; ETFs first-class → P4; distill/cohesion → P3,P5,P7;drop short-horizon → P3; live intraday for viewed fund → P6.### Phase 1 — Yahoo-only data layer ✅ DONE (deployed 2026-05-30, `76f38f4`)Kill the rate-limit problem at the root.- Remove the Stooq provider, the `stooq` `EndpointGuard`, `STOOQ_APIKEY` config, the per-symbol Stooq history job, and the seed's Stooq backfill path.- Add `YahooProvider::daily_history` (`interval=1d&range=max`) → full daily OHLCV; route through the `yahoo` guard with pacing.- Seed/backfill `daily_prices` from Yahoo for symbols missing deep history (one-time, paced under 1000/hr).- `daily_close` appends the day's bar to `daily_prices` from data it already fetches (no extra requests), so there's no recurring per-symbol history sweep.- Verify: guard never trips during a full backfill; `daily_prices` populates; charts render; `/health` shows no stooq endpoint.### Phase 2 — Universe curation ✅ DONE on dev (commit + deploy pending)- ✅ Reconciled the 503 stocks to the current S&P 500 (fetch + diff: exact match).- ✅ Curated ETFs to iShares + Vanguard (43 total): dropped other-issuer thematic/sector funds, kept SPY/QQQ/DIA/GLD/SLV proxies, added the most-common Vanguard + iShares funds incl. core holdings IAU/IBIT/VBIL. Issuer/category tags reuse the Yahoo `fund_metadata` (no new schema).- ✅ Kept the 6 indexes and all 10 futures (4 index-futures + 6 commodities).- ✅ `seed::sync_universe` now reconciles (upsert + prune) on every boot, so CSV edits take effect on deploy. Pruned 7 stale non-S&P leftovers in passing.- ✅ Fixed Yahoo `range=max` downsampling (provider 10y fallback + `run_history` self-heal); every seeded symbol now holds true daily bars.### Phase 3 — Drop short-horizon prediction → quality leaderboard + home de-dup ✅ DONE (deployed 2026-05-30)- ✅ Removed all four pick rankers (Day/Week *and* Month/Quarter), the backtest machinery, the scheduler snapshot job, and the backtest-only models helpers.- ✅ Merged Top picks + Stock health + Strongest & weakest into a single non-advice **Quality leaderboard** (Healthiest / Most concerning) driven by `compute::health_read`, trailing-year return folded into each row.- ✅ Dropped the `picks` table via migration `0013_drop_picks.sql` (+ swept stale status rows). `compute::standing` retained for the movers badge + symbol/search pages.- Note: the leaderboard's home-page placement is intentionally provisional — Phase 5 rebuilds the dashboard (hero + bands) around it.### Phase 4 — ETFs as true first-class citizens ✅ DONE (deployed 2026-05-30, `19d0b14`)- ✅ Blended ETF **quality score** (`compute::etf_quality`): cost-weighted blend of cost (40%) / tracking (25%) / diversification (20%) / size (15%), composite −1..1 → percent, four sub-reading chips — structurally mirrors the stock `health_read` donut. Renormalises over gradeable factors (commodity trusts with no holdings drop diversification cleanly); shows only with ≥2 factors.- ✅ Distinct ETF symbol-page identity: the quality donut anchors the header top-right (reusing the `health-badge` styling), hover reveals the four sub-readings + non-advice note. The ETF already shows fund sections (holdings, expense/yield, NAV/premium, sector/geo, trailing returns vs benchmark) and no stock fundamentals — so the badge + `ETF` tag complete the distinct identity.- ✅ **Tracking now backed by a real daily NAV.** Discovered the 30-day metadata cadence made premium/discount unreliable; added a dedicated daily `fund_nav` scheduler job + `nav_synced_at` column (migration `0014`) so NAV is current, and a freshness gate that drops the tracking factor to `—` when NAV is stale rather than assert a bogus premium. (See Decisions log for the full story.)- ⏸ **Distinct ETF band on the dashboard — deferred to Phase 5** (user's call): the dashboard is fully redesigned in P5, so the ETF band is built there once rather than built twice.- **Verified on dev:** `cargo build` clean; migration `0014` applies on boot; `/health` lists the new "ETF NAV" job; every ETF page renders the quality donut (VOO 89% Strong, GLD 64% with diversification correctly `—`, BND 100% on its two graded factors); the freshness gate correctly drops tracking to `—` while NAV is stale (screenshot reviewed). **One runtime path still unverified live:** the daily `fund_nav` Yahoo fetch — the dev `yahoo` guard's breaker was open (normal post-deploy back-off) so the job stopped early 0/43; the fetch reuses the proven `quoteSummary` NAV parse, and prod exercises it on first deploy once its guard is healthy.### Phase 5 — Dashboard redesign: "how is the market doing TODAY" ✅ DONE (deployed 2026-05-30, `656e21d`)- ✅ Hero: two-line plain-language verdict (blended index move + breadth + VIX tone) + compact index strip + headline figures + non-advice note. Direction tracks the broad index sign so it never contradicts the figure shown.- ✅ Breadth band: advancers/decliners + % S&P green + a proportion bar. (Sector leaders/laggards stay in their **own** Industries band, per the user's call — not folded into breadth.)- ✅ ETF band (the Phase-4 deferral): 5 curated quality cards + a gainers/losers strip, each carrying the Phase-4 quality verdict.- ✅ Clearly labeled bands, dual-first density: Hero · Indexes · Breadth · ETFs · Stock movers · Industries · Risk & commodities · Quality leaderboard.### Phase 6 — Symbol-page distillation + live intraday on chart ✅ DONE on dev (commit + deploy pending)- ✅ Live intraday surfaced as **1D / 1W range buttons** (the user's call over a single stitched-on candle): 15m bars from `intraday_bars` on a minute axis, daily-only overlays/toggles suppressed, a dashed prev-close reference line.- ✅ Live tick of the trailing bar via a re-broadcast `finance:quote` event + a 60s local-DB re-fetch — no second EventSource. (Live path runs first in prod; a closed-market weekend can't stream quotes on dev.)- ✅ Mobile above-the-fold reorder: full chart **promoted** (no separate mini chart, user's call); donut is desktop-only and a flattened full-width **verdict line** (verdict · % · trajectory) sits under the price on mobile. Order: price/change → verdict → chart.- ✅ Fixed a latent `.ind-btn[hidden]` override bug found in passing.### Phase 7 — Health/systems page distillation + final polish pass ✅ DONE (deployed 2026-05-30, `645b351`)- ✅ **`/health` distilled with a top systems verdict** (the chosen shape): a one-line plain read above the existing detail, computed live in `health.js`.- ✅ **Two-tier footer built** (`footer__grid` + `.footer-bar`, Paper Ledger), data attribution folded into the About column.- ✅ **Live market summary** (the Phase-5 hero/breadth deferral): `src/summary.rs` + `StreamEvent::Summary` + scheduler publishes + `home.js` patches. Live path runs first in prod (closed-market weekend can't stream on dev).- Light cohesion only (per the scope decision); no full cross-page audit.- See Status + the 2026-05-30 Phase 7 decisions log for the full outcome.Original Phase 7 brief (kept for reference):- Distill `/health` and overall cross-page cohesion; one closing UI polish pass.- **Expand the footer to match the sibling-project pattern.** Finance currently has a single-line footer; the user wants it grown "a lot" to match how the other apps do footers. The house pattern (analytics, status, blog, repos) is a **two-tier footer**: a multi-column upper `<footer>` (columns like About / Pages / Links — nav links, cross-project links, Portfolio/GitHub/LinkedIn, and a "Source" link to `github.com/overshard/finance`) over a slim `.footer-bar` with `© {{ now.year }} Isaac Bythewood · Some rights reserved` + a GitHub icon link. `repos`'s `footer__grid` with `// LABEL` column headers is the closest fit for the Paper Ledger aesthetic — model finance's on that. Fold the data attribution below into one of the columns.- **Data-attribution (partially done early):** the stale **Stooq** credit was a factual bug (Stooq removed Phase 1), so the one-line footer was corrected on 2026-05-30 to "Market data via Yahoo Finance · Fundamentals via SEC EDGAR · not investment advice" ahead of schedule. The full footer build above still belongs to this polish pass. Yahoo's chart endpoint is unofficial (no published ToS/attribution requirement), but a tasteful credit is honest and suits the professional face the user wants.ends verified + committed + deployed.### Phase A — Strip background sweeps + remove Industries ✅ DONE on dev (commit + deploy pending)Make the app demand-only and delete the dead surface, before building the new one.See the Status block above for the full outcome.- **Gut the scheduler.** Remove the `daily_close`, `sec`, `dividends`, `fund_metadata`, `fund_nav`, `earnings_calendar`, `asset_profile`, and periodic `history` jobs and their bring-forward calls. The loop keeps: market-session broadcast, `run_intraday` (demand-driven), and `run_prune_if_due`. The `EndpointGuard` and `data_status` / `fetch_log` plumbing stay (the on-demand fetches in Phases B/C still record through them).- **Stop the seed's automatic history backfill.** First-run seed still creates the universe rows (metadata only, local) via `sync_universe`, but no longer fetches any history over the network. A symbol's history is filled the first time it is viewed (Phase B).- **Remove Industries entirely:** the `industries` route + `/api/industries`, the `industries` frontend entry, the `industries_*` templates, the home Industries band, and the nav link. Drop the industries-only compute if unused elsewhere.- **Verify:** with no browser connected, the app makes zero outbound calls over a several-minute idle window (watch `fetch_log` / guard counters); `/industries` → 404; home still renders (old bands, trimmed of Industries) until Phase C replaces it; `/health` no longer lists the removed jobs.### Phase B — On-demand symbol data (loading bar + data age + manual refresh) ✅ DONE on dev (commit + deploy pending)Turn the symbol page into a pull-on-demand surface. See the Status block above forthe full outcome.- **On load:** if the symbol's quote / intraday / daily history is stale or absent, trigger a fresh Yahoo pull. Slow SEC data (fundamentals, filings, holdings, NAV) is pulled only when missing or past its staleness window. A manual **Refresh** button re-pulls everything.- **Loading bar over the whole pull.** A clear progress indicator that names each step as it runs (prices → history → fundamentals → filings → ...), driven by the existing SSE stream or a dedicated per-symbol refresh endpoint that reports progress. Guard-denied steps surface plainly ("rate-limited, showing cached").- **Data-age on every block.** Each figure/section shows a plain age read derived from its `*_synced_at` timestamp ("live" / "3m ago" / "as of Fri close" / "stale").- **Respect the guard.** All pulls route through the `yahoo` / `sec` guards; an open breaker degrades gracefully to cached + a clear note, never a hammer.- **Verify:** loading a cold symbol fills its chart + stats live with the bar advancing; data-age reads correctly; manual refresh re-pulls; a tripped guard shows cached-with-note; no console errors at mobile + desktop.### Phase C — Dashboard rework: session watchlist + %-vs-SPX hero graph ✅ DONE on dev (commit + deploy pending)See the Status block above for the full outcome.Rebuild home around the watchlist and the day graph.- **Session watchlist.** An opaque `fin_sid` cookie identifies a browser session (no accounts); a `watchlist(sid, ticker, position, added_at)` table holds its symbols, seeded with VTI, VXUS, BND, IAU, IBIT on first visit. Add/remove UI on the dashboard. Clearing cookies loses the list (acceptable, by design). SPX is a fixed, non-removable baseline.- **Hero graph = normalized %-vs-SPX overlay.** All watchlist symbols + SPX on one intraday chart, each as % change from today's open (TradingView/Google compare style), SPX as the visual baseline. The graph delineates pre / regular / after-hours and shows "closed" state clearly. Falls back to the most recent trading day on weekends/holidays.- **Watchlist rows/cards** beneath: price, day change, a sparkline, and a data-age read each. Refresh ~every 5 minutes while the dashboard is open (demand-driven: the page registers its watchlist tickers with the interest registry; the intraday poll honors a ~5-minute per-symbol cadence for dashboard viewers).- **Keep these market reads on the dashboard (user's call, 2026-06-03):** - **Overall market volume.** An aggregate market-volume read for the day (source TBD during build: the lead index's / a broad-ETF's volume, since we no longer sweep the whole universe for a true summed figure). Shown with its data-age. - **VIX, kept tracking.** ^VIX stays a first-class dashboard read (the risk tone), not dropped with the old hero verdict. It is one of the always-on reads even though it is not a watchlist symbol. - **SMA 50 / SMA 200.** Interesting in general. They stay as chart overlays on symbol pages (Phase B, daily ranges). On the dashboard they only make sense on a *daily* SPX chart, not on the intraday %-vs-SPX hero overlay; decide during the build whether the dashboard carries a small SPX daily chart with the 50/200 SMAs or whether 50/200 live only on symbol pages. (Open sub-decision.)- **Strip the rest of the old home bands** that depended on whole-universe daily data we no longer fetch (breadth, stock movers, the quality leaderboard, the hero verdict sentence). The new dashboard is: session/hours banner + hero comparison graph + the volume/VIX reads + editable watchlist. (Revisit whether any other old band is worth keeping during the build.)- **Verify:** first visit seeds the 5 starters; add/remove persists across reloads in the same browser and resets in a fresh one; the hero overlay normalizes all lines to % from open with SPX as baseline; market-hours labeling is correct in each session; watchlist refreshes on the ~5-minute cadence with live data-age; nothing polls once the dashboard tab is closed.### Phase D — Cohesion + polish pass ✅ DONE on dev (commit + deploy pending)Removed the dead Summary machinery, trimmed `/health` to the demand-only job set +copy, added a "Prices as of" age to the dashboard reads, and fixed a verdictpluralization. (Industries links were already removed in Phase A; the loading bar,market-hours banner, and mobile hierarchy landed in Phases B/C.) See the Statusblock above for the full outcome. **This completes the demand-only roadmap.**### Backlog / parked- Watchlists (tables exist, unused — user wants an opinionated no-customization view for now).- **Popular non-S&P watchlist.** Phase 2 narrowed stocks to exactly the S&P 500, which dropped some popular names the universe used to carry (MSTR, NET, RIVN, SHOP, SNOW, SOFI, MRVL). Add them back as an opt-in "popular / most-watched" band if the user wants them.- Issuer-direct ETF feeds (iShares/Vanguard) if Yahoo/SEC prove thin.- Deep pre-2000 history (lost with Stooq; revisit only if charts feel thin). Note: index/futures daily history via Yahoo caps at ~10y (the `range=10y` fallback); ^SPX/^DJI/^NDX still carry deep daily history from before.- **Scrub Claude/Anthropic trailers from git history (cross-repo, force-push).** User wants every `Co-Authored-By: Claude`, `🤖 Generated with [Claude Code]`, and any Anthropic ad line removed from *all* commit messages in *all* repos. **Survey 2026-05-30: history is already clean** — a strict scan across all 9 `~/code` repos (analytics, blog, darkfurrow, finance, isaacbythewood, repos, status, taproot, timelite) found **zero** such trailers; the only "claude" hits are legitimate references to the `CLAUDE.md` *filename* in commit messages, which must NOT be scrubbed. So this is a no-op today and only matters if a trailer ever slips in. Procedure if needed (do it as its own focused session — it is irreversible): 1. Per repo, rewrite history dropping the offending trailer lines, e.g. `git filter-repo --message-callback` (preferred) or a `filter-branch` fallback, stripping only `Co-Authored-By: Claude*`, `🤖 Generated with*`, and `Generated with [Claude Code]*` lines — never the `CLAUDE.md` mentions. 2. `git push --force-with-lease` to **every** remote (GitHub `origin` *and* the deploy remote `server`). 3. **Server fixup:** the deploy is a bare repo + post-receive hook that builds into the project dir. After a history rewrite the server's checkout will be on an orphaned commit, so SSH to the alpine host (`taproot` manages it) and reset the bare repo's `master` + re-run the deploy (`docker compose up --build --detach`) so `/srv` tracks the rewritten history; verify the container is healthy and reattached to `bythewood-edge`. Coordinate via the `taproot` repo (its CLAUDE.md is off-limits to auto-edits per user rule).- Named multiple watchlists (only a single session list is planned for now).- A "popular non-S&P" quick-add set on the dashboard, if the curated catalog feels thin for adding symbols.- Server-rendered fallback when JS is off (the hero graph + loading bar are JS).- Whether to keep *any* whole-market read (breadth/movers) as an optional, on-demand-only panel rather than deleting it outright.---## Decisions log**2026-05-30 — Phase 7 (health distillation + footer + live breadth) design forksresolved.** Answered 3 clarifying questions before building:1. **/health distillation = add a top "verdict" summary** (like the dashboard hero): a one-line plain-language systems read at the top (`● All systems normal · both data sources healthy · N jobs on schedule · last fetch 4m ago`), keeping the existing Endpoints / Background jobs / Activity-log detail below. Computed client-side in `health.js` from the snapshot so it stays live.2. **Build live breadth now** (not parked). The Phase-5 hero verdict + breadth were page-load snapshots; add a server-pushed market-summary event on the stream hub so the verdict sentence, the headline figures, and the breadth counts/bar recompute intraday instead of going stale beside the live-ticking index chips. (Runs first in prod during market hours — a closed-market weekend can't stream quotes on dev, like Phase 6's live tick.)3. **Polish scope = footer + health properly, light cohesion only.** Build the two-tier footer (modeled on `repos`' `footer__grid`, adapted to Paper Ledger) and the health verdict; do a light spacing/heading cohesion sweep only where quick, rather than a full cross-page audit.Implementation shape: a new `src/summary.rs` owns the verdict vocabulary(`market_verdict` + `vix_tone`, moved out of `home.rs` so page-load and live pushshare one wording) and a cheap `market_summary(pool, session)` (two aggregatereads: breadth over curated stocks + the lead index / VIX). A new`StreamEvent::Summary` carries it; the scheduler publishes it after an intradaysweep that touched a pulse ticker (the broad index or ^VIX) and after daily_close.`base/stream.js` re-broadcasts it as a `finance:summary` window event; a new`home.js` patches the hero + breadth DOM in place.**2026-05-30 — Phase 6 (symbol-page distillation + live intraday).** Answered 3design forks before building:1. **Intraday = dedicated 1D + 1W range buttons** (not a single live candle stitched onto every daily range, and not both). 15m bars on their own minute axis, the conventional Yahoo/Google shape; the daily ranges stay daily. The plan's "stitched onto the daily series" wording was reconciled this way — 15m granularity is invisible on a 1Y view, so a separate intraday tab is the honest surface, with a prev-close reference line tying it to the daily basis.2. **Mobile = promote the full chart** above the fold (no separate mini chart), so there is one chart with its range/indicator controls, just lifted higher.3. **Mobile health = a full-width verdict line** under the price (`● Verdict · NN% · trajectory ↑`); the donut ring stays the desktop treatment. Works for a stock (with the trajectory clause) or an ETF (without).Implementation notes worth keeping: `intraday_bars.ts` is epoch-**ms**, so it is÷1000 for lightweight-charts' UNIX-**seconds** intraday axis; a `BarTime`untagged enum lets one `candles` field serialise a date string (daily) or anumber (intraday). The live tick reuses the existing SSE quotes via are-broadcast `finance:quote` window event — no second connection — plus a 60slocal-DB re-fetch for new bars. Found + fixed in passing: `.ind-btn`'s`display: inline-flex` was beating the `[hidden]` attribute, so neither thebenchmark toggle (on benchmark-less symbols) nor the daily-only toggles (onintraday) actually hid; an `.ind-btn[hidden]` rule re-asserts it. The live tickcouldn't be exercised on a closed-market weekend; it runs first in prod.**2026-05-30 — Phase 5 (dashboard redesign).** Answered 4 design forks beforebuilding:1. **Hero verdict = blended + a touch more.** A two-line read: a punchy lead ("Higher, but narrow.") plus a clause ("Markets higher with narrow participation."), blending the broad index move, breadth (% green), and the VIX risk tone, with the headline figures and a non-advice note beneath.2. **Sectors stay separate.** Breadth band = advancers/decliners + % green only; the existing Industries (sector up/down) panel keeps its own band rather than folding "sector leaders/laggards" into breadth.3. **ETF band = both.** Curated quality cards (VOO/VTI/QQQ/BND/GLD) *and* a compact ETF gainers/losers strip — this is the Phase-4 deferral, now built.4. **Breadth viz = stat strip + proportion bar** (advancers/decliners counts, % green, one up/flat/down bar), not a distribution histogram.Implemented entirely in `src/routes/home.rs` + `home.html`/`macros.html` +`home.scss`, reusing the already-loaded card/stock scans so the hero and breadthadd **zero** extra queries (only the ETF band adds two: the ETF roll-up and atop-10-holdings window query). Found + fixed during the screenshot review: theverdict could contradict the figure beside it — a +0.12% S&P with weak breadthread "Broadly lower" because the ±0.15% direction deadband swallowed the move andlet breadth flip the sign. Fixed by tightening the deadband to ±0.05% and makingdirection track the broad index's sign, with breadth breaking ties only when noindex price exists. Deferred live updates for the hero/breadth (page-loadsnapshots) to the Phase 7 polish pass.**2026-05-30 — Phase 4 build + the NAV-staleness discovery.** Built the ETFquality read (`compute::etf_quality`, mirroring `health_read`) and thesymbol-page quality donut. **Mid-build discovery:** the tracking factor(premium/discount to NAV, the option the user picked) was reading "Wide gap" onfunds that track perfectly (VOO, IVV) because Yahoo's NAV is only refreshedevery 30 days (`FUND_METADATA_STALE_SECS`) — so a live price compared to aweeks-old NAV showed a bogus premium. This broke the premise behind the user'stracking choice. Surfaced it and asked; user chose **refresh NAV daily** sotracking becomes a true daily read. Implemented as:- a dedicated daily `fund_nav` scheduler job (separate from the 30-day static metadata sweep so the slow fields keep their cadence) that fetches NAV only (`YahooProvider::fund_nav`, two `quoteSummary` modules) through the `yahoo` guard — ~43 req/day, negligible vs the 1000/hr budget;- a `nav_synced_at` column (migration `0014`) the daily job stamps;- a **freshness gate** on the symbol page: the tracking factor is graded only when NAV was synced ≤3 days ago, else it drops to `—` and the cost/div/size blend renormalises — so a stale NAV never drives a false tracking verdict.Also deferred the **dashboard ETF band to Phase 5** (it rebuilds the dashboardanyway) and corrected the footer's stale Stooq credit early (a factual bug).The `fund_nav` live fetch wasn't exercised on dev (the guard's breaker was openpost-deploy), but it reuses the proven NAV parse and runs on first prod deploy.**2026-05-30 — Phase 4 (ETFs as first-class citizens) design forks resolved.**Answered 3 clarifying questions before building:1. **ETF quality score = cost-weighted, all four factors.** Cost (expense ratio) ~40%, Tracking ~25%, Diversification ~20%, Size (AUM) ~15%. Composite in [-1,1] → percent, with four sub-reading chips — structurally mirrors the stock `health_read` donut. Weights renormalize over whichever factors are gradeable (commodity trusts like GLD/SLV/IBIT have no holdings, so diversification drops out and the rest reweight). Show the badge only when ≥2 factors are gradeable.2. **Tracking factor = premium/discount to NAV**, reusing the existing `compute::premium_discount_pct` + `premium_grade` (price vs Yahoo NAV). No new compute, no benchmark-alignment work. (True tracking error vs benchmark was considered and parked.)3. **Dashboard ETF band deferred to Phase 5.** Phase 4 ships the quality score + the distinct ETF symbol-page identity (own quality donut + sub-readings in the header, alongside the fund sections that already exist). The dashboard ETF band gets built into the Phase 5 dashboard redesign rather than built twice.Grading bands (initial, tune later): cost cheap ≤0.10% / pricey >0.50%;tracking via `premium_grade` (±0.25% tight, ±1% wide); diversification on top-10holdings concentration (≤~25% broad, >~50% concentrated); size on log10(AUM)centered ~ $2B ok, ≥ $20B large.**2026-05-30 — Phase 3 (drop short-horizon prediction → quality leaderboard).**Executed the kickoff decision to drop prediction. Resolved the one open designfork (what the unified leaderboard ranks by) without blocking: used the existing`health_read` composite — it is the "healthiest" read the plan named and alreadyblends strength + trajectory + leadership stability — rather than inventing a newscore or reusing `standing`. Folded the old strongest/weakest panel'strailing-year return into each leaderboard row so no useful signal was lost.Kept the home-page treatment deliberately light (a straight three-into-onede-dup) because Phase 5 redesigns the dashboard around this band anyway. Foundduring the work: `movers` reuses `compute::standing` for its strength badge, so`standing` stays (only the strongest/weakest *panel* was removed); and therunning dev server had to be restarted to pick up the new binary + run thedrop-picks migration.**2026-05-30 — Phase 2 (universe curation).** Answered 4 clarifying questions:1. **ETF roster:** keep iShares + Vanguard + the SPY/QQQ/DIA/GLD/SLV proxies; drop other-issuer thematic/sector funds; add the most-common Vanguard + iShares funds. Landed at 43 ETFs.2. **Core holdings:** "get the most common Vanguard and iShares funds" (not just the named IAU/IBIT/VBIL) — drove the broadened ETF set above.3. **ETF tags:** reuse Yahoo `fund_metadata.category` / `fund_family` for the dashboard band separation; no new schema/migration.4. **S&P 500 recon:** fetch the live constituent list and diff to match exactly (result: already an exact match).Plus, surfaced and fixed during verification: the seed never reconciled onre-boot (now does, via `sync_universe`), and Yahoo `range=max` silentlydownsamples `interval=1d` to weekly/monthly for long-history symbols (fixed inthe provider + a `run_history` self-heal). See Status for the full outcome.**2026-05-30 — The "distill + ETF-first" refactor kickoff.** User steered abroad refactor; answered 10 clarifying questions. Decisions:1. **Drop next-day/next-week prediction.** Confirmed it's a gamble and the main driver of live-data demand. Day/Week picks removed; Month/Quarter reframed as a non-advice quality leaderboard.2. **Universe = S&P 500 + major indexes + major commodities + iShares/Vanguard ETFs.** Commodities stay.3. **Data goes Yahoo-only; Stooq removed.** Investigated the Stooq bulk download: it's CAPTCHA-gated (authorizes the PHP session, not a reusable token — verified live), so it can't be cron'd; the per-symbol apikey path is reusable but carries the undocumented "Exceeded the daily hits limit" cap that was blocking us. Yahoo `interval=1d&range=max` gives full daily history in one call and `daily_close` already touches every symbol, so Yahoo covers both backfill and updates with no rate-limit exposure. Trade-off: lose Stooq's ultra-deep pre-2000 history (a chart curiosity).4. **Freshness tiers:** live intraday only for dashboard indexes + the open symbol; daily close for everything else.5. **ETF separation:** distinct dashboard bands + distinct ETF symbol-page identity.6. **ETF read:** a blended "quality" score (not a fake "company health").7. **Dashboard hero:** one-line verdict + index strip + breadth.8. **Mobile stock order:** price/change → health verdict → mini chart → trajectory.9. **Cadence:** commit + auto-deploy each verified phase.10. **This session:** rewrite PLAN.md (done), then start Phase 1.**Pre-2026-05-30 history (condensed).** The app shipped Phases 0–31 of theoriginal roadmap: MVP (universe, Stooq history, scheduler+guard, Paper Ledgerredesign, live quotes+SSE, health page, SEC fundamentals+filings, chartindicators, search+add-symbol, commodities, home redesign, ship/Docker), thenpost-MVP work (company leadership, industry trends, ETF profiles, strongest/weakest, data-age captions, financials table, dividends, earnings dates,per-ticker anomaly feed, stock health read, ETF first-class v1, top picks +backtest, and a full UI polish pass). It deployed to production atfinance.bythewood.me. This refactor reworks the prediction, data-source, ETF,and distillation layers on top of that base. The blow-by-blow phase historyfrom before this rewrite lives in git history; it was intentionally droppedfrom this doc to keep it scannable.**2026-06-03 — The "demand-only refocus" kickoff.** The user steered a focus shiftaway from the broad always-on dashboard: the app could not source enough live datato show what they wanted, and the timed sweeps were spending API budget on symbolsnobody was viewing. Answered 4 clarifying questions:1. **Dashboard = a session-scoped editable watchlist.** Cookie-based, no accounts; custom to a browser "session" however we manage it (chosen impl: an opaque `fin_sid` cookie + a `watchlist` table). Clearing the browser loses it, which is fine. Seeded with VTI, VXUS, BND, IAU, IBIT; SPX is the fixed baseline.2. **Hero graph = normalized %-vs-SPX overlay** (TradingView/Google compare style): every watchlist symbol + SPX as % change from today's open, SPX the baseline.3. **Symbol pull scope = prices auto, deep data if stale.** On load, auto-pull fast Yahoo data (quote / intraday / daily history) live with a loading bar; pull slow SEC data (fundamentals / filings / holdings / NAV) only when missing or stale, each carrying a data-age read. Manual refresh re-pulls everything.4. **Drop all timed network sweeps** (including `daily_close`). Nothing happens automatically. History is fetched on loading a page whose data is stale. Keep only the local prune and demand-driven polling. The universe CSV stays as a searchable catalog (metadata only; data fetched on first view).Then riffed three reads to keep on the dashboard despite stripping the old bands:**overall market volume**, **^VIX** (keep tracking), and **SMA 50/200** (interestingin general). Budgeted into Phase C: volume + VIX are always-on dashboard reads; the50/200 SMAs stay as symbol-page chart overlays, with an open sub-decision on whetherthe dashboard also carries a daily SPX chart to host them (the intraday %-vs-SPX herooverlay can't show daily SMAs meaningfully).Roadmap rewritten into Phases A-D above. The previous "distill + ETF-first"roadmap (Phases 1-7) is **superseded** but its outcomes remain live in prod untileach new phase replaces them; that history is condensed below and in git.**Superseded — the "distill + ETF-first" rewrite (2026-05-30, Phases 1-7, deployed`645b351`).** Condensed, since the demand-only refocus reworks much of it:- **P1 Yahoo-only data layer:** removed the Stooq provider/guard/config; Yahoo `interval=1d&range=max` serves deep history. Fixed Yahoo `range=max` downsampling (provider 10y fallback + `run_history` self-heal). *Yahoo-only stays; the periodic history job it added is now being removed in Phase A.*- **P2 Universe curation:** reconciled stocks to the current S&P 500 (exact match); curated ETFs to iShares + Vanguard (43); `seed::sync_universe` reconciles on every boot. *Universe stays as the search catalog; the boot history backfill is being dropped in Phase A.*- **P3 Dropped short-horizon prediction → quality leaderboard:** removed the four pick rankers, `src/picks.rs`, `/backtest`, the snapshot job; dropped the `picks` table (migration `0013`). *The leaderboard band is being removed in Phase C.*- **P4 ETFs first-class:** `compute::etf_quality` (cost/tracking/diversification/ size blend), the ETF symbol-page quality donut, a daily `fund_nav` job + `nav_synced_at` (migration `0014`) + a NAV-freshness gate. *The quality read + ETF page identity stay; the daily `fund_nav` sweep is being removed in Phase A (NAV now fetched on demand for a viewed ETF).*- **P5 Dashboard "how is the market TODAY":** hero verdict + breadth band + ETF band + bands layout. *Being replaced by the watchlist + %-vs-SPX dashboard in Phase C.*- **P6 Symbol-page distillation + live intraday:** 1D/1W intraday range buttons (15m bars on a minute axis, prev-close reference line), live tick via a re-broadcast `finance:quote` event, mobile above-the-fold reorder. *The intraday chart + live tick stay and are the base for Phase B's on-demand pull.*- **P7 Health distillation + footer + live breadth:** `/health` top systems verdict, two-tier Paper Ledger footer, `src/summary.rs` + `StreamEvent::Summary` live market summary. *The footer + `/health` stay (Phase D trims `/health` to the smaller job set); the live market summary/breadth push is being removed with the old dashboard.***Pre-2026-05-30 history (condensed).** The app shipped the original Phases 0-31:MVP (universe, Stooq history, scheduler+guard, Paper Ledger redesign, livequotes+SSE, health page, SEC fundamentals+filings, chart indicators, search+add-symbol, commodities, home redesign, ship/Docker), then post-MVP work (leadership,industry trends, ETF profiles, strongest/weakest, data-age captions, financialstable, dividends, earnings dates, anomaly feed, stock health read, ETF first-classv1, top picks + backtest, UI polish). Blow-by-blow lives in git history.---## Hard-won lessons (don't relearn these)- **Yahoo `range=max` silently downsamples `interval=1d`.** For symbols with long histories (futures, ^RUT/^VIX, multi-year ETFs) the v8 chart endpoint returns weekly/monthly bars even when a daily interval is asked for — and for some it returned a single bar. A *bounded* window (`range=10y`, or explicit `period1`/`period2`) is served at true daily granularity. The provider detects a downsampled `range=max` response (median timestamp gap > 4 days) and refetches 10y; `run_history` self-heals already-stored coarse/single-bar series (it flags a symbol with < 30 bars in its trailing 90 days, replacing them with a daily re-fetch). Detect coarseness from *recent* spacing, not whole-span density — ^SPX has daily data for decades but a sparse pre-1900 tail, and a whole-span test wrongly flags it.- **The seed must reconcile on every boot, not only first-run.** `seed_completed` gates the history *backfill*, but `sync_universe` (upsert + prune) runs each boot so a `starter.csv` edit (symbols added or dropped) takes effect on deploy. Pruning keys on `is_seeded = 1` so user-added symbols are safe.- **Yahoo `range=max` silently downsamples `interval=1d`.** For symbols with long histories (futures, ^RUT/^VIX, multi-year ETFs) the v8 chart endpoint returns weekly/monthly bars even when a daily interval is asked for, and for some a single bar. A *bounded* window (`range=10y`, or explicit `period1`/`period2`) is served at true daily granularity. The provider detects a downsampled `range=max` response (median gap > 4 days) and refetches 10y; detect coarseness from *recent* spacing, not whole-span density (^SPX has daily data for decades but a sparse pre-1900 tail). The on-demand history fetch in Phase B reuses this provider logic.- **The seed reconciles the universe on every boot, not only first-run.** `sync_universe` (upsert + prune) runs each boot so a `starter.csv` edit takes effect on deploy. Pruning keys on `is_seeded = 1` so user-added symbols are safe. (Phase A keeps this; it removes only the *history backfill*, not the metadata sync.)- **Yahoo `quoteSummary` is crumb-gated.** Must prime `fc.yahoo.com` + fetch `/v1/test/getcrumb` with cookies, cache the crumb, rotate on 401/403. Already implemented in `src/providers/yahoo.rs`.- **SEC's `fy` field tags the *filing's* fiscal year, not the period's.** Derive fiscal year from period-end + the company's fiscal-year-end month, or comparatives shift by years. Keep only clean full-year/discrete-quarter durations; skip quarterly balance-sheet figures (10-Q mis-tags prior-year comparatives).- **"No data" / historyless symbols are a clean empty, not a failure** — never feed the breaker for a symbol Yahoo legitimately has no history for. `/v1/test/getcrumb` with cookies, cache the crumb, rotate on 401/403. Already in `src/providers/yahoo.rs`.- **SEC's `fy` field tags the *filing's* fiscal year, not the period's.** Derive fiscal year from period-end + the company's fiscal-year-end month. Keep only clean full-year/discrete-quarter durations; skip quarterly balance-sheet figures.- **"No data" / historyless symbols are a clean empty, not a failure** — never feed the breaker for a symbol Yahoo legitimately has no history for.- **Guard state is shared across server + `seed` subcommand** via SQLite, and survives restarts. A boot-time breaker trip is normal after a deploy that adds a new upstream-backed job; it recovers via the half-open probe.- **Chart indicator lines use a non-semantic palette on purpose** — green/amber/ red are reserved for good/ok/bad; candles own green/red.- **Yahoo NAV is only as fresh as you keep it.** The 30-day `fund_metadata` sweep (`FUND_METADATA_STALE_SECS`) carries a NAV, but comparing a live price to a weeks-old NAV yields a meaningless premium/discount — it reads as a huge fake "premium" on funds that actually track to the basis point. NAV is struck once per trading day; anything that reads a price-vs-NAV premium (the ETF quality read's tracking factor) must run off the daily `fund_nav` refresh and gate on `nav_synced_at` freshness. Don't trust the metadata sweep's NAV for any real-time comparison.- **`cargo` isn't on PATH in this dev container** — use `~/.cargo/bin/cargo`, and run the dev binary with `FINANCE_ROOT=/home/dev/code/finance` (or from the project dir so paths resolve from cwd). survives restarts. A boot-time breaker trip is normal after a deploy; it recovers via the half-open probe.- **Chart indicator lines use a non-semantic palette on purpose** — green/amber/red are reserved for good/ok/bad; candles own green/red.- **Yahoo NAV is only as fresh as you keep it.** Comparing a live price to a weeks-old NAV yields a meaningless premium/discount. The ETF quality read's tracking factor must gate on `nav_synced_at` freshness. In the demand-only model NAV is fetched when an ETF page is viewed and stale, and the freshness gate still drops the tracking factor to `—` rather than assert a bogus premium.- **`cargo` isn't on PATH in this dev container** — use `~/.cargo/bin/cargo`, and run the dev binary with `FINANCE_ROOT=/home/dev/code/finance` (or from the project dir so paths resolve from cwd).
modified
frontend/static_src/base/scripts/stream.js
@@ -244,18 +244,6 @@ export function initStream() { // it is purely a nudge to re-pull /api/health. window.dispatchEvent(new Event("finance:health")); }); es.addEventListener("summary", (e) => { // Re-broadcast the recomputed dashboard summary (Phase 7) so the home // page can patch its hero verdict + breadth without a second connection. // Only the dashboard listens; every other page ignores it. try { window.dispatchEvent( new CustomEvent("finance:summary", { detail: JSON.parse(e.data) }), ); } catch { /* ignore a malformed frame */ } }); } connect();
modified
frontend/static_src/health/scripts/health.js
@@ -218,7 +218,8 @@ function renderVerdict(snap) { } else if (jobErr || jobStale) { jobPart = `${jobs.length - jobErr - jobStale}/${jobs.length} jobs on schedule`; } else { jobPart = `${jobs.length} jobs on schedule`; const noun = jobs.length === 1 ? "job" : "jobs"; jobPart = `${jobs.length} ${noun} on schedule`; } const parts = [srcPart, jobPart];
modified
frontend/static_src/home/index.js
@@ -1,7 +1,10 @@// Markets dashboard. The live stream runs from the base entry and patches the// sparkline cards in place; this entry ships the dashboard styles and the// Phase-7 live patcher for the hero verdict + breadth band.// Markets dashboard (Phase C). The base entry runs the live stream and patches// the watchlist sparkline cards in place; this entry ships the dashboard styles,// the normalized %-vs-S&P-500 hero graph + market reads, and the watchlist add /// remove controls.import "./styles/home.scss";import { initSummary } from "./scripts/home.js";import { initHero } from "./scripts/hero.js";import { initWatchlist } from "./scripts/watchlist.js";initSummary();initHero();initWatchlist();
added
frontend/static_src/home/scripts/hero.js
@@ -0,0 +1,239 @@// The dashboard's hero day graph + market reads (Phase C).//// Draws every watchlist symbol plus the S&P 500 on one chart, each as % change// from today's open (the TradingView/Google "compare" shape), and fills the// headline reads. Both come from /api/dashboard, re-fetched ~every minute (and// on tab focus) so the chart and reads stay live without a reload — the// watchlist cards below already live-tick via the base stream client.import { createChart, LineSeries, ColorType } from "lightweight-charts";// Non-semantic line palette for the watchlist (green/amber/red stay reserved// for good/ok/bad reads elsewhere). Chosen to spread across the wheel so// adjacent lines stay tellable apart; the S&P baseline is drawn in ink.const PALETTE = [ "#2f6fb0", // blue "#e07b29", // orange "#8a4fb3", // purple "#0f8b8d", // teal "#c2407a", // magenta "#8c6239", // brown "#3b4a9c", // indigo "#6b8e23", // olive];const INK = "#211f1a";const DASH = "·";// Display name for a ticker on the axis label and legend.const displayName = (ticker) => (ticker === "^SPX" ? "S&P 500" : ticker);const SESSION_LABELS = { regular: "Regular session", pre: "Pre-market", post: "After hours", closed: "Market closed",};const pctFmt = (v) => (v >= 0 ? "+" : "") + v.toFixed(2) + "%";function fmtMoney(n) { if (n == null || Number.isNaN(n)) return DASH; return "$" + n.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2 });}function fmtPct(n) { if (n == null || Number.isNaN(n)) return DASH; return n.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, signDisplay: "exceptZero", }) + "%";}// Compact volume: 1.2M / 853K, matching the server-side `compact` filter shape.function fmtCompact(n) { if (n == null || Number.isNaN(n)) return DASH; const abs = Math.abs(n); if (abs >= 1e9) return (n / 1e9).toFixed(1).replace(/\.0$/, "") + "B"; if (abs >= 1e6) return (n / 1e6).toFixed(1).replace(/\.0$/, "") + "M"; if (abs >= 1e3) return (n / 1e3).toFixed(1).replace(/\.0$/, "") + "K"; return String(n);}const cap = (s) => (s ? s.charAt(0).toUpperCase() + s.slice(1) : s);// A fixed timestamp as an ET clock time, matching the server `asof` filter.function fmtClock(ms) { if (!ms) return null; return new Date(ms) .toLocaleTimeString("en-US", { timeZone: "America/New_York", hour: "numeric", minute: "2-digit", }) .replace(/\s/g, "") .toLowerCase();}function setText(role, text) { const el = document.querySelector(`[data-role="${role}"]`); if (el && text != null) el.textContent = text;}function setTone(role, tone, prefix) { const el = document.querySelector(`[data-role="${role}"]`); if (!el || !tone) return; [...el.classList].forEach((c) => { if (c.startsWith(prefix)) el.classList.remove(c); }); el.classList.add(prefix + tone);}export function initHero() { const mount = document.querySelector('[data-role="hero-chart"]'); if (!mount) return; const chart = createChart(mount, { autoSize: true, handleScroll: false, handleScale: false, layout: { background: { type: ColorType.Solid, color: "transparent" }, textColor: "#6b6456", fontFamily: "'JetBrains Mono', monospace", attributionLogo: false, }, grid: { vertLines: { color: "rgba(33,31,26,0.06)" }, horzLines: { color: "rgba(33,31,26,0.07)" }, }, rightPriceScale: { borderColor: "rgba(33,31,26,0.16)" }, timeScale: { borderColor: "rgba(33,31,26,0.16)", timeVisible: true, secondsVisible: false, }, crosshair: { mode: 1 }, localization: { priceFormatter: pctFmt }, }); const seriesByTicker = new Map(); // ticker -> { series } function drawSeries(list) { const empty = document.querySelector('[data-role="hero-empty"]'); if (!list || !list.length) { if (empty) empty.hidden = false; return; } if (empty) empty.hidden = true; const seen = new Set(); const legend = []; let ci = 0; for (const s of list) { seen.add(s.ticker); const color = s.baseline ? INK : PALETTE[ci++ % PALETTE.length]; let entry = seriesByTicker.get(s.ticker); if (!entry) { const series = chart.addSeries(LineSeries, { color, lineWidth: s.baseline ? 2 : 1.75, // The title labels the line at its last value on the price axis, so // each line is identifiable without decoding the colour. title: displayName(s.ticker), priceLineVisible: false, lastValueVisible: true, crosshairMarkerRadius: 3, }); entry = { series }; seriesByTicker.set(s.ticker, entry); } else { entry.series.applyOptions({ color, title: displayName(s.ticker) }); } entry.series.setData(s.points.map((p) => ({ time: p.t, value: p.v }))); const last = s.points.length ? s.points[s.points.length - 1].v : null; legend.push({ ticker: s.ticker, color, baseline: s.baseline, last }); } // Drop series whose ticker is no longer in the payload. for (const [t, entry] of seriesByTicker) { if (!seen.has(t)) { chart.removeSeries(entry.series); seriesByTicker.delete(t); } } chart.timeScale().fitContent(); renderLegend(legend); } function renderLegend(items) { const box = document.querySelector('[data-role="hero-legend"]'); if (!box) return; box.innerHTML = ""; for (const it of items) { const el = document.createElement("span"); el.className = "legend-item" + (it.baseline ? " legend-item--baseline" : ""); const sw = document.createElement("span"); sw.className = "legend-item__swatch"; sw.style.background = it.color; const name = document.createElement("span"); name.textContent = it.ticker === "^SPX" ? "S&P 500" : it.ticker; const pct = document.createElement("span"); pct.className = "legend-item__pct"; pct.textContent = it.last == null ? "" : pctFmt(it.last); el.append(sw, name, pct); box.appendChild(el); } } function patchReads(r) { if (!r) return; setText("vix-level", r.vix_level != null ? r.vix_level.toFixed(2) : DASH); if (r.vix_tone) { setText("vix-tone", cap(r.vix_tone)); setTone("vix-tone", r.vix_tone, "read__tone--"); } setText("volume", fmtCompact(r.volume)); setText("volume-label", r.volume_label ? r.volume_label + " vs avg" : DASH); if (r.sma_read) { setText("sma-read", r.sma_read); setTone("sma-read", r.sma_tone, "read__tone--"); } const clock = fmtClock(r.asof); if (clock) setText("reads-asof", "Prices as of " + clock); } function patchSession(session) { const banner = document.querySelector('[data-role="session-banner"]'); if (banner && session) banner.dataset.session = session; setText("session-label", SESSION_LABELS[session] || "Market closed"); setText( "session-note", session === "closed" ? "Prices update during market hours " + DASH + " ET" : "US equities " + DASH + " all times ET", ); } async function refresh() { let data; try { const res = await fetch("/api/dashboard", { headers: { Accept: "application/json" } }); if (!res.ok) return; data = await res.json(); } catch { return; } drawSeries(data.series); patchReads(data.reads); patchSession(data.session); } // Draw immediately from stored data, then pull fresh quotes once on open (the // dashboard otherwise only updates via the market-hours intraday poll, so it // can look stale on open — especially after the close) and redraw. The 60s // loop keeps the chart/reads live; the watchlist cards live-tick over the // stream from the quotes the open refresh publishes. refresh(); fetch("/api/dashboard/refresh") .catch(() => {}) .finally(() => refresh()); const timer = setInterval(refresh, 60000); document.addEventListener("visibilitychange", () => { if (!document.hidden) refresh(); }); window.addEventListener("pagehide", () => clearInterval(timer));}
deleted
frontend/static_src/home/scripts/home.js
@@ -1,89 +0,0 @@// Markets dashboard — live hero verdict + breadth (Phase 7).//// The hero's plain-language verdict, its headline figures, and the breadth band// are server-rendered at page load and then kept live: the scheduler recomputes// a market `summary` as the indexes tick intraday, base/stream.js re-broadcasts// it as a `finance:summary` window event, and this patches those nodes in place// from it — no second EventSource. The sparkline cards already stream live via// the base quote patcher; this closes the gap the verdict + breadth left (they// used to be frozen page-load snapshots, see PLAN.md Phase 5's known limitation).//// Patch targets carry data-role hooks in home.html. A node that wasn't rendered// at load (a figure that was null then) is simply skipped — the patcher refines// what's on the page, it doesn't build missing rows.const DASH = "·";// Mirrors the server-side `pct` filter and base/stream.js's fmtPct.function fmtPct(n) { if (n == null || Number.isNaN(n)) return DASH; return ( n.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, signDisplay: "exceptZero", }) + "%" );}function fmtInt(n) { return (n ?? 0).toLocaleString("en-US");}function setText(role, text) { const el = document.querySelector(`[data-role="${role}"]`); if (el) el.textContent = text;}function setWidth(role, pct) { const el = document.querySelector(`[data-role="${role}"]`); if (el && pct != null) el.style.width = `${pct}%`;}// Set the semantic move class (green/red/flat) from a percentage change.function setMove(el, pct) { el.classList.remove("is-up", "is-down", "is-flat"); if (pct == null || Number.isNaN(pct)) el.classList.add("is-flat"); else el.classList.add(pct >= 0 ? "is-up" : "is-down");}function applySummary(s) { if (!s) return; // Hero verdict + supporting clause. setText("hero-verdict", s.verdict); setText("hero-detail", s.detail); // Headline figures. const broad = document.querySelector('[data-role="hero-broad"]'); if (broad && s.broad_pct != null) { broad.textContent = fmtPct(s.broad_pct); setMove(broad, s.broad_pct); } const b = s.breadth || {}; if (b.pct_green != null) setText("hero-green", `${b.pct_green}%`); if (s.vix_label) setText("hero-vix", s.vix_label); // Breadth band: the counts, the share-green label, and the proportion bar. setText("breadth-adv", fmtInt(b.advancers)); setText("breadth-dec", fmtInt(b.decliners)); setText("breadth-flat", fmtInt(b.unchanged)); if (b.pct_green != null) setText("breadth-pct", `${b.pct_green}% green`); setWidth("breadth-up", b.up_w); setWidth("breadth-flat-seg", b.flat_w); setWidth("breadth-down", b.down_w); const bar = document.querySelector('[data-role="breadth-bar"]'); if (bar && b.total != null) { bar.setAttribute( "aria-label", `${b.advancers} advancing, ${b.decliners} declining of ${b.total} stocks`, ); }}export function initSummary() { // Only the dashboard carries the hero; elsewhere this is a no-op. if (!document.querySelector('[data-role="hero"]')) return; window.addEventListener("finance:summary", (e) => applySummary(e.detail));}
added
frontend/static_src/home/scripts/watchlist.js
@@ -0,0 +1,61 @@// Dashboard watchlist editing (Phase C).//// Add and remove post to /api/watchlist; on success the page reloads so the// server re-renders the cards, re-registers the live stream tickers, and the// hero graph re-fetches with the new set. A brand-new symbol is validated and// backfilled server-side before it returns, so the reloaded page is complete.async function post(url, ticker) { try { const res = await fetch(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ ticker }), }); return await res.json(); } catch { return { ok: false, error: "Network error — try again." }; }}export function initWatchlist() { const form = document.querySelector('[data-role="watch-add"]'); const msg = document.querySelector('[data-role="watch-msg"]'); function showMsg(text, isError) { if (!msg) return; msg.textContent = text; msg.hidden = false; msg.classList.toggle("is-error", !!isError); } if (form) { form.addEventListener("submit", async (e) => { e.preventDefault(); const input = form.querySelector('[name="ticker"]'); const ticker = (input.value || "").trim(); if (!ticker) return; const btn = form.querySelector("button"); if (btn) btn.disabled = true; showMsg("Adding " + ticker.toUpperCase() + "…", false); const res = await post("/api/watchlist", ticker); if (res.ok) { location.reload(); } else { if (btn) btn.disabled = false; showMsg(res.error || "Could not add that symbol.", true); } }); } document.querySelectorAll("[data-remove]").forEach((btn) => { btn.addEventListener("click", async () => { const ticker = btn.dataset.ticker; if (!ticker) return; btn.disabled = true; const res = await post("/api/watchlist/remove", ticker); if (res.ok) location.reload(); else btn.disabled = false; }); });}
modified
frontend/static_src/home/styles/home.scss
@@ -1,321 +1,330 @@@use "../../base/styles/variables" as *;@use "../../base/styles/mixins" as *;/* ============================================================================ Markets dashboard (Phase 11) An opinionated, no-customization read of the market: a row of intraday sparkline cards for the indexes and headline commodities, then the day's biggest movers among the curated large-cap stocks. ========================================================================== *//* ---------- "browse all" link beside the page heading ---------- */.page-head__link { font-size: 0.82rem; color: var(--ink-dim); white-space: nowrap;}.page-head__link::after { content: " \2192"; /* → */}.page-head__link:hover { color: var(--ink); text-decoration: underline; text-underline-offset: 2px;}/* ============================================================================ Hero (Phase 5) The dashboard's "how is the market doing TODAY" read: a one-line verdict blending the broad move, breadth, and the VIX tone, over a compact index strip. A page-load snapshot, so it reads as a plain description, not a feed. ========================================================================== */.hero { @include card;/* =========================================================================== Markets dashboard (Phase C): a market-hours banner, the normalized %-vs-S&P-500 day graph, the headline market reads, and the editable session watchlist. The sparkline card styles at the bottom are shared by the watchlist cards and the base stream client's live repaint. ======================================================================== */.dash { display: flex; flex-direction: column; gap: var(--sp-4); margin-top: var(--sp-4); padding: var(--sp-4) var(--sp-4) var(--sp-5); gap: var(--sp-5);}.hero__verdict { @include serif; font-size: var(--fs-2xl); line-height: 1.15; color: var(--ink);/* ---------- market-hours banner ---------- */.dash-banner { display: flex; align-items: center; gap: var(--sp-2); padding: var(--sp-2) var(--sp-3); border: 1px solid var(--rule); border-radius: var(--radius); background: var(--surface); font-size: var(--fs-sm);}.hero__detail { margin-top: var(--sp-1); font-size: var(--fs-md); color: var(--ink-dim);.dash-banner__dot { width: 9px; height: 9px; border-radius: 50%; background: var(--ink-faint); flex: none;}/* the headline figures behind the verdict — a quiet middot-separated row */.hero__stats { display: flex; flex-wrap: wrap; gap: 4px var(--sp-3); margin-top: var(--sp-3); font-size: var(--fs-sm); color: var(--ink-dim);.dash-banner[data-session="regular"] .dash-banner__dot { background: var(--up); box-shadow: 0 0 0 3px var(--up-soft);}.dash-banner[data-session="pre"] .dash-banner__dot,.dash-banner[data-session="post"] .dash-banner__dot { background: var(--warn); box-shadow: 0 0 0 3px var(--warn-soft);}.hero__stat {.dash-banner__label { @include mono; display: inline-flex; align-items: baseline; gap: 5px; white-space: nowrap; font-weight: 700;}/* a thin middot between stats, drawn so it does not need markup */.hero__stat + .hero__stat::before { content: "\00b7"; /* · */ margin-right: var(--sp-3);.dash-banner__sub { color: var(--ink-faint);}.hero__note { margin-top: var(--sp-3); font-size: var(--fs-xs); font-style: italic; color: var(--ink-faint);}/* the compact index strip: clickable chips with ticker + day move */.hero__chips { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr));/* ---------- hero day graph ---------- */.hero-graph { @include card; padding: var(--sp-3) var(--sp-3) var(--sp-2); display: flex; flex-direction: column; gap: var(--sp-2);}.hero__chip {.hero-graph__head { display: flex; align-items: baseline; justify-content: space-between; gap: var(--sp-2); padding: 7px 10px; border-radius: var(--radius-sm); background: var(--well); color: var(--ink); transition: background 0.14s, transform 0.14s; flex-wrap: wrap; gap: var(--sp-1) var(--sp-3);}.hero__chip:hover { color: var(--ink); transform: translateY(-1px);.hero-graph__title { @include serif; font-size: var(--fs-lg); font-weight: 600;}.hero__chip-sym { @include mono; font-weight: 700; font-size: var(--fs-sm);.hero-graph__note { font-size: var(--fs-xs); color: var(--ink-faint);}.hero__chip-chg { font-size: var(--fs-sm); font-weight: 700;.hero-graph__base { color: var(--ink-dim); font-weight: 600;}.hero__chip.is-up .hero__chip-chg { color: var(--up);}.hero__chip.is-down .hero__chip-chg { color: var(--down);.hero-graph__chart { position: relative; width: 100%; height: 300px;}@media (min-width: $bp-md) { .hero { flex-direction: row; align-items: center; justify-content: space-between; gap: var(--sp-6); } .hero__read { flex: 1 1 auto; min-width: 0; } /* the strip sits to the right; five chips in a single column read as a compact ledger of the indexes beside the verdict */ .hero__chips { flex: 0 0 auto; grid-template-columns: 1fr; min-width: 168px;@media (min-width: $bp-sm) { .hero-graph__chart { height: 380px; }}/* ============================================================================ Breadth band (Phase 5) Advancer / decliner counts and the share green, over a single proportion bar of the up / flat / down split. ========================================================================== */.breadth { @include card;.hero-graph__empty { position: absolute; inset: 0; display: flex; flex-direction: column; gap: var(--sp-3); padding: var(--sp-4); align-items: center; justify-content: center; color: var(--ink-faint); font-size: var(--fs-sm);}/* Re-assert [hidden] — the display:flex above otherwise beats the UA rule, so the placeholder would linger over the drawn chart (cf. the symbol page's .ind-btn[hidden] fix). */.hero-graph__empty[hidden] { display: none;}.breadth__stats {.hero-graph__legend { display: flex; flex-wrap: wrap; align-items: baseline; gap: 6px var(--sp-4); font-size: var(--fs-sm); color: var(--ink-dim); gap: var(--sp-1) var(--sp-3); padding-top: var(--sp-1);}.breadth__num {.legend-item { display: inline-flex; align-items: center; gap: 0.4em; font-size: var(--fs-xs); color: var(--ink-dim);}.legend-item--baseline { font-weight: 700; font-size: var(--fs-lg); color: var(--ink);}.legend-item__swatch { width: 14px; height: 3px; border-radius: 2px; flex: none;}.legend-item__pct { @include mono; font-weight: 600;}.breadth__stat--up .breadth__num { color: var(--up);/* ---------- headline market reads ---------- */.reads-head { display: flex; justify-content: flex-end; margin-bottom: calc(-1 * var(--sp-3) + 4px);}.breadth__stat--down .breadth__num { color: var(--down);.reads-head__asof { font-size: var(--fs-xs); color: var(--ink-faint);}/* the share green sits at the right end as the headline figure */.breadth__pct { margin-left: auto; font-weight: 700; font-size: var(--fs-md); color: var(--ink);.reads { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 11px;}@media (min-width: $bp-sm) { .reads { grid-template-columns: repeat(4, minmax(0, 1fr)); }}/* the proportion bar: three segments in a sunken well, no gaps */.breadth__bar {.read { @include card; padding: var(--sp-2) var(--sp-3); display: flex; height: 12px; border-radius: 999px; overflow: hidden; background: var(--well); flex-direction: column; gap: 3px;}.breadth__seg { height: 100%;.read__label { @include eyebrow;}.breadth__seg--up { background: var(--up); }.breadth__seg--flat { background: var(--ink-faint); }.breadth__seg--down { background: var(--down); }/* ============================================================================ ETF band quality pieces (Phase 5) ========================================================================== *//* the quality verdict pill in an ETF card head — pushed to the right end */.qchip { margin-left: auto; flex: none; align-self: center; padding: 2px 7px; border-radius: 999px; font-size: 0.56rem;.read__value { font-size: var(--fs-lg); font-weight: 700;}.read__value--text { font-size: var(--fs-sm); font-weight: 600; text-transform: uppercase; letter-spacing: 0.1em; line-height: 1.25;}.qchip--good { color: var(--up); background: var(--up-soft); }.qchip--ok { color: var(--warn); background: var(--warn-soft); }.qchip--bad { color: var(--down); background: var(--down-soft); }.qchip--unknown { color: var(--ink-faint); background: var(--well); }.read__sub { font-size: var(--fs-xs); color: var(--ink-dim);}/* the compact gainers / losers strip below the cards */.etf-movers { display: grid; grid-template-columns: 1fr; gap: var(--sp-2); margin-top: var(--sp-3);/* tone classes for the VIX / trend reads (semantic, sparing) */.read__tone--calm,.read__tone--up { color: var(--up);}.read__tone--steady { color: var(--ink-dim);}.read__tone--elevated,.read__tone--warn { color: var(--warn);}.read__tone--stressed,.read__tone--down { color: var(--down);}.etf-movers__group {/* ---------- watchlist ---------- */.watch__head { display: flex; flex-wrap: wrap; align-items: center; gap: var(--sp-2); justify-content: space-between; flex-wrap: wrap; gap: var(--sp-2) var(--sp-3);}.etf-movers__label { @include eyebrow; font-size: var(--fs-2xs); color: var(--ink-faint); min-width: 4.5ch;.watch-add { display: flex; gap: var(--sp-1);}.etf-pill { display: inline-flex; align-items: center; gap: 6px; padding: 4px 9px; border-radius: 999px; background: var(--well);.watch-add__input { width: 14rem; max-width: 48vw; padding: 0.4rem 0.6rem; font: inherit; font-size: var(--fs-sm); color: var(--ink); font-size: var(--fs-xs); transition: transform 0.14s; background: var(--surface); border: 1px solid var(--rule-strong); border-radius: var(--radius-sm);}.etf-pill:hover { color: var(--ink); transform: translateY(-1px);.watch-add__input:focus-visible { outline: 2px solid var(--ink); outline-offset: 1px;}.etf-pill__dot { width: 6px; height: 6px; border-radius: 999px; flex: none;.watch-add__btn { @include tap; padding: 0.4rem 0.9rem; font: 500 var(--fs-sm) / 1 inherit; color: var(--surface); background: var(--ink); border: 1px solid var(--ink); border-radius: var(--radius-sm); cursor: pointer;}.etf-pill__dot--good { background: var(--up); }.etf-pill__dot--ok { background: var(--warn); }.etf-pill__dot--bad { background: var(--down); }.etf-pill__dot--unknown { background: var(--ink-faint); }.etf-pill__sym { @include mono; font-weight: 700;.watch-add__btn:disabled { opacity: 0.6; cursor: default;}.etf-pill__chg { font-weight: 700;.watch-msg { margin-top: var(--sp-2); font-size: var(--fs-sm); color: var(--ink-dim);}.watch-msg.is-error { color: var(--down);}.etf-pill.is-up .etf-pill__chg { color: var(--up); }.etf-pill.is-down .etf-pill__chg { color: var(--down); }@media (min-width: $bp-md) { .etf-movers { grid-template-columns: 1fr 1fr; gap: var(--sp-4); }.watch-empty { @include card; padding: var(--sp-4); text-align: center; color: var(--ink-dim); font-size: var(--fs-sm);}/* ---------- sparkline cards ---------- */.spark-grid {.watch-grid { margin-top: var(--sp-3); display: grid; /* phone: a tight 2-up grid so the nine cards are not one long column. minmax(0, 1fr) lets a card shrink under its mono price rather than overflow. Wider screens get the auto-fill flow in the media query. */ grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 11px;}@media (min-width: $bp-sm) { .spark-grid { .watch-grid { grid-template-columns: repeat(auto-fill, minmax(184px, 1fr)); }}/* The watch card wraps a sparkline card (`.spark-card` styles below) in a relative box so a remove button can float in its corner without nesting an interactive control inside the navigating link. */.watch-card { position: relative;}.watch-card__link { display: flex; flex-direction: column; gap: 9px; color: inherit;}.watch-card__remove { position: absolute; top: 6px; right: 6px; width: 22px; height: 22px; display: flex; align-items: center; justify-content: center; font-size: 1.05rem; line-height: 1; color: var(--ink-faint); background: var(--surface); border: 1px solid transparent; border-radius: 50%; cursor: pointer; opacity: 0; transition: opacity 0.12s, color 0.12s, border-color 0.12s;}.watch-card:hover .watch-card__remove,.watch-card__remove:focus-visible { opacity: 1;}.watch-card__remove:hover { color: var(--down); border-color: var(--down);}.watch-card__remove:disabled { cursor: default; opacity: 0.4;}/* ---------- shared sparkline card (also used by the live stream client) ---- */.spark-card { @include card; display: flex;
@@ -325,7 +334,6 @@ color: var(--ink); transition: border-color 0.14s, transform 0.14s, box-shadow 0.14s;}.spark-card:hover { color: var(--ink); border-color: var(--rule-strong);
@@ -338,13 +346,11 @@ align-items: baseline; gap: var(--sp-2);}.spark-card__sym { @include mono; font-weight: 700; font-size: var(--fs-md);}.spark-card__name { font-size: var(--fs-2xs); color: var(--ink-dim);
@@ -354,14 +360,11 @@ min-width: 0;}/* the sparkline svg is stretched to the card width (preserveAspectRatio none); strokes keep a constant screen weight via non-scaling-stroke. */.spark { display: block; width: 100%; height: 46px;}.spark__line { fill: none; stroke-width: 1.6;
@@ -369,12 +372,9 @@ stroke-linecap: round; vector-effect: non-scaling-stroke;}.spark__area { stroke: none;}/* faint dashed rule at the previous close, so above/below reads at a glance */.spark__base { stroke: var(--ink-faint); stroke-width: 1;
@@ -382,7 +382,6 @@ opacity: 0.7; vector-effect: non-scaling-stroke;}.is-up-card .spark__line { stroke: var(--up);}
@@ -395,7 +394,6 @@.is-down-card .spark__area { fill: var(--down-soft);}.spark--empty { display: flex; align-items: center;
@@ -413,222 +411,18 @@ padding-top: 8px; border-top: 1px solid var(--rule);}.spark-card__price { font-size: var(--fs-md); font-weight: 700;}.spark-card__chg { font-size: var(--fs-sm); font-weight: 700;}/* update flash, toggled by the live stream client (mirrors .ticker-card) */.spark-card.flash-up { animation: flash-up 0.7s ease-out;}.spark-card.flash-down { animation: flash-down 0.7s ease-out;}/* ---------- movers panels ---------- */.movers { display: grid; grid-template-columns: 1fr; gap: 12px;}.movers__panel { @include card; padding: 4px var(--sp-3) var(--sp-2);}/* Panel header: a small serif label (echoes the page-level section titles one tier down) with a hairline rule below. */.movers__title { font-family: var(--font-serif); font-weight: 600; font-size: var(--fs-md); letter-spacing: -0.01em; display: block; padding: var(--sp-3) 2px var(--sp-2); border-bottom: 1px solid var(--rule); color: var(--ink);}.movers__empty { color: var(--ink-faint); font-size: var(--fs-sm); padding: var(--sp-4) 4px;}/* The four ranked-row classes (mover / standing / hrow / industry-row) share the chrome via the `ranked-row` mixin; each only owns its grid template and per-column type. */.mover { @include ranked-row; /* sym, name, standing badge, price, day change */ grid-template-columns: auto minmax(0, 1fr) auto auto auto;}.mover--up::before { background: var(--up-soft); }.mover--down::before { background: var(--down-soft); }.mover__sym { font-family: var(--font-mono); font-weight: 700; font-size: var(--fs-sm);}.mover__name { font-size: var(--fs-xs); color: var(--ink-dim); white-space: nowrap; overflow: hidden; text-overflow: ellipsis;}.mover__price { font-family: var(--font-mono); font-size: var(--fs-xs); color: var(--ink-faint);}.mover__chg { font-family: var(--font-mono); font-size: var(--fs-sm); font-weight: 700; text-align: right; min-width: 4.4ch;}/* the badge cell: always present (empty for a not-yet-graded stock) so the row's grid keeps a fixed column count */.mover__badge { display: flex;}/* ---------- quality leaderboard: healthiest / most-concerning (Phase 17, reframed Phase 3) ---------- *//* Each row carries the symbol, the name with three sub-component chips underneath (fundamentals / trajectory / leadership), the trailing-year return as a quiet price anchor, and the overall Healthy / Mixed / Concerning verdict on the right. The chips read as coloured pills so the synthesis is legible without expanding. */.hrow { @include ranked-row; /* sym, body (name + chips), trailing-year return, verdict badge */ grid-template-columns: auto minmax(0, 1fr) auto auto;}.hrow--up::before { background: var(--up-soft); }.hrow--down::before { background: var(--down-soft); }.hrow__sym { font-family: var(--font-mono); font-weight: 700; font-size: var(--fs-sm);}.hrow__body { display: flex; flex-direction: column; gap: var(--sp-1); min-width: 0;}.hrow__name { font-size: var(--fs-xs); color: var(--ink-dim); white-space: nowrap; overflow: hidden; text-overflow: ellipsis;}.hrow__sub { display: flex; flex-wrap: wrap; gap: var(--sp-1);}/* the three sub-component pills — fundamentals / trajectory / leadership */.hrow__chip { display: inline-flex; align-items: center; padding: 2px 7px; border-radius: 999px; text-transform: uppercase; letter-spacing: 0.12em; font-weight: 600; font-size: 0.54rem; /* a hair larger than before (0.48 → 0.54): the chips were too small to scan as discrete pills. Still well under the eyebrow voice. */}.hrow__chip--good { color: var(--up); background: var(--up-soft); }.hrow__chip--ok { color: var(--warn); background: var(--warn-soft); }.hrow__chip--bad { color: var(--down); background: var(--down-soft); }.hrow__chip--unknown { color: var(--ink-faint); background: var(--well); }/* the trailing-year return, a quiet context figure beside the verdict */.hrow__ret { font-family: var(--font-mono); font-size: var(--fs-xs); color: var(--ink-faint); text-align: right; min-width: 5ch;}/* ---------- desktop layering ---------- */@media (min-width: $bp-md) { .movers { grid-template-columns: 1fr 1fr; }}/* on a narrow phone the mover price yields to the badge and day change, the row's key figures (mirrors the .hold row's phone treatment) */@media (max-width: $bp-sm) { .mover { grid-template-columns: auto minmax(0, 1fr) auto auto; } .mover__price { display: none; }}/* ---------- Today's industries panel (Phase 15) ---------- *//* Each row is one sector: name on the left, member count, and the day composite %. Same chrome as `.mover` and `.hrow` via the ranked-row mixin. */.industry-row { @include ranked-row; grid-template-columns: minmax(0, 1fr) auto auto;}.industry-row--up::before { background: var(--up-soft); }.industry-row--down::before { background: var(--down-soft); }.industry-row__name { font-size: var(--fs-md); font-weight: 500; white-space: nowrap; overflow: hidden; text-overflow: ellipsis;}.industry-row__count { font-size: var(--fs-2xs); color: var(--ink-faint);}.industry-row__chg { font-family: var(--font-mono); font-size: var(--fs-md); font-weight: 700; text-align: right; min-width: 5.5ch;}
deleted
frontend/static_src/industries/index.js
@@ -1,84 +0,0 @@// Industries page (Phase 15).//// On `/industries` the page is server-rendered HTML; nothing to do here.// On a sector / industry detail page a `#ind-chart` element carries// `data-sector` (and optionally `data-industry`); we fetch the equal-weight// composite history + ^SPX benchmark and draw both with lightweight-charts.import "./styles/industries.scss";import { createChart, AreaSeries, LineSeries, ColorType } from "lightweight-charts";const COMP_LINE = "#2f7d4f";const COMP_FILL_TOP = "rgba(47, 125, 79, 0.20)";const COMP_FILL_BTM = "rgba(47, 125, 79, 0.00)";const BENCH_LINE = "#7a5237";const chartOptions = { autoSize: true, handleScroll: false, handleScale: false, layout: { background: { type: ColorType.Solid, color: "transparent" }, textColor: "#6b6456", fontFamily: "'JetBrains Mono', monospace", attributionLogo: false, }, grid: { vertLines: { color: "rgba(33,31,26,0.07)" }, horzLines: { color: "rgba(33,31,26,0.07)" }, }, rightPriceScale: { borderColor: "rgba(33,31,26,0.16)" }, timeScale: { borderColor: "rgba(33,31,26,0.16)", rightOffset: 0 }, crosshair: { mode: 1 },};const el = document.getElementById("ind-chart");if (el) { const sector = el.dataset.sector; const industry = el.dataset.industry; // `/api/industries/{sector}/{industry?}/history` returns a composite + benchmark // series, each anchored at 100 on the first shared trading date. const url = industry ? `/api/industries/${encodeURIComponent(sector)}/${encodeURIComponent(industry)}/history` : `/api/industries/${encodeURIComponent(sector)}/history`; fetch(url) .then((res) => { if (!res.ok) throw new Error(`industries chart ${res.status}`); return res.json(); }) .then((d) => { if (!d.composite || d.composite.length < 2) { el.innerHTML = '<p class="ind-chart__empty">No composite history available yet.</p>'; return; } const chart = createChart(el, chartOptions); const comp = chart.addSeries(AreaSeries, { lineColor: COMP_LINE, topColor: COMP_FILL_TOP, bottomColor: COMP_FILL_BTM, lineWidth: 2, priceFormat: { type: "custom", formatter: fmtIndex, minMove: 0.01 }, }); comp.setData(d.composite.map((p) => ({ time: p.d, value: p.v }))); if (d.benchmark && d.benchmark.length > 1) { const bench = chart.addSeries(LineSeries, { color: BENCH_LINE, lineWidth: 2, lineStyle: 2, priceFormat: { type: "custom", formatter: fmtIndex, minMove: 0.01 }, priceLineVisible: false, }); bench.setData(d.benchmark.map((p) => ({ time: p.d, value: p.v }))); } chart.timeScale().fitContent(); }) .catch((err) => { el.innerHTML = `<p class="ind-chart__empty">Chart failed: ${err.message}</p>`; console.error(err); });}function fmtIndex(n) { if (n == null || !Number.isFinite(n)) return "—"; return n.toFixed(1);}
deleted
frontend/static_src/industries/styles/industries.scss
@@ -1,356 +0,0 @@@use "../../base/styles/_variables" as *;@use "../../base/styles/_mixins" as *;/* ============================================================================ Industries (Phase 15) Three surfaces share this stylesheet: - `/industries` sector index (a sortable table + a blurb) - `/industries/{sector}` and `/industries/{sector}/{industry}` detail pages (stat row + composite chart + seasonality + members + breakdown) - The home page also reuses these tokens for its `.industry-row`, but that lives in home.scss to stay co-located with the panel layout. ========================================================================== */.ind-blurb { margin: var(--sp-3) 0 var(--sp-4); max-width: 78ch; color: var(--ink-dim); font-size: var(--fs-sm); line-height: 1.55;}.ind-crumbs { margin: calc(var(--sp-2) * -1) 0 var(--sp-3); color: var(--ink-faint); font-size: var(--fs-xs); a { color: var(--ink-dim); text-decoration: none; } a:hover { color: var(--ink); text-decoration: underline; text-underline-offset: 2px; }}/* shared sector index / industry-inside-sector table */.ind-table { width: 100%; border-collapse: collapse; margin-bottom: var(--sp-6); th, td { padding: var(--sp-2) var(--sp-3); border-bottom: 1px solid var(--rule); text-align: right; font-size: var(--fs-sm); } thead th { @include eyebrow; font-size: var(--eye-cap); color: var(--ink-faint); text-align: right; border-bottom: 1px solid var(--rule-strong); } th.ind-table__sec, td.ind-table__sec { text-align: left; } td.num { @include mono; } tbody th.ind-table__sec { font-weight: 500; } a { color: var(--ink); text-decoration: none; } a:hover { color: var(--accent, var(--ink)); text-decoration: underline; }}/* Phone: the 8-column sector index table is too wide for a 390px viewport — relayout each row as a 5-column card. Sector header sits on row 1, a quiet "<n> members · <n> industries" line on row 2, and the five horizon percentages with their own eyebrow label on row 3. nth-child selectors key off the template's column order: sec | d1 | d5 | d21 | d63 | d252 | members | industries. Scoped to `--sectors` so the 3-column "industries inside this sector" table on the detail page keeps its native table layout (which fits on a phone fine). */@media (max-width: $bp-sm) { .ind-table--sectors { display: block; border-collapse: separate; margin-bottom: var(--sp-4); thead { display: none; } tbody { display: block; } tr { display: grid; grid-template-columns: repeat(5, 1fr); column-gap: 8px; row-gap: 4px; padding: 10px 12px; margin-bottom: 8px; background: var(--surface); border: 1px solid var(--rule); border-radius: var(--radius); } th.ind-table__sec, td.ind-table__sec { grid-column: 1 / -1; padding: 0; border: none; font-weight: 500; font-size: var(--fs-md); } td.num { border: none; padding: 4px 0 0; text-align: left; font-size: var(--fs-xs); min-width: 0; &::before { display: block; font-family: var(--font-sans); font-size: 10px; color: var(--ink-faint); letter-spacing: 0.08em; font-weight: 700; text-transform: uppercase; margin-bottom: 1px; } } td:nth-child(2)::before { content: "Today"; } td:nth-child(3)::before { content: "5d"; } td:nth-child(4)::before { content: "1m"; } td:nth-child(5)::before { content: "3m"; } td:nth-child(6)::before { content: "1y"; } /* Members + Industries: a quiet caption pair on row 2, left + right */ td:nth-child(7), td:nth-child(8) { font-family: var(--font-mono); font-size: 11px; color: var(--ink-faint); border: none; padding: 0; grid-row: 2; margin-bottom: 4px; } td:nth-child(7) { grid-column: 1 / span 3; text-align: left; &::after { content: " members"; font-family: var(--font-sans); } } td:nth-child(8) { grid-column: 4 / -1; text-align: right; &::after { content: " industries"; font-family: var(--font-sans); } } }}/* stat row on the detail pages */.ind-stats { display: grid; grid-template-columns: repeat(6, minmax(0, 1fr)); gap: 10px; margin: 14px 0 24px;}@media (max-width: 720px) { .ind-stats { grid-template-columns: repeat(3, minmax(0, 1fr)); }}.ind-stat { @include card; padding: 10px 12px;}.ind-stat__label { @include eyebrow; color: var(--ink-faint); margin-bottom: 4px;}.ind-stat__val { @include mono; font-size: 1.05rem;}/* composite chart */.ind-chart-panel { padding: 14px; margin-bottom: 22px;}#ind-chart { height: 320px; width: 100%;}.ind-chart__empty { padding: 60px 0; text-align: center; color: var(--ink-faint); font-size: 0.88rem;}/* seasonality bar grid: one column per calendar month */.ind-season { padding: 14px 14px 18px; margin-bottom: 24px;}.season-grid { display: grid; grid-template-columns: repeat(12, minmax(0, 1fr)); gap: 6px; align-items: end; height: 160px;}.season-cell { display: flex; flex-direction: column; align-items: stretch; justify-content: flex-end; height: 100%; position: relative;}.season-cell__bar { width: 100%; min-height: 1px; background: var(--ink-faint); border-radius: 2px 2px 0 0;}.season-cell--up .season-cell__bar { background: var(--up, #2f7d4f);}.season-cell--down .season-cell__bar { background: var(--down, #b3402b);}.season-cell--empty .season-cell__bar { background: transparent; border-bottom: 1px dashed var(--rule);}.season-cell__lbl { @include eyebrow; text-align: center; margin-top: 6px; color: var(--ink-faint);}.season-cell__val { @include mono; font-size: 0.62rem; text-align: center; color: var(--ink-faint); margin-top: 2px;}@media (max-width: 520px) { // Phone: the 12-month grid stays one row but the value lines wrap-out, so // hide them and lean on the bar height + month label only. .season-cell__val { display: none; }}/* members list */.ind-members { display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); gap: 6px;}.ind-member { display: grid; grid-template-columns: 64px 1fr auto auto; gap: 10px; align-items: baseline; padding: 9px 12px; text-decoration: none; color: var(--ink); background: var(--surface); border: 1px solid var(--rule); border-radius: var(--radius); position: relative; overflow: hidden; // Magnitude tint behind the row. &::before { content: ""; position: absolute; inset: 0 auto 0 0; width: var(--bar, 0%); background: var(--ink-faint); opacity: 0.06; }}.ind-member--up::before { background: var(--up, #2f7d4f); opacity: 0.09;}.ind-member--down::before { background: var(--down, #b3402b); opacity: 0.09;}.ind-member__sym { @include mono; font-weight: 600;}.ind-member__name { color: var(--ink-dim); font-size: 0.86rem; white-space: nowrap; overflow: hidden; text-overflow: ellipsis;}.ind-member__industry { @include eyebrow; color: var(--ink-faint); font-size: 0.6rem;}.ind-member__price { @include mono; font-size: 0.86rem;}.ind-member__chg { @include mono; font-size: 0.86rem; text-align: right; min-width: 60px;}
modified
frontend/static_src/symbol/index.js
@@ -4,7 +4,9 @@ import "./styles/symbol.scss";import { initChart } from "./scripts/chart.js";import { initFundamentals } from "./scripts/fundamentals.js";import { initGrowth } from "./scripts/growth.js";import { initRefresh } from "./scripts/refresh.js";initChart();initFundamentals();initGrowth();initRefresh();
added
frontend/static_src/symbol/scripts/refresh.js
@@ -0,0 +1,107 @@// On-demand data refresh for the symbol page (Phase B).//// The server fetches nothing on a timer any more: a symbol's data is pulled// here, when its page is open. On load this auto-runs a refresh (the live price// always; slow SEC / metadata only when their stored copy is stale); the// Refresh button re-pulls everything. Progress streams over SSE from// `/api/symbols/{ticker}/refresh` and drives the bar.//// When a "deep" (server-rendered) section was refreshed, the server's `done`// event asks us to reload so the new data and its "synced … ago" age render;// when only the live price moved, the stream client has already patched it in// place, so no reload is needed. A one-shot sessionStorage flag set just before// our own reload keeps that reload from re-triggering the auto-refresh.export function initRefresh() { const root = document.querySelector("[data-refresh-root]"); if (!root) return; const ticker = root.dataset.ticker; const btn = root.querySelector("[data-refresh]"); const bar = root.querySelector("[data-refresh-bar]"); const fill = root.querySelector("[data-refresh-fill]"); const status = root.querySelector("[data-refresh-status]"); const skipKey = "fin:skipnext:" + ticker; let running = false; const setStatus = (t) => { if (status) status.textContent = t; }; const setFill = (pct) => { if (fill) fill.style.width = Math.max(0, Math.min(100, pct)) + "%"; }; function run(force) { if (running) return; running = true; root.classList.add("is-running"); if (bar) bar.hidden = false; setFill(2); if (btn) btn.disabled = true; setStatus("Updating…"); const url = `/api/symbols/${encodeURIComponent(ticker)}/refresh?force=${force ? 1 : 0}`; const es = new EventSource(url); let done = false; es.addEventListener("step", (e) => { let d; try { d = JSON.parse(e.data); } catch { return; } // A step counts as half-done while running, full when it reports back, so // the bar advances smoothly across the step list. const progressed = d.i - (d.state === "running" ? 0.5 : 0); setFill((progressed / Math.max(1, d.n)) * 100); setStatus("Updating: " + d.label); }); es.addEventListener("done", (e) => { done = true; es.close(); running = false; setFill(100); let reload = false; try { reload = !!JSON.parse(e.data).reload; } catch {} if (reload) { // Show the new server-rendered data + ages. Mark this so the load it // triggers does not auto-refresh again. sessionStorage.setItem(skipKey, "1"); location.reload(); return; } root.classList.remove("is-running"); if (btn) btn.disabled = false; setStatus("Up to date"); window.setTimeout(() => { if (bar) bar.hidden = true; }, 700); }); es.onerror = () => { if (done) return; // normal end-of-stream after `done` es.close(); running = false; root.classList.remove("is-running"); if (btn) btn.disabled = false; if (bar) bar.hidden = true; setStatus("Couldn’t refresh — showing stored data"); }; } if (btn) btn.addEventListener("click", () => run(true)); // Auto-refresh on load, unless this very load was triggered by our own reload // after a refresh (the one-shot skip flag). if (sessionStorage.getItem(skipKey)) { sessionStorage.removeItem(skipKey); setStatus("Updated just now"); } else { run(false); }}
modified
frontend/static_src/symbol/styles/symbol.scss
@@ -1871,3 +1871,92 @@ a.anomaly__link:hover .anomaly__body { font-size: 0.74rem; color: var(--ink-faint);}/* ---------- on-demand refresh control (Phase B) ---------- A button + live status + a thin progress bar that fills as the symbol's data is pulled. The fill is neutral ink, not semantic green: it is progress, not a "good" reading. The bar reserves its row height even when hidden so the header does not jump when a refresh starts. */.refresh { display: flex; align-items: center; gap: var(--sp-3); margin: 0 0 var(--sp-3); min-height: 1.9rem;}.refresh__btn { display: inline-flex; align-items: center; gap: 0.4em; padding: 0.3rem 0.7rem; font: 500 var(--fs-sm) / 1 inherit; color: var(--ink-dim); background: var(--surface); border: 1px solid var(--rule-strong); border-radius: var(--radius-sm); cursor: pointer; transition: color 0.12s, border-color 0.12s, background 0.12s; svg { width: 1em; height: 1em; transition: transform 0.4s; } &:hover:not(:disabled) { color: var(--ink); border-color: var(--ink); } &:disabled { cursor: default; opacity: 0.7; }}/* Spin the icon while a refresh is in flight. */.refresh.is-running .refresh__btn svg { animation: refresh-spin 0.9s linear infinite;}@keyframes refresh-spin { to { transform: rotate(360deg); }}.refresh__status { flex: 1 1 auto; min-width: 0; font-size: var(--fs-xs); color: var(--ink-faint); white-space: nowrap; overflow: hidden; text-overflow: ellipsis;}.refresh__bar { flex: 0 0 30%; max-width: 12rem; height: 3px; background: var(--rule); border-radius: 999px; overflow: hidden;}.refresh__fill { width: 0; height: 100%; background: var(--ink-dim); border-radius: inherit; transition: width 0.3s ease;}@media (prefers-reduced-motion: reduce) { .refresh.is-running .refresh__btn svg { animation: none; } .refresh__fill { transition: none; }}
modified
frontend/vite.config.js
@@ -18,7 +18,6 @@ export default defineConfig({ symbol: resolve(__dirname, "static_src/symbol/index.js"), health: resolve(__dirname, "static_src/health/index.js"), search: resolve(__dirname, "static_src/search/index.js"), industries: resolve(__dirname, "static_src/industries/index.js"), }, }, },
added
migrations/0015_watchlist.sql
@@ -0,0 +1,20 @@-- Phase C: session-scoped dashboard watchlists.---- The demand-only refocus made the dashboard a personal, editable watchlist-- rather than a fixed view. There are no accounts: a browser is identified by-- an opaque `fin_sid` cookie, and its watchlist symbols live here keyed on that-- sid. Clearing the browser's cookies loses the list (acceptable by design).-- The S&P 500 baseline is not stored — it is always shown by the dashboard.---- `position` orders the list as the user arranged it; `added_at` is a tiebreak.-- The FK cascades a symbol prune (seed reconcile) into the watchlists, so a-- dropped universe symbol cannot linger as a dangling watchlist row.CREATE TABLE watchlist ( sid TEXT NOT NULL, ticker TEXT NOT NULL REFERENCES symbols(ticker) ON DELETE CASCADE, position INTEGER NOT NULL DEFAULT 0, added_at INTEGER NOT NULL, PRIMARY KEY (sid, ticker));CREATE INDEX idx_watchlist_sid ON watchlist (sid, position);
@@ -101,8 +101,8 @@ pub fn router(state: AppState) -> Router { Router::new() .merge(routes::home::router()) .merge(routes::industries::router()) .merge(routes::symbols::router()) .merge(routes::watchlist::router()) .merge(routes::search::router()) .merge(routes::stream::router()) .merge(routes::health::router())
@@ -21,6 +21,18 @@ pub fn change(last: f64, prev: f64) -> Change { Change { abs, pct }}/// A VIX level read into one plain word, for the dashboard's volatility tone./// The bands suit the ^VIX cash gauge: sub-14 is a placid tape, the teens are/// normal, the low-20s start to show stress, and 28+ is outright fear.pub fn vix_tone(level: f64) -> &'static str { match level { v if v < 14.0 => "calm", v if v < 20.0 => "steady", v if v < 28.0 => "elevated", _ => "stressed", }}/// Position of `value` along the `[lo, hi]` range, as a 0..100 percent for/// placing a marker on a track. Clamped to the ends; a zero-width range maps/// to the midpoint. Rounded to 2 dp so it inlines cleanly into a `style`.
@@ -819,17 +831,6 @@ fn score_grade(score: f64) -> Grade { }}/// Trailing return (percent) over the price-trend window, for display beside a/// stock's standing. `None` with less than a few months of history.pub fn trailing_return(closes: &[f64]) -> Option<f64> { if closes.len() < TREND_MIN { return None; } let window = &closes[closes.len().saturating_sub(TREND_WINDOW)..]; let (&first, &last) = (window.first()?, window.last()?); (first > 0.0).then(|| (last - first) / first * 100.0)}/// Score the trailing-year price trend in [-1, 1]: a trailing return blended/// with how steady the climb was — the share of ~monthly sub-blocks that did/// not fall. `None` with too little history to judge.
@@ -1654,176 +1655,6 @@ pub fn next_earnings_estimate(dates: &[&str]) -> Option<String> { Some(next.format("%Y-%m-%d").to_string())}// ────────────────────────── industry aggregation (Phase 15) ─────────────────//// Pure helpers that aggregate per-stock daily-close series into one industry// composite, and roll a single member's history into a 12-month seasonality// average. No DB, no I/O — `routes/industries.rs` does the SQL and hands the// numbers in./// Equal-weight composite return read for one industry / sector aggregation./// Each return is the average across the industry's members of that member's/// own simple return over the same trailing window. `None` when too few/// members carry enough history for the period.#[derive(Debug, Clone, Default, Serialize)]pub struct IndustryReturns { /// One-day return (latest close vs prior close), percent. pub d1: Option<f64>, /// Trailing five-day return, percent. pub d5: Option<f64>, /// Trailing ~21-day (one month) return, percent. pub d21: Option<f64>, /// Trailing ~63-day (one quarter) return, percent. pub d63: Option<f64>, /// Trailing ~252-day (one year) return, percent. pub d252: Option<f64>, /// Number of members that contributed at least one return figure. pub members: usize,}/// Compute one industry's equal-weight composite trailing returns from each/// member's oldest-first daily close series. A member with too short a/// history simply drops out of that period's average rather than skewing it;/// the average is over the members that *did* qualify for that period.////// Every period is computed independently — a stock with 200 bars contributes/// to d5 / d21 / d63 but not d252. `None` for a period where no member/// qualifies. The `members` count is the union: any member that contributed/// to at least one period.pub fn industry_returns(member_closes: &[&[f64]]) -> IndustryReturns { // Lookback offsets keyed in trading days. d5 is "one trading week" — five // bars back — matching how the existing Phase 8 / 30 maths counts. let mean_of_returns = |lookback: usize| -> Option<f64> { let rets: Vec<f64> = member_closes .iter() .filter_map(|closes| { if closes.len() <= lookback { return None; } let last = *closes.last()?; let prev = closes[closes.len() - 1 - lookback]; if prev <= 0.0 { return None; } Some((last - prev) / prev * 100.0) }) .collect(); if rets.is_empty() { return None; } Some(rets.iter().sum::<f64>() / rets.len() as f64) }; let members = member_closes .iter() .filter(|closes| closes.len() >= 2) .count(); IndustryReturns { d1: mean_of_returns(1), d5: mean_of_returns(5), d21: mean_of_returns(21), d63: mean_of_returns(63), d252: mean_of_returns(252), members, }}/// One twelve-cell seasonality reading: the average daily return for each/// calendar month across all years of history. Percent per day, not per/// month — kept comparable to the trailing returns above and avoids a/// compounding artefact from uneven trading-day counts. `None` for months/// with no observations.#[derive(Debug, Clone, Serialize)]pub struct Seasonality { /// Twelve entries, Jan → Dec. pub months: [Option<f64>; 12],}/// Per-month average daily return computed across all years of `closes` //// `dates`. Both slices must be the same length and aligned, oldest first./// `dates` are `YYYY-MM-DD`; an unparseable date is skipped.pub fn seasonality(closes: &[f64], dates: &[&str]) -> Option<Seasonality> { if closes.len() != dates.len() || closes.len() < 2 { return None; } let mut sums = [0.0_f64; 12]; let mut counts = [0u32; 12]; for i in 1..closes.len() { let (prev, cur) = (closes[i - 1], closes[i]); if prev <= 0.0 { continue; } // Month bucketed from the *current* bar's date — the return earned by // sitting through that bar, attributed to the month it landed in. let Some(d) = chrono::NaiveDate::parse_from_str(dates[i], "%Y-%m-%d").ok() else { continue; }; let m = d.month0() as usize; sums[m] += (cur - prev) / prev * 100.0; counts[m] += 1; } let mut months: [Option<f64>; 12] = Default::default(); for i in 0..12 { if counts[i] > 0 { months[i] = Some(sums[i] / counts[i] as f64); } } Some(Seasonality { months })}#[cfg(test)]mod phase15_tests { use super::*; #[test] fn industry_returns_equal_weights_each_member() { // Two members, identical 252-day flat-to-step pattern. The composite // should be the same as each member's individual return. let mut a: Vec<f64> = (0..260).map(|i| 100.0 + i as f64 * 0.1).collect(); let b = a.clone(); // Bump the last bar to make d1 non-trivial. *a.last_mut().unwrap() = 110.0; let mut b = b.clone(); *b.last_mut().unwrap() = 110.0; let r = industry_returns(&[&a[..], &b[..]]); assert_eq!(r.members, 2); // d1: prior bar was 100 + 258*0.1 = 125.8 -> 110, return is negative. let prior = 100.0 + 258.0 * 0.1; let expected = (110.0 - prior) / prior * 100.0; assert!((r.d1.unwrap() - expected).abs() < 1e-9); } #[test] fn industry_returns_drops_too_short_members_per_period() { // One full-history member, one too-short member. d252 should reflect // only the full-history member; d1 should reflect both (each carries // at least 2 bars). let long: Vec<f64> = (0..260).map(|i| 100.0 + i as f64).collect(); let short = vec![200.0, 201.0]; let r = industry_returns(&[&long[..], &short[..]]); assert!(r.d252.is_some(), "long member should qualify for d252"); assert!(r.d1.is_some(), "both members qualify for d1"); // members = 2 (both ≥ 2 bars), even though only one carried d252. assert_eq!(r.members, 2); } #[test] fn seasonality_averages_by_calendar_month() { // Two consecutive January bars (one return), plus two consecutive // February bars (one return). Both deterministic. let dates = ["2025-01-15", "2025-01-16", "2025-02-15", "2025-02-16"]; let closes = [100.0, 101.0, 200.0, 196.0]; // Jan +1.0%, Feb −2.0% let date_refs: Vec<&str> = dates.iter().copied().collect(); let s = seasonality(&closes, &date_refs).unwrap(); assert!((s.months[0].unwrap() - 1.0).abs() < 1e-9); assert!((s.months[1].unwrap() - (-2.0)).abs() < 1e-9); for m in 2..12 { assert!(s.months[m].is_none()); } }}#[cfg(test)]mod phase25_tests { use super::*;
@@ -49,7 +49,9 @@ pub async fn set_meta(pool: &SqlitePool, key: &str, value: &str) -> sqlx::Result Ok(())}/// Read a `meta` setting, if present./// Read a `meta` setting, if present. (Paired with `set_meta`; retained as the/// generic meta accessor even though Phase A removed its last caller.)#[allow(dead_code)]pub async fn get_meta(pool: &SqlitePool, key: &str) -> sqlx::Result<Option<String>> { sqlx::query_scalar("SELECT value FROM meta WHERE key = ?") .bind(key)
@@ -11,8 +11,8 @@ mod routes;mod scheduler;mod seed;mod stream;mod summary;mod templates;mod watchlist;pub use app::{AppState, Config};
@@ -65,14 +65,17 @@ pub fn session_at(now: DateTime<Utc>) -> Session { }}/// The `America/New_York` calendar date (`YYYY-MM-DD`) at `now`. This is the/// trading day the daily-close job keys its once-per-day guard on./// The `America/New_York` calendar date (`YYYY-MM-DD`) at `now`.// Retained past the Phase-A removal of the daily-close job: the Phase-C// dashboard resolves "today" / the most-recent trading day for the day graph.#[allow(dead_code)]pub fn et_date(now: DateTime<Utc>) -> String { now.with_timezone(&New_York).format("%Y-%m-%d").to_string()}/// Whether `now` falls on a weekday in ET (no holiday calendar; see the/// module note).#[allow(dead_code)] // see et_date: Phase-C market-hours logic.pub fn is_et_weekday(now: DateTime<Utc>) -> bool { !matches!( now.with_timezone(&New_York).weekday(),
@@ -81,8 +84,8 @@ pub fn is_et_weekday(now: DateTime<Utc>) -> bool {}/// Whether the regular session has closed for the current ET day: time is at/// or past 16:05 ET. The five-minute cushion lets the closing print settle/// before the daily-close job snapshots it./// or past 16:05 ET.#[allow(dead_code)] // see et_date: Phase-C market-hours logic.pub fn after_close(now: DateTime<Utc>) -> bool { now.with_timezone(&New_York).time() >= at(16, 5)}
modified
src/routes/health.rs
@@ -214,75 +214,33 @@ fn endpoint_label(endpoint: &str) -> &str { }}/// Human label and one-line description per scheduler job. An unknown job (a/// future one) falls back to its raw id and no description./// Human label and one-line description per scheduler job. Since the demand-only/// refocus the only timed job is the intraday poll (all the old sweeps were/// removed; deep data is fetched on demand when a page is viewed — see the guard/// usage in the Endpoints section above). An unknown job falls back to its raw/// id and no description.fn job_meta(job: &str) -> (&str, &str) { match job { "seed" => ( "Universe seed", "First-run import of the curated symbol list and its deep daily history.", ), "history" => ( "Daily history", "Backstop incremental daily-bar refresh from Yahoo, roughly every 6 hours.", ), "intraday" => ( "Intraday quotes", "Live quotes from Yahoo for the symbols on screen, during market hours.", ), "daily_close" => ( "Daily close", "Once-a-day closing snapshot of the whole universe, just after the bell.", ), "sec" => ( "Fundamentals & filings", "SEC EDGAR company facts and filing history for each stock, refreshed weekly.", ), "dividends" => ( "Dividend payouts", "Per-payout dividend / distribution history from Yahoo for each \ stock and ETF, refreshed weekly.", ), "fund_metadata" => ( "ETF fund metadata", "Yahoo quoteSummary snapshot for each ETF — expense ratio, yield, \ NAV, inception, category, fund family, strategy. Refreshed monthly.", ), "fund_nav" => ( "ETF NAV", "Yahoo quoteSummary NAV for each ETF, refreshed daily — keeps the \ price-vs-NAV premium behind the ETF quality read's tracking factor \ current (the monthly metadata sweep's NAV is too stale for that).", ), "earnings_calendar" => ( "Earnings calendar", "Yahoo quoteSummary `calendarEvents` for each stock — the next \ expected earnings date. Refreshed monthly and whenever the \ stored date passes.", "Live quotes from Yahoo for the symbols a browser is viewing, on a \ ~5-minute cadence. Nothing is polled when nobody is on the site.", ), "asset_profile" => ( "Stock sector & industry", "Yahoo quoteSummary `assetProfile` for each stock — sector and \ industry classification. Refreshed monthly.", "prune" => ( "Prune", "Local cleanup of aged intraday bars and fetch-log rows (no network).", ), other => (other, ""), }}/// Display order for the jobs list: the data pipeline's own order./// Display order for the jobs list. Only the intraday poll reports a status row/// now; the fallback keeps any future job at the end.fn job_rank(job: &str) -> u8 { match job { "seed" => 0, "history" => 1, "sec" => 2, "fund_metadata" => 3, "fund_nav" => 4, "asset_profile" => 5, "earnings_calendar" => 6, "dividends" => 7, "intraday" => 8, "daily_close" => 9, _ => 10, "intraday" => 0, "prune" => 1, _ => 2, }}
modified
src/routes/home.rs
@@ -1,88 +1,57 @@//! `GET /` — the markets dashboard.//! `GET /` — the markets dashboard (Phase C), and `GET /api/dashboard` behind it.//!//! An opinionated, no-customization read of the market: a row of sparkline//! cards for the major US indexes and the headline commodities, the day's//! biggest movers, and a strongest / weakest read over the curated large-cap//! stocks. There is deliberately no per-user layout — the app decides what//! matters (see PLAN.md Phases 11 and 20). The full, browsable universe lives//! on `/search`.use std::cmp::Ordering;//! A TradingView-style read of the day: a normalized %-vs-S&P-500 day graph over//! the session's market reads (S&P, volume, VIX, the 50/200-day trend) and the//! browser's personal, editable watchlist. The watchlist is session-scoped (a//! `fin_sid` cookie; see `crate::watchlist`), seeded with starters on a first//! visit. Data is demand-driven: opening this page is what makes the server poll//! the watchlist + baseline symbols (via the stream interest registry); nothing//! is polled when nobody is here.use std::collections::HashMap;use axum::{extract::State, response::Response, routing::get, Router};use axum::{ extract::State, http::{header, HeaderMap}, response::{Html, IntoResponse, Response}, routing::get, Json, Router,};use serde::Serialize;use crate::compute::{self, Sparkline};use crate::market;use crate::models;use crate::render::render;use crate::summary::{market_verdict, vix_tone};use crate::AppState;use crate::render::render_to_string;use crate::{watchlist, AppState};pub fn router() -> Router<AppState> { Router::new().route("/", get(home)) Router::new() .route("/", get(home)) .route("/api/dashboard", get(dashboard_api)) .route("/api/dashboard/refresh", get(dashboard_refresh))}/// The dashboard's index cards: each cash index paired with its index future./// Outside the regular cash session the future is shown in the card's place/// (it trades nearly around the clock, while the cash index sits frozen on its/// last close; see PLAN.md Phase 21). The Nasdaq Composite (`^NDQ`) has no/// clean tradable future, so it always shows the cash index. Hardcoded on/// purpose: the home page is a fixed, opinionated view, not a user-built/// watchlist. Five cards = one clean desktop row.const INDEXES: &[(&str, Option<&str>)] = &[ ("^SPX", Some("ES=F")), ("^DJI", Some("YM=F")), ("^NDX", Some("NQ=F")), ("^RUT", Some("RTY=F")), ("^NDQ", None),];/// The dashboard's risk + commodity cards: the volatility gauge first, then/// WTI crude, gold, and natural gas. Shown as the futures themselves for the/// three commodities (no cash instrument to swap to) and as the cash index/// for `^VIX` (no tradable future for it on Yahoo). VIX leads here rather/// than living in the Indexes row because it is a derived sentiment gauge,/// not a price index — putting it first turns the section into a quick/// "what is the market worried about today" panel.const COMMODITIES: &[&str] = &["^VIX", "CL=F", "GC=F", "NG=F"];/// The dashboard's curated ETF cards (Phase 5): one broad-equity anchor/// (`VOO`), a total-market and a growth/tech tilt (`VTI`, `QQQ`), a bond core/// (`BND`), and a commodity (`GLD`) — five cards spanning the asset classes a/// watcher scans first, mirroring the hardcoded `INDEXES` row. Each shows an/// intraday sparkline plus its Phase-4 quality verdict. The full ETF universe/// (and the movers strip below) lives on `/search?kind=etf`.const ETF_CARDS: &[&str] = &["VOO", "VTI", "QQQ", "BND", "GLD"];/// How many ETF gainers / losers the dashboard's compact movers strip lists.const ETF_MOVERS_LIMIT: usize = 5;/// NAV freshness gate for the ETF quality read's tracking factor (Phase 4):/// NAV is struck daily, so a price-vs-NAV premium read against a NAV older than/// this is meaningless and the factor drops out. Mirrors the symbol page.const NAV_FRESH_MS: i64 = 3 * 24 * 3600 * 1000;/// How many gainers and how many losers each movers panel lists.const MOVERS_LIMIT: usize = 8;/// How many stocks each side of the quality leaderboard (healthiest //// most-concerning) lists. Mirrors `MOVERS_LIMIT` so the panels read alike.const HEALTH_LIMIT: usize = 8;/// A symbol's latest session counts as the bars within this window of its most/// recent intraday bar. The regular-plus-extended session spans ~16h, while/// the prior session's bars sit a full ~24h earlier, so 23h cleanly isolates/// just the latest day./// The S&P 500 cash index — the day graph's baseline and the headline read.const BASELINE: &str = "^SPX";/// The volatility gauge behind the VIX read.const VIX: &str = "^VIX";/// A liquid S&P 500 ETF used as the "market volume" proxy: cash indexes carry no/// real share volume on Yahoo, so the dashboard reads volume off SPY. Polled/// while the dashboard is open (it carries a `data-ticker`) so it stays fresh.const VOLUME_PROXY: &str = "SPY";/// A symbol's latest session = the intraday bars within this window of its most/// recent bar (regular+extended spans ~16h; the prior session sits ~24h back).const SESSION_WINDOW_MS: i64 = 23 * 3600 * 1000;/// Calendar days of daily closes to pull for the trajectory read. Comfortably/// over the ~252 trading days `compute`'s trend window needs.const TREND_LOOKBACK_DAYS: i64 = 400;/// Calendar days of daily closes to pull for the 50/200-day SMA trend read.const SMA_LOOKBACK_DAYS: i64 = 320;/// One sparkline card on the dashboard's top row.#[derive(Serialize)]/// Volume vs its recent average: this many trading days form the baseline.const VOLUME_AVG_DAYS: i64 = 65;/// One card on the dashboard: a symbol's price, day move, and intraday spark.#[derive(Serialize, Clone)]struct SparkCard { ticker: String, name: String,
@@ -94,300 +63,337 @@ struct SparkCard { up: bool,}/// One sparkline section of the dashboard (Indexes, Commodities): its cards and/// the freshest quote behind them, so the section heading can carry a quiet/// "prices as of ..." freshness caption (PLAN.md Phase 22).#[derive(Serialize)]struct SparkSection { cards: Vec<SparkCard>, /// The most recent `last_quote_at` (epoch-ms) across the section's symbols; /// `None` until at least one has ever been quoted./// The dashboard's headline market reads, server-rendered and then refreshed by/// the `/api/dashboard` poll. Every field is best-effort: a missing one renders/// as a dash and is simply skipped by the live patcher.#[derive(Serialize, Default)]struct MarketReads { /// S&P 500 level + day move. spx_price: Option<f64>, spx_pct: Option<f64>, /// VIX level, its tone bucket (calm/steady/elevated/stressed), and a label. vix_level: Option<f64>, vix_tone: Option<String>, /// Market volume proxy (SPY): today's volume, the ratio to its recent /// average, a heavy/normal/light label, and when it was last quoted. volume: Option<i64>, volume_ratio: Option<f64>, volume_label: Option<String>, volume_asof: Option<i64>, /// The S&P's stance vs its 50- and 200-day averages, plus a tone. sma_read: Option<String>, sma_tone: Option<String>, /// Freshest quote time (epoch-ms) across the baseline reads, for the /// "prices as of …" caption. asof: Option<i64>,}/// The dashboard hero (Phase 5): a one-line plain-language read of the day/// blending the broad index move, market breadth, and the VIX risk tone, with/// the headline figures behind it and a compact index strip. The verdict is a/// descriptive read of today's tape, not a forecast — the non-advice note rides/// with it./// One overlaid line on the day graph: a symbol's intraday move as % from the/// session's open, on a shared time axis.#[derive(Serialize)]struct Hero { /// The punchy lead, e.g. "Risk-on, and broad." verdict: String, /// The supporting clause, e.g. "Markets higher with wide participation." detail: String, /// Compact index chips beside the verdict (the resolved index cards' moves). chips: Vec<HeroChip>, /// The broad-market day change (the lead index card's move); drives the /// "S&P +0.9%" stat and feeds the verdict. `None` until that card prices. broad_pct: Option<f64>, /// Share of curated stocks trading green, 0..100, rounded; "78% green". green_pct: Option<u8>, /// The VIX read folded into one phrase, e.g. "calm at 13.2". `None` until /// the VIX card prices. vix_label: Option<String>,struct Series { ticker: String, name: String, /// True for the S&P baseline (drawn distinctly). baseline: bool, points: Vec<SeriesPoint>,}/// One index chip in the hero strip: the resolved index card's ticker and day/// move (cash index during the regular session, its future outside it).#[derive(Serialize)]struct HeroChip { ticker: String, change_pct: Option<f64>, up: bool,struct SeriesPoint { /// UNIX seconds (lightweight-charts wants seconds, not ms). t: i64, /// Percent change from the session's first bar. v: f64,}/// Market breadth across the curated large-cap stocks (Phase 5): how many are/// advancing vs declining today and the share green, plus the proportion-bar/// segment widths. Drives the Breadth band and feeds the hero verdict.#[derive(Serialize, Default)]struct Breadth { advancers: usize, decliners: usize, unchanged: usize, /// Stocks with a computable day change (advancers + decliners + unchanged). total: usize, /// Advancers as a percent of `total`, rounded; `None` when `total` is 0. pct_green: Option<u8>, /// Proportion-bar segment widths (percent of `total`): green, flat, red. up_w: f64, flat_w: f64, down_w: f64,/// What `/api/dashboard` returns and what `home` seeds the page with.#[derive(Serialize)]struct DashboardData { session: String, reads: MarketReads, series: Vec<Series>,}/// One curated ETF with everything the dashboard ETF band needs: its price, the/// close it is changing against, and its Phase-4 quality read. Built once and/// fed to both the curated cards and the movers strip.#[derive(Serialize, Clone)]struct EtfRow { ticker: String, name: String, last: Option<f64>, prev: Option<f64>, change_pct: Option<f64>, last_quote_at: Option<i64>, quality: Option<compute::EtfQuality>,}async fn home(State(state): State<AppState>, headers: HeaderMap) -> Response { let session = watchlist::resolve(&state.pool, &headers).await; let tickers = watchlist::list(&state.pool, &session.sid).await;/// One curated ETF card: its sparkline tile plus the quality verdict (Phase 5).#[derive(Serialize)]struct EtfCard { card: SparkCard, quality: Option<compute::EtfQuality>,} let cards = spark_cards_for( &state, &tickers.iter().map(String::as_str).collect::<Vec<_>>(), ) .await; let reads = market_reads(&state).await; let market_session = market::session_at(chrono::Utc::now());/// One pill in the compact ETF movers strip: ticker, day move, quality grade.#[derive(Serialize, Clone)]struct EtfMover { ticker: String, change_pct: f64, quality: Option<compute::EtfQuality>, let extra = minijinja::context! { title => "Markets", cards => cards, empty => tickers.is_empty(), reads => reads, baseline => BASELINE, vix => VIX, volume_proxy => VOLUME_PROXY, session => market_session.as_str(), session_label => session_label(market_session), }; match render_to_string(&state, "pages/home.html", "/", extra) { Ok(html) => { let mut resp = Html(html).into_response(); if let Some(c) = session.set_cookie { if let Ok(v) = header::HeaderValue::from_str(&c) { resp.headers_mut().insert(header::SET_COOKIE, v); } } resp } Err(resp) => resp, }}/// One row in a movers panel.#[derive(Serialize, Clone)]struct Mover { ticker: String, name: String, price: f64, change_abs: f64, change_pct: f64, /// Width (0..100) of the row's magnitude tint, scaled to the largest /// absolute move shown across both panels. bar: f64, /// The stock's rolled-up strong / fair / weak badge (Phase 20); `None` /// until its SEC fundamentals have synced. strength: Option<compute::Standing>,/// `GET /api/dashboard` — the day graph series + the market reads, polled by the/// page (~every minute) so the chart and reads stay live without a reload. The/// series are normalized %-from-open so the watchlist and the S&P baseline share/// one axis (the TradingView/Google "compare" shape).async fn dashboard_api(State(state): State<AppState>, headers: HeaderMap) -> Response { let session = watchlist::resolve(&state.pool, &headers).await; let tickers = watchlist::list(&state.pool, &session.sid).await; // The baseline leads, then each watchlist symbol. let mut series = Vec::with_capacity(tickers.len() + 1); if let Some(s) = pct_series(&state, BASELINE, "S&P 500", true).await { series.push(s); } for t in &tickers { if let Some(s) = pct_series(&state, t, t, false).await { series.push(s); } } let data = DashboardData { session: market::session_at(chrono::Utc::now()).as_str().to_string(), reads: market_reads(&state).await, series, }; let mut resp = Json(data).into_response(); if let Some(c) = session.set_cookie { if let Ok(v) = header::HeaderValue::from_str(&c) { resp.headers_mut().insert(header::SET_COOKIE, v); } } resp}/// One row in the quality leaderboard (healthiest / most-concerning panels,/// Phase 17 / reframed in Phase 3). Carries the `HealthRead` directly so the/// row can show the overall verdict alongside the three sub-readings, plus the/// trailing-year return as a quiet price-performance anchor (folded in from the/// old strongest / weakest panel when those merged into the leaderboard).#[derive(Serialize, Clone)]struct HealthRow { ticker: String, name: String, health: compute::HealthRead, /// Trailing 12-month return, percent; `None` when history is too short. ret_12m: Option<f64>, /// Width (0..100) of the row's magnitude tint, scaled to the largest /// absolute score shown across both panels. bar: f64, /// Colour hook: true when the composite score is not negative. up: bool,/// `GET /api/dashboard/refresh` — the dashboard's on-open refresh. Pulls fresh/// quotes for the watchlist + the baseline reads (^SPX/^VIX/SPY) once, so opening/// the page always shows current data rather than whatever was last stored. The/// scheduler's staleness gate skips anything quoted in the last few minutes, so a/// reload doesn't re-hit Yahoo. Published quotes live-tick the open cards.async fn dashboard_refresh(State(state): State<AppState>, headers: HeaderMap) -> Response { let session = watchlist::resolve(&state.pool, &headers).await; let mut tickers = watchlist::list(&state.pool, &session.sid).await; for b in [BASELINE, VIX, VOLUME_PROXY] { tickers.push(b.to_string()); } let refreshed = crate::scheduler::refresh_quotes(&state.pool, &state.config, &state.hub, &tickers).await; let mut resp = Json(serde_json::json!({ "refreshed": refreshed })).into_response(); if let Some(c) = session.set_cookie { if let Ok(v) = header::HeaderValue::from_str(&c) { resp.headers_mut().insert(header::SET_COOKIE, v); } } resp}/// One curated large-cap stock with everything the home panels need: its/// price, the close it is changing against, its rolled-up standing, its/// trailing-year return, and the health read (Phase 17). Built once per/// render and fed to the movers, strongest / weakest, and health panels.struct StockRow { ticker: String, name: String, last: Option<f64>, prev: Option<f64>, standing: Option<compute::Standing>, health: Option<compute::HealthRead>, ret_12m: Option<f64>, /// When this stock was last quoted (epoch-ms); feeds the movers panel's /// freshness caption (PLAN.md Phase 22). last_quote_at: Option<i64>, /// When this stock's SEC fundamentals last synced (epoch-ms); feeds the /// strongest / weakest panels' freshness caption. fundamentals_synced_at: Option<i64>, /// When this stock's SEC leadership roster last synced (epoch-ms); feeds /// the health panels' freshness caption. leadership_synced_at: Option<i64>, /// Yahoo `assetProfile` sector (Phase 15). `None` until the asset_profile /// sweep has reached this stock; the sector panels drop those rows. sector: Option<String>, /// When the asset_profile sweep last touched this stock (Phase 15). asset_profile_synced_at: Option<i64>,/// A human session label for the dashboard's market-hours banner.fn session_label(s: market::Session) -> &'static str { match s { market::Session::Regular => "Regular session", market::Session::Pre => "Pre-market", market::Session::Post => "After hours", market::Session::Closed => "Market closed", }}/// One row in the home page's "Today's industries" panel: a sector with its/// composite day move, member count, and the magnitude tint sized against/// the largest absolute composite shown across the top / bottom rows.#[derive(Serialize, Clone)]struct IndustryRow { name: String, slug: String, members: usize, change_pct: f64, bar: f64, up: bool,/// Build one symbol's normalized intraday series: its latest session's 15-minute/// bars as % change from the session's first bar's open. `None` when the symbol/// has no intraday bars (e.g. never polled, or a holiday).async fn pct_series(state: &AppState, ticker: &str, name: &str, baseline: bool) -> Option<Series> { let rows: Vec<(i64, f64, f64)> = sqlx::query_as( "SELECT ts, open, close FROM intraday_bars \ WHERE ticker = ? \ AND ts >= (SELECT MAX(ts) FROM intraday_bars WHERE ticker = ?) - ? \ ORDER BY ts", ) .bind(ticker) .bind(ticker) .bind(SESSION_WINDOW_MS) .fetch_all(&state.pool) .await .unwrap_or_default(); // Base off the first bar's open; fall back to its close if open is zero. let base = rows.first().map(|r| if r.1 > 0.0 { r.1 } else { r.2 })?; if base <= 0.0 { return None; } let points: Vec<SeriesPoint> = rows .iter() .map(|(ts, _open, close)| SeriesPoint { t: ts / 1000, v: (close / base - 1.0) * 100.0, }) .collect(); if points.is_empty() { return None; } Some(Series { ticker: ticker.to_string(), name: name.to_string(), baseline, points, })}/// How many sectors to show in each of the top / bottom industry panels.const INDUSTRY_LIMIT: usize = 3;/// Build the dashboard's headline reads from stored data (no network): the S&P/// level/move, the VIX, the SPY-proxied market volume, and the S&P's 50/200-day/// stance.async fn market_reads(state: &AppState) -> MarketReads { let mut r = MarketReads::default(); // S&P 500 level + day move. if let Some((last, prev)) = last_and_prev(state, BASELINE).await { r.spx_price = last; if let (Some(l), Some(p)) = (last, prev) { r.spx_pct = Some(compute::change(l, p).pct); } }async fn home(State(state): State<AppState>) -> Response { let seeded: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM symbols WHERE is_seeded = 1") .fetch_one(&state.pool) .await .unwrap_or(0); // VIX level + tone. if let Some((Some(level), _)) = last_and_prev(state, VIX).await { r.vix_level = Some(level); r.vix_tone = Some(compute::vix_tone(level).to_string()); } // No curated universe yet: the seed has not run. Show the same guidance // the page carried before the redesign. if seeded == 0 { let extra = minijinja::context! { title => "Markets", empty => true }; return render(&state, "pages/home.html", "/", extra); // Market volume proxy (SPY): today's volume vs its recent average. let vol: Option<(Option<i64>, Option<i64>)> = sqlx::query_as("SELECT volume, fetched_at FROM quotes WHERE ticker = ?") .bind(VOLUME_PROXY) .fetch_optional(&state.pool) .await .ok() .flatten(); if let Some((Some(today), asof)) = vol { if today > 0 { r.volume = Some(today); r.volume_asof = asof; let avg: Option<f64> = sqlx::query_scalar( "SELECT AVG(volume) FROM (SELECT volume FROM daily_prices \ WHERE ticker = ? AND volume > 0 ORDER BY d DESC LIMIT ?)", ) .bind(VOLUME_PROXY) .bind(VOLUME_AVG_DAYS) .fetch_optional(&state.pool) .await .ok() .flatten(); if let Some(avg) = avg.filter(|a| *a > 0.0) { let ratio = today as f64 / avg; r.volume_ratio = Some(ratio); r.volume_label = Some( if ratio >= 1.15 { "Heavy" } else if ratio <= 0.85 { "Light" } else { "Normal" } .to_string(), ); } } } let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM symbols") .fetch_one(&state.pool) .await .unwrap_or(seeded); let (index_section, commodity_section) = dashboard_cards(&state).await; // One scan of the curated stocks feeds breadth, the movers, the industry // composites, and the quality leaderboard. let stocks = load_stocks(&state).await; let breadth = breadth(&stocks); // The hero blends the lead index card's move, breadth, and the VIX card into // a one-line read; both sections are already loaded, so it is free. let hero = build_hero(&index_section, &commodity_section, &breadth); // The ETF band: curated quality cards plus a compact gainers / losers strip. let (etf_cards, etf_gainers, etf_losers, etf_asof) = etf_band(&state).await; let (gainers, losers) = movers(&stocks); let (healthiest, concerning) = health_panels(&stocks); let (top_industries, bottom_industries) = industry_panels(&stocks); let industries_asof = stocks .iter() .filter_map(|s| s.asset_profile_synced_at) .max(); // Section freshness (PLAN.md Phase 22): the movers panels date off the // freshest stock quote — the data each panel actually leans on. let movers_asof = stocks.iter().filter_map(|s| s.last_quote_at).max(); // The quality leaderboard leans on both fundamentals and leadership, so its // freshness caption tracks whichever sync ran later. let health_asof = stocks .iter() .filter_map(|s| { [s.fundamentals_synced_at, s.leadership_synced_at] .iter() .filter_map(|x| *x) .max() }) .max(); // S&P stance vs its 50- and 200-day moving averages. let closes_desc: Vec<f64> = sqlx::query_scalar( "SELECT close FROM daily_prices WHERE ticker = ? ORDER BY d DESC LIMIT ?", ) .bind(BASELINE) .bind(SMA_LOOKBACK_DAYS) .fetch_all(&state.pool) .await .unwrap_or_default(); if let Some(&last) = closes_desc.first() { let closes: Vec<f64> = closes_desc.iter().rev().copied().collect(); let sma50 = compute::sma(&closes, 50).last().copied().flatten(); let sma200 = compute::sma(&closes, 200).last().copied().flatten(); if let (Some(s50), Some(s200)) = (sma50, sma200) { let (read, tone) = match (last >= s50, last >= s200) { (true, true) => ("Above its 50- and 200-day average", "up"), (false, false) => ("Below its 50- and 200-day average", "down"), (true, false) => ("Above its 50-day, below its 200-day", "warn"), (false, true) => ("Below its 50-day, above its 200-day", "warn"), }; r.sma_read = Some(read.to_string()); r.sma_tone = Some(tone.to_string()); } } let extra = minijinja::context! { title => "Markets", empty => false, hero => hero, breadth => breadth, etf_cards => etf_cards, etf_gainers => etf_gainers, etf_losers => etf_losers, etf_asof => etf_asof, index_cards => index_section.cards, index_asof => index_section.asof, commodity_cards => commodity_section.cards, commodity_asof => commodity_section.asof, gainers => gainers, losers => losers, movers_asof => movers_asof, healthiest => healthiest, concerning => concerning, health_asof => health_asof, top_industries => top_industries, bottom_industries => bottom_industries, industries_asof => industries_asof, total => total, }; render(&state, "pages/home.html", "/", extra) // Freshest quote across the baseline reads, for the "prices as of" caption. r.asof = sqlx::query_scalar( "SELECT MAX(fetched_at) FROM quotes WHERE ticker IN (?, ?, ?)", ) .bind(BASELINE) .bind(VIX) .bind(VOLUME_PROXY) .fetch_optional(&state.pool) .await .ok() .flatten() .flatten(); r}/// The dashboard's index and commodity sparkline cards.////// Outside the regular cash session each index card resolves to its index/// future (see `INDEXES`): the future trades nearly around the clock, so the/// card stays live overnight instead of freezing on the 16:00 ET close.async fn dashboard_cards(state: &AppState) -> (SparkSection, SparkSection) { let regular = matches!( market::session_at(chrono::Utc::now()), market::Session::Regular ); // During the regular cash session show each index itself; outside it, // swap in the index future where one exists. let index_tickers: Vec<&str> = INDEXES .iter() .map(|&(index, future)| match future { Some(fut) if !regular => fut, _ => index, }) .collect(); let indexes = spark_cards_for(state, &index_tickers).await; let commodities = spark_cards_for(state, COMMODITIES).await; (indexes, commodities)/// The latest price and the prior close for one symbol: the live last price/// (else the latest stored daily close) and the close before it.async fn last_and_prev(state: &AppState, ticker: &str) -> Option<(Option<f64>, Option<f64>)> { sqlx::query_as( "SELECT \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)) \ FROM symbols s WHERE s.ticker = ?", ) .bind(ticker) .fetch_optional(&state.pool) .await .ok() .flatten()}/// Build a sparkline section for `tickers`, in that order: a card per ticker/// (current price, the day's change, a sparkline of the latest session's bars)/// plus the freshest quote across them. A ticker the universe does not hold is/// skipped.async fn spark_cards_for(state: &AppState, tickers: &[&str]) -> SparkSection {/// Build a sparkline card per ticker, in order: current price, the day's change,/// and a sparkline of the latest session's bars. A ticker the universe does not/// hold is skipped.async fn spark_cards_for(state: &AppState, tickers: &[&str]) -> Vec<SparkCard> { if tickers.is_empty() { return SparkSection { cards: Vec::new(), asof: None, }; return Vec::new(); } // One query for the price rows. The `IN` list is built from the hardcoded // dashboard consts — never user input — so the placeholder count is fixed // and safe. The trailing `last_quote_at` feeds the section's freshness. type SparkRow = (String, String, String, Option<f64>, Option<f64>, Option<i64>); // One query for the price rows; the `IN` placeholder count matches `tickers`. type SparkRow = (String, String, Option<f64>, Option<f64>); let placeholders = vec!["?"; tickers.len()].join(","); let sql = format!( "SELECT s.ticker, s.name, s.kind, \ "SELECT s.ticker, s.name, \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)), \ s.last_quote_at \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)) \ FROM symbols s WHERE s.ticker IN ({placeholders})" ); let mut q = sqlx::query_as::<_, SparkRow>(&sql);
@@ -395,19 +401,15 @@ async fn spark_cards_for(state: &AppState, tickers: &[&str]) -> SparkSection { q = q.bind(*t); } let rows: Vec<SparkRow> = q.fetch_all(&state.pool).await.unwrap_or_default(); let asof = rows.iter().filter_map(|r| r.5).max(); let mut by_ticker: HashMap<String, SparkRow> = rows.into_iter().map(|r| (r.0.clone(), r)).collect(); let mut cards = Vec::with_capacity(tickers.len()); for &t in tickers { // Skip a dashboard symbol the universe somehow does not hold. let Some((ticker, name, _kind, last, prev, _quote_at)) = by_ticker.remove(t) else { let Some((ticker, name, last, prev)) = by_ticker.remove(t) else { continue; }; // The latest session's intraday closes, oldest first. The window keys // off this symbol's own most recent bar (see SESSION_WINDOW_MS). // The latest session's intraday closes, oldest first. let closes: Vec<f64> = sqlx::query_scalar( "SELECT close FROM intraday_bars \ WHERE ticker = ? \
@@ -434,547 +436,5 @@ async fn spark_cards_for(state: &AppState, tickers: &[&str]) -> SparkSection { up: change_pct.map_or(true, |p| p >= 0.0), }); } SparkSection { cards, asof }}/// Market breadth across the curated large-cap stocks: advancers vs decliners/// today and the share green, plus the proportion-bar segment widths. Reuses/// the already-loaded [`StockRow`] scan — no extra query. A stock without a/// computable change (no price, or a non-positive prior close) is left out of/// every count so a missing quote never reads as "flat".fn breadth(stocks: &[StockRow]) -> Breadth { let mut b = Breadth::default(); for s in stocks { let (Some(last), Some(prev)) = (s.last, s.prev) else { continue; }; if prev <= 0.0 { continue; } b.total += 1; match last.partial_cmp(&prev) { Some(Ordering::Greater) => b.advancers += 1, Some(Ordering::Less) => b.decliners += 1, _ => b.unchanged += 1, } } if b.total > 0 { let total = b.total as f64; b.pct_green = Some((b.advancers as f64 / total * 100.0).round() as u8); b.up_w = b.advancers as f64 / total * 100.0; b.down_w = b.decliners as f64 / total * 100.0; b.flat_w = (100.0 - b.up_w - b.down_w).max(0.0); } b}/// Assemble the hero from the already-loaded index + commodity cards and the/// breadth read. The broad-market move is the lead index card (resolved to the/// cash S&P or its future by `dashboard_cards`); the VIX read is the lead/// commodity card (`^VIX` always sits first). The chips mirror the index cards.fn build_hero(index: &SparkSection, commodity: &SparkSection, breadth: &Breadth) -> Hero { let broad_pct = index.cards.first().and_then(|c| c.change_pct); let vix = commodity.cards.first(); let vix_level = vix.and_then(|c| c.price); let vix_pct = vix.and_then(|c| c.change_pct); let green_pct = breadth.pct_green; let (verdict, detail) = market_verdict(broad_pct, green_pct, vix_level, vix_pct); let vix_label = vix_level.map(|v| format!("{} at {:.1}", vix_tone(v), v)); let chips = index .cards .iter() .map(|c| HeroChip { ticker: c.ticker.clone(), change_pct: c.change_pct, up: c.up, }) .collect(); Hero { verdict, detail, chips, broad_pct, green_pct, vix_label, }}/// The dashboard ETF band: the curated quality cards and a compact gainers //// losers strip, plus the freshest quote behind them. Builds the full curated/// ETF read once ([`load_etfs`]); the cards reuse the sparkline query for their/// intraday tiles and attach each ETF's quality, and the strip ranks the whole/// curated ETF set by day move.async fn etf_band(state: &AppState) -> (Vec<EtfCard>, Vec<EtfMover>, Vec<EtfMover>, Option<i64>) { let etfs = load_etfs(state).await; let quality_by: HashMap<&str, compute::EtfQuality> = etfs .iter() .filter_map(|e| e.quality.map(|q| (e.ticker.as_str(), q))) .collect(); // Curated cards: sparkline tiles for the hardcoded set, each carrying its // quality verdict. `spark_cards_for` skips a ticker the universe lacks. let spark = spark_cards_for(state, ETF_CARDS).await; let etf_cards: Vec<EtfCard> = spark .cards .into_iter() .map(|card| { let quality = quality_by.get(card.ticker.as_str()).copied(); EtfCard { card, quality } }) .collect(); let (gainers, losers) = etf_movers(&etfs); // Freshest quote across the band: the cards' section asof, else the widest // quote across the curated ETF set. let etf_asof = spark .asof .or_else(|| etfs.iter().filter_map(|e| e.last_quote_at).max()); (etf_cards, gainers, losers, etf_asof)}/// Every curated ETF, each rolled into an [`EtfRow`] with its Phase-4 quality/// read. Mirrors `load_stocks` but for funds: one query for price + the Yahoo/// metadata (expense ratio, NAV) + the SEC AUM, a second for top-10 holdings/// concentration, then `compute::etf_quality`. The tracking factor reads a/// price-vs-NAV premium only against a *fresh* NAV (the daily `fund_nav` job),/// else it drops out — a stale NAV must never assert a bogus premium.async fn load_etfs(state: &AppState) -> Vec<EtfRow> { type EtfPriceRow = ( String, // ticker String, // name Option<f64>, // last Option<f64>, // prev Option<i64>, // last_quote_at Option<f64>, // expense_ratio Option<f64>, // nav_price Option<i64>, // nav_synced_at Option<f64>, // net_assets ); let rows: Vec<EtfPriceRow> = sqlx::query_as( "SELECT s.ticker, s.name, \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)), \ s.last_quote_at, m.expense_ratio, m.nav_price, m.nav_synced_at, fp.net_assets \ FROM symbols s \ LEFT JOIN fund_metadata m ON m.ticker = s.ticker \ LEFT JOIN fund_profiles fp ON fp.ticker = s.ticker \ WHERE s.is_seeded = 1 AND s.kind = 'etf'", ) .fetch_all(&state.pool) .await .unwrap_or_default(); if rows.is_empty() { return Vec::new(); } // Top-10 holdings concentration per ETF (summed weight of the ten largest // holdings, percent) for the diversification factor. A fund with no // holdings (a commodity trust) simply misses the map, so that factor drops // out of its blend rather than reading as zero. let top10_rows: Vec<(String, Option<f64>)> = sqlx::query_as( "SELECT ticker, SUM(pct) FROM ( \ SELECT ticker, pct, ROW_NUMBER() OVER (PARTITION BY ticker ORDER BY rank) AS rn \ FROM fund_holdings \ ) WHERE rn <= 10 AND pct IS NOT NULL GROUP BY ticker", ) .fetch_all(&state.pool) .await .unwrap_or_default(); let top10_by: HashMap<String, f64> = top10_rows .into_iter() .filter_map(|(t, p)| p.filter(|v| *v > 0.0).map(|v| (t, v))) .collect(); let now = crate::db::now_ms(); rows.into_iter() .map( |(ticker, name, last, prev, last_quote_at, expense_ratio, nav_price, nav_synced_at, net_assets)| { let change_pct = match (last, prev) { (Some(l), Some(p)) if p > 0.0 => Some(compute::change(l, p).pct), _ => None, }; // Tracking factor: read the price-vs-NAV premium only against a // NAV synced within NAV_FRESH_MS; else let the factor drop out. let nav_fresh = nav_synced_at.is_some_and(|t| now - t <= NAV_FRESH_MS); let premium_pct = if nav_fresh { last.and_then(|p| compute::premium_discount_pct(p, nav_price)) } else { None }; let top10_pct = top10_by.get(&ticker).copied(); let quality = compute::etf_quality(expense_ratio, premium_pct, top10_pct, net_assets); EtfRow { ticker, name, last, prev, change_pct, last_quote_at, quality, } }, ) .collect()}/// The day's biggest ETF gainers and losers among the curated funds, each/// carrying its quality grade for a quiet chip. A compact strip (top/// [`ETF_MOVERS_LIMIT`] each), so the band stays scannable beneath the cards.fn etf_movers(etfs: &[EtfRow]) -> (Vec<EtfMover>, Vec<EtfMover>) { let mut all: Vec<EtfMover> = etfs .iter() .filter_map(|e| { Some(EtfMover { ticker: e.ticker.clone(), change_pct: e.change_pct?, quality: e.quality, }) }) .collect(); if all.is_empty() { return (Vec::new(), Vec::new()); } all.sort_by(|a, b| { b.change_pct .partial_cmp(&a.change_pct) .unwrap_or(Ordering::Equal) }); let gainers: Vec<EtfMover> = all.iter().take(ETF_MOVERS_LIMIT).cloned().collect(); let losers: Vec<EtfMover> = all.iter().rev().take(ETF_MOVERS_LIMIT).cloned().collect(); (gainers, losers)}/// Every curated large-cap stock, each graded into a [`StockRow`].////// Restricted to `is_seeded` stocks on purpose (see PLAN.md Phase 11): the/// home panels are meant to show names worth noticing, not a small user-added/// symbol's noise. Three queries — price, fundamentals, trailing-year closes —/// then each stock is graded in `compute`. ETFs, indexes and futures are/// excluded: only single stocks have the SEC fundamentals a standing needs.async fn load_stocks(state: &AppState) -> Vec<StockRow> { // 1. Price per curated stock: the live last price, else the latest daily // close; plus the prior close it is changing against. Also pulls each // stock's fundamentals and leadership sync times (for the panels' // freshness captions, Phase 22 + Phase 17) and the Phase 15 sector + // asset-profile sync time (for the Today's industries panel). type PriceRow = ( String, String, Option<f64>, Option<f64>, Option<i64>, Option<i64>, Option<i64>, Option<String>, Option<i64>, ); let price_rows: Vec<PriceRow> = sqlx::query_as( "SELECT s.ticker, s.name, \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)), \ s.last_quote_at, s.fundamentals_synced_at, s.leadership_synced_at, \ s.sector, s.asset_profile_synced_at \ FROM symbols s WHERE s.is_seeded = 1 AND s.kind = 'stock'", ) .fetch_all(&state.pool) .await .unwrap_or_default(); if price_rows.is_empty() { return Vec::new(); } // 2. Every stored fundamentals fact for the curated stocks, grouped by // ticker — the basis for each stock's graded ratios. let fact_rows: Vec<(String, String, String, i64, Option<i64>, f64, String)> = sqlx::query_as( "SELECT f.ticker, f.metric, f.period, f.fiscal_year, f.fiscal_qtr, f.value, f.period_end \ FROM fundamentals f JOIN symbols s ON s.ticker = f.ticker \ WHERE s.is_seeded = 1 AND s.kind = 'stock'", ) .fetch_all(&state.pool) .await .unwrap_or_default(); let mut facts: HashMap<String, Vec<models::FundFact>> = HashMap::new(); for (ticker, metric, period, fiscal_year, fiscal_qtr, value, period_end) in fact_rows { facts.entry(ticker).or_default().push(models::FundFact { metric, period, fiscal_year, fiscal_qtr, value, period_end, }); } // 3. The trailing-year daily closes per curated stock, oldest first. let cutoff = (chrono::Utc::now().date_naive() - chrono::Duration::days(TREND_LOOKBACK_DAYS)) .to_string(); let close_rows: Vec<(String, f64)> = sqlx::query_as( "SELECT p.ticker, p.close FROM daily_prices p JOIN symbols s ON s.ticker = p.ticker \ WHERE s.is_seeded = 1 AND s.kind = 'stock' AND p.d >= ? ORDER BY p.ticker, p.d", ) .bind(&cutoff) .fetch_all(&state.pool) .await .unwrap_or_default(); let mut closes: HashMap<String, Vec<f64>> = HashMap::new(); for (ticker, close) in close_rows { closes.entry(ticker).or_default().push(close); } // 4. Recent 8-K item-5.02 leadership-change count per curated stock, // inside the Phase 17 stability window. One bulk query: the GROUP BY // returns only stocks with at least one match, so the lookup misses // cleanly for a fully-stable company (read below as 0). let lead_cutoff = (chrono::Utc::now().date_naive() - chrono::Duration::days(compute::LEADERSHIP_STABILITY_DAYS)) .to_string(); let change_rows: Vec<(String, i64)> = sqlx::query_as( "SELECT f.ticker, COUNT(*) FROM filings f JOIN symbols s ON s.ticker = f.ticker \ WHERE s.is_seeded = 1 AND s.kind = 'stock' \ AND f.form LIKE '8-K%' AND f.items LIKE '%5.02%' \ AND f.filed_at >= ? \ GROUP BY f.ticker", ) .bind(&lead_cutoff) .fetch_all(&state.pool) .await .unwrap_or_default(); let lead_counts: HashMap<String, usize> = change_rows .into_iter() .map(|(t, n)| (t, n.max(0) as usize)) .collect(); // Assemble: grade each stock off its facts and price, read its trajectory // off its closes, and read its leadership stability off the change count. // A stock with no fundamentals stored yet simply has no standing or health // and is left out of the strongest / weakest and healthiest rankings. price_rows .into_iter() .map( |( ticker, name, last, prev, last_quote_at, fundamentals_synced_at, leadership_synced_at, sector, asset_profile_synced_at, )| { let stock_closes = closes.get(&ticker).map(Vec::as_slice).unwrap_or(&[]); let ratios = facts.get(&ticker).and_then(|f| { let inputs = models::latest_annual_inputs(f, last)?; Some(compute::compute_ratios(&inputs)) }); let standing = ratios .as_ref() .and_then(|r| compute::standing(r, stock_closes)); // Leadership-change count is `None` until the leadership sweep // has reached this stock; the composite then drops that // component instead of penalising an unsynced company. let recent_changes = leadership_synced_at .map(|_| lead_counts.get(&ticker).copied().unwrap_or(0)); let health = ratios .as_ref() .and_then(|r| compute::health_read(r, stock_closes, recent_changes)); StockRow { ticker, name, last, prev, standing, health, ret_12m: compute::trailing_return(stock_closes), last_quote_at, fundamentals_synced_at, leadership_synced_at, sector: sector.filter(|s| !s.is_empty()), asset_profile_synced_at, } }, ) .collect()}/// Today's biggest sector composites (top / bottom equal-weight day moves)/// for the home page. Curated `is_seeded` stocks only — same universe as/// movers / strongest — so a small user-added stock's noise does not crowd/// out a recognised sector. Sectors only (industry-level moves are noisier/// and live on `/industries`).fn industry_panels(stocks: &[StockRow]) -> (Vec<IndustryRow>, Vec<IndustryRow>) { // Bucket each stock's day move by sector. Drop a stock without both a // sector classification and a computable change. let mut by_sector: HashMap<String, Vec<f64>> = HashMap::new(); for s in stocks { let Some(sector) = s.sector.as_deref() else { continue; }; let (Some(last), Some(prev)) = (s.last, s.prev) else { continue; }; if prev <= 0.0 { continue; } by_sector .entry(sector.to_string()) .or_default() .push((last - prev) / prev * 100.0); } if by_sector.is_empty() { return (Vec::new(), Vec::new()); } let mut rows: Vec<IndustryRow> = by_sector .into_iter() .map(|(name, pcts)| { let avg = pcts.iter().sum::<f64>() / pcts.len() as f64; IndustryRow { slug: crate::routes::industries::slug(&name), name, members: pcts.len(), change_pct: avg, bar: 0.0, up: avg >= 0.0, } }) .collect(); rows.sort_by(|a, b| { b.change_pct .partial_cmp(&a.change_pct) .unwrap_or(Ordering::Equal) }); let mut top: Vec<IndustryRow> = rows.iter().take(INDUSTRY_LIMIT).cloned().collect(); let mut bottom: Vec<IndustryRow> = rows.iter().rev().take(INDUSTRY_LIMIT).cloned().collect(); // Scale every magnitude tint to the largest absolute move shown across // both panels (mirrors movers / strongest / health). let max_abs = top .iter() .chain(bottom.iter()) .map(|r| r.change_pct.abs()) .fold(0.0_f64, f64::max); for r in top.iter_mut().chain(bottom.iter_mut()) { r.bar = if max_abs > 0.0 { (r.change_pct.abs() / max_abs * 100.0).clamp(0.0, 100.0) } else { 0.0 }; } (top, bottom)}/// The day's biggest gainers and losers among the curated large-cap stocks./// Each row also carries the stock's strong / fair / weak badge (Phase 20).fn movers(stocks: &[StockRow]) -> (Vec<Mover>, Vec<Mover>) { // Keep only stocks with a computable change. let mut all: Vec<Mover> = stocks .iter() .filter_map(|s| { let (last, prev) = (s.last?, s.prev?); if prev == 0.0 { return None; } let c = compute::change(last, prev); Some(Mover { ticker: s.ticker.clone(), name: s.name.clone(), price: last, change_abs: c.abs, change_pct: c.pct, bar: 0.0, strength: s.standing, }) }) .collect(); if all.is_empty() { return (Vec::new(), Vec::new()); } // Sorted by the day's % change: gainers from the top, losers from the // bottom (most negative first). all.sort_by(|a, b| { b.change_pct .partial_cmp(&a.change_pct) .unwrap_or(Ordering::Equal) }); let mut gainers: Vec<Mover> = all.iter().take(MOVERS_LIMIT).cloned().collect(); let mut losers: Vec<Mover> = all.iter().rev().take(MOVERS_LIMIT).cloned().collect(); // Scale every magnitude tint to the largest absolute move on display, so a // +1% and a -1% row read the same width. let max_abs = gainers .iter() .chain(losers.iter()) .map(|m| m.change_pct.abs()) .fold(0.0_f64, f64::max); for m in gainers.iter_mut().chain(losers.iter_mut()) { m.bar = if max_abs > 0.0 { (m.change_pct.abs() / max_abs * 100.0).clamp(0.0, 100.0) } else { 0.0 }; } (gainers, losers)}/// The quality leaderboard: the healthiest and most-concerning curated stocks/// by their Phase 17 composite — fundamentals + recent trajectory + leadership/// stability rolled into one read. The single non-advice "best / worst quality/// right now" surface that replaced the old Top picks, Strongest & weakest, and/// Stock health panels (Phase 3). Each row also carries its trailing-year/// return as a price-performance anchor. Stocks without a health read/// (fundamentals not synced) are left out; the panels are a fixed page-load/// snapshot.fn health_panels(stocks: &[StockRow]) -> (Vec<HealthRow>, Vec<HealthRow>) { let mut ranked: Vec<&StockRow> = stocks.iter().filter(|s| s.health.is_some()).collect(); if ranked.is_empty() { return (Vec::new(), Vec::new()); } ranked.sort_by(|a, b| { let (sa, sb) = (a.health.unwrap().score, b.health.unwrap().score); sb.partial_cmp(&sa).unwrap_or(Ordering::Equal) }); let row = |s: &StockRow| { let health = s.health.unwrap(); HealthRow { ticker: s.ticker.clone(), name: s.name.clone(), health, ret_12m: s.ret_12m, bar: 0.0, up: health.score >= 0.0, } }; let mut healthiest: Vec<HealthRow> = ranked.iter().copied().take(HEALTH_LIMIT).map(&row).collect(); let mut concerning: Vec<HealthRow> = ranked .iter() .copied() .rev() .take(HEALTH_LIMIT) .map(&row) .collect(); let max_abs = healthiest .iter() .chain(concerning.iter()) .map(|r| r.health.score.abs()) .fold(0.0_f64, f64::max); for r in healthiest.iter_mut().chain(concerning.iter_mut()) { r.bar = if max_abs > 0.0 { (r.health.score.abs() / max_abs * 100.0).clamp(0.0, 100.0) } else { 0.0 }; } (healthiest, concerning) cards}
deleted
src/routes/industries.rs
@@ -1,670 +0,0 @@//! Industry trends (Phase 15).//!//! Treats sectors and industries as first-class. Each stock is classified by//! Yahoo's `quoteSummary.assetProfile`, populated by the `asset_profile`//! scheduler job; this module aggregates the classifications into://!//! - `GET /industries` — sector index page.//! - `GET /industries/{sector-slug}` — sector detail page.//! - `GET /industries/{sector-slug}/{industry-slug}` — industry detail page.//! - `GET /api/industries/{sector-slug}/{industry-slug?}/history` —//! equal-weight composite price series (capped to 5 years) plus the `^SPX`//! benchmark anchored to the same first date.//!//! Aggregation is in-memory, on every render — same pattern as the home//! Strongest & Weakest panels (Phase 20). The data the page leans on//! (current prices and trailing daily closes for the curated stocks) is//! already loaded by other panels in a similar shape, so the cost is small.//! ETFs / indexes / futures are excluded; they carry no asset profile.use std::cmp::Ordering;use std::collections::HashMap;use axum::{ extract::{Path, State}, http::StatusCode, response::{IntoResponse, Response}, routing::get, Json, Router,};use serde::Serialize;use serde_json::json;use crate::compute::{self, IndustryReturns};use crate::render::render;use crate::AppState;/// Trading-day cap on the composite price chart. Five years is long enough/// to span a full market cycle, short enough that the equal-weight scan over/// dozens of members is cheap.const COMPOSITE_LOOKBACK_DAYS: i64 = 5 * 365 + 2;/// One row per stock with classification + the figures the page needs./// Loaded once and reused for the sector index, the detail pages, and (with/// a follow-up SQL pass for daily closes) the chart API.#[derive(Clone)]struct StockClass { ticker: String, name: String, sector: String, industry: String, last: Option<f64>, prev: Option<f64>,}/// Load every stock with both a sector and an industry populated, alongside/// its live last-price and prev-close (for the day move).async fn load_classified(state: &AppState) -> Vec<StockClass> { type Row = (String, String, String, String, Option<f64>, Option<f64>); sqlx::query_as::<_, Row>( "SELECT s.ticker, s.name, s.sector, s.industry, \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)) \ FROM symbols s \ WHERE s.kind = 'stock' \ AND s.sector IS NOT NULL AND s.sector <> '' \ AND s.industry IS NOT NULL AND s.industry <> ''", ) .fetch_all(&state.pool) .await .unwrap_or_default() .into_iter() .map(|(ticker, name, sector, industry, last, prev)| StockClass { ticker, name, sector, industry, last, prev, }) .collect()}/// Group `rows` by `key(row)`, returning each group's name and members in a/// deterministic order (groups alphabetical, members alphabetical by ticker).fn group_by<K, F>(rows: &[StockClass], key: F) -> Vec<(String, Vec<StockClass>)>where K: Into<String>, F: Fn(&StockClass) -> K,{ let mut map: HashMap<String, Vec<StockClass>> = HashMap::new(); for r in rows { map.entry(key(r).into()).or_default().push(r.clone()); } let mut groups: Vec<(String, Vec<StockClass>)> = map.into_iter().collect(); for (_, members) in groups.iter_mut() { members.sort_by(|a, b| a.ticker.cmp(&b.ticker)); } groups.sort_by(|a, b| a.0.cmp(&b.0)); groups}/// Equal-weight day-move composite across `members`. `None` when no member/// has both `last` and `prev`.fn day_composite(members: &[StockClass]) -> Option<f64> { let pcts: Vec<f64> = members .iter() .filter_map(|m| { let (last, prev) = (m.last?, m.prev?); if prev <= 0.0 { return None; } Some((last - prev) / prev * 100.0) }) .collect(); if pcts.is_empty() { return None; } Some(pcts.iter().sum::<f64>() / pcts.len() as f64)}/// Lowercase, hyphenated slug for a sector / industry name. ASCII-only —/// our universe is US-listed so this is enough. Whitespace and `&` collapse;/// anything not `[a-z0-9-]` is dropped. Empty input maps to `"other"`.pub fn slug(s: &str) -> String { let mut out = String::with_capacity(s.len()); let mut prev_dash = false; for c in s.chars() { let mapped = if c.is_ascii_alphanumeric() { prev_dash = false; c.to_ascii_lowercase() } else if c.is_whitespace() || c == '-' || c == '&' || c == '/' || c == ',' { if prev_dash { continue; } prev_dash = true; '-' } else { // Drop punctuation entirely so "Industrials, Inc." doesn't gain // an extra dash from the comma + the space after it. continue; }; out.push(mapped); } let trimmed = out.trim_matches('-').to_string(); if trimmed.is_empty() { "other".to_string() } else { trimmed }}pub fn router() -> Router<AppState> { Router::new() .route("/industries", get(index_page)) .route("/industries/{sector}", get(sector_page)) .route("/industries/{sector}/{industry}", get(industry_page)) .route( "/api/industries/{sector}/history", get(sector_history_api), ) .route( "/api/industries/{sector}/{industry}/history", get(industry_history_api), )}/// One row in the `/industries` sector index.#[derive(Serialize)]struct SectorIndexRow { name: String, slug: String, members: usize, industries: usize, /// Composite returns (day / month / quarter / year). Each is built from /// equal-weight averages over the sector's members. returns: IndustryReturns,}async fn index_page(State(state): State<AppState>) -> Response { let rows = load_classified(&state).await; if rows.is_empty() { let extra = minijinja::context! { title => "Industries", empty => true, }; return render(&state, "pages/industries_index.html", "/industries", extra); } let tickers: Vec<&str> = rows.iter().map(|r| r.ticker.as_str()).collect(); let closes_map = load_closes_map(&state, &tickers).await; let groups = group_by(&rows, |r| r.sector.clone()); let mut sectors: Vec<SectorIndexRow> = groups .into_iter() .map(|(name, members)| { let industries = members .iter() .map(|m| m.industry.as_str()) .collect::<std::collections::HashSet<_>>() .len(); let member_closes: Vec<&[f64]> = members .iter() .filter_map(|m| closes_map.get(&m.ticker).map(Vec::as_slice)) .collect(); let returns = compute::industry_returns(&member_closes); SectorIndexRow { slug: slug(&name), name, members: members.len(), industries, returns, } }) .collect(); // Default order: best day move first, but keep a stable secondary sort by // name so a no-data day still reads alphabetically. sectors.sort_by(|a, b| { let (ar, br) = (a.returns.d1.unwrap_or(f64::NEG_INFINITY), b.returns.d1.unwrap_or(f64::NEG_INFINITY)); br.partial_cmp(&ar).unwrap_or(Ordering::Equal).then(a.name.cmp(&b.name)) }); let total = sectors.iter().map(|s| s.members).sum::<usize>(); let total_industries: usize = sectors.iter().map(|s| s.industries).sum(); let extra = minijinja::context! { title => "Industries", empty => false, sectors => sectors, total => total, total_industries => total_industries, }; render(&state, "pages/industries_index.html", "/industries", extra)}/// Bulk-load the trailing-N-day daily closes for every ticker in `tickers`,/// grouped by ticker, oldest first. One scan over `daily_prices` joined to/// the symbol list.async fn load_closes_map(state: &AppState, tickers: &[&str]) -> HashMap<String, Vec<f64>> { if tickers.is_empty() { return HashMap::new(); } let cutoff = (chrono::Utc::now().date_naive() - chrono::Duration::days(COMPOSITE_LOOKBACK_DAYS)) .to_string(); let placeholders = vec!["?"; tickers.len()].join(","); let sql = format!( "SELECT p.ticker, p.close FROM daily_prices p \ WHERE p.ticker IN ({placeholders}) AND p.d >= ? ORDER BY p.ticker, p.d" ); let mut q = sqlx::query_as::<_, (String, f64)>(&sql); for t in tickers { q = q.bind(*t); } q = q.bind(&cutoff); let rows = q.fetch_all(&state.pool).await.unwrap_or_default(); let mut out: HashMap<String, Vec<f64>> = HashMap::new(); for (t, c) in rows { out.entry(t).or_default().push(c); } out}/// Bulk-load every ticker's dated daily closes (oldest first, paired with the/// trading-date string). Used by the chart API where we need each bar's date/// to align the composite.async fn load_dated_closes_map( state: &AppState, tickers: &[&str],) -> HashMap<String, Vec<(String, f64)>> { if tickers.is_empty() { return HashMap::new(); } let cutoff = (chrono::Utc::now().date_naive() - chrono::Duration::days(COMPOSITE_LOOKBACK_DAYS)) .to_string(); let placeholders = vec!["?"; tickers.len()].join(","); let sql = format!( "SELECT p.ticker, p.d, p.close FROM daily_prices p \ WHERE p.ticker IN ({placeholders}) AND p.d >= ? ORDER BY p.ticker, p.d" ); let mut q = sqlx::query_as::<_, (String, String, f64)>(&sql); for t in tickers { q = q.bind(*t); } q = q.bind(&cutoff); let rows = q.fetch_all(&state.pool).await.unwrap_or_default(); let mut out: HashMap<String, Vec<(String, f64)>> = HashMap::new(); for (t, d, c) in rows { out.entry(t).or_default().push((d, c)); } out}/// One member row on a sector / industry detail page.#[derive(Serialize, Clone)]struct MemberRow { ticker: String, name: String, industry: String, price: Option<f64>, change_pct: Option<f64>, /// Magnitude tint (0..100), scaled to the largest day move shown. bar: f64,}/// Build a list of member rows, scaling each row's tint to the largest/// absolute day-move shown.fn member_rows(members: &[StockClass]) -> Vec<MemberRow> { let mut rows: Vec<MemberRow> = members .iter() .map(|m| { let change_pct = match (m.last, m.prev) { (Some(l), Some(p)) if p > 0.0 => Some((l - p) / p * 100.0), _ => None, }; MemberRow { ticker: m.ticker.clone(), name: m.name.clone(), industry: m.industry.clone(), price: m.last, change_pct, bar: 0.0, } }) .collect(); let max_abs = rows .iter() .filter_map(|r| r.change_pct.map(f64::abs)) .fold(0.0_f64, f64::max); for r in rows.iter_mut() { if let (Some(p), true) = (r.change_pct, max_abs > 0.0) { r.bar = (p.abs() / max_abs * 100.0).clamp(0.0, 100.0); } } // Best-first by day move; missing values sink to the bottom. rows.sort_by(|a, b| { let (ax, bx) = ( a.change_pct.unwrap_or(f64::NEG_INFINITY), b.change_pct.unwrap_or(f64::NEG_INFINITY), ); bx.partial_cmp(&ax).unwrap_or(Ordering::Equal) }); rows}/// Roll the per-industry breakdown out for a sector page.#[derive(Serialize)]struct IndustryBreakdownRow { name: String, slug: String, members: usize, day_pct: Option<f64>,}async fn sector_page( State(state): State<AppState>, Path(sector_slug): Path<String>,) -> Response { let rows = load_classified(&state).await; let members: Vec<StockClass> = rows .into_iter() .filter(|r| slug(&r.sector) == sector_slug) .collect(); if members.is_empty() { return not_found(&state, &format!("Sector '{sector_slug}' not found")); } let sector_name = members[0].sector.clone(); let tickers: Vec<&str> = members.iter().map(|m| m.ticker.as_str()).collect(); let closes_map = load_closes_map(&state, &tickers).await; let member_closes: Vec<&[f64]> = members .iter() .filter_map(|m| closes_map.get(&m.ticker).map(Vec::as_slice)) .collect(); let returns = compute::industry_returns(&member_closes); // Per-industry breakdown inside this sector. let by_industry = group_by(&members, |r| r.industry.clone()); let mut breakdown: Vec<IndustryBreakdownRow> = by_industry .into_iter() .map(|(name, group)| IndustryBreakdownRow { slug: slug(&name), name, members: group.len(), day_pct: day_composite(&group), }) .collect(); breakdown.sort_by(|a, b| { let (ar, br) = (a.day_pct.unwrap_or(f64::NEG_INFINITY), b.day_pct.unwrap_or(f64::NEG_INFINITY)); br.partial_cmp(&ar).unwrap_or(Ordering::Equal).then(a.name.cmp(&b.name)) }); // Seasonality computed on the composite — average across members of each // member's per-month average daily return. A member with too short a // history contributes whichever months it covers. let dated_map = load_dated_closes_map(&state, &tickers).await; let seasonality_months = sector_seasonality(&members, &dated_map); let seasonality_max = seasonality_months .iter() .filter_map(|v| v.map(f64::abs)) .fold(0.0_f64, f64::max); let member_view = member_rows(&members); let extra = minijinja::context! { title => sector_name.clone(), scope => "sector", sector_name => sector_name, sector_slug => sector_slug, industry_name => "", industry_slug => "", returns => returns, breakdown => breakdown, seasonality => seasonality_months, seasonality_max => seasonality_max, members => member_view, }; render(&state, "pages/industries_detail.html", "/industries", extra)}async fn industry_page( State(state): State<AppState>, Path((sector_slug, industry_slug)): Path<(String, String)>,) -> Response { let rows = load_classified(&state).await; let members: Vec<StockClass> = rows .into_iter() .filter(|r| slug(&r.sector) == sector_slug && slug(&r.industry) == industry_slug) .collect(); if members.is_empty() { return not_found( &state, &format!("Industry '{sector_slug}/{industry_slug}' not found"), ); } let sector_name = members[0].sector.clone(); let industry_name = members[0].industry.clone(); let tickers: Vec<&str> = members.iter().map(|m| m.ticker.as_str()).collect(); let closes_map = load_closes_map(&state, &tickers).await; let member_closes: Vec<&[f64]> = members .iter() .filter_map(|m| closes_map.get(&m.ticker).map(Vec::as_slice)) .collect(); let returns = compute::industry_returns(&member_closes); let dated_map = load_dated_closes_map(&state, &tickers).await; let seasonality_months = sector_seasonality(&members, &dated_map); let seasonality_max = seasonality_months .iter() .filter_map(|v| v.map(f64::abs)) .fold(0.0_f64, f64::max); let member_view = member_rows(&members); let extra = minijinja::context! { title => format!("{industry_name} · {sector_name}"), scope => "industry", sector_name => sector_name, sector_slug => sector_slug, industry_name => industry_name, industry_slug => industry_slug, returns => returns, seasonality => seasonality_months, seasonality_max => seasonality_max, members => member_view, breakdown => Vec::<IndustryBreakdownRow>::new(), }; render(&state, "pages/industries_detail.html", "/industries", extra)}/// Average each member's per-month seasonality into one 12-cell composite. A/// member that can't produce a reading for a given month simply does not vote/// in that month; an empty month maps to `None`.fn sector_seasonality( members: &[StockClass], dated_map: &HashMap<String, Vec<(String, f64)>>,) -> Vec<Option<f64>> { let mut sums = [0.0_f64; 12]; let mut counts = [0u32; 12]; for m in members { let Some(rows) = dated_map.get(&m.ticker) else { continue; }; let dates: Vec<&str> = rows.iter().map(|(d, _)| d.as_str()).collect(); let closes: Vec<f64> = rows.iter().map(|(_, c)| *c).collect(); let Some(s) = compute::seasonality(&closes, &dates) else { continue; }; for (i, v) in s.months.iter().enumerate() { if let Some(v) = v { sums[i] += *v; counts[i] += 1; } } } (0..12) .map(|i| { if counts[i] > 0 { Some(sums[i] / counts[i] as f64) } else { None } }) .collect()}// ──────────────────────────── chart API ────────────────────────────────────async fn sector_history_api( State(state): State<AppState>, Path(sector_slug): Path<String>,) -> Response { composite_history(state, sector_slug, None).await}async fn industry_history_api( State(state): State<AppState>, Path((sector_slug, industry_slug)): Path<(String, String)>,) -> Response { composite_history(state, sector_slug, Some(industry_slug)).await}/// Build the equal-weight composite daily series for a sector (or industry,/// when `industry_slug` is `Some`) plus the `^SPX` benchmark scaled to the/// composite's first date. JSON shape:////// ```json/// { "composite": [{"d": "YYYY-MM-DD", "v": 100.0}, ...],/// "benchmark": [{"d": "YYYY-MM-DD", "v": 100.0}, ...] }/// ```async fn composite_history( state: AppState, sector_slug: String, industry_slug: Option<String>,) -> Response { let rows = load_classified(&state).await; let members: Vec<StockClass> = rows .into_iter() .filter(|r| { slug(&r.sector) == sector_slug && industry_slug .as_ref() .map_or(true, |is| slug(&r.industry) == *is) }) .collect(); if members.is_empty() { return ( StatusCode::NOT_FOUND, Json(json!({"error": "no members"})), ) .into_response(); } let tickers: Vec<&str> = members.iter().map(|m| m.ticker.as_str()).collect(); let dated = load_dated_closes_map(&state, &tickers).await; // Build a per-date equal-weight indexed series: for each trading date the // composite is the average across members of (member_close / member_base), // where each member's base is its first close inside the lookback window. // Members whose history starts later simply do not contribute until their // first bar; the composite continues with the available members. This // mirrors how an equal-weight index handles a new constituent. let mut bases: HashMap<&str, f64> = HashMap::new(); for m in &members { if let Some(rows) = dated.get(&m.ticker) { if let Some((_, first_close)) = rows.first() { if *first_close > 0.0 { bases.insert(m.ticker.as_str(), *first_close); } } } } // Union of every trading date seen in the lookback window. let mut date_set: std::collections::BTreeSet<String> = std::collections::BTreeSet::new(); for rows in dated.values() { for (d, _) in rows { date_set.insert(d.clone()); } } // Per-member, per-date close lookup. let mut member_dates: HashMap<&str, HashMap<&str, f64>> = HashMap::new(); for m in &members { let Some(rows) = dated.get(&m.ticker) else { continue; }; let mut by_date: HashMap<&str, f64> = HashMap::new(); for (d, c) in rows.iter() { by_date.insert(d.as_str(), *c); } member_dates.insert(m.ticker.as_str(), by_date); } #[derive(Serialize)] struct Point { d: String, v: f64, } let mut composite: Vec<Point> = Vec::with_capacity(date_set.len()); for d in &date_set { let mut sum = 0.0; let mut n = 0u32; for m in &members { let Some(by_date) = member_dates.get(m.ticker.as_str()) else { continue; }; let Some(close) = by_date.get(d.as_str()) else { continue; }; let Some(base) = bases.get(m.ticker.as_str()) else { continue; }; sum += close / base * 100.0; n += 1; } if n > 0 { composite.push(Point { d: d.clone(), v: sum / n as f64, }); } } // Benchmark: ^SPX scaled to 100 at the composite's first date. let benchmark = load_benchmark_series(&state, composite.first().map(|p| p.d.as_str())).await; Json(json!({ "composite": composite, "benchmark": benchmark, })) .into_response()}/// Pull `^SPX` daily closes from `anchor_date` forward, scaled so the first/// bar at or after the anchor reads `100`. Empty when the anchor is missing/// or the index has no bars there.async fn load_benchmark_series( state: &AppState, anchor_date: Option<&str>,) -> Vec<serde_json::Value> { let Some(anchor) = anchor_date else { return Vec::new(); }; let rows: Vec<(String, f64)> = sqlx::query_as( "SELECT d, close FROM daily_prices WHERE ticker = '^SPX' AND d >= ? ORDER BY d", ) .bind(anchor) .fetch_all(&state.pool) .await .unwrap_or_default(); let Some(base) = rows.first().map(|(_, c)| *c) else { return Vec::new(); }; if base <= 0.0 { return Vec::new(); } rows.into_iter() .map(|(d, c)| json!({"d": d, "v": c / base * 100.0})) .collect()}fn not_found(state: &AppState, msg: &str) -> Response { let extra = minijinja::context! { title => "Industries", message => msg, }; let mut response = render(state, "pages/not_found.html", "/industries", extra); *response.status_mut() = StatusCode::NOT_FOUND; response}
modified
src/routes/mod.rs
@@ -1,7 +1,7 @@pub mod health;pub mod home;pub mod industries;pub mod search;pub mod seo;pub mod stream;pub mod symbols;pub mod watchlist;
modified
src/routes/stream.rs
@@ -99,9 +99,6 @@ async fn stream( Ok(StreamEvent::Market { session }) => { yield Ok(sse_market(&session)); } Ok(StreamEvent::Summary(s)) => { yield Ok(sse_summary(&s)); } Ok(StreamEvent::Health) => { yield Ok(sse_health()); }
@@ -170,14 +167,6 @@ fn sse_market(session: &str) -> Event { .data(format!("{{\"session\":\"{session}\"}}"))}/// The recomputed dashboard market summary (Phase 7). An open `/` page patches/// its hero verdict + headline figures + breadth from this payload.fn sse_summary(s: &crate::summary::MarketSummary) -> Event { Event::default() .event("summary") .data(serde_json::to_string(s).unwrap_or_default())}/// A content-free nudge: an open `/health` page answers it by pulling a fresh/// snapshot from `/api/health`. See `routes::health`.fn sse_health() -> Event {
modified
src/routes/symbols.rs
@@ -3,6 +3,7 @@ use std::collections::HashMap;use axum::{ extract::{Path, Query, State}, http::StatusCode, response::sse::Sse, response::{IntoResponse, Response}, routing::{get, post}, Json, Router,
@@ -26,6 +27,7 @@ pub fn router() -> Router<AppState> { .route("/api/symbols", post(add_symbol)) .route("/api/symbols/{ticker}/history", get(history_api)) .route("/api/symbols/{ticker}/growth", get(growth_api)) .route("/api/symbols/{ticker}/refresh", get(refresh_stream))}/// Stats for the symbol page header and key-stats visualizations. Until
@@ -1494,19 +1496,6 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) None }; // Phase 15: slugs for the sector / industry header tags' links. Computed // here so the template stays declarative. let sector_slug = symbol .sector .as_deref() .map(crate::routes::industries::slug) .unwrap_or_default(); let industry_slug = symbol .industry .as_deref() .map(crate::routes::industries::slug) .unwrap_or_default(); let extra = minijinja::context! { title => ticker, symbol => symbol,
@@ -1524,8 +1513,6 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) anomalies => anomalies, earnings => earnings, filings => filings, sector_slug => sector_slug, industry_slug => industry_slug, }; render(&state, "pages/symbol.html", &format!("/s/{ticker}"), extra)}
@@ -2074,12 +2061,33 @@ fn add_err(status: StatusCode, msg: impl Into<String>) -> Response {/// before the response, so its page is complete the moment the add returns/// (PLAN.md Phase 21; see `scheduler::backfill_symbol`). Every outbound call/// goes through the shared endpoint guard (see PLAN.md's anti-spam policy).async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBody>) -> Response { let Some(ticker) = valid_ticker(&body.ticker) else { return add_err(/// The outcome of ensuring a symbol is in the tracked universe.pub(crate) struct EnsureOutcome { pub ticker: String, pub name: String, pub kind: String, /// True when this call created the symbol; false when it already existed. pub added: bool,}/// Ensure `ticker` is a tracked symbol, adding it to the universe if missing.////// Idempotent: an already-tracked symbol returns at once. A new one is validated/// against Yahoo (one guarded lookup that also yields its name / kind / exchange/// / currency), inserted as a user-added (`is_seeded = 0`) row, has the lookup's/// quote + bars stored, and its full backfill (deep history + SEC data) pulled/// synchronously, so its page is complete on return. Errors come back as a/// (status, message) pair the caller surfaces. Shared by `POST /api/symbols`/// (the Search "Add") and the dashboard watchlist add.pub(crate) async fn ensure_symbol( state: &AppState, raw_ticker: &str,) -> Result<EnsureOutcome, (StatusCode, String)> { let Some(ticker) = valid_ticker(raw_ticker) else { return Err(( StatusCode::BAD_REQUEST, "That does not look like a ticker symbol.", ); "That does not look like a ticker symbol.".into(), )); }; // Already tracked? Idempotent — report it so the caller can just navigate.
@@ -2091,15 +2099,7 @@ async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBod .ok() .flatten(); if let Some((name, kind)) = existing { return Json(AddSymbolResponse { ok: true, ticker: Some(ticker), name: Some(name), kind: Some(kind), added: false, error: None, }) .into_response(); return Ok(EnsureOutcome { ticker, name, kind, added: false }); } // One guarded Yahoo lookup: validates the symbol and describes it.
@@ -2108,28 +2108,28 @@ async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBod match guard.acquire().await { Ok(Permit::Granted) => {} Ok(Permit::Denied(_)) => { return add_err( return Err(( StatusCode::SERVICE_UNAVAILABLE, "The market data source is busy right now. Try again in a few minutes.", ); "The market data source is busy right now. Try again in a few minutes.".into(), )); } Err(e) => { tracing::error!("add_symbol guard for {ticker}: {e:#}"); return add_err( tracing::error!("ensure_symbol guard for {ticker}: {e:#}"); return Err(( StatusCode::INTERNAL_SERVER_ERROR, "Something went wrong. Try again shortly.", ); "Something went wrong. Try again shortly.".into(), )); } } let (info, data) = match yahoo.lookup(&ticker).await { Err(e) => { let _ = guard.record_failure(&e).await; tracing::warn!("add_symbol lookup {ticker}: {e:#}"); return add_err( tracing::warn!("ensure_symbol lookup {ticker}: {e:#}"); return Err(( StatusCode::BAD_GATEWAY, "Could not reach the market data source. Try again shortly.", ); "Could not reach the market data source. Try again shortly.".into(), )); } Ok(outcome) => { // The endpoint answered — even an "unknown symbol" is a healthy
@@ -2138,19 +2138,16 @@ async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBod match outcome { SymbolLookup::Found { info, data } => (info, data), SymbolLookup::Unknown => { return add_err( return Err(( StatusCode::NOT_FOUND, format!("No symbol called {ticker} was found."), ); )); } SymbolLookup::Unsupported(raw_kind) => { let what = raw_kind.to_lowercase(); return add_err( StatusCode::UNPROCESSABLE_ENTITY, format!( "{ticker} is a {what}. Only stocks, ETFs, indexes, and futures can be added right now." ), ); return Err((StatusCode::UNPROCESSABLE_ENTITY, format!( "{ticker} is a {what}. Only stocks, ETFs, indexes, and futures can be added right now." ))); } } }
@@ -2172,38 +2169,122 @@ async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBod .execute(&state.pool) .await; if let Err(e) = inserted { tracing::error!("add_symbol insert {ticker}: {e:#}"); return add_err( tracing::error!("ensure_symbol insert {ticker}: {e:#}"); return Err(( StatusCode::INTERNAL_SERVER_ERROR, "Could not save the symbol. Try again shortly.", ); "Could not save the symbol. Try again shortly.".into(), )); } // Store the quote (and bars) the lookup already paid for, so the symbol's // page shows a live price at once. Best-effort: a hiccup here does not // undo a successful add. // Store the quote (and bars) the lookup already paid for, then pull the full // backfill before returning, so the symbol's page is complete at once. if let Err(e) = scheduler::store_quote(&state.pool, &ticker, &data.quote).await { tracing::warn!("add_symbol store_quote {ticker}: {e:#}"); tracing::warn!("ensure_symbol store_quote {ticker}: {e:#}"); } if !data.bars.is_empty() { if let Err(e) = scheduler::store_intraday(&state.pool, &ticker, &data.bars).await { tracing::warn!("add_symbol store_intraday {ticker}: {e:#}"); tracing::warn!("ensure_symbol store_intraday {ticker}: {e:#}"); } } // Pull the full backfill — deep daily history and all SEC data — before // responding, so the new symbol's page is complete the moment the add // returns rather than filling in over later scheduler cycles. Best-effort // and guard-routed; see `scheduler::backfill_symbol`. scheduler::backfill_symbol(&state.pool, &state.config, &ticker, &info.kind).await; tracing::info!("add_symbol: added {ticker} ({}, {})", info.name, info.kind); Json(AddSymbolResponse { ok: true, ticker: Some(ticker), name: Some(info.name), kind: Some(info.kind), added: true, error: None, }) .into_response() tracing::info!("ensure_symbol: added {ticker} ({}, {})", info.name, info.kind); Ok(EnsureOutcome { ticker, name: info.name, kind: info.kind, added: true })}/// `POST /api/symbols` — add a symbol to the tracked universe (the Search "Add").async fn add_symbol(State(state): State<AppState>, Json(body): Json<AddSymbolBody>) -> Response { match ensure_symbol(&state, &body.ticker).await { Ok(o) => Json(AddSymbolResponse { ok: true, ticker: Some(o.ticker), name: Some(o.name), kind: Some(o.kind), added: o.added, error: None, }) .into_response(), Err((status, msg)) => add_err(status, msg), }}#[derive(Deserialize)]struct RefreshQuery { /// `1` = the manual Refresh button: re-pull every source regardless of /// staleness. Default `0` = a page load: pull the live price always, the /// slow SEC / metadata sources only when their stored copy is stale. #[serde(default)] force: u8,}/// `GET /api/symbols/{ticker}/refresh` — Server-Sent Events driving the symbol/// page's loading bar. Asks the scheduler which steps to run for this symbol/// (kind + staleness + `force`), then runs each in turn, emitting a `step`/// event before and after so the bar advances and names what it is doing. A/// final `done` event tells the page whether a deep (server-rendered) section/// changed — if so it reloads to show it; otherwise the live price was already/// patched in place over the stream and no reload is needed.async fn refresh_stream( Path(ticker): Path<String>, Query(q): Query<RefreshQuery>, State(state): State<AppState>,) -> Response { let ticker = ticker.to_uppercase(); let force = q.force != 0; let kind: Option<String> = sqlx::query_scalar("SELECT kind FROM symbols WHERE ticker = ?") .bind(&ticker) .fetch_optional(&state.pool) .await .ok() .flatten(); let Some(kind) = kind else { return not_found(&state); }; let steps = scheduler::refresh_plan(&state.pool, &state.config, &ticker, &kind, force).await; let any_deep = steps.iter().any(|s| s.deep); let body = async_stream::stream! { let total = steps.len(); // Announce the plan up front so the bar can size itself. yield sse_json("plan", &format!("{{\"total\":{total}}}")); for (i, st) in steps.iter().enumerate() { yield sse_json( "step", &format!( "{{\"i\":{},\"n\":{},\"label\":{},\"state\":\"running\"}}", i + 1, total, json_str(st.label) ), ); let status = scheduler::refresh_step(&state.pool, &state.config, &state.hub, &ticker, &kind, st.key) .await; yield sse_json( "step", &format!( "{{\"i\":{},\"n\":{},\"label\":{},\"state\":{}}}", i + 1, total, json_str(st.label), json_str(status) ), ); } yield sse_json("done", &format!("{{\"reload\":{}}}", any_deep)); }; Sse::new(body) .keep_alive(axum::response::sse::KeepAlive::default()) .into_response()}/// Build a named SSE event with a raw JSON `data` payload.fn sse_json(event: &str, data: &str) -> Result<axum::response::sse::Event, std::convert::Infallible> { Ok(axum::response::sse::Event::default().event(event).data(data))}/// Minimal JSON string escaper for the short, known labels/statuses streamed/// above (no control characters in play; just quote the value safely).fn json_str(s: &str) -> String { serde_json::to_string(s).unwrap_or_else(|_| "\"\"".to_string())}
added
src/routes/watchlist.rs
@@ -0,0 +1,117 @@//! `POST /api/watchlist` (add) and `POST /api/watchlist/remove` — edit the//! current browser session's dashboard watchlist (Phase C).//!//! Both resolve the session from the `fin_sid` cookie (minting one on a first//! visit), so they work even if a script calls them before the dashboard set//! the cookie. Add ensures the symbol is in the universe first (validating ///! backfilling a brand-new ticker via `ensure_symbol`), then appends it.use axum::{ extract::State, http::{header, HeaderMap, StatusCode}, response::{IntoResponse, Response}, routing::post, Json, Router,};use serde::{Deserialize, Serialize};use crate::routes::symbols::ensure_symbol;use crate::{watchlist, AppState};pub fn router() -> Router<AppState> { Router::new() .route("/api/watchlist", post(add)) .route("/api/watchlist/remove", post(remove))}#[derive(Deserialize)]struct Body { ticker: String,}#[derive(Serialize)]struct Resp { ok: bool, #[serde(skip_serializing_if = "Option::is_none")] ticker: Option<String>, #[serde(skip_serializing_if = "Option::is_none")] name: Option<String>, #[serde(skip_serializing_if = "Option::is_none")] error: Option<String>,}/// Add a symbol to the session's watchlist, validating / backfilling it into/// the universe first if it is not tracked yet.async fn add(State(state): State<AppState>, headers: HeaderMap, Json(body): Json<Body>) -> Response { let session = watchlist::resolve(&state.pool, &headers).await; match ensure_symbol(&state, &body.ticker).await { Ok(o) => { if let Err(e) = watchlist::add_ticker(&state.pool, &session.sid, &o.ticker).await { tracing::warn!("watchlist add {}: {e:#}", o.ticker); return reply( &session, StatusCode::INTERNAL_SERVER_ERROR, Resp { ok: false, ticker: None, name: None, error: Some("Could not save to your watchlist.".into()), }, ); } reply( &session, StatusCode::OK, Resp { ok: true, ticker: Some(o.ticker), name: Some(o.name), error: None, }, ) } Err((status, msg)) => reply( &session, status, Resp { ok: false, ticker: None, name: None, error: Some(msg), }, ), }}/// Remove a symbol from the session's watchlist. Idempotent.async fn remove( State(state): State<AppState>, headers: HeaderMap, Json(body): Json<Body>,) -> Response { let session = watchlist::resolve(&state.pool, &headers).await; let ticker = body.ticker.trim().to_uppercase(); let _ = watchlist::remove_ticker(&state.pool, &session.sid, &ticker).await; reply( &session, StatusCode::OK, Resp { ok: true, ticker: Some(ticker), name: None, error: None, }, )}/// Build the JSON response, attaching the `Set-Cookie` header when the session/// was just minted.fn reply(session: &watchlist::Session, status: StatusCode, body: Resp) -> Response { let mut resp = (status, Json(body)).into_response(); if let Some(c) = &session.set_cookie { if let Ok(v) = header::HeaderValue::from_str(c) { resp.headers_mut().insert(header::SET_COOKIE, v); } } resp}
modified
src/scheduler.rs
@@ -1,26 +1,25 @@//! Background job scheduler.//! Background job scheduler (demand-only).//!//! One long-lived tokio task wakes on a fixed tick and runs the market-data//! maintenance jobs as they fall due://! - the first-run universe seed, once, while `meta.seed_completed` is unset;//! - an incremental daily-history refresh from Stooq (~every 6h), touching//! only symbols whose stored history has gone stale;//! - market-hours intraday quotes from Yahoo for the symbols a browser is//! actually viewing right now — demand-driven via the stream hub's interest//! registry, so nothing is polled when nobody is watching;//! - a once-a-day close fetch that snapshots the whole universe from Yahoo//! shortly after the regular session ends;//! - a prune of aged `intraday_bars` and `fetch_log` rows (~daily).//! One long-lived tokio task wakes on a fixed tick. Since the demand-only//! refocus (PLAN.md 2026-06-03) it does **no** timed network fetching: with//! nobody on the site it makes zero outbound calls. Each tick it only://! - broadcasts a market-session change so open pages update their pill (and//! re-pushes the dashboard summary, a local DB read, on a session flip);//! - runs the demand-driven intraday quote poll — Yahoo quotes for just the//! symbols a browser is currently viewing (the stream hub's interest//! registry), so nothing is polled when nobody is watching;//! - prunes aged `intraday_bars` and `fetch_log` rows (~daily, local only).//!//! Every data job records a `fetch_log` row, refreshes its `data_status` row,//! and pings the stream hub so the `/health` page reflects it live. Outbound//! Stooq calls go through the persistent `EndpointGuard` (see `src/guard.rs`),//! which paces requests and stops a job early when the circuit breaker is open//! or the hourly request budget is spent.//! All the old timed sweeps (daily-close, SEC, dividends, fund metadata, NAV,//! earnings, asset profile, periodic history) were removed in Phase A. The data//! they fetched is now pulled **on demand** when a symbol's page is viewed and//! its stored copy is stale — see `backfill_symbol` (the synchronous per-symbol//! pull the add-symbol route and the on-demand refresh use) and Phase B.//!//! Modelled on `status/src/scheduler.rs`, scaled down: finance's jobs run//! hours apart, not seconds, and are inherently sequential (the pacing is the//! point), so they run inline in the loop task rather than via semaphores.//! The boot seed only reconciles the universe rows from the curated CSV (local,//! no network); a symbol's history and deep data fill in the first time it is//! viewed. Outbound calls still pass through the persistent `EndpointGuard`//! (see `src/guard.rs`), which paces requests and trips on rate limits.use std::collections::{HashMap, HashSet};use std::sync::Arc;
@@ -29,18 +28,17 @@ use std::time::{Duration, Instant};use sqlx::SqlitePool;use tokio::task::JoinHandle;use crate::db::{get_meta, now_ms, set_meta};use crate::db::now_ms;use crate::guard::{EndpointGuard, Permit};use crate::market;use crate::providers::sec::SecProvider;use crate::providers::yahoo::YahooProvider;use crate::providers::{ self, DailyBar, DividendEvent, Fact, FilingRecord, FundId, FundMetadata, self, DividendEvent, Fact, FilingRecord, FundId, FundMetadata, FundShape, FundamentalsProvider, HistoryProvider, IntradayBar, OwnershipPerson, PortfolioData, Quote, QuoteProvider,};use crate::stream::{Hub, QuoteUpdate, StreamEvent};use crate::summary;use crate::{seed, Config};/// How often the loop wakes to check whether a job is due. The jobs themselves
@@ -48,22 +46,12 @@ use crate::{seed, Config};/// (two small SELECTs per wake).const TICK: Duration = Duration::from_secs(60);/// Incremental daily-history refresh cadence.const HISTORY_INTERVAL_SECS: i64 = 6 * 3600;/// Retry cadence after a history run stops early (guard breaker open or hourly/// budget spent) with work still pending — far shorter than the full interval/// so a large backlog (e.g. the symbols a deploy just added, plus any coarse/// series being healed) drains over the next hour rather than over days. The/// guard still gates every request, so a too-soon retry is a cheap no-op while/// the breaker is open.const HISTORY_RETRY_SECS: i64 = 30 * 60;/// A symbol's daily history counts as stale once `history_synced_at` is older/// than this. Kept under 24h so the ~6-hourly job refreshes each symbol about/// once per trading day (markets emit one new daily bar a day) without/// re-fetching symbols already touched a few hours ago.const HISTORY_STALE_SECS: i64 = 20 * 3600;/// Minimum seconds between intraday polls of the same viewed symbol. With the/// 60s tick this yields a ~5-minute per-symbol cadence (PLAN.md Phase C): the/// dashboard's watchlist and an open symbol page refresh about every five/// minutes while watched, not every minute. Just under 5 min so a symbol due at/// the 5-minute mark isn't skipped to the next tick.const INTRADAY_MIN_INTERVAL_SECS: i64 = 4 * 60 + 45;/// Prune cadence and the two retention windows it enforces.const PRUNE_INTERVAL_SECS: i64 = 24 * 3600;
@@ -78,12 +66,6 @@ const FETCH_LOG_RETENTION_DAYS: i64 = 30;/// ceiling (see `routes::symbols`).pub(crate) const YAHOO_BUDGET: i64 = 1000;/// SEC fundamentals job: how often the loop checks whether the sweep is due,/// and how stale a company's SEC data may go before it is re-fetched. SEC/// filings land quarterly, so a weekly refresh is ample.const SEC_INTERVAL_SECS: i64 = 24 * 3600;const SEC_STALE_SECS: i64 = 7 * 24 * 3600;/// Per-hour request ceiling for the SEC endpoint guard. A first-run full sweep/// is one bulk ticker-map call plus two calls per stock (~220 for the starter/// universe); 600 clears that in a single budget hour while still capping a
@@ -103,56 +85,8 @@ const SEC_BUDGET: i64 = 600;/// universe, churned through the SEC endpoint's hourly budget during the/// Phase 14 backfill (markets-closed weekend burn). Smaller chunks spread/// that initial fill across more sweeps without changing the eventual roster.const LEADERSHIP_STALE_SECS: i64 = 30 * 24 * 3600;const LEADERSHIP_MAX_FILINGS: usize = 10;/// Dividend history refresh (Phase 26). Declared dividends are confirmed and/// stable once an ex-date passes, so the data changes slowly: a stock pays out/// at most a handful of times a year. A weekly cadence is plenty to land each/// new payment within a few days while keeping the Yahoo budget light.const DIVIDENDS_INTERVAL_SECS: i64 = 24 * 3600;const DIVIDENDS_STALE_SECS: i64 = 7 * 24 * 3600;/// ETF fund metadata refresh (Phase 28). The Yahoo `quoteSummary` figures —/// expense ratio, distribution yield, NAV, inception, category, fund family,/// strategy summary — change slowly (the prospectus is updated once a year),/// so a monthly staleness window is enough. The due-check is daily so a/// newly-stale ETF is picked up within a day, but the steady-state cost is/// one request per ETF per month.const FUND_METADATA_INTERVAL_SECS: i64 = 24 * 3600;const FUND_METADATA_STALE_SECS: i64 = 30 * 24 * 3600;/// ETF NAV refresh (Phase 4). NAV is struck once per trading day, and the/// price-vs-NAV premium behind the ETF quality read's "tracking" factor is only/// meaningful against a *fresh* NAV — so unlike the 30-day `fund_metadata`/// sweep (which owns the slow static fields, NAV included but far too stale to/// read a premium against), this job refreshes *only* nav_price, daily. One/// lightweight `quoteSummary` request per ETF (~43/day) through the shared/// `yahoo` `EndpointGuard` — negligible against the hourly budget. The 20h/// staleness window (< the 24h interval) means every ETF refreshes once per/// daily run.const FUND_NAV_INTERVAL_SECS: i64 = 24 * 3600;const FUND_NAV_STALE_SECS: i64 = 20 * 3600;/// Earnings calendar refresh (Phase 25). Yahoo's `calendarEvents.earnings`/// rolls forward one quarter at a time as each print lands, and a stock's/// next date is irrelevant before it shifts — so a monthly cadence is/// enough to land each new date within a few weeks of when Yahoo learns it./// Daily due-check; one request per stock through the shared `yahoo`/// `EndpointGuard`. The whole sweep also re-runs once the stored/// `next_earnings_at` passes, so a missed roll-forward never sits stale.const EARNINGS_INTERVAL_SECS: i64 = 24 * 3600;const EARNINGS_STALE_SECS: i64 = 30 * 24 * 3600;/// Asset-profile refresh (Phase 15). A company's sector and industry/// classification changes only on a structural shift (a reverse-merger or a/// reclassification by the index provider), so a monthly cadence is plenty./// Daily due-check; one request per stock through the shared `yahoo`/// `EndpointGuard`. ~512 stocks × monthly = ~17 requests/day in steady/// state, well below the 1000/hr budget.const ASSET_PROFILE_INTERVAL_SECS: i64 = 24 * 3600;const ASSET_PROFILE_STALE_SECS: i64 = 30 * 24 * 3600;/// Spawn the scheduler. The returned handle is normally dropped: dropping it/// detaches the task, which then runs for the lifetime of the process.pub fn spawn(pool: SqlitePool, config: Arc<Config>, hub: Arc<Hub>) -> JoinHandle<()> {
@@ -166,69 +100,13 @@ pub fn spawn(pool: SqlitePool, config: Arc<Config>, hub: Arc<Hub>) -> JoinHandle tracing::warn!("[scheduler] register endpoints: {e}"); } // History comes from Yahoo now (no API key needed), so the seed and // incremental refresh always run. if let Err(e) = run_boot_seed(&pool, &config, &hub).await { // Reconcile the universe rows from the curated CSV (local, no network). // No history is fetched here any more — a symbol's data fills in on // demand the first time its page is viewed (Phase B). if let Err(e) = run_boot_seed(&pool, &config).await { tracing::warn!("[scheduler] boot seed: {e:#}"); } // SEC's fair-access policy asks consumers to identify themselves; with // no contact email configured the SEC job stays off rather than make // anonymous requests. let sec_enabled = !config.sec_contact_email.is_empty(); if !sec_enabled { tracing::warn!( "[scheduler] SEC_CONTACT_EMAIL unset: SEC fundamentals & filings job disabled" ); } else if let Err(e) = schedule_next(&pool, "sec", now_ms()).await { // Bring the SEC job forward to the first tick. The sweep is // resumable and cheap when nothing is stale, so running it on // each boot is harmless — and it means a deploy that introduces // new SEC-backed data (e.g. the Phase 18 ETF profiles) backfills // within a tick instead of waiting out the ~24h interval. tracing::warn!("[scheduler] bring sec job forward: {e}"); } // Dividends job (Phase 26): bring it forward the same way the SEC job // is, so a deploy adding the table backfills the universe within a // tick rather than waiting out the daily interval. The sweep is // resumable and the no-stale fast path is free. if let Err(e) = schedule_next(&pool, "dividends", now_ms()).await { tracing::warn!("[scheduler] bring dividends job forward: {e}"); } // Fund-metadata job (Phase 28): same pattern as the SEC and dividends // jobs — bring it forward to the first tick so a deploy that adds the // table backfills the ETF universe within a tick rather than waiting // out the daily interval. Resumable; the no-stale fast path is free. if let Err(e) = schedule_next(&pool, "fund_metadata", now_ms()).await { tracing::warn!("[scheduler] bring fund_metadata job forward: {e}"); } // Fund-NAV job (Phase 4): same bring-forward pattern, so a deploy adding // the daily NAV refresh populates fresh NAVs — and thus the ETF quality // read's tracking factor — within a tick rather than waiting out the // daily interval. Resumable; the no-stale fast path is free. if let Err(e) = schedule_next(&pool, "fund_nav", now_ms()).await { tracing::warn!("[scheduler] bring fund_nav job forward: {e}"); } // Earnings calendar job (Phase 25): same bring-forward pattern, so a // deploy adding the columns backfills the stock universe within a // tick rather than the daily interval. Resumable; no-stale fast path // is free. if let Err(e) = schedule_next(&pool, "earnings_calendar", now_ms()).await { tracing::warn!("[scheduler] bring earnings_calendar job forward: {e}"); } // Asset profile job (Phase 15): same bring-forward pattern. A fresh // deploy populates each curated stock's sector and industry within // hours rather than the daily interval; resumable, the no-stale fast // path is free. if let Err(e) = schedule_next(&pool, "asset_profile", now_ms()).await { tracing::warn!("[scheduler] bring asset_profile job forward: {e}"); } // Prune's last-run time is loop-local: a restart simply re-prunes once, // which is harmless (local-only DELETEs, no network). let mut last_prune: Option<i64> = None;
@@ -249,117 +127,17 @@ pub fn spawn(pool: SqlitePool, config: Arc<Config>, hub: Arc<Hub>) -> JoinHandle hub.publish(StreamEvent::Market { session: session.as_str().to_string(), }); // The lead index swaps between the cash S&P and its future on a // session change, so the verdict's basis moves — push a fresh // summary so an open dashboard re-reads against the new lead. hub.publish(StreamEvent::Summary( summary::market_summary(&pool, session).await, )); last_session = Some(session); } match is_due(&pool, "history", now_ms()).await { Ok(true) => { if let Err(e) = run_history(&pool, &config, &hub).await { tracing::warn!("[scheduler] history: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] history due-check: {e}"), } if sec_enabled { match is_due(&pool, "sec", now_ms()).await { Ok(true) => { if let Err(e) = run_sec(&pool, &config, &hub).await { tracing::warn!("[scheduler] sec: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] sec due-check: {e}"), } } // Dividend payouts (Phase 26 + 28): sweep stocks AND ETFs whose // dividend / distribution history has gone stale (weekly). Runs // only when Yahoo is reachable; gated nowhere else — declared // events drift after the ex-date passes, so a fresh pull every // week is enough. match is_due(&pool, "dividends", now_ms()).await { Ok(true) => { if let Err(e) = run_dividends(&pool, &config, &hub).await { tracing::warn!("[scheduler] dividends: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] dividends due-check: {e}"), } // ETF fund metadata (Phase 28): sweep ETFs whose Yahoo // quoteSummary snapshot has gone stale (monthly). Same Yahoo // guard as intraday + daily-close + dividends; expense / yield / // NAV / strategy change rarely, so this is cheap in steady state. match is_due(&pool, "fund_metadata", now_ms()).await { Ok(true) => { if let Err(e) = run_fund_metadata(&pool, &config, &hub).await { tracing::warn!("[scheduler] fund_metadata: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] fund_metadata due-check: {e}"), } // ETF NAV (Phase 4): refresh every ETF's NAV daily so the // price-vs-NAV premium behind the quality read's tracking factor // stays current. Same Yahoo guard; one lightweight request per ETF. match is_due(&pool, "fund_nav", now_ms()).await { Ok(true) => { if let Err(e) = run_fund_nav(&pool, &config, &hub).await { tracing::warn!("[scheduler] fund_nav: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] fund_nav due-check: {e}"), } // Earnings calendar (Phase 25): sweep stocks whose next-expected // earnings date has gone stale or already passed. Same Yahoo // guard; one request per stock per month in steady state. match is_due(&pool, "earnings_calendar", now_ms()).await { Ok(true) => { if let Err(e) = run_earnings_calendar(&pool, &config, &hub).await { tracing::warn!("[scheduler] earnings_calendar: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] earnings_calendar due-check: {e}"), } // Asset profile (Phase 15): sweep stocks whose sector / industry // classification has gone stale (monthly). Same Yahoo guard; one // request per stock per month in steady state. match is_due(&pool, "asset_profile", now_ms()).await { Ok(true) => { if let Err(e) = run_asset_profile(&pool, &config, &hub).await { tracing::warn!("[scheduler] asset_profile: {e:#}"); } } Ok(false) => {} Err(e) => tracing::warn!("[scheduler] asset_profile due-check: {e}"), } // Intraday quotes: demand-driven (only symbols a browser is // viewing). Inside a trading session every viewed symbol is // polled; outside it, only viewed futures, which trade nearly // around the clock. Does no network work when neither applies. // around the clock. Does no network work when nobody is watching. if let Err(e) = run_intraday(&pool, &config, &hub, session).await { tracing::warn!("[scheduler] intraday: {e:#}"); } if let Err(e) = run_daily_close_if_due(&pool, &config, &hub).await { tracing::warn!("[scheduler] daily close: {e:#}"); } if let Err(e) = run_prune_if_due(&pool, &mut last_prune, &hub).await { tracing::warn!("[scheduler] prune: {e:#}"); }
@@ -378,6 +156,13 @@ async fn register_endpoints(pool: &SqlitePool) -> anyhow::Result<()> { sqlx::query("DELETE FROM endpoint_guard WHERE endpoint = 'stooq'") .execute(pool) .await?; // The demand-only refocus (Phase A) removed every timed sweep. Drop their // leftover `data_status` rows so `/health` lists only the jobs that still // run (the demand-driven intraday poll and the local prune); a prod DB // carries rows from the old jobs that would otherwise show as stale. sqlx::query("DELETE FROM data_status WHERE job NOT IN ('intraday', 'prune')") .execute(pool) .await?; EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET) .ensure_registered() .await?;
@@ -397,251 +182,23 @@ async fn reset_states_on_boot(pool: &SqlitePool) -> sqlx::Result<()> { Ok(())}/// Run the first-run universe seed while `meta.seed_completed` is unset.////// `seed::run` is itself idempotent (symbols are upserted) and resumable/// (symbols that already hold history are skipped), so re-running it on each/// boot until the seed completes is cheap. Afterwards the incremental history/// job is deferred one full interval so it does not re-fetch, on this same/// boot, the handful of symbols the seed just touched.async fn run_boot_seed(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { if get_meta(pool, "seed_completed").await?.as_deref() == Some("1") { // The full backfill is done, but still reconcile the curated list each // boot so a CSV edit (symbols added to or dropped from the universe) // takes effect on deploy without a manual re-seed. This is local-only // and cheap; any newly-added symbol's history is backfilled by the // incremental `run_history` job (it picks up `history_synced_at IS NULL`). match seed::sync_universe(pool, config).await { Ok(r) if r.pruned > 0 => { tracing::info!("[scheduler] universe sync: {} symbols, {} pruned", r.total, r.pruned) } Ok(_) => {} Err(e) => tracing::warn!("[scheduler] universe sync: {e:#}"), } // Run the history job promptly (next tick) rather than waiting out the // remaining interval from before the restart. A deploy that adds symbols // or ships a history fix should reconcile data soon, not up to // HISTORY_INTERVAL_SECS later: the job backfills freshly-added symbols // and self-heals any coarse stored series. It is a cheap no-op when // nothing needs work. schedule_next(pool, "history", now_ms()).await?; return Ok(()); } tracing::info!("[scheduler] seed_completed unset: running first-run seed"); let started = now_ms(); mark_fetching(pool, "seed").await?; notify_health(hub); let yahoo = YahooProvider::new(providers::http::build_client(config)); let t0 = Instant::now(); let result = seed::run(pool, config, &yahoo).await; let dur = t0.elapsed().as_millis() as i64; match &result { Ok(()) => { let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM symbols WHERE is_seeded = 1") .fetch_one(pool) .await?; let with_history: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM symbols WHERE is_seeded = 1 AND history_last_date IS NOT NULL", ) .fetch_one(pool) .await?; let detail = format!("{with_history}/{total} seeded symbols have history"); log_fetch(pool, "seed", "yahoo", "ok", Some(&detail), Some(with_history), dur, started) .await?; mark_ok(pool, "seed", None).await?; } Err(e) => { let msg = format!("{e:#}"); log_fetch(pool, "seed", "yahoo", "error", Some(&msg), None, dur, started).await?; mark_error(pool, "seed", &msg, None).await?; } } // Defer the first incremental refresh: it would otherwise re-fetch the same // still-stale symbols the seed just handled. schedule_next(pool, "history", now_ms() + HISTORY_INTERVAL_SECS * 1000).await?; notify_health(hub); result}/// Incremental daily-history refresh: re-fetch only the symbols whose stored/// history has gone stale, asking Yahoo for the window since each symbol's/// last stored bar. Paced and circuit-broken like the seed./// Reconcile the universe rows from the curated CSV on every boot — local only,/// no network. Upserts every listed symbol and prunes curated rows dropped from/// the CSV, so a `starter.csv` edit takes effect on deploy without a manual/// re-seed (user-added `is_seeded = 0` rows are never touched).////// This is a backstop: on a normal trading day the `daily_close` job already/// appends each symbol's bar (and stamps `history_synced_at`), so nothing is/// stale here. It still earns its keep for symbols missed while the server was/// down at the close, weekends, and freshly added tickers./// True when a stored daily series is actually coarser than daily, judged from/// its *recent* density: `recent_bars` is the number of bars in the trailing 90/// days of the series, which is ~63 for a genuine daily series (trading days),/// ~13 for weekly and ~3 for monthly — so under 30 means coarser than daily./// Only judged once the series spans more than 180 days, so a young-but-daily/// symbol is not flagged for merely having a short history. Mirrors the SQL test/// in [`run_history`]'s selection query.fn is_coarser_than_daily(first: &Option<String>, last: &Option<String>, recent_bars: i64) -> bool { let (Some(f), Some(l)) = (first, last) else { return false; }; if f == l { return false; } let (Ok(fd), Ok(ld)) = ( chrono::NaiveDate::parse_from_str(f, "%Y-%m-%d"), chrono::NaiveDate::parse_from_str(l, "%Y-%m-%d"), ) else { return false; }; (ld - fd).num_days() > 180 && recent_bars < 30}async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + HISTORY_INTERVAL_SECS * 1000; mark_fetching(pool, "history").await?; notify_health(hub); // Every symbol we track, curated or user-added, is eligible: a symbol // added through the Phase 9 add-symbol flow needs its history backfilled // exactly as a seeded one does. (The seed itself stays curated-list-only; // it is the only job that keys on `is_seeded`.) Futures are included now — // Yahoo serves `=F` daily history, unlike the Stooq source this replaced. // A symbol is refreshed when any of these hold: // - its history has gone stale (the normal incremental case); // - it holds a single stored bar (`history_first_date = history_last_date`) // — a symbol added after the initial seed whose `history_last_date` was // stamped by a `daily_close` snapshot before any range=max backfill ran, // so the incremental window can never reach its deep history; // - its stored history is coarser than daily. Yahoo silently downsamples // `interval=1d` to weekly / monthly bars when a range spans many years // (see `YahooProvider::is_downsampled`), so a symbol first backfilled via // range=max can hold monthly bars. Detect it by *recent* density — the // bar count in the trailing 90 days of the series: a daily series carries // ~63 (trading days), a weekly one ~13, a monthly one ~3, so fewer than // 30 means coarser than daily. The trailing window (rather than whole- // span density) is what keeps a deep symbol with a sparse pre-modern tail // — e.g. ^SPX, daily for decades but with century-old gaps — from being // misread as coarse. Only judged once a symbol spans over 180 days, so a // young-but-daily symbol is not flagged for its short history. // The last two are selected regardless of staleness so the heal is not gated // behind the stale cutoff; both are repaired with a full daily re-fetch (the // `deep` branch below). `recent_bars` rides along so the same test can pick // the fetch window without a second query. let cutoff = started - HISTORY_STALE_SECS * 1000; let stale: Vec<(String, Option<String>, Option<String>, i64)> = sqlx::query_as( "SELECT s.ticker, s.history_first_date, s.history_last_date, \ (SELECT COUNT(*) FROM daily_prices d WHERE d.ticker = s.ticker \ AND d.d >= date(s.history_last_date, '-90 days')) AS recent_bars \ FROM symbols s \ WHERE s.history_synced_at IS NULL OR s.history_synced_at < ? \ OR s.history_first_date = s.history_last_date \ OR (s.history_last_date IS NOT NULL \ AND julianday(s.history_last_date) - julianday(s.history_first_date) > 180 \ AND (SELECT COUNT(*) FROM daily_prices d WHERE d.ticker = s.ticker \ AND d.d >= date(s.history_last_date, '-90 days')) < 30) \ ORDER BY s.ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { log_fetch(pool, "history", "yahoo", "ok", Some("no stale symbols"), Some(0), 0, started) .await?; mark_ok(pool, "history", Some(next)).await?; notify_health(hub); return Ok(()); } tracing::info!("[scheduler] history: refreshing {} stale symbols", stale.len()); let yahoo = YahooProvider::new(providers::http::build_client(config)); let t0 = Instant::now(); // Route every request through the persistent endpoint guard: it paces the // loop and refuses requests once the breaker opens or the hourly budget is // spent, so the job stops cleanly instead of hammering a guarded endpoint. // History shares the `yahoo` guard with live quotes, so build it with the // same budget rather than the 200-default `new` ceiling. let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let mut ok = 0usize; let mut total_bars = 0i64; let mut stopped: Option<String> = None; for (ticker, first_date, last_date, recent_bars) in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } // Decide the fetch window. A symbol with no full daily history yet — // none stored, a single bar, or a coarser-than-daily series — needs a // deep re-fetch, so ask for range=max (`None`); the provider re-asks a // bounded window when Yahoo downsamples it. Everything else just wants // the incremental window since its last stored bar. let single_bar = first_date.is_some() && first_date == last_date; let coarse = is_coarser_than_daily(first_date, last_date, *recent_bars); let deep = last_date.is_none() || single_bar || coarse; let since = if deep { None } else { last_date.as_deref() }; match yahoo.daily(ticker, since).await { Ok(bars) if !bars.is_empty() => { guard.record_success().await?; // A coarse series is replaced, not merged: drop its weekly / // monthly bars so the fresh daily ones do not interleave with // leftover old-granularity bars. Done only once we have new // bars in hand, so the symbol is never left empty. if coarse { sqlx::query("DELETE FROM daily_prices WHERE ticker = ?") .bind(ticker) .execute(pool) .await?; } total_bars += bars.len() as i64; seed::store_daily(pool, ticker, &bars).await?; ok += 1; } Ok(_) => { // A valid but empty response: the request succeeded and the // endpoint simply had nothing new. Stamp the symbol checked so // it is not re-fetched until it goes stale again. guard.record_success().await?; mark_history_checked(pool, ticker).await?; } Err(e) => { guard.record_failure(&e).await?; tracing::warn!("[scheduler] history {ticker} failed: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; match stopped { Some(why) => { // The guard cut the run short (breaker open or hourly budget // spent). That is the guard doing its job, not a failure of this // job: record it as `skipped` and let the next cycle retry. let detail = format!( "stopped early ({why}); {ok}/{} refreshed, {total_bars} bars", stale.len() ); tracing::warn!("[scheduler] history: {detail}"); log_fetch(pool, "history", "yahoo", "skipped", Some(&detail), Some(total_bars), dur, started) .await?; // Work remains, so retry soon rather than after the full interval. mark_ok(pool, "history", Some(started + HISTORY_RETRY_SECS * 1000)).await?; } None => { let detail = format!("{ok}/{} symbols refreshed, {total_bars} bars", stale.len()); tracing::info!("[scheduler] history: {detail}"); log_fetch(pool, "history", "yahoo", "ok", Some(&detail), Some(total_bars), dur, started) .await?; mark_ok(pool, "history", Some(next)).await?; }/// Unlike before the demand-only refocus, this no longer backfills any history:/// a symbol's deep daily history and SEC data fill in the first time its page is/// viewed and found stale (Phase B), or via an explicit `make seed` run.async fn run_boot_seed(pool: &SqlitePool, config: &Config) -> anyhow::Result<()> { match seed::sync_universe(pool, config).await { Ok(r) => tracing::info!( "[scheduler] universe sync: {} symbols, {} pruned", r.total, r.pruned ), Err(e) => tracing::warn!("[scheduler] universe sync: {e:#}"), } notify_health(hub); Ok(())}
@@ -673,7 +230,7 @@ async fn run_intraday( return Ok(()); } // Inside a session, poll every viewed symbol; outside it, only the futures. let targets: Vec<String> = if session.is_open() { let mut targets: Vec<String> = if session.is_open() { viewed } else { let futures: HashSet<String> =
@@ -687,6 +244,25 @@ async fn run_intraday( if targets.is_empty() { return Ok(()); } // Throttle to a ~5-minute per-symbol cadence (PLAN.md Phase C): the loop // ticks every 60s, but a symbol quoted within the last few minutes is left // alone, so a dashboard left open polls each watchlist symbol about once // every five minutes rather than every minute — light on the budget, and // plenty "real-time" for delayed data. A symbol never quoted is always // eligible. The set is small (only viewed symbols), so the lookup is cheap. let throttle_cutoff = now_ms() - INTRADAY_MIN_INTERVAL_SECS * 1000; let recent: HashSet<String> = sqlx::query_scalar( "SELECT ticker FROM symbols WHERE last_quote_at IS NOT NULL AND last_quote_at >= ?", ) .bind(throttle_cutoff) .fetch_all(pool) .await? .into_iter() .collect(); targets.retain(|t| !recent.contains(t)); if targets.is_empty() { return Ok(()); } let started = now_ms(); mark_fetching(pool, "intraday").await?; notify_health(hub);
@@ -750,618 +326,6 @@ async fn run_intraday( mark_ok(pool, "intraday", None).await?; } // If this sweep touched a symbol that moves the dashboard verdict (the broad // index or ^VIX), recompute and push the market summary so an open dashboard // re-reads its hero + figures live. The curated stocks behind breadth don't // move intraday (they only price at the daily close), so a sweep that hit // only a viewed stock page leaves the summary unchanged and we skip it. if ok > 0 { let pulse = summary::pulse_tickers(session); if targets.iter().any(|t| pulse.contains(&t.as_str())) { hub.publish(StreamEvent::Summary(summary::market_summary(pool, session).await)); } } notify_health(hub); Ok(())}/// Build a daily OHLCV bar from a closing quote. `close` is the quote's last/// price; open/high/low fall back to the close when Yahoo's `meta` left them/// null (some indexes carry only a level), and volume defaults to 0.fn daily_bar_from_quote(date: &str, q: &Quote) -> DailyBar { let close = q.price; DailyBar { d: date.to_string(), open: q.open.unwrap_or(close), high: q.day_high.unwrap_or(close), low: q.day_low.unwrap_or(close), close, volume: q.volume.unwrap_or(0), }}/// Once-a-day closing snapshot of the whole universe.////// Shortly after the regular session closes (>= 16:05 ET on a weekday), fetch/// a Yahoo quote for every seeded symbol so each one carries a same-day close/// even if nobody viewed it. Keyed on the ET trading date in `meta` so it runs/// exactly once per day; a guard stop leaves the date unset so the next cycle/// finishes the rest.async fn run_daily_close_if_due( pool: &SqlitePool, config: &Config, hub: &Hub,) -> anyhow::Result<()> { let now = chrono::Utc::now(); if !market::is_et_weekday(now) || !market::after_close(now) { return Ok(()); } let date = market::et_date(now); if get_meta(pool, "daily_close_date").await?.as_deref() == Some(date.as_str()) { return Ok(()); } // The whole tracked universe, curated and user-added alike, so every // symbol carries a same-day close even if nobody viewed it. let symbols: Vec<String> = sqlx::query_scalar("SELECT ticker FROM symbols ORDER BY ticker") .fetch_all(pool) .await?; if symbols.is_empty() { return Ok(()); } tracing::info!("[scheduler] daily close: snapshotting {} symbols for {date}", symbols.len()); let started = now_ms(); mark_fetching(pool, "daily_close").await?; notify_health(hub); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &symbols { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.quote(ticker).await { Ok(data) => { guard.record_success().await?; store_quote(pool, ticker, &data.quote).await?; if !data.bars.is_empty() { store_intraday(pool, ticker, &data.bars).await?; } // Append today's daily bar from the same quote (no extra // request), so `daily_prices` stays current with Yahoo as the // sole price source — no separate per-symbol history sweep. seed::store_daily(pool, ticker, &[daily_bar_from_quote(&date, &data.quote)]) .await?; hub.publish(StreamEvent::Quote(QuoteUpdate::new( ticker.clone(), data.quote.price, data.quote.prev_close, data.quote.market_state.clone(), ))); ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] daily close {ticker} failed: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; match stopped { Some(why) => { // Leave `daily_close_date` unset: the next cycle retries and // finishes the symbols this run did not reach. let detail = format!("stopped early ({why}); {ok}/{} done", symbols.len()); tracing::warn!("[scheduler] daily close: {detail}"); log_fetch(pool, "daily_close", "yahoo", "skipped", Some(&detail), Some(ok), dur, started) .await?; mark_ok(pool, "daily_close", None).await?; } None => { set_meta(pool, "daily_close_date", &date).await?; let detail = format!("{ok}/{} symbols, {errors} errors", symbols.len()); tracing::info!("[scheduler] daily close: {detail}"); log_fetch(pool, "daily_close", "yahoo", "ok", Some(&detail), Some(ok), dur, started) .await?; mark_ok(pool, "daily_close", None).await?; } } // The close just re-priced the whole universe, so breadth and the verdict // have moved — push a fresh summary for any dashboard still open after the // bell. Use the live session so the lead index resolves the same way the // page would (the close runs just after 16:00 ET, in the post session). if ok > 0 { let session = market::session_at(chrono::Utc::now()); hub.publish(StreamEvent::Summary(summary::market_summary(pool, session).await)); } notify_health(hub); Ok(())}/// SEC fundamentals, filings & ETF fund-profile sweep.////// On the first run (and whenever new symbols appear) two bulk ticker-map/// fetches fill in CIKs — `company_tickers.json` for stocks,/// `company_tickers_mf.json` for ETFs. Then every symbol whose SEC data has/// gone stale is refreshed:/// - a stock's XBRL `companyfacts` into `fundamentals`, its submission/// history into `filings`;/// - an ETF's latest N-PORT into `fund_profiles` + `fund_holdings`, its/// filing history into `filings`. A physical-commodity grantor trust files/// no N-PORT, so its AUM comes from `companyfacts` instead./// - a stock's officer/board roster into `leadership`, parsed from its recent/// Form 3/4/5 ownership filings (Phase 14, on a slower monthly cadence)./// Indexes are skipped; they do not file with the SEC.////// Resumable like the history job: each symbol's sync timestamps are stamped/// only on a successful fetch, so a guard stop simply leaves the rest for the/// next cycle.async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + SEC_INTERVAL_SECS * 1000; mark_fetching(pool, "sec").await?; notify_health(hub); let sec = SecProvider::new(providers::http::build_sec_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), sec.name(), SEC_BUDGET); let t0 = Instant::now(); // 1. Stock CIK resolution. One bulk call maps the whole market; only // needed while some stock still lacks a CIK. let missing: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM symbols WHERE kind = 'stock' AND cik IS NULL", ) .fetch_one(pool) .await?; if missing > 0 { match guard.acquire().await? { Permit::Granted => match sec.cik_map().await { Ok(map) => { guard.record_success().await?; let resolved = resolve_ciks(pool, &map).await?; tracing::info!("[scheduler] sec: resolved {resolved}/{missing} CIKs"); } Err(e) => { guard.record_failure(&e).await?; let msg = format!("CIK map: {e:#}"); let dur = t0.elapsed().as_millis() as i64; log_fetch(pool, "sec", "sec", "error", Some(&msg), None, dur, started).await?; mark_error(pool, "sec", &msg, Some(next)).await?; notify_health(hub); return Ok(()); } }, Permit::Denied(why) => { let dur = t0.elapsed().as_millis() as i64; let detail = format!("stopped before CIK map ({why})"); log_fetch(pool, "sec", "sec", "skipped", Some(&detail), Some(0), dur, started) .await?; mark_ok(pool, "sec", Some(next)).await?; notify_health(hub); return Ok(()); } } } // 1b. ETF fund-CIK resolution from the mutual-fund ticker map. Best-effort: // a guard stop or error here only leaves the ETFs for a later cycle, // rather than aborting the stock sweep below. let etfs_missing: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM symbols WHERE kind = 'etf' AND cik IS NULL", ) .fetch_one(pool) .await?; if etfs_missing > 0 { match guard.acquire().await? { Permit::Granted => match sec.fund_ticker_map().await { Ok(map) => { guard.record_success().await?; let resolved = resolve_fund_ciks(pool, &map).await?; tracing::info!("[scheduler] sec: resolved {resolved}/{etfs_missing} fund CIKs"); } Err(e) => { guard.record_failure(&e).await?; tracing::warn!("[scheduler] sec fund CIK map: {e:#}"); } }, Permit::Denied(why) => { tracing::info!("[scheduler] sec: fund CIK map skipped ({why})"); } } } // 2. Stale sweep. A symbol is due when one of its SEC timestamps is unset // or older than the staleness window. let cutoff = started - SEC_STALE_SECS * 1000; let stale_stocks: Vec<(String, String, Option<i64>, Option<i64>)> = sqlx::query_as( "SELECT ticker, cik, fundamentals_synced_at, filings_synced_at FROM symbols \ WHERE kind = 'stock' AND cik IS NOT NULL \ AND (fundamentals_synced_at IS NULL OR filings_synced_at IS NULL \ OR fundamentals_synced_at < ? OR filings_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .bind(cutoff) .fetch_all(pool) .await?; let stale_etfs: Vec<(String, String, Option<String>)> = sqlx::query_as( "SELECT ticker, cik, series_id FROM symbols \ WHERE kind = 'etf' AND cik IS NOT NULL \ AND (fund_synced_at IS NULL OR fund_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .fetch_all(pool) .await?; // Leadership has its own, longer staleness window (see LEADERSHIP_STALE_SECS). let leadership_cutoff = started - LEADERSHIP_STALE_SECS * 1000; let stale_leadership: Vec<(String, String, Option<i64>)> = sqlx::query_as( "SELECT ticker, cik, leadership_synced_at FROM symbols \ WHERE kind = 'stock' AND cik IS NOT NULL \ AND (leadership_synced_at IS NULL OR leadership_synced_at < ?) \ ORDER BY ticker", ) .bind(leadership_cutoff) .fetch_all(pool) .await?; if stale_stocks.is_empty() && stale_etfs.is_empty() && stale_leadership.is_empty() { log_fetch(pool, "sec", "sec", "ok", Some("no stale companies or funds"), Some(0), 0, started) .await?; mark_ok(pool, "sec", Some(next)).await?; notify_health(hub); return Ok(()); } tracing::info!( "[scheduler] sec: refreshing {} companies, {} funds, {} rosters", stale_stocks.len(), stale_etfs.len(), stale_leadership.len() ); // A metric is due when its timestamp is unset or past the cutoff. let due = |at: Option<i64>| at.map_or(true, |t| t < cutoff); let mut funds_ok = 0i64; let mut filings_ok = 0i64; let mut etfs_ok = 0i64; let mut leaders_ok = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; 'stocks: for (ticker, cik, f_at, fl_at) in &stale_stocks { if due(*f_at) { match guard.acquire().await? { Permit::Granted => match sec.facts(cik).await { Ok(facts) => { guard.record_success().await?; store_fundamentals(pool, ticker, &facts).await?; mark_sec_synced(pool, ticker, "fundamentals_synced_at").await?; funds_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec facts {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'stocks; } } } if due(*fl_at) { match guard.acquire().await? { Permit::Granted => match sec.filings(cik).await { Ok(filings) => { guard.record_success().await?; store_filings(pool, ticker, &filings).await?; mark_sec_synced(pool, ticker, "filings_synced_at").await?; filings_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec filings {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'stocks; } } } } // 3. ETF fund-profile sweep, sharing the guard and the early-exit. Skipped // wholesale if the stock sweep above already hit a guard stop. if stopped.is_none() { 'funds: for (ticker, cik, series_id) in &stale_etfs { let id = FundId { cik: cik.clone(), series_id: series_id.clone(), }; // 3a. The filing list, and what it says about the fund's shape. let shape = match guard.acquire().await? { Permit::Granted => match sec.fund_filings(&id).await { Ok(ff) => { guard.record_success().await?; store_filings(pool, ticker, &ff.filings).await?; ff.shape } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_filings {ticker} failed: {e:#}"); continue 'funds; } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }; // 3b. Holdings from N-PORT, or AUM for a commodity trust. match shape { FundShape::Portfolio { nport_href } => match guard.acquire().await? { Permit::Granted => match sec.fund_portfolio(&nport_href).await { Ok(portfolio) => { guard.record_success().await?; store_fund_portfolio(pool, ticker, &portfolio).await?; mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_portfolio {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }, FundShape::CommodityTrust => match guard.acquire().await? { Permit::Granted => match sec.fund_aum(cik).await { Ok(aum) => { guard.record_success().await?; store_fund_commodity(pool, ticker, aum).await?; mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_aum {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }, FundShape::Unknown => { // The filing list synced but there is no portfolio to // record. Stamp it so it is not retried until next stale. mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } } } } // 4. Leadership sweep (Phase 14): for each stale stock, parse a window of // its recent Form 3/4/5 ownership filings into the officer/board roster. // Shares the guard and the early-exit; skipped wholesale once the sweeps // above have already hit a guard stop. if stopped.is_none() { 'leaders: for (ticker, cik, lead_at) in &stale_leadership { // The company's recent ownership filings, newest first. let index = match guard.acquire().await? { Permit::Granted => match sec.ownership_index(cik).await { Ok(idx) => { guard.record_success().await?; idx } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec ownership_index {ticker} failed: {e:#}"); continue 'leaders; } }, Permit::Denied(why) => { stopped = Some(why); break 'leaders; } }; // First sweep (no prior sync): the most recent filings. Later // sweeps: only filings since the last sync, with a few days' slack, // so the steady-state cost is a handful of requests per company. let since: Option<String> = lead_at.and_then(|ms| { chrono::DateTime::from_timestamp_millis(ms - 5 * 86_400_000) .map(|dt| dt.format("%Y-%m-%d").to_string()) }); let to_parse: Vec<_> = index .into_iter() .filter(|f| since.as_deref().map_or(true, |s| f.filed_at.as_str() >= s)) .take(LEADERSHIP_MAX_FILINGS) .collect(); // Parse each filing's XML, keeping the directors and officers (a // filer who is only a >10% owner is not leadership and is dropped). let mut roster: Vec<(OwnershipPerson, String)> = Vec::new(); for f in &to_parse { match guard.acquire().await? { Permit::Granted => { match sec.ownership_doc(cik, &f.accession, &f.primary_doc).await { Ok(people) => { guard.record_success().await?; for p in people { if p.is_director || p.is_officer { roster.push((p, f.filed_at.clone())); } } } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!( "[scheduler] sec ownership_doc {ticker} failed: {e:#}" ); } } } Permit::Denied(why) => { stopped = Some(why); break; } } } // Upsert what was gathered. A guard stop mid-company still stores // the partial roster (the upsert is idempotent) but leaves // `leadership_synced_at` unset so the next cycle finishes the rest. store_leadership(pool, ticker, &roster).await?; if stopped.is_some() { break 'leaders; } mark_sec_synced(pool, ticker, "leadership_synced_at").await?; leaders_ok += 1; } } let dur = t0.elapsed().as_millis() as i64; let counts = format!( "{funds_ok} fundamentals, {filings_ok} filings, {etfs_ok} fund profiles, \ {leaders_ok} rosters, {errors} errors" ); match stopped { Some(why) => { let detail = format!("stopped early ({why}); {counts}"); tracing::warn!("[scheduler] sec: {detail}"); log_fetch(pool, "sec", "sec", "skipped", Some(&detail), Some(funds_ok), dur, started) .await?; } None => { tracing::info!("[scheduler] sec: {counts}"); log_fetch(pool, "sec", "sec", "ok", Some(&counts), Some(funds_ok), dur, started) .await?; } } mark_ok(pool, "sec", Some(next)).await?; notify_health(hub); Ok(())}/// Dividend payout sweep (Phase 26).////// For every stock whose dividend history is stale, ask Yahoo for the last/// five years of declared dividends and upsert them. Stocks only — ETFs,/// indexes and futures do not pay regular dividends in this app's sense; an/// ETF's distributions live in the fund profile, not here. Routed through the/// shared `yahoo` `EndpointGuard` so it shares pacing and the per-hour budget/// with the intraday and daily-close jobs.////// Resumable in the same way as the SEC job: each stock's/// `dividends_synced_at` is stamped only on a successful fetch, so a guard/// stop leaves the rest for the next cycle.async fn run_dividends(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + DIVIDENDS_INTERVAL_SECS * 1000; let cutoff = started - DIVIDENDS_STALE_SECS * 1000; // Phase 28: dividends now covers ETF distributions too — same Yahoo // event series, same store path. Indexes and futures still skipped (the // former do not pay anything, the latter have no concept of dividends). let stale: Vec<String> = sqlx::query_scalar( "SELECT ticker FROM symbols \ WHERE kind IN ('stock', 'etf') \ AND (dividends_synced_at IS NULL OR dividends_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { // The fast path: nothing stale, nothing to do. No fetching banner. mark_ok(pool, "dividends", Some(next)).await?; return Ok(()); } mark_fetching(pool, "dividends").await?; notify_health(hub); tracing::info!("[scheduler] dividends: refreshing {} symbols", stale.len()); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut payouts = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.dividends(ticker).await { Ok(events) => { guard.record_success().await?; payouts += events.len() as i64; if let Err(e) = store_dividends(pool, ticker, &events).await { tracing::warn!("[scheduler] dividends store {ticker}: {e:#}"); errors += 1; continue; } mark_dividends_synced(pool, ticker).await?; ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] dividends {ticker}: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; let detail = format!("{ok}/{} symbols, {payouts} payouts, {errors} errors", stale.len()); match stopped { Some(why) => { let full = format!("stopped early ({why}); {detail}"); tracing::warn!("[scheduler] dividends: {full}"); log_fetch(pool, "dividends", "yahoo", "skipped", Some(&full), Some(ok), dur, started) .await?; } None => { tracing::info!("[scheduler] dividends: {detail}"); log_fetch(pool, "dividends", "yahoo", "ok", Some(&detail), Some(ok), dur, started) .await?; } } mark_ok(pool, "dividends", Some(next)).await?; notify_health(hub); Ok(())}
@@ -1407,108 +371,6 @@ async fn mark_dividends_synced(pool: &SqlitePool, ticker: &str) -> sqlx::Result< Ok(())}// ────────────────── ETF fund_metadata sweep (Phase 28) ────────────────────/// Sweep ETFs whose Yahoo `quoteSummary` snapshot has gone stale (monthly)./// One request per ETF through the shared `yahoo` `EndpointGuard`; expense/// ratio, distribution yield, NAV, inception, category, fund family, and/// the strategy paragraph are all returned by one request, so even the full/// 28-ETF sweep is well inside the hourly budget. Mirrors `run_dividends`'s/// shape — guard-paced, resumable, broadcast-to-/health.async fn run_fund_metadata(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + FUND_METADATA_INTERVAL_SECS * 1000; let cutoff = started - FUND_METADATA_STALE_SECS * 1000; let stale: Vec<String> = sqlx::query_scalar( "SELECT ticker FROM symbols \ WHERE kind = 'etf' \ AND (fund_metadata_synced_at IS NULL OR fund_metadata_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { // Fast path: no ETFs need refreshing. No fetching banner, no log row. mark_ok(pool, "fund_metadata", Some(next)).await?; return Ok(()); } mark_fetching(pool, "fund_metadata").await?; notify_health(hub); tracing::info!("[scheduler] fund_metadata: refreshing {} ETFs", stale.len()); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut empty = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.fund_metadata(ticker).await { Ok(Some(meta)) => { guard.record_success().await?; if let Err(e) = store_fund_metadata(pool, ticker, &meta).await { tracing::warn!("[scheduler] fund_metadata store {ticker}: {e:#}"); errors += 1; continue; } mark_fund_metadata_synced(pool, ticker).await?; ok += 1; } // Yahoo answered cleanly but the ETF has no fund modules (a tiny // or obscure ticker): stamp it checked so we do not re-fetch the // same empty answer next tick, but log the empty. Ok(None) => { guard.record_success().await?; mark_fund_metadata_synced(pool, ticker).await?; empty += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] fund_metadata {ticker}: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; let detail = format!( "{ok}/{} ETFs ({empty} empty, {errors} errors)", stale.len() ); match stopped { Some(why) => { let full = format!("stopped early ({why}); {detail}"); tracing::warn!("[scheduler] fund_metadata: {full}"); log_fetch( pool, "fund_metadata", "yahoo", "skipped", Some(&full), Some(ok), dur, started, ).await?; } None => { tracing::info!("[scheduler] fund_metadata: {detail}"); log_fetch( pool, "fund_metadata", "yahoo", "ok", Some(&detail), Some(ok), dur, started, ).await?; } } mark_ok(pool, "fund_metadata", Some(next)).await?; notify_health(hub); Ok(())}/// Upsert one ETF's Yahoo `quoteSummary` metadata. Yahoo serves the full/// current snapshot each call, so the row is replaced wholesale: a field/// Yahoo no longer carries on a refresh becomes `NULL` here, rather than
@@ -1519,155 +381,31 @@ pub(crate) async fn store_fund_metadata( m: &FundMetadata,) -> sqlx::Result<()> { sqlx::query( "INSERT INTO fund_metadata \ (ticker, expense_ratio, yield_pct, trailing_yield_pct, nav_price, \ inception_date, category, fund_family, strategy_summary, updated_at) \ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) \ ON CONFLICT(ticker) DO UPDATE SET \ expense_ratio = excluded.expense_ratio, \ yield_pct = excluded.yield_pct, \ trailing_yield_pct = excluded.trailing_yield_pct, \ nav_price = excluded.nav_price, \ inception_date = excluded.inception_date, \ category = excluded.category, \ fund_family = excluded.fund_family, \ strategy_summary = excluded.strategy_summary, \ updated_at = excluded.updated_at", ) .bind(ticker) .bind(m.expense_ratio) .bind(m.yield_pct) .bind(m.trailing_yield_pct) .bind(m.nav_price) .bind(&m.inception_date) .bind(&m.category) .bind(&m.fund_family) .bind(&m.strategy_summary) .bind(now_ms()) .execute(pool) .await?; Ok(())}// ─────────────────────── ETF NAV daily refresh (Phase 4) ──────────────────/// Refresh every ETF's NAV daily so the price-vs-NAV premium behind the ETF/// quality read's "tracking" factor (Phase 4) stays current. NAV is struck once/// per trading day; the 30-day `fund_metadata` sweep carries a NAV too but far/// too stale to read a premium against, so this job owns the daily nav_price +/// nav_synced_at on its own cadence, leaving the slow static fields alone. One/// lightweight `quoteSummary` request per ETF through the shared `yahoo` guard./// Mirrors `run_fund_metadata`'s shape — guard-paced, resumable, /health-aware.async fn run_fund_nav(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + FUND_NAV_INTERVAL_SECS * 1000; let cutoff = started - FUND_NAV_STALE_SECS * 1000; let stale: Vec<String> = sqlx::query_scalar( "SELECT s.ticker FROM symbols s \ LEFT JOIN fund_metadata m ON m.ticker = s.ticker \ WHERE s.kind = 'etf' \ AND (m.nav_synced_at IS NULL OR m.nav_synced_at < ?) \ ORDER BY s.ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { // Fast path: every ETF's NAV is fresh. No fetching banner, no log row. mark_ok(pool, "fund_nav", Some(next)).await?; return Ok(()); } mark_fetching(pool, "fund_nav").await?; notify_health(hub); tracing::info!("[scheduler] fund_nav: refreshing {} ETFs", stale.len()); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut empty = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.fund_nav(ticker).await { Ok(nav) => { guard.record_success().await?; // `nav` may be None (Yahoo knows the symbol but carries no NAV // today): still stamp nav_synced_at so we do not re-fetch the // same empty next tick — the freshness gate then simply finds no // premium and the tracking factor drops out. if let Err(e) = store_fund_nav(pool, ticker, nav).await { tracing::warn!("[scheduler] fund_nav store {ticker}: {e:#}"); errors += 1; continue; } if nav.is_some() { ok += 1; } else { empty += 1; } } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] fund_nav {ticker}: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; let detail = format!("{ok}/{} ETFs ({empty} no NAV, {errors} errors)", stale.len()); match stopped { Some(why) => { let full = format!("stopped early ({why}); {detail}"); tracing::warn!("[scheduler] fund_nav: {full}"); log_fetch( pool, "fund_nav", "yahoo", "skipped", Some(&full), Some(ok), dur, started, ).await?; } None => { tracing::info!("[scheduler] fund_nav: {detail}"); log_fetch( pool, "fund_nav", "yahoo", "ok", Some(&detail), Some(ok), dur, started, ).await?; } } mark_ok(pool, "fund_nav", Some(next)).await?; notify_health(hub); Ok(())}/// Upsert one ETF's freshly-fetched NAV + its sync stamp, touching only the NAV/// columns so the static fields the 30-day `fund_metadata` sweep owns are left/// intact. Inserts a NAV-only row if the metadata sweep has not reached this/// ETF yet (the other columns fill in on its next run). A `None` nav clears any/// prior NAV — honest: we have no fresh value to read a premium against.async fn store_fund_nav(pool: &SqlitePool, ticker: &str, nav: Option<f64>) -> sqlx::Result<()> { let now = now_ms(); sqlx::query( "INSERT INTO fund_metadata (ticker, nav_price, nav_synced_at, updated_at) \ VALUES (?, ?, ?, ?) \ "INSERT INTO fund_metadata \ (ticker, expense_ratio, yield_pct, trailing_yield_pct, nav_price, \ inception_date, category, fund_family, strategy_summary, updated_at) \ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) \ ON CONFLICT(ticker) DO UPDATE SET \ expense_ratio = excluded.expense_ratio, \ yield_pct = excluded.yield_pct, \ trailing_yield_pct = excluded.trailing_yield_pct, \ nav_price = excluded.nav_price, \ nav_synced_at = excluded.nav_synced_at, \ inception_date = excluded.inception_date, \ category = excluded.category, \ fund_family = excluded.fund_family, \ strategy_summary = excluded.strategy_summary, \ updated_at = excluded.updated_at", ) .bind(ticker) .bind(nav) .bind(now) .bind(now) .bind(m.expense_ratio) .bind(m.yield_pct) .bind(m.trailing_yield_pct) .bind(m.nav_price) .bind(&m.inception_date) .bind(&m.category) .bind(&m.fund_family) .bind(&m.strategy_summary) .bind(now_ms()) .execute(pool) .await?; Ok(())
@@ -1688,125 +426,6 @@ async fn mark_fund_metadata_synced(pool: &SqlitePool, ticker: &str) -> sqlx::Res Ok(())}// ────────────── stock earnings calendar sweep (Phase 25) ──────────────────/// Sweep stocks whose next-expected earnings date has gone stale (monthly)/// or already passed. One request per stock through the shared `yahoo`/// `EndpointGuard`; mirrors `run_fund_metadata`'s shape — guard-paced,/// resumable, broadcast-to-/health.async fn run_earnings_calendar( pool: &SqlitePool, config: &Config, hub: &Hub,) -> anyhow::Result<()> { let started = now_ms(); let next = started + EARNINGS_INTERVAL_SECS * 1000; let cutoff = started - EARNINGS_STALE_SECS * 1000; // Refresh a stock when either its sync-timestamp has aged out OR its // stored next date has already passed (the print landed; Yahoo should // carry the following quarter's date by now). let stale: Vec<String> = sqlx::query_scalar( "SELECT ticker FROM symbols \ WHERE kind = 'stock' \ AND ( \ earnings_synced_at IS NULL OR earnings_synced_at < ? \ OR (next_earnings_at IS NOT NULL AND next_earnings_at < ?) \ ) \ ORDER BY ticker", ) .bind(cutoff) .bind(started) .fetch_all(pool) .await?; if stale.is_empty() { mark_ok(pool, "earnings_calendar", Some(next)).await?; return Ok(()); } mark_fetching(pool, "earnings_calendar").await?; notify_health(hub); tracing::info!( "[scheduler] earnings_calendar: refreshing {} stocks", stale.len() ); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut empty = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.earnings_calendar(ticker).await { Ok(Some(ts_ms)) => { guard.record_success().await?; if let Err(e) = store_earnings_next(pool, ticker, Some(ts_ms)).await { tracing::warn!("[scheduler] earnings_calendar store {ticker}: {e:#}"); errors += 1; continue; } ok += 1; } // Yahoo answered cleanly but has no upcoming date for this // stock (uneven coverage). Clear the stored next date so the // symbol page falls back to a cadence estimate, and stamp the // sync so the next sweep does not re-fetch the same empty. Ok(None) => { guard.record_success().await?; if let Err(e) = store_earnings_next(pool, ticker, None).await { tracing::warn!("[scheduler] earnings_calendar store {ticker}: {e:#}"); errors += 1; continue; } empty += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] earnings_calendar {ticker}: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; let detail = format!( "{ok}/{} stocks ({empty} empty, {errors} errors)", stale.len() ); match stopped { Some(why) => { let full = format!("stopped early ({why}); {detail}"); tracing::warn!("[scheduler] earnings_calendar: {full}"); log_fetch( pool, "earnings_calendar", "yahoo", "skipped", Some(&full), Some(ok), dur, started, ) .await?; } None => { tracing::info!("[scheduler] earnings_calendar: {detail}"); log_fetch( pool, "earnings_calendar", "yahoo", "ok", Some(&detail), Some(ok), dur, started, ) .await?; } } mark_ok(pool, "earnings_calendar", Some(next)).await?; notify_health(hub); Ok(())}/// Write one stock's next-earnings date and stamp it as freshly synced./// `next` is `None` when Yahoo has no upcoming date (the stored value is/// cleared so the page falls back to a cadence estimate). `pub(crate)`:
@@ -1831,115 +450,6 @@ pub(crate) async fn store_earnings_next( Ok(())}// ────────────── stock asset profile sweep (Phase 15) ──────────────────────/// Sweep stocks whose Yahoo `quoteSummary.assetProfile` snapshot has gone/// stale (monthly), refreshing each stock's sector and industry. One request/// per stock through the shared `yahoo` `EndpointGuard`; mirrors the/// `earnings_calendar` shape — guard-paced, resumable, broadcast-to-/health.////// A stock Yahoo cleanly knows but has no `assetProfile` module for (uneven/// coverage on small caps) is still stamped synced, so the sweep does not/// re-fetch the same empty answer every cycle.async fn run_asset_profile( pool: &SqlitePool, config: &Config, hub: &Hub,) -> anyhow::Result<()> { let started = now_ms(); let next = started + ASSET_PROFILE_INTERVAL_SECS * 1000; let cutoff = started - ASSET_PROFILE_STALE_SECS * 1000; let stale: Vec<String> = sqlx::query_scalar( "SELECT ticker FROM symbols \ WHERE kind = 'stock' \ AND (asset_profile_synced_at IS NULL OR asset_profile_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { mark_ok(pool, "asset_profile", Some(next)).await?; return Ok(()); } mark_fetching(pool, "asset_profile").await?; notify_health(hub); tracing::info!( "[scheduler] asset_profile: refreshing {} stocks", stale.len() ); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); let t0 = Instant::now(); let mut ok = 0i64; let mut empty = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &stale { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => { stopped = Some(why); break; } } match yahoo.asset_profile(ticker).await { Ok(Some(profile)) => { guard.record_success().await?; if let Err(e) = store_asset_profile(pool, ticker, &profile).await { tracing::warn!("[scheduler] asset_profile store {ticker}: {e:#}"); errors += 1; continue; } ok += 1; } // Yahoo answered cleanly but had no profile for this stock — // stamp it checked so the next sweep does not re-fetch the // same empty answer. Ok(None) => { guard.record_success().await?; let _ = mark_asset_profile_synced(pool, ticker).await; empty += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] asset_profile {ticker}: {e:#}"); } } } let dur = t0.elapsed().as_millis() as i64; let detail = format!( "{ok}/{} stocks ({empty} empty, {errors} errors)", stale.len() ); match stopped { Some(why) => { let full = format!("stopped early ({why}); {detail}"); tracing::warn!("[scheduler] asset_profile: {full}"); log_fetch( pool, "asset_profile", "yahoo", "skipped", Some(&full), Some(ok), dur, started, ).await?; } None => { tracing::info!("[scheduler] asset_profile: {detail}"); log_fetch( pool, "asset_profile", "yahoo", "ok", Some(&detail), Some(ok), dur, started, ).await?; } } mark_ok(pool, "asset_profile", Some(next)).await?; notify_health(hub); Ok(())}/// Write one stock's sector / industry classification and stamp it freshly/// synced. Either field may be `None` when Yahoo's coverage is partial; an/// empty / whitespace value is dropped at parse time, not stored. `pub(crate)`:
@@ -2773,42 +1283,6 @@ async fn mark_error( Ok(())}/// Set only a job's `next_run_at` (creating an `idle` row if absent), leaving/// any existing state untouched. `pub(crate)`: the add-symbol route uses it to/// bring the history job forward so a newly added symbol is backfilled within/// a tick instead of waiting out the ~6h interval.pub(crate) async fn schedule_next( pool: &SqlitePool, job: &str, next_run_at: i64,) -> sqlx::Result<()> { let now = now_ms(); sqlx::query( "INSERT INTO data_status (job, state, next_run_at, updated_at) VALUES (?, 'idle', ?, ?) \ ON CONFLICT(job) DO UPDATE SET \ next_run_at = excluded.next_run_at, updated_at = excluded.updated_at", ) .bind(job) .bind(next_run_at) .bind(now) .execute(pool) .await?; Ok(())}/// Whether `job` is due: no row yet, a null `next_run_at`, or one in the past.async fn is_due(pool: &SqlitePool, job: &str, now: i64) -> sqlx::Result<bool> { let next: Option<Option<i64>> = sqlx::query_scalar("SELECT next_run_at FROM data_status WHERE job = ?") .bind(job) .fetch_optional(pool) .await?; Ok(match next { None | Some(None) => true, Some(Some(t)) => t <= now, })}/// Stamp a symbol as history-checked without storing bars: used when the/// upstream returned a valid response that simply held nothing new.async fn mark_history_checked(pool: &SqlitePool, ticker: &str) -> sqlx::Result<()> {
@@ -2852,3 +1326,286 @@ async fn log_fetch( .await?; Ok(())}// ───────────────────── on-demand refresh pipeline (Phase B) ─────────────────//// Since the demand-only refocus there are no timed sweeps: a viewed symbol's// data is pulled here, on demand, when its page loads (the fast price steps// always run; the slow SEC / metadata steps run only when their stored copy is// stale) or when the user hits Refresh (`force`, which runs everything). The// symbol-page SSE route (`routes::symbols::refresh_stream`) drives it: it asks// `refresh_plan` which steps will run, then runs each via `refresh_step`,// streaming progress to the page's loading bar. Each step reuses the same// guarded `backfill_*` helpers the add-symbol flow already uses./// On-demand staleness windows for the gated (slow) steps. The always-run price/// steps (quote + history) carry no window — they run on every load.const REFRESH_SEC_STALE_SECS: i64 = 7 * 24 * 3600;const REFRESH_LEADERSHIP_STALE_SECS: i64 = 30 * 24 * 3600;const REFRESH_META_STALE_SECS: i64 = 7 * 24 * 3600;/// One step in a symbol's refresh, shown on the page's loading bar.pub(crate) struct RefreshStep { /// Stable key the route passes back to `refresh_step`. pub key: &'static str, /// Human label for the loading bar. pub label: &'static str, /// Whether this refreshes a server-rendered "deep" section — if any deep /// step ran, the page reloads to show it; a load that runs only the (live) /// price steps patches the price in place instead. pub deep: bool,}const fn step(key: &'static str, label: &'static str, deep: bool) -> RefreshStep { RefreshStep { key, label, deep }}/// Decide which steps a symbol's refresh will run. The two price steps always/// run; the slow steps are included only when stale (or `force`). An index //// future / unknown kind gets just the price steps.pub(crate) async fn refresh_plan( pool: &SqlitePool, config: &Config, ticker: &str, kind: &str, force: bool,) -> Vec<RefreshStep> { let mut steps = vec![ step("quote", "Live quote", false), step("history", "Daily history", false), ]; let Some(s) = sqlx::query_as::<_, crate::models::SymbolRow>("SELECT * FROM symbols WHERE ticker = ?") .bind(ticker) .fetch_optional(pool) .await .ok() .flatten() else { return steps; }; let now = now_ms(); let stale = |at: Option<i64>, secs: i64| force || at.map_or(true, |t| now - t > secs * 1000); let sec_ok = !config.sec_contact_email.is_empty(); match kind { "stock" => { if sec_ok && (stale(s.fundamentals_synced_at, REFRESH_SEC_STALE_SECS) || stale(s.filings_synced_at, REFRESH_SEC_STALE_SECS) || stale(s.leadership_synced_at, REFRESH_LEADERSHIP_STALE_SECS)) { steps.push(step("sec", "Fundamentals, filings & leadership", true)); } if stale(s.earnings_synced_at, REFRESH_META_STALE_SECS) { steps.push(step("earnings", "Earnings date", true)); } if stale(s.asset_profile_synced_at, REFRESH_META_STALE_SECS) { steps.push(step("profile", "Sector & industry", true)); } if stale(s.dividends_synced_at, REFRESH_META_STALE_SECS) { steps.push(step("dividends", "Dividends", true)); } } "etf" => { if sec_ok && stale(s.fund_synced_at, REFRESH_SEC_STALE_SECS) { steps.push(step("fund_sec", "Holdings & filings", true)); } if stale(s.fund_metadata_synced_at, REFRESH_META_STALE_SECS) { steps.push(step("fund_meta", "Fund details & NAV", true)); } if stale(s.dividends_synced_at, REFRESH_META_STALE_SECS) { steps.push(step("dividends", "Distributions", true)); } } _ => {} } steps}/// Run one refresh step by key. Returns a short status for the loading bar:/// "ok" when it ran, "skipped" when the guard denied it (breaker open / budget/// spent). The backfill helpers are best-effort and swallow their own errors,/// so the deep steps report "ok" once attempted; the price steps, which this/// runs inline through the guard, distinguish a guard denial.pub(crate) async fn refresh_step( pool: &SqlitePool, config: &Config, hub: &Hub, ticker: &str, kind: &str, key: &str,) -> &'static str { let _ = kind; match key { "quote" => refresh_quote(pool, config, hub, ticker).await, "history" => refresh_history_incremental(pool, config, ticker).await, "sec" => refresh_sec(pool, config, ticker, false).await, "fund_sec" => refresh_sec(pool, config, ticker, true).await, "earnings" => { backfill_earnings_calendar(pool, config, ticker).await; "ok" } "profile" => { backfill_asset_profile(pool, config, ticker).await; "ok" } "dividends" => { backfill_dividends(pool, config, ticker).await; "ok" } "fund_meta" => refresh_fund_meta(pool, config, ticker).await, _ => "ok", }}/// Pull fresh quotes for a set of dashboard symbols on demand — the dashboard's/// on-open refresh (PLAN.md Phase C polish). A symbol quoted within the last few/// minutes is skipped, so a reload (or the add/remove reload) does not re-hit/// Yahoo, and overnight the gate keeps a re-open from re-polling the same frozen/// close. Runs regardless of session: opening the dashboard after the close/// should still confirm the latest (closing) prices rather than show a stale/// snapshot. Each fetch publishes to the hub so open cards live-tick. Returns how/// many symbols were actually refreshed.pub(crate) async fn refresh_quotes( pool: &SqlitePool, config: &Config, hub: &Hub, tickers: &[String],) -> usize { let cutoff = now_ms() - INTRADAY_MIN_INTERVAL_SECS * 1000; let mut refreshed = 0; for t in tickers { // Skip a symbol with a quote younger than the throttle window. let fresh: Option<i64> = sqlx::query_scalar( "SELECT last_quote_at FROM symbols WHERE ticker = ? AND last_quote_at >= ?", ) .bind(t) .bind(cutoff) .fetch_optional(pool) .await .ok() .flatten(); if fresh.is_some() { continue; } if refresh_quote(pool, config, hub, t).await == "ok" { refreshed += 1; } } refreshed}/// Pull one live quote + its intraday bars and publish it to the hub so an open/// page patches its price in place (mirrors `run_intraday`'s per-symbol body).async fn refresh_quote(pool: &SqlitePool, config: &Config, hub: &Hub, ticker: &str) -> &'static str { let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); match guarded(&guard, yahoo.quote(ticker)).await { Some(Ok(data)) => { let _ = store_quote(pool, ticker, &data.quote).await; if !data.bars.is_empty() { let _ = store_intraday(pool, ticker, &data.bars).await; } hub.publish(StreamEvent::Quote(QuoteUpdate::new( ticker.to_string(), data.quote.price, data.quote.prev_close, data.quote.market_state.clone(), ))); "ok" } Some(Err(e)) => { tracing::warn!("[refresh] quote {ticker}: {e:#}"); "error" } None => "skipped", }}/// Pull the daily history a viewed symbol is missing: the window since its last/// stored bar (incremental) when it already has history, else a full/// `range=max` backfill. Cheaper than the deep re-fetch on a routine load.async fn refresh_history_incremental( pool: &SqlitePool, config: &Config, ticker: &str,) -> &'static str { let last: Option<String> = sqlx::query_scalar("SELECT history_last_date FROM symbols WHERE ticker = ?") .bind(ticker) .fetch_optional(pool) .await .ok() .flatten(); let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); match guarded(&guard, yahoo.daily(ticker, last.as_deref())).await { Some(Ok(bars)) if !bars.is_empty() => { let _ = seed::store_daily(pool, ticker, &bars).await; "ok" } Some(Ok(_)) => { let _ = mark_history_checked(pool, ticker).await; "ok" } Some(Err(e)) => { tracing::warn!("[refresh] history {ticker}: {e:#}"); "error" } None => "skipped", }}/// Backfill a viewed symbol's SEC data on demand (stock fundamentals/filings//// leadership, or an ETF's holdings/filings). Skipped cleanly with no contact/// email configured, as the old sweep was.async fn refresh_sec(pool: &SqlitePool, config: &Config, ticker: &str, fund: bool) -> &'static str { if config.sec_contact_email.is_empty() { return "skipped"; } let sec = SecProvider::new(providers::http::build_sec_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), sec.name(), SEC_BUDGET); if fund { backfill_etf_sec(pool, &sec, &guard, ticker).await; } else { backfill_stock_sec(pool, &sec, &guard, ticker).await; } "ok"}/// Refresh an ETF's Yahoo fund metadata and its NAV (the price-vs-NAV premium/// behind the quality read's tracking factor needs a fresh NAV; see the/// hard-won lesson in PLAN.md). Two cheap `quoteSummary` calls through the/// Yahoo guard.async fn refresh_fund_meta(pool: &SqlitePool, config: &Config, ticker: &str) -> &'static str { backfill_fund_metadata(pool, config, ticker).await; let yahoo = YahooProvider::new(providers::http::build_client(config)); let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET); if let Some(Ok(nav)) = guarded(&guard, yahoo.fund_nav(ticker)).await { let _ = store_fund_nav(pool, ticker, nav).await; } "ok"}/// Upsert an ETF's freshly-fetched NAV + its sync stamp, touching only the NAV/// columns so the static fields stay intact. A `None` nav clears any prior NAV/// (honest: no fresh value to read a premium against). Re-added for the Phase-B/// on-demand NAV pull after the daily `fund_nav` job was removed in Phase A.async fn store_fund_nav(pool: &SqlitePool, ticker: &str, nav: Option<f64>) -> sqlx::Result<()> { let now = now_ms(); sqlx::query( "INSERT INTO fund_metadata (ticker, nav_price, nav_synced_at, updated_at) \ VALUES (?, ?, ?, ?) \ ON CONFLICT(ticker) DO UPDATE SET \ nav_price = excluded.nav_price, \ nav_synced_at = excluded.nav_synced_at, \ updated_at = excluded.updated_at", ) .bind(ticker) .bind(nav) .bind(now) .bind(now) .execute(pool) .await?; Ok(())}
@@ -21,8 +21,6 @@ use std::sync::Mutex;use serde::Serialize;use tokio::sync::broadcast;use crate::summary::MarketSummary;/// Broadcast channel depth. One full universe sweep (~144 quote events) sits/// well under this; a subscriber that still lags is handled by skipping the/// gap (see `routes::stream`), never by stalling a publisher.
@@ -36,11 +34,6 @@ const CHANNEL_CAPACITY: usize = 1024;pub enum StreamEvent { Quote(QuoteUpdate), Market { session: String }, /// A recomputed dashboard market summary — the hero verdict, headline /// figures, and breadth — pushed as index quotes tick intraday so the /// home page's verdict and breadth stay live (Phase 7). Published by the /// scheduler; an open `/` page patches the hero + breadth DOM in place. Summary(MarketSummary), /// A background-data state change — a job started or finished, a /// `fetch_log` row landed. Carries no payload: it is a nudge telling an /// open `/health` page to pull a fresh snapshot from `/api/health`.
@@ -1,264 +0,0 @@//! Live market-summary snapshot — the dashboard hero verdict + market breadth,//! computed straight from stored prices.//!//! This is the one source of truth for the dashboard's plain-language read of//! the day. It is used two ways://! - the home route renders it at page load (via the verdict vocabulary below,//! [`market_verdict`] / [`vix_tone`]); and//! - the scheduler recomputes it as index quotes tick intraday and pushes it//! over the stream as a [`crate::stream::StreamEvent::Summary`], so the//! verdict sentence, the headline figures, and the breadth counts stay live//! instead of going stale beside the live-ticking index chips (Phase 7).//!//! For a Python reader: think of [`market_summary`] as a small read-model query//! — two cheap aggregate reads (breadth across the curated stocks, plus the//! lead index and ^VIX) folded into one serializable struct the browser patches//! into the page.use serde::Serialize;use sqlx::SqlitePool;use crate::compute;use crate::market::Session;/// The broad-market index the hero verdict reads: the cash S&P during the/// regular session, its E-mini future outside it (so the verdict stays live/// overnight, matching how `home::dashboard_cards` resolves the lead card).const BROAD_CASH: &str = "^SPX";const BROAD_FUTURE: &str = "ES=F";/// The volatility gauge folded into the risk tone. No tradable future on Yahoo,/// so it is always the cash level.const VIX: &str = "^VIX";/// The dashboard hero verdict + market breadth in one payload, shaped for the/// browser to patch in place. Mirrors the fields the home template renders at/// load so a pushed update and the server render agree.#[derive(Debug, Clone, Serialize, Default)]pub struct MarketSummary { /// The punchy lead, e.g. "Higher, but narrow." pub verdict: String, /// The supporting clause, e.g. "Markets higher with narrow participation." pub detail: String, /// The broad-market day change (the lead index move); drives the "S&P +x%" /// stat. `None` until that symbol prices. pub broad_pct: Option<f64>, /// The VIX read folded into one phrase, e.g. "calm at 13.2". `None` until /// ^VIX prices. pub vix_label: Option<String>, /// Advancers vs decliners across the curated large-cap stocks. pub breadth: BreadthCounts,}/// Market breadth across the curated large-cap stocks: how many are advancing/// vs declining today, the share green, and the proportion-bar segment widths./// Field names match the home template's `breadth.*` so the page render and a/// pushed update patch the same nodes.#[derive(Debug, Clone, Serialize, Default)]pub struct BreadthCounts { pub advancers: usize, pub decliners: usize, pub unchanged: usize, /// Stocks with a computable day change (advancers + decliners + unchanged). pub total: usize, /// Advancers as a percent of `total`, rounded; `None` when `total` is 0. pub pct_green: Option<u8>, /// Proportion-bar segment widths (percent of `total`): green, flat, red. pub up_w: f64, pub flat_w: f64, pub down_w: f64,}/// The dashboard symbols whose live quotes move the hero verdict: the broad/// index (cash or its future, by session) and ^VIX. The scheduler uses this to/// decide whether an intraday sweep touched anything worth re-pushing a summary/// for (the curated stocks behind breadth only move at the daily close).pub fn pulse_tickers(session: Session) -> [&'static str; 2] { let broad = if matches!(session, Session::Regular) { BROAD_CASH } else { BROAD_FUTURE }; [broad, VIX]}/// Build the full market summary from stored prices. Two cheap reads: a breadth/// scan over the curated stocks and a two-symbol pull for the lead index + VIX./// Both resolve the live last price, falling back to the latest stored daily/// close, exactly as the home route's card/breadth queries do — so a pushed/// update reconciles with what the page rendered.pub async fn market_summary(pool: &SqlitePool, session: Session) -> MarketSummary { let breadth = breadth_snapshot(pool).await; let [broad_ticker, _] = pulse_tickers(session); let broad_pct = day_change_pct(pool, broad_ticker).await; let vix_level = latest_price(pool, VIX).await; let vix_pct = day_change_pct(pool, VIX).await; let (verdict, detail) = market_verdict(broad_pct, breadth.pct_green, vix_level, vix_pct); let vix_label = vix_level.map(|v| format!("{} at {:.1}", vix_tone(v), v)); MarketSummary { verdict, detail, broad_pct, vix_label, breadth, }}/// Market breadth across the curated large-cap stocks. One read of each stock's/// resolved last/prev price (live quote, else latest daily close), folded the/// same way [`crate::routes::home`]'s in-memory breadth does: a stock without a/// computable change (no price, or a non-positive prior close) is left out of/// every count so a missing quote never reads as "flat".async fn breadth_snapshot(pool: &SqlitePool) -> BreadthCounts { let rows: Vec<(Option<f64>, Option<f64>)> = sqlx::query_as( "SELECT \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)) \ FROM symbols s WHERE s.is_seeded = 1 AND s.kind = 'stock'", ) .fetch_all(pool) .await .unwrap_or_default(); let mut b = BreadthCounts::default(); for (last, prev) in rows { let (Some(last), Some(prev)) = (last, prev) else { continue; }; if prev <= 0.0 { continue; } b.total += 1; if last > prev { b.advancers += 1; } else if last < prev { b.decliners += 1; } else { b.unchanged += 1; } } if b.total > 0 { let total = b.total as f64; b.pct_green = Some((b.advancers as f64 / total * 100.0).round() as u8); b.up_w = b.advancers as f64 / total * 100.0; b.down_w = b.decliners as f64 / total * 100.0; b.flat_w = (100.0 - b.up_w - b.down_w).max(0.0); } b}/// One symbol's resolved last price (live quote, else latest daily close).async fn latest_price(pool: &SqlitePool, ticker: &str) -> Option<f64> { sqlx::query_scalar( "SELECT COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)) \ FROM symbols s WHERE s.ticker = ?", ) .bind(ticker) .fetch_optional(pool) .await .ok() .flatten() .flatten()}/// One symbol's day change percent from its resolved last vs prior close.async fn day_change_pct(pool: &SqlitePool, ticker: &str) -> Option<f64> { let row: Option<(Option<f64>, Option<f64>)> = sqlx::query_as( "SELECT \ COALESCE(s.last_price, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1)), \ COALESCE(s.prev_close, \ (SELECT close FROM daily_prices p WHERE p.ticker = s.ticker ORDER BY d DESC LIMIT 1 OFFSET 1)) \ FROM symbols s WHERE s.ticker = ?", ) .bind(ticker) .fetch_optional(pool) .await .ok() .flatten(); match row { Some((Some(last), Some(prev))) => Some(compute::change(last, prev).pct), _ => None, }}/// A VIX level read into one plain word. The bands suit the ^VIX cash gauge:/// sub-14 is a placid tape, the teens are normal, the low-20s start to show/// stress, and 28+ is outright fear.pub fn vix_tone(level: f64) -> &'static str { match level { v if v < 14.0 => "calm", v if v < 20.0 => "steady", v if v < 28.0 => "elevated", _ => "stressed", }}/// Blend the broad-market move, breadth, and the VIX read into the hero's/// two-line verdict. `broad_pct` is the lead index card's day change (the cash/// S&P during the regular session, its future outside it); `green_pct` the/// share of curated stocks green; `vix_level` / `vix_pct` the volatility gauge/// and its move. Returns `(lead, detail)`. A descriptive read of the tape, not/// a forecast — direction comes from the broad move, falling back to breadth/// when the index is flat; width and risk tone colour the wording.pub fn market_verdict( broad_pct: Option<f64>, green_pct: Option<u8>, vix_level: Option<f64>, vix_pct: Option<f64>,) -> (String, String) { // Direction: the broad index move is the headline truth, so the verdict's // direction tracks its sign — never the opposite of the "S&P +x%" figure // shown beside it. A near-flat index (|move| < 0.05%) reads as mixed even if // breadth skews, since that is genuinely a directionless tape. Breadth only // sets direction when there is no index price at all (e.g. it never quoted). // 1 up / -1 down / 0 flat. let dir = match broad_pct { Some(p) if p > 0.05 => 1, Some(p) if p < -0.05 => -1, Some(_) => 0, None => match green_pct { Some(g) if g >= 55 => 1, Some(g) if g <= 45 => -1, _ => 0, }, }; // Breadth width: 2 broad / 1 split / 0 narrow. let width = match green_pct { Some(g) if g >= 60 => 2, Some(g) if g <= 40 => 0, _ => 1, }; let vix_rising = vix_pct.is_some_and(|p| p > 4.0); let vix_elevated = vix_level.is_some_and(|v| v >= 20.0); let lead = match (dir, width) { (1, 2) if !vix_elevated => "Risk-on, and broad.", (1, 2) => "Higher across the board.", (1, 0) => "Higher, but narrow.", (1, _) => "Modestly higher.", (-1, _) if vix_rising || vix_elevated => "Risk-off.", (-1, 0) => "Broadly lower.", (-1, _) => "Softer today.", _ => "Quiet, mixed tape.", }; let move_word = match dir { 1 => "higher", -1 => "lower", _ => "little changed", }; let part_word = match width { 2 => "wide participation", 0 => "narrow participation", _ => "mixed participation", }; ( lead.to_string(), format!("Markets {move_word} with {part_word}."), )}
@@ -0,0 +1,131 @@//! Session-scoped dashboard watchlists (Phase C).//!//! The dashboard is a personal, editable watchlist with no accounts: a browser//! is identified by an opaque `fin_sid` cookie, and its symbols live in the//! `watchlist` table keyed on that sid (see migration 0015). A brand-new//! browser (no cookie) is minted a sid and seeded with the [`STARTERS`]; an//! existing cookie's list is used as-is, even when empty, so a user who removes//! everything is not re-seeded. Clearing cookies loses the list, by design.use axum::http::{header, HeaderMap};use sqlx::SqlitePool;use crate::db::now_ms;/// The symbols a brand-new browser's watchlist is seeded with. The S&P 500/// baseline is *not* here — the dashboard always shows it as the comparison/// baseline; these are the user's editable rows.pub const STARTERS: &[&str] = &["VTI", "VXUS", "BND", "IAU", "IBIT"];/// The cookie name that carries the opaque session id.pub const COOKIE: &str = "fin_sid";/// The session cookie's lifetime: one year.const COOKIE_MAX_AGE_SECS: i64 = 365 * 24 * 3600;/// A resolved browser session: its sid, and the `Set-Cookie` value to send when/// the session was just minted (a first visit).pub struct Session { pub sid: String, pub set_cookie: Option<String>,}/// Parse the `fin_sid` value out of the request's `Cookie` header, if present./// Accepts only a short hex string (what we mint), so a hand-crafted cookie/// can't smuggle anything unexpected into the sid.pub fn sid_from_headers(headers: &HeaderMap) -> Option<String> { let raw = headers.get(header::COOKIE)?.to_str().ok()?; let prefix = format!("{COOKIE}="); for part in raw.split(';') { let part = part.trim(); if let Some(v) = part.strip_prefix(&prefix) { let v = v.trim(); if !v.is_empty() && v.len() <= 64 && v.bytes().all(|b| b.is_ascii_hexdigit()) { return Some(v.to_string()); } } } None}/// The `Set-Cookie` header value that persists `sid` for a year.pub fn set_cookie_value(sid: &str) -> String { format!("{COOKIE}={sid}; Path=/; Max-Age={COOKIE_MAX_AGE_SECS}; HttpOnly; SameSite=Lax")}/// Resolve the browser's session. With a cookie present, use that sid as-is/// (its list is whatever the browser arranged, even if empty). With no cookie,/// mint a new opaque sid, seed the starter watchlist, and return the cookie to/// set.pub async fn resolve(pool: &SqlitePool, headers: &HeaderMap) -> Session { if let Some(sid) = sid_from_headers(headers) { return Session { sid, set_cookie: None }; } // Mint 16 random bytes as hex via SQLite, so no extra crate is needed; fall // back to a timestamp-derived id only if that ever fails. let sid: String = sqlx::query_scalar("SELECT lower(hex(randomblob(16)))") .fetch_one(pool) .await .unwrap_or_else(|_| format!("{:032x}", now_ms())); seed_starters(pool, &sid).await; let set_cookie = Some(set_cookie_value(&sid)); Session { sid, set_cookie }}/// Seed a fresh session with the starter symbols (only those that exist in the/// universe — they all do, but the guard keeps a missing one from inserting a/// dangling row).async fn seed_starters(pool: &SqlitePool, sid: &str) { let now = now_ms(); for (i, t) in STARTERS.iter().enumerate() { let _ = sqlx::query( "INSERT INTO watchlist (sid, ticker, position, added_at) \ SELECT ?, ?, ?, ? WHERE EXISTS (SELECT 1 FROM symbols WHERE ticker = ?) \ ON CONFLICT(sid, ticker) DO NOTHING", ) .bind(sid) .bind(t) .bind(i as i64) .bind(now) .bind(t) .execute(pool) .await; }}/// The watchlist tickers for `sid`, in display order.pub async fn list(pool: &SqlitePool, sid: &str) -> Vec<String> { sqlx::query_scalar("SELECT ticker FROM watchlist WHERE sid = ? ORDER BY position, added_at") .bind(sid) .fetch_all(pool) .await .unwrap_or_default()}/// Append `ticker` to `sid`'s watchlist. Assumes the symbol already exists in/// the universe (the route ensures it first). Idempotent on (sid, ticker).pub async fn add_ticker(pool: &SqlitePool, sid: &str, ticker: &str) -> sqlx::Result<()> { let now = now_ms(); sqlx::query( "INSERT INTO watchlist (sid, ticker, position, added_at) \ VALUES (?, ?, COALESCE((SELECT MAX(position) + 1 FROM watchlist WHERE sid = ?), 0), ?) \ ON CONFLICT(sid, ticker) DO NOTHING", ) .bind(sid) .bind(ticker) .bind(sid) .bind(now) .execute(pool) .await?; Ok(())}/// Remove `ticker` from `sid`'s watchlist.pub async fn remove_ticker(pool: &SqlitePool, sid: &str, ticker: &str) -> sqlx::Result<()> { sqlx::query("DELETE FROM watchlist WHERE sid = ? AND ticker = ?") .bind(sid) .bind(ticker) .execute(pool) .await?; Ok(())}
modified
templates/base.html
@@ -31,7 +31,6 @@ </form> <nav class="topnav"> <a href="/"{% if request.path == '/' %} class="is-active"{% endif %}>Markets</a> <a href="/industries"{% if request.path == '/industries' %} class="is-active"{% endif %}>Industries</a> <a href="/search"{% if request.path == '/search' %} class="is-active"{% endif %}>Search</a> </nav> {% include "includes/status_indicator.html" %}
@@ -47,10 +46,6 @@ <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M4 19V11M10 19V5M16 19v-6M21 19H3"/></svg> <span>Markets</span> </a> <a href="/industries"{% if request.path == '/industries' %} class="is-active"{% endif %}> <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M3 21h18M5 21V9l7-5 7 5v12M9 21v-6h6v6"/></svg> <span>Industries</span> </a> <a href="/search"{% if request.path == '/search' %} class="is-active"{% endif %}> <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="11" cy="11" r="7"/><path d="M21 21l-4.3-4.3"/></svg> <span>Search</span>
@@ -77,7 +72,6 @@ <div class="footer__label">// Pages</div> <ul> <li><a href="/">Markets</a></li> <li><a href="/industries">Industries</a></li> <li><a href="/search">Search</a></li> <li><a href="/health">Data health</a></li> <li><a href="https://github.com/overshard/finance" target="_blank" rel="noopener">Source</a></li>
modified
templates/pages/health.html
@@ -1,6 +1,6 @@{% extends "base.html" %}{% block title %}Data health{% endblock %}{% block description %}Live status of the background data jobs, the endpoint guards, and the fetch log.{% endblock %}{% block description %}Live status of the on-demand data fetches, the endpoint guards, and the fetch log.{% endblock %}{% block extra_css %}<link rel="stylesheet" href="{{ vite_asset('static_src/health/index.js', 'css') }}">{% endblock %}{% block main %}
@@ -23,9 +23,10 @@ </div> <p class="health-lede"> Every background fetch this app makes, laid open: each upstream's request guard and hourly budget, what each scheduler job is doing, and a live tail of the fetch log. This app fetches market data <strong>on demand</strong> — when you open a page — through a request guard per upstream; the only timed job is the live intraday poll for the symbols on screen. Laid open here: each guard and its hourly budget, that poll, and a live tail of the fetch log. </p> {# Shown only while a job is fetching; the page script toggles `hidden`. #}
modified
templates/pages/home.html
@@ -1,206 +1,116 @@{% extends "base.html" %}{% block title %}Markets{% endblock %}{% block description %}A live, normalized view of your watchlist against the S&P 500, with market hours, volume, and volatility.{% endblock %}{% block extra_css %}<link rel="stylesheet" href="{{ vite_asset('static_src/home/index.js', 'css') }}">{% endblock %}{% block main %}<div class="wrap"> {% if empty %} <section class="empty"> <svg viewBox="0 0 64 64" fill="none" aria-hidden="true"> <polyline points="4,44 18,30 28,38 44,14 52,24 60,12" stroke="currentColor" stroke-width="3.5" stroke-linecap="round" stroke-linejoin="round"/> <circle cx="60" cy="12" r="3.6" fill="currentColor"/> </svg> <h1>The market universe is empty</h1> <p>No symbols are seeded yet. Run <code>make seed</code> to import the curated starter list and its deep price history. This dashboard fills in once data lands.</p> </section> {% else %} {% from "includes/macros.html" import spark_card, etf_card, etf_pill, mover_row, health_row %}<div class="wrap dash"> <div class="page-head"> <h1>Markets</h1> <a class="page-head__link" href="/search">Browse all {{ total }} symbols</a> {# Market-hours banner: makes the current session unmistakable. The dot colour keys off data-session; hero.js keeps it live as the session flips. #} <div class="dash-banner" data-role="session-banner" data-session="{{ session }}"> <span class="dash-banner__dot"></span> <span class="dash-banner__label" data-role="session-label">{{ session_label }}</span> <span class="dash-banner__sub" data-role="session-note">{% if session == 'closed' %}Prices update during market hours · ET{% else %}US equities · all times ET{% endif %}</span> </div> {# Hero (Phase 5): a one-line plain read of the day — the broad move, breadth, and the VIX risk tone blended into a verdict — over a compact index strip. Rendered at page load, then kept live (Phase 7): the scheduler pushes a recomputed `summary` over the stream as the indexes tick, and home.js patches the verdict, the figures, and the breadth band below in place. The data-role hooks are those patch targets. #} <section class="hero" data-role="hero"> <div class="hero__read"> <p class="hero__verdict" data-role="hero-verdict">{{ hero.verdict }}</p> <p class="hero__detail" data-role="hero-detail">{{ hero.detail }}</p> <p class="hero__stats"> {% if hero.broad_pct is not none %}<span class="hero__stat">S&P <span class="num {{ 'is-up' if hero.broad_pct >= 0 else 'is-down' }}" data-role="hero-broad">{{ hero.broad_pct|pct }}</span></span>{% endif %} {% if hero.green_pct is not none %}<span class="hero__stat"><span class="num" data-role="hero-green">{{ hero.green_pct }}%</span> of the S&P green</span>{% endif %} {% if hero.vix_label %}<span class="hero__stat">VIX <span data-role="hero-vix">{{ hero.vix_label }}</span></span>{% endif %} </p> <p class="hero__note">A plain read of today’s tape — not investment advice.</p> {# The hero day graph: every watchlist symbol plus the S&P 500, each as % change from today's open, on one axis. Drawn by hero.js from /api/dashboard; this is its mount point and a no-JS fallback line. #} <section class="hero-graph"> <div class="hero-graph__head"> <h1 class="hero-graph__title">Today, vs the S&P 500</h1> <span class="hero-graph__note">% change from the open · <span class="hero-graph__base">S&P 500 baseline</span></span> </div> {% if hero.chips %} <div class="hero__chips"> {% for c in hero.chips %} <a class="hero__chip {{ 'is-up' if c.up else 'is-down' }}" href="/s/{{ c.ticker|urlencode }}"> <span class="hero__chip-sym">{{ c.ticker }}</span> <span class="hero__chip-chg num">{{ c.change_pct|pct }}</span> </a> {% endfor %} <div class="hero-graph__chart" data-role="hero-chart"> <p class="hero-graph__empty" data-role="hero-empty">Loading today’s session…</p> </div> {% endif %} <div class="hero-graph__legend" data-role="hero-legend"></div> </section> {# The sparkline-card sections stream live, so their freshness time rides in a data-field the stream client refreshes as quotes land (Phase 22). The movers / standings panels below are fixed page-load snapshots, so their captions are plain server-rendered text. #} <h2 class="section-title">Indexes{% if index_asof %}<span class="section-title__asof">prices as of <span data-field="spark-asof">{{ index_asof|asof }}</span></span>{% endif %}</h2> <div class="spark-grid"> {% for c in index_cards %}{{ spark_card(c) }}{% endfor %} {# Headline market reads. S&P carries data-ticker/data-field so the base stream client live-patches its price; VIX and SPY carry data-ticker only (to keep them polled) and are patched by hero.js, which also fills volume + trend. #} <div class="reads-head"> <span class="reads-head__asof" data-role="reads-asof">{% if reads.asof %}Prices as of {{ reads.asof|asof }}{% else %}Awaiting first quotes…{% endif %}</span> </div> {# Breadth band (Phase 5): how many curated S&P stocks are up vs down today, the share green, and a proportion bar of the split. A page-load snapshot. #} {% if breadth.total %} <h2 class="section-title">Breadth{% if movers_asof %}<span class="section-title__asof">prices as of {{ movers_asof|asof }}</span>{% endif %}</h2> <p class="section-note">How many of the curated large-cap stocks are advancing versus declining today, and the share trading green.</p> <div class="breadth" data-role="breadth"> <div class="breadth__stats"> <span class="breadth__stat breadth__stat--up"><span class="breadth__num num" data-role="breadth-adv">{{ breadth.advancers }}</span> advancing</span> <span class="breadth__stat breadth__stat--down"><span class="breadth__num num" data-role="breadth-dec">{{ breadth.decliners }}</span> declining</span> {% if breadth.unchanged %}<span class="breadth__stat breadth__stat--flat"><span class="breadth__num num" data-role="breadth-flat">{{ breadth.unchanged }}</span> flat</span>{% endif %} {% if breadth.pct_green is not none %}<span class="breadth__pct num" data-role="breadth-pct">{{ breadth.pct_green }}% green</span>{% endif %} <section class="reads"> <div class="read" data-ticker="{{ baseline }}"> <div class="read__label">S&P 500</div> <div class="read__value num" data-field="price">{{ reads.spx_price|money }}</div> <div class="read__sub num{% if reads.spx_pct is none %} is-flat{% elif reads.spx_pct >= 0 %} is-up{% else %} is-down{% endif %}" data-field="change_pct">{{ reads.spx_pct|pct }}</div> </div> <div class="breadth__bar" role="img" data-role="breadth-bar" aria-label="{{ breadth.advancers }} advancing, {{ breadth.decliners }} declining of {{ breadth.total }} stocks"> <span class="breadth__seg breadth__seg--up" style="width: {{ breadth.up_w }}%" data-role="breadth-up"></span> <span class="breadth__seg breadth__seg--flat" style="width: {{ breadth.flat_w }}%" data-role="breadth-flat-seg"></span> <span class="breadth__seg breadth__seg--down" style="width: {{ breadth.down_w }}%" data-role="breadth-down"></span> <div class="read" data-ticker="{{ vix }}"> <div class="read__label">Volatility · VIX</div> <div class="read__value num" data-role="vix-level">{{ reads.vix_level|round(2) if reads.vix_level is not none else "—" }}</div> <div class="read__sub read__tone read__tone--{{ reads.vix_tone or 'steady' }}" data-role="vix-tone">{{ reads.vix_tone|capitalize if reads.vix_tone else "—" }}</div> </div> </div> {% endif %} {# ETF band (Phase 5): curated quality cards spanning the asset classes, then a compact gainers / losers strip across the whole curated ETF set. #} <h2 class="section-title">ETFs{% if etf_asof %}<span class="section-title__asof">prices as of <span data-field="spark-asof">{{ etf_asof|asof }}</span></span>{% endif %}<span class="section-title__asof"><a href="/search?kind=etf">All ETFs →</a></span></h2> <p class="section-note">A quality read per fund — cost, tracking, diversification, and size, blended. Hover the verdict for the breakdown.</p> {% if etf_cards %} <div class="spark-grid"> {% for e in etf_cards %}{{ etf_card(e) }}{% endfor %} </div> {% else %} <p class="movers__empty">No curated ETFs yet.</p> {% endif %} {% if etf_gainers or etf_losers %} <div class="etf-movers"> <div class="etf-movers__group etf-movers__group--up"> <span class="etf-movers__label">Gainers</span> {% for m in etf_gainers %}{{ etf_pill(m) }}{% endfor %} <div class="read" data-ticker="{{ volume_proxy }}"> <div class="read__label">Market volume</div> <div class="read__value num" data-role="volume">{{ reads.volume|compact if reads.volume is not none else "—" }}</div> <div class="read__sub" data-role="volume-label">{% if reads.volume_label %}{{ reads.volume_label }} vs avg{% else %}—{% endif %}</div> </div> <div class="etf-movers__group etf-movers__group--down"> <span class="etf-movers__label">Losers</span> {% for m in etf_losers %}{{ etf_pill(m) }}{% endfor %} <div class="read"> <div class="read__label">S&P trend</div> <div class="read__value read__value--text read__tone--{{ reads.sma_tone or 'steady' }}" data-role="sma-read">{{ reads.sma_read or "—" }}</div> <div class="read__sub">50 & 200-day average</div> </div> </div> {% endif %} </section> <h2 class="section-title">Today’s movers{% if movers_asof %}<span class="section-title__asof">prices as of {{ movers_asof|asof }}</span>{% endif %}</h2> <div class="movers"> <section class="movers__panel"> <h3 class="movers__title">Top gainers</h3> {% if gainers %} <div class="movers__list">{% for m in gainers %}{{ mover_row(m) }}{% endfor %}</div> {% else %} <p class="movers__empty">No price data yet.</p> {% endif %} </section> <section class="movers__panel"> <h3 class="movers__title">Top losers</h3> {% if losers %} <div class="movers__list">{% for m in losers %}{{ mover_row(m) }}{% endfor %}</div> {% else %} <p class="movers__empty">No price data yet.</p> {% endif %} </section> </div> {# The editable, session-scoped watchlist. #} <section class="watch"> <div class="watch__head"> <h2 class="section-title">Watchlist</h2> <form class="watch-add" data-role="watch-add" autocomplete="off"> <input class="watch-add__input" name="ticker" type="text" inputmode="text" placeholder="Add a symbol — e.g. AAPL" aria-label="Add a symbol to your watchlist" maxlength="15"> <button class="watch-add__btn" type="submit">Add</button> </form> </div> <p class="watch-msg" data-role="watch-msg" role="status" hidden></p> <h2 class="section-title">Today’s industries{% if industries_asof %}<span class="section-title__asof">classifications synced {{ industries_asof|ago }}</span>{% endif %}<span class="section-title__asof"><a href="/industries">All industries →</a></span></h2> <p class="section-note">Equal-weight day move of the curated stocks inside each sector. The classification comes from Yahoo’s <code>assetProfile</code> and lands in the background; a sector waits for at least one of its members to sync.</p> <div class="movers"> <section class="movers__panel"> <h3 class="movers__title">Sectors moving up</h3> {% if top_industries %} <div class="movers__list"> {% for r in top_industries %} <a class="industry-row {{ 'industry-row--up' if r.up else 'industry-row--down' }}" href="/industries/{{ r.slug }}" style="--bar: {{ r.bar }}%"> <span class="industry-row__name">{{ r.name }}</span> <span class="industry-row__count">{{ r.members }} stocks</span> <span class="industry-row__chg num {{ 'is-up' if r.up else 'is-down' }}">{{ r.change_pct|pct }}</span> {% if cards %} <div class="watch-grid"> {% for c in cards %} {# data-ticker drives the live stream; the `spark-card` class lets the base client repaint the sparkline tip on each quote. The remove button sits outside the link so the whole card stays navigable. #} <div class="watch-card spark-card{% if c.up %} is-up-card{% else %} is-down-card{% endif %}" data-ticker="{{ c.ticker }}"> <a class="watch-card__link" href="/s/{{ c.ticker|urlencode }}"> <div class="spark-card__head"> <span class="spark-card__sym">{{ c.ticker }}</span> <span class="spark-card__name">{{ c.name }}</span> </div> {% if c.spark %} <svg class="spark" viewBox="0 0 100 36" preserveAspectRatio="none" data-lo="{{ c.spark.lo }}" data-hi="{{ c.spark.hi }}" aria-hidden="true"> <polygon class="spark__area" points="{{ c.spark.area }}"/> {% if c.spark.baseline is not none %} <line class="spark__base" x1="0" x2="100" y1="{{ c.spark.baseline }}" y2="{{ c.spark.baseline }}"/> {% endif %} <polyline class="spark__line" points="{{ c.spark.line }}"/> </svg> {% else %} <div class="spark spark--empty">no intraday data</div> {% endif %} <div class="spark-card__foot"> <span class="spark-card__price num" data-field="price">{{ c.price|money }}</span> <span class="spark-card__chg num{% if c.change_pct is none %} is-flat{% elif c.change_pct >= 0 %} is-up{% else %} is-down{% endif %}" data-field="change_pct">{{ c.change_pct|pct }}</span> </div> </a> {% endfor %} <button class="watch-card__remove" type="button" data-remove data-ticker="{{ c.ticker }}" aria-label="Remove {{ c.ticker }} from watchlist" title="Remove">×</button> </div> {% else %} <p class="movers__empty">No sector classifications synced yet.</p> {% endif %} </section> <section class="movers__panel"> <h3 class="movers__title">Sectors moving down</h3> {% if bottom_industries %} <div class="movers__list"> {% for r in bottom_industries %} <a class="industry-row {{ 'industry-row--up' if r.up else 'industry-row--down' }}" href="/industries/{{ r.slug }}" style="--bar: {{ r.bar }}%"> <span class="industry-row__name">{{ r.name }}</span> <span class="industry-row__count">{{ r.members }} stocks</span> <span class="industry-row__chg num {{ 'is-up' if r.up else 'is-down' }}">{{ r.change_pct|pct }}</span> </a> {% endfor %} </div> {% else %} <p class="movers__empty">No sector classifications synced yet.</p> {% endif %} </section> </div> <h2 class="section-title">Risk & commodities{% if commodity_asof %}<span class="section-title__asof">prices as of <span data-field="spark-asof">{{ commodity_asof|asof }}</span></span>{% endif %}</h2> <div class="spark-grid"> {% for c in commodity_cards %}{{ spark_card(c) }}{% endfor %} </div> <h2 class="section-title">Quality leaderboard{% if health_asof %}<span class="section-title__asof">last sync {{ health_asof|ago }}</span>{% endif %}</h2> <p class="section-note">One quality read per curated stock: fundamentals + recent price and growth trajectory + leadership stability, rolled together. The figure beside each row is its trailing-year return. Ranks the healthiest names against the most concerning right now — it is not a forecast of what any of them will do next.</p> <p class="disclaimer">For reading at a glance. Not investment advice and not a buy or sell signal.</p> <div class="movers"> <section class="movers__panel"> <h3 class="movers__title">Healthiest</h3> {% if healthiest %} <div class="movers__list">{% for r in healthiest %}{{ health_row(r) }}{% endfor %}</div> {% else %} <p class="movers__empty">No graded fundamentals yet.</p> {% endif %} </section> <section class="movers__panel"> <h3 class="movers__title">Most concerning</h3> {% if concerning %} <div class="movers__list">{% for r in concerning %}{{ health_row(r) }}{% endfor %}</div> {% else %} <p class="movers__empty">No graded fundamentals yet.</p> {% endif %} </section> </div> {% endif %} {% endfor %} </div> {% else %} <p class="watch-empty">Your watchlist is empty. Add a stock or ETF above, or <a href="/search">browse the universe</a> to find one. The S&P 500 stays on the graph as the baseline.</p> {% endif %} </section></div>{% endblock %}
deleted
templates/pages/industries_detail.html
@@ -1,136 +0,0 @@{% extends "base.html" %}{% block title %}{{ title }}{% endblock %}{% block description %}{{ sector_name }}{% if industry_name %} · {{ industry_name }}{% endif %} — composite trailing returns, seasonality, and members.{% endblock %}{% block extra_css %}<link rel="stylesheet" href="{{ vite_asset('static_src/industries/index.js', 'css') }}">{% endblock %}{% block main %}<div class="wrap"> <div class="page-head"> <h1> {% if scope == 'industry' %}{{ industry_name }}{% else %}{{ sector_name }}{% endif %} </h1> <a class="page-head__link" href="/industries">All industries</a> </div> <p class="ind-crumbs"> {% if scope == 'industry' %} <a href="/industries">Industries</a> › <a href="/industries/{{ sector_slug }}">{{ sector_name }}</a> › <span>{{ industry_name }}</span> {% else %} <a href="/industries">Industries</a> › <span>{{ sector_name }}</span> {% endif %} </p> <section class="ind-stats"> {% set r = returns %} <div class="ind-stat"> <div class="ind-stat__label">Today</div> <div class="ind-stat__val num{% if r.d1 is not none and r.d1 >= 0 %} is-up{% elif r.d1 is not none %} is-down{% endif %}">{{ r.d1|pct }}</div> </div> <div class="ind-stat"> <div class="ind-stat__label">5d</div> <div class="ind-stat__val num{% if r.d5 is not none and r.d5 >= 0 %} is-up{% elif r.d5 is not none %} is-down{% endif %}">{{ r.d5|pct }}</div> </div> <div class="ind-stat"> <div class="ind-stat__label">1m</div> <div class="ind-stat__val num{% if r.d21 is not none and r.d21 >= 0 %} is-up{% elif r.d21 is not none %} is-down{% endif %}">{{ r.d21|pct }}</div> </div> <div class="ind-stat"> <div class="ind-stat__label">3m</div> <div class="ind-stat__val num{% if r.d63 is not none and r.d63 >= 0 %} is-up{% elif r.d63 is not none %} is-down{% endif %}">{{ r.d63|pct }}</div> </div> <div class="ind-stat"> <div class="ind-stat__label">1y</div> <div class="ind-stat__val num{% if r.d252 is not none and r.d252 >= 0 %} is-up{% elif r.d252 is not none %} is-down{% endif %}">{{ r.d252|pct }}</div> </div> <div class="ind-stat"> <div class="ind-stat__label">Members</div> <div class="ind-stat__val num">{{ r.members }}</div> </div> </section> <h2 class="section-title">Equal-weight composite</h2> <p class="section-note">A 5-year price-only index, rebased to 100 at the first trading date all members share. <code>^SPX</code> is overlaid on the same scale for comparison.</p> <section class="panel chart-panel ind-chart-panel"> <div id="ind-chart" data-sector="{{ sector_slug }}" {% if industry_slug %}data-industry="{{ industry_slug }}"{% endif %}> </div> </section> <h2 class="section-title">Seasonality</h2> <p class="section-note">Average daily return by calendar month, across all years of stored history for the members. Tall green bars are months the basket historically printed positive average days; tall red bars the opposite. Reads at a glance, not a forecast.</p> <section class="panel ind-season"> {% set months = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'] %} <div class="season-grid"> {% for v in seasonality %} {% if v is none %} <div class="season-cell season-cell--empty"> <div class="season-cell__bar"></div> <div class="season-cell__lbl">{{ months[loop.index0] }}</div> <div class="season-cell__val num">—</div> </div> {% else %} {% set abs = v|abs %} {% set h = (abs / seasonality_max * 100) if seasonality_max > 0 else 0 %} <div class="season-cell {% if v >= 0 %}season-cell--up{% else %}season-cell--down{% endif %}"> <div class="season-cell__bar" style="height: {{ h }}%"></div> <div class="season-cell__lbl">{{ months[loop.index0] }}</div> <div class="season-cell__val num">{{ v|pct }}</div> </div> {% endif %} {% endfor %} </div> </section> {% if scope == 'sector' and breakdown %} <h2 class="section-title">Industries inside this sector</h2> <table class="ind-table"> <thead> <tr> <th scope="col" class="ind-table__sec">Industry</th> <th scope="col" class="num">Today</th> <th scope="col" class="num">Members</th> </tr> </thead> <tbody> {% for i in breakdown %} <tr> <th scope="row" class="ind-table__sec"> <a href="/industries/{{ sector_slug }}/{{ i.slug }}">{{ i.name }}</a> </th> <td class="num{% if i.day_pct is not none and i.day_pct >= 0 %} is-up{% elif i.day_pct is not none %} is-down{% endif %}">{{ i.day_pct|pct }}</td> <td class="num">{{ i.members }}</td> </tr> {% endfor %} </tbody> </table> {% endif %} <h2 class="section-title">Members</h2> <div class="ind-members"> {% for m in members %} <a class="ind-member {% if m.change_pct is not none and m.change_pct >= 0 %}ind-member--up{% elif m.change_pct is not none %}ind-member--down{% endif %}" href="/s/{{ m.ticker|urlencode }}" style="--bar: {{ m.bar }}%"> <span class="ind-member__sym num">{{ m.ticker }}</span> <span class="ind-member__name">{{ m.name }}</span> {% if scope == 'sector' %}<span class="ind-member__industry">{{ m.industry }}</span>{% endif %} <span class="ind-member__price num">{{ m.price|money }}</span> <span class="ind-member__chg num{% if m.change_pct is not none and m.change_pct >= 0 %} is-up{% elif m.change_pct is not none %} is-down{% endif %}">{{ m.change_pct|pct }}</span> </a> {% endfor %} </div></div>{% endblock %}{% block extra_js %}<script type="module" src="{{ vite_asset('static_src/industries/index.js') }}"></script>{% endblock %}
deleted
templates/pages/industries_index.html
@@ -1,69 +0,0 @@{% extends "base.html" %}{% block title %}Industries{% endblock %}{% block description %}Stocks aggregated by sector and industry: day, month, and year-trailing composite moves at a glance.{% endblock %}{% block extra_css %}<link rel="stylesheet" href="{{ vite_asset('static_src/industries/index.js', 'css') }}">{% endblock %}{% block main %}<div class="wrap"> <div class="page-head"> <h1>Industries</h1> <a class="page-head__link" href="/">Back to home</a> </div> {% if empty %} <section class="empty"> <svg viewBox="0 0 64 64" fill="none" aria-hidden="true"> {# stacked ledger columns — a quiet hint at the per-sector composition this page will show once classifications land. #} <rect x="6" y="40" width="8" height="18" stroke="currentColor" stroke-width="2.4" rx="1.5"/> <rect x="18" y="28" width="8" height="30" stroke="currentColor" stroke-width="2.4" rx="1.5"/> <rect x="30" y="18" width="8" height="40" stroke="currentColor" stroke-width="2.4" rx="1.5"/> <rect x="42" y="32" width="8" height="26" stroke="currentColor" stroke-width="2.4" rx="1.5"/> <rect x="54" y="44" width="8" height="14" stroke="currentColor" stroke-width="2.4" rx="1.5"/> <line x1="2" y1="60" x2="62" y2="60" stroke="currentColor" stroke-width="2" stroke-linecap="round"/> </svg> <h1>Industries are still classifying</h1> <p>The Yahoo <code>assetProfile</code> sweep runs in the background and fills in over time. Once a stock has a sector and industry it shows up here; follow the sweep on the <a href="/health">data health page</a>.</p> </section> {% else %} <p class="ind-blurb">{{ total }} curated stocks across {{ sectors|length }} sectors and {{ total_industries }} industries. Each row’s composite is an equal-weight average of its members’ returns. Click a sector for a deeper read, the seasonality, and the members.</p> <table class="ind-table ind-table--sectors"> <thead> <tr> <th scope="col" class="ind-table__sec">Sector</th> <th scope="col" class="num">Today</th> <th scope="col" class="num">5d</th> <th scope="col" class="num">1m</th> <th scope="col" class="num">3m</th> <th scope="col" class="num">1y</th> <th scope="col" class="num">Members</th> <th scope="col" class="num">Industries</th> </tr> </thead> <tbody> {% for s in sectors %} <tr> <th scope="row" class="ind-table__sec"> <a href="/industries/{{ s.slug }}">{{ s.name }}</a> </th> {% set r = s.returns %} <td class="num{% if r.d1 is not none and r.d1 >= 0 %} is-up{% elif r.d1 is not none %} is-down{% endif %}">{{ r.d1|pct }}</td> <td class="num{% if r.d5 is not none and r.d5 >= 0 %} is-up{% elif r.d5 is not none %} is-down{% endif %}">{{ r.d5|pct }}</td> <td class="num{% if r.d21 is not none and r.d21 >= 0 %} is-up{% elif r.d21 is not none %} is-down{% endif %}">{{ r.d21|pct }}</td> <td class="num{% if r.d63 is not none and r.d63 >= 0 %} is-up{% elif r.d63 is not none %} is-down{% endif %}">{{ r.d63|pct }}</td> <td class="num{% if r.d252 is not none and r.d252 >= 0 %} is-up{% elif r.d252 is not none %} is-down{% endif %}">{{ r.d252|pct }}</td> <td class="num">{{ s.members }}</td> <td class="num">{{ s.industries }}</td> </tr> {% endfor %} </tbody> </table> {% endif %}</div>{% endblock %}
modified
templates/pages/symbol.html
@@ -33,13 +33,25 @@{% block main %}<div class="wrap"> {# On-demand refresh control (Phase B). On load the page pulls the latest data for this symbol (live price always; slow SEC / metadata only when stale), showing the progress bar; the button re-pulls everything. The data for each section carries its own "synced … ago" age in its heading. #} <div class="refresh" data-refresh-root data-ticker="{{ symbol.ticker }}"> <button type="button" class="refresh__btn" data-refresh aria-label="Refresh all data"> <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M21 12a9 9 0 1 1-2.64-6.36M21 3v6h-6"/></svg> <span>Refresh</span> </button> <span class="refresh__status" data-refresh-status aria-live="polite"></span> <div class="refresh__bar" data-refresh-bar hidden><div class="refresh__fill" data-refresh-fill></div></div> </div> <header class="sym-head{% if health or etf_quality %} sym-head--has-health{% endif %}"> <div class="sym-head__id"> <h1 class="sym-head__ticker">{{ symbol.ticker }}</h1> <span class="sym-head__tag">{{ symbol.kind }}</span> {% if symbol.exchange %}<span class="sym-head__tag">{{ symbol.exchange }}</span>{% endif %} {% if symbol.sector %}<a class="sym-head__tag sym-head__tag--link" href="/industries/{{ sector_slug }}">{{ symbol.sector }}</a>{% endif %} {% if symbol.industry %}<a class="sym-head__tag sym-head__tag--link" href="/industries/{{ sector_slug }}/{{ industry_slug }}">{{ symbol.industry }}</a>{% endif %} {% if symbol.sector %}<span class="sym-head__tag">{{ symbol.sector }}</span>{% endif %} {% if symbol.industry %}<span class="sym-head__tag">{{ symbol.industry }}</span>{% endif %} </div> <div class="sym-head__name">{{ symbol.name }}</div> {# Prefer the live quote; fall back to the most recent daily close. The
@@ -403,8 +415,8 @@ </section> {% else %} <div class="fund-pending"> SEC fundamentals for {{ symbol.ticker }} have not synced yet. They land on the next data refresh; see <a href="/health">data health</a>. SEC fundamentals for {{ symbol.ticker }} have not synced yet. They are pulled on demand — hit <strong>Refresh</strong> above to fetch them now. </div> {% endif %}
@@ -458,8 +470,8 @@ <div class="fund-pending">No leadership data is available for {{ symbol.ticker }}.</div> {% else %} <div class="fund-pending"> Leadership data for {{ symbol.ticker }} has not synced yet. It lands on the next SEC data refresh; see <a href="/health">data health</a>. Leadership data for {{ symbol.ticker }} has not synced yet. It is pulled on demand — hit <strong>Refresh</strong> above to fetch it now. </div> {% endif %}
@@ -541,8 +553,8 @@ </section> {% else %} <div class="fund-pending"> Fund metadata for {{ symbol.ticker }} has not synced yet. It lands on the next refresh; see <a href="/health">data health</a>. Fund metadata for {{ symbol.ticker }} has not synced yet. It is pulled on demand — hit <strong>Refresh</strong> above to fetch it now. </div> {% endif %}
@@ -676,8 +688,8 @@ {% endif %} {% else %} <div class="fund-pending"> The fund profile for {{ symbol.ticker }} has not synced yet. It lands on the next SEC data refresh; see <a href="/health">data health</a>. The fund profile for {{ symbol.ticker }} has not synced yet. It is pulled on demand — hit <strong>Refresh</strong> above to fetch it now. </div> {% endif %} {% endif %}
@@ -730,8 +742,8 @@ <div class="fund-pending">{{ symbol.ticker }} has not {% if symbol.kind == 'etf' %}made a distribution{% else %}paid a dividend{% endif %} in the past five years.</div> {% else %} <div class="fund-pending"> {{ div_label }} history for {{ symbol.ticker }} has not synced yet. It lands on the next data refresh; see <a href="/health">data health</a>. {{ div_label }} history for {{ symbol.ticker }} has not synced yet. It is pulled on demand — hit <strong>Refresh</strong> above to fetch it now. </div> {% endif %} {% endif %}