@@ -52,7 +52,7 @@ There are no tests or linters configured.finance/├── Cargo.toml, Cargo.lock # rust deps├── Makefile, README.md, PLAN.md # top-level (PLAN.md is the living design doc)├── migrations/ # sqlx migrations 0001-0004, applied on boot├── migrations/ # sqlx migrations 0001-0005, applied on boot├── universe/starter.csv # curated seed list (~150 symbols)├── src/│ ├── main.rs # entry: env init, `seed` subcommand, server boot
@@ -84,7 +84,7 @@ The binary reads `templates/`, `dist/`, `migrations/`, and `universe/` from cwd## Key Routes- `/`: home dashboard — index/commodity sparkline cards over the day's top movers- `/s/{ticker}`: symbol page — candlestick chart with indicators, key stats, fundamentals, filings- `/s/{ticker}`: symbol page — candlestick chart with indicators, key stats; a stock also shows fundamentals, an ETF a fund profile (holdings, AUM), both show SEC filings- `/api/symbols/{ticker}/history`: candle + indicator series JSON for the chart- `/api/symbols` (POST): add a symbol not yet in the universe (validated against Yahoo)- `/search`: browse and search the whole universe (filter by kind, match ticker and company name)
@@ -99,7 +99,7 @@ All free, no account. See `PLAN.md` for the full anti-spam / caching policy.- **Historical daily OHLCV — Stooq.** One call returns a symbol's entire daily history. Gated behind a free apikey (`STOOQ_APIKEY`, in `.env`, gitignored).- **Intraday bars + live quotes — Yahoo Finance.** `v8/finance/chart`; no key, just a browser User-Agent.- **Fundamentals + filings — SEC EDGAR.** `company_tickers.json`, `companyfacts`, `submissions`; no key, a contact email (`SEC_CONTACT_EMAIL`) in the User-Agent. Stocks only — ETFs and indexes do not file.- **Fundamentals, filings + ETF profiles — SEC EDGAR.** `company_tickers.json` / `companyfacts` / `submissions` for stock fundamentals and filings; `company_tickers_mf.json` plus quarterly N-PORT filings for ETF fund profiles (holdings, AUM, asset mix). No key; a contact email (`SEC_CONTACT_EMAIL`) rides in the User-Agent. Indexes do not file with the SEC.## Tooling
@@ -537,6 +537,7 @@ dependencies = [ "dotenvy", "futures-util", "minijinja", "quick-xml", "reqwest", "serde", "serde_json",
@@ -1396,6 +1397,15 @@ dependencies = [ "unicode-ident",][[package]]name = "quick-xml"version = "0.37.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "331e97a1af0bf59823e6eadffe373d7b27f485be8748f71471c662c1f269b7fb"dependencies = [ "memchr",][[package]]name = "quinn"version = "0.11.9"
@@ -27,6 +27,7 @@ async-stream = "0.3"csv = "1"zip = { version = "2", default-features = false, features = ["deflate"] }urlencoding = "2"quick-xml = "0.37"[profile.release]lto = true
@@ -32,14 +32,16 @@ and resume cleanly from this file alone, keeping token use low._Last updated: 2026-05-22_**Current phase:** Phases 0 through 12 complete and verified. The MVP isshipped and **live in production at https://finance.bythewood.me** (deployed2026-05-22). Next work is the post-MVP backlog, phases 13 through 19.**Current phase:** Phases 0 through 12 (the MVP) plus Phase 18 (ETF profiles)are complete, verified, and **live in production athttps://finance.bythewood.me** (Phase 18 deployed 2026-05-22). Remainingpost-MVP backlog: phases 13 through 17 and 19.**Roadmap (restructured 2026-05-22, see decisions log):** the home-pageredesign and commodities are pre-ship MVP phases. Order: 9 Search +add-symbol, 10 Commodities & futures, 11 Home dashboard redesign, 12 Polish +ship. Post-MVP backlog is phases 13 through 19.ship. Post-MVP backlog is phases 13 through 19; the user picked Phase 18 (ETFprofiles) as the first post-MVP phase, done 2026-05-22.**Watchlists dropped from the MVP (2026-05-22, see decisions log):** the userno longer wants watchlists for now and wants the app to stay an opinionated,
@@ -467,14 +469,70 @@ schema, unused for now. return 200 over HTTPS with a valid cert and the page renders the Paper Ledger UI; the first-run seed backfills on the live box.- **Phase 18 ETF profiles.** Complete, verified, deployed to production. - The first post-MVP phase (the user picked it over the rest of the 13-19 backlog). ETFs are now first-class: each ETF page carries a fund profile section — net assets (AUM), holdings count, top 25 holdings by weight, asset-class mix — plus its own SEC filing history. - **Data source: SEC N-PORT.** An ETF files as a registered fund, so its portfolio is not in the XBRL `companyfacts` behind the stock fundamentals; it is in quarterly N-PORT filings, one large XML each. `providers/sec.rs` gained inherent fund methods on `SecProvider` (N-PORT is wholly SEC- specific, no second source to trait over): `fund_ticker_map` (the bulk `company_tickers_mf.json`, ticker -> CIK + series id), `fund_filings` (one browse-edgar Atom request -> filing list + the fund's shape), and `fund_portfolio` (fetch + stream-parse one N-PORT `primary_doc.xml`). Migration `0005` adds `symbols.series_id` / `fund_synced_at` and the `fund_profiles` + `fund_holdings` tables. `quick-xml` is a new dependency (streaming parser — a bond-fund N-PORT runs to 15+ MB / 13k positions). - **Series filtering.** One registrant CIK can host dozens of fund series (the Vanguard and iShares trusts). Lookups are keyed on the SEC *series id* via browse-edgar, so a fund never picks up a sibling fund's filings or N-PORT. The unit investment trusts (SPY, DIA) and the commodity grantor trusts (GLD, SLV) are absent from `company_tickers_mf.json`, so a small hardcoded CIK fallback covers them. - **Filing-agent N-PORTs.** An N-PORT's accession number leads with the *filer's* CIK, which for a fund using a filing agent is not the registrant's; the Archives path needs the registrant CIK. The N-PORT XML is located from the filing's browse-edgar index-page URL (which always carries the registrant CIK) rather than from the accession. - **Commodity trusts.** GLD and SLV hold physical bullion, not a securities portfolio, so they file no N-PORT. They are detected by shape (files 10-K, no N-PORT, no N-CEN) and get a minimal profile: AUM from their 10-K `companyfacts` `Assets`, the filing list, and a "holds physical bullion" note — no holdings table. - **Scheduler.** `run_sec` now also resolves ETF fund CIKs and sweeps stale ETF profiles (weekly staleness on `fund_synced_at`), through the same `sec` `EndpointGuard` — 2 requests per ETF, well inside the 600/hr budget. - **No expense ratio, no category** (user decision, see decisions log): those are not in SEC structured data, only in prospectus HTML, so they are dropped. The asset mix derived from N-PORT holdings stands in. - **UI.** `symbol.html` gained a "Fund profile" panel (AUM / holdings / as-of + a thin ink-shaded asset-mix bar), a "Top holdings" list (each row a weight bar scaled to the largest holding), and the filing list is now shown for ETFs as well as stocks. Holdings display the N-PORT issue `title` (clean mixed-case) over the issuer `name` (often truncated caps). - Verified: a full 28-ETF sweep stored 28/28 profiles with 0 errors (26 portfolio funds, 2 commodity trusts) in ~83s. N-PORT parsing is correct — QQQ 101 holdings (NVIDIA 9.0%, Apple 8.0%, ...), VOO 518, SPY 503, AGG 13,186 from a 15.8 MB XML; AUM figures plausible (VTI $2.06T, GLD $155B). `/s/QQQ`, `/s/GLD`, `/s/AGG` render the profile, mix bar and holdings in the Paper Ledger look at 1280px and 390px with no overflow and zero console errors; `/s/AAPL` (stock) and `/s/^VIX` (index) are unchanged.**Resuming, next action****The MVP is shipped and live at https://finance.bythewood.me.** Phases 0through 12 are complete, verified, and deployed. No phase is in progress.Future work is the post-MVP backlog, phases 13 through 19 below. Redeploysare `git push server master` from this repo (the `server` remote is set).There is still no GitHub repo for finance — the user deferred that; if one iscreated later, add it as `origin` and the `overshard/finance` slug already intaproot's `projects.conf` lines up.**The MVP plus Phase 18 (ETF profiles) are live athttps://finance.bythewood.me.** No phase is in progress. Phase 18 was deployedon 2026-05-22 via `git push server master`; migration `0005` applies on thebox and the `sec` job backfills the 28 ETF profiles on its first due cycle.Remaining future work is the post-MVP backlog, phases 13 through 17 and 19below; ordering among them is loose. There is still no GitHub repo forfinance: the user deferred that; if one is created later, add it as `origin`and the `overshard/finance` slug already in taproot's `projects.conf` lines up.Note: Phase 18 added the `quick-xml` crate (N-PORT XML streaming parser) andmigration `0005`. A fresh `make run` applies `0005`; the ETF fund profilespopulate on the first `sec` job cycle that finds them stale.Note: because `^RUT`/`^VIX` stay historyless, `meta.seed_completed` is neverset, so the boot seed re-runs on every restart (cheap: ~2 Stooq calls that
@@ -727,12 +785,13 @@ depend on Phase 5 (live quotes) and Phase 7 (SEC data). the industry, solid fundamentals, consistent gains with the occasional acceptable setback)? Explicitly NOT buy or sell advice, and labelled as such in the UI. Builds on Phases 7, 14 and 15.- [ ] **Phase 18: ETF profiles.** Treat ETFs as first-class rather than the Phase 7 stocks-only treatment: top holdings and their weights (from SEC N-PORT), expense ratio, fund category, and the fund's own filing history (prospectus / 485BPOS, N-CEN). Needs a fund ticker-to-CIK source distinct from the operating-company `company_tickers.json`. Captured 2026-05-22 from a user question about ETF filings; see decisions log.- [x] **Phase 18: ETF profiles.** Complete and verified (2026-05-22) — the first post-MVP phase, see the Phase 18 entry in Status and the decisions log. ETFs are first-class: a fund profile (AUM, holdings count, top 25 holdings by weight, asset mix) and a fund filing history, sourced from SEC N-PORT via the new `company_tickers_mf.json` ticker map. Expense ratio and fund category were dropped (not in SEC structured data — user decision). Commodity grantor trusts (GLD, SLV) get a minimal AUM-only profile.- [ ] **Phase 19: Watchlists.** Named lists of symbols the user curates: watchlist and per-list pages plus mutation APIs (create / delete / rename
@@ -750,7 +809,7 @@ depend on Phase 5 (live quotes) and Phase 7 (SEC data).finance/ Cargo.toml Makefile migrations/ 0001_initial.sql 0002_endpoint_guard.sql 0003_guard_budget.sql 0004_fundamentals_unique.sql 0004_fundamentals_unique.sql 0005_fund_profiles.sql universe/starter.csv curated seed list src/ main.rs entry + `seed` subcommand
@@ -1063,6 +1122,60 @@ finance/ renders, and the first-run seed backfills on the live box. This closes Phase 12; the MVP is shipped. Remaining work is the post-MVP backlog (phases 13-19).- **2026-05-22: Phase 18 picked as the first post-MVP phase.** Asked which of the loose-ordered backlog (13-19) to take first, the user chose 18, ETF profiles. It is the most self-contained: it builds only on data sources already shipped (SEC EDGAR) and needs no prior backlog phase.- **2026-05-22: expense ratio and fund category dropped from Phase 18.** Research found neither is in SEC structured data — N-PORT carries holdings, net assets and an asset-class breakdown, but the expense ratio and a Morningstar-style category live only in prospectus (485BPOS) HTML, with no clean machine-readable field. Offered to curate them in `starter.csv`, parse the prospectus, or drop them; the user chose to drop both and show only what N-PORT provides. The asset-class mix computed from N-PORT holdings stands in for a category label.- **2026-05-22: GLD/SLV get a minimal commodity-trust profile.** GLD and SLV are grantor trusts holding physical metal: they file 10-Ks, not N-PORT, and have no securities portfolio. The user chose a minimal profile for them — AUM (from the 10-K `companyfacts`), the filing list, and a plain note that the trust holds bullion directly — over either a bare filings list or the no-section treatment indexes get.- **2026-05-22: Phase 18 ETF profiles shipped.** SEC N-PORT is now a fourth use of the EDGAR source. Design calls made during the build: (1) the fund methods are inherent to `SecProvider`, not behind a trait — N-PORT is wholly SEC-specific with no second source to abstract over, unlike the `HistoryProvider` / `QuoteProvider` / `FundamentalsProvider` concerns. (2) N-PORT lookups are keyed on the SEC *series id* (via the legacy browse-edgar Atom interface, validated to still work and routed through the `sec` guard), because one registrant CIK hosts many fund series and the modern `submissions` JSON cannot filter by series. (3) The N-PORT XML is located from the filing's browse-edgar index-page URL, not from the accession number, because a fund that files through a filing agent has the agent's CIK on the accession while the Archives path needs the registrant's — the index URL always carries the registrant CIK (this bit during verification: AGG and SPY 404'd until the fix). (4) `quick-xml` was added as a streaming parser: a bond aggregate fund's N-PORT runs to 15+ MB and 13k positions, so a DOM parse is wrong; only the top 25 holdings, the count and the asset mix are kept. (5) Holdings display the N-PORT issue `title` over the issuer `name`, since `name` often arrives truncated and all-caps. (6) The asset-mix bar uses ink shades, not semantic green/amber/red — a fund's composition is not a good/ok/bad judgement (the same exception the Phase 8 chart-indicator palette took). Phase 18 was deployed to production on 2026-05-22 via `git push server master` (migration `0005` applies on the box; the `sec` job backfills the 28 ETF profiles on its first due cycle).- **2026-05-22: dashboard futures cards now update overnight.** The user noticed the home-page sparkline cards showing stale "last night" numbers while futures were trading. Cause: the demand-driven `intraday` poll was gated to the US *equity* session (`session.is_open()`: pre, regular, post), so through the overnight `Closed` window nothing was polled and the commodity-futures cards (CL=F, GC=F, NG=F) sat frozen on the 16:00 ET daily-close snapshot. Fix: `run_intraday` now always runs; inside a trading session it polls every viewed symbol as before, but outside one it polls only viewed *futures*, which trade nearly around the clock. Indexes, stocks and ETFs stay correctly frozen off-hours. Still demand-driven and guarded: nothing is polled unless a browser is viewing it. No futures-hours calendar is modelled (a closed futures market just returns a flat quote), consistent with the no-holiday-calendar decision in `market.rs`.---
@@ -1079,6 +1192,9 @@ finance/- `/` has a "Futures & commodities" section; futures (`kind = 'future'`) are never sent to Stooq and carry a live quote only — `/s/GC=F` renders with no daily chart, like `^VIX`.- An ETF page (`/s/QQQ`) shows a Fund profile (AUM, holdings count, asset mix) and a Top holdings list, plus its SEC filing history; a commodity-trust ETF (`/s/GLD`) shows AUM and filings only, with no holdings.- Phone (~360 px) and desktop are both fully usable: no unintended horizontal scroll, chart resizes, every feature reachable.- No automated test suite or linter, matching the sibling projects.
modified
frontend/static_src/symbol/styles/symbol.scss
@@ -658,6 +658,214 @@ color: var(--ink-faint);}/* ---------- ETF fund profile ---------- */.fund { padding: 16px 18px;}.fund__stats { display: flex; flex-wrap: wrap; gap: 12px 36px;}.fund__stat { display: flex; flex-direction: column; gap: 3px;}.fund__stat-cap { @include eyebrow; font-size: 0.58rem;}.fund__stat-val { font-size: 1.4rem; font-weight: 700;}/* the physical-bullion note for a commodity-trust ETF (GLD, SLV) */.fund__note { margin-top: 15px; padding-top: 15px; border-top: 1px solid var(--rule); color: var(--ink-dim); font-size: 0.86rem;}/* asset-class mix: a thin segmented bar plus a compact legend. Segments are ink shades, not semantic green/amber/red — a fund's composition is not a good/ok/bad judgement (same reasoning as the chart indicator palette). */.fund__mix { margin-top: 16px; padding-top: 16px; border-top: 1px solid var(--rule);}.mixbar { display: flex; height: 10px; border-radius: 999px; overflow: hidden; background: var(--well);}.mixbar__seg { height: 100%; border-right: 1px solid var(--surface);}.mixbar__seg:last-child { border-right: none;}.mixbar__seg:nth-child(1) { background: var(--ink);}.mixbar__seg:nth-child(2) { background: var(--ink-dim);}.mixbar__seg:nth-child(3) { background: var(--ink-faint);}.mixbar__seg:nth-child(n + 4) { background: var(--rule-strong);}.mixlegend { display: flex; flex-wrap: wrap; gap: 6px 22px; margin-top: 13px;}.mixlegend__item { display: flex; align-items: baseline; gap: 9px;}.mixlegend__item dt { display: flex; align-items: center; gap: 7px; font-size: 0.83rem; color: var(--ink-dim);}.mixlegend__item dd { font-size: 0.83rem; font-weight: 600;}.mixlegend__dot { width: 9px; height: 9px; border-radius: 2px; flex: none; background: var(--ink-faint);}.mixlegend__item:nth-child(1) .mixlegend__dot { background: var(--ink);}.mixlegend__item:nth-child(2) .mixlegend__dot { background: var(--ink-dim);}.mixlegend__item:nth-child(3) .mixlegend__dot { background: var(--ink-faint);}.mixlegend__item:nth-child(n + 4) .mixlegend__dot { background: var(--rule-strong);}/* ---------- top holdings ---------- */.holdings { padding: 2px 16px;}.hold-list { margin: 0; padding: 0; list-style: none;}/* each row carries a faint left-anchored fill sized to the holding's weight, scaled so the largest holding shown fills the rail */.hold { position: relative; display: grid; grid-template-columns: 2.6ch 1fr auto auto; align-items: baseline; gap: 2px 12px; padding: 10px 7px; border-top: 1px solid var(--rule);}.hold:first-child { border-top: none;}.hold__fill { position: absolute; left: 0; top: 0; bottom: 0; z-index: 0; background: var(--well); border-radius: 3px;}/* the row text rides above the weight fill */.hold > :not(.hold__fill) { position: relative; z-index: 1;}.hold__rank { font-size: 0.74rem; color: var(--ink-faint);}.hold__name { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 0.88rem; font-weight: 500;}.hold__wt { font-size: 0.88rem; font-weight: 700;}.hold__val { min-width: 6.5ch; text-align: right; font-size: 0.82rem; color: var(--ink-dim);}/* on a narrow phone the position value yields to the weight, the key figure */@media (max-width: $bp-sm) { .hold { grid-template-columns: 2.6ch 1fr auto; } .hold__val { display: none; }}/* ---------- desktop layering ---------- */@media (min-width: $bp-md) { #chart {
added
migrations/0005_fund_profiles.sql
@@ -0,0 +1,40 @@-- finance migration 0005: ETF fund profiles (Phase 18).---- An ETF files with the SEC as a registered fund, not an operating company,-- so its portfolio comes from quarterly N-PORT filings rather than the XBRL-- companyfacts that back the stock fundamentals. These two tables hold the-- parsed snapshot of an ETF's latest N-PORT; a physical-commodity grantor-- trust (GLD, SLV) files 10-Ks instead and gets the degenerate `kind` below.-- The fund's SEC series id, e.g. S000002839. One registrant CIK can host many-- fund series (the Vanguard and iShares trusts host dozens), so the series id-- is what pins an N-PORT lookup to a single ETF. NULL for a single-fund trust-- (SPY, DIA) and for every non-ETF symbol.ALTER TABLE symbols ADD COLUMN series_id TEXT;-- When this ETF's fund profile was last refreshed from SEC. NULL = never.ALTER TABLE symbols ADD COLUMN fund_synced_at INTEGER;-- One profile row per ETF: the headline figures from its latest N-PORT, or,-- for a commodity trust, the AUM from its 10-K companyfacts.CREATE TABLE fund_profiles ( ticker TEXT PRIMARY KEY REFERENCES symbols(ticker) ON DELETE CASCADE, kind TEXT NOT NULL, -- 'portfolio' | 'commodity_trust' net_assets REAL, -- total net assets (AUM), USD total_assets REAL, -- gross assets, USD holdings_count INTEGER, -- positions in the full portfolio report_date TEXT, -- N-PORT "as of" date, YYYY-MM-DD asset_mix TEXT, -- JSON [[bucket, percent], ...]; portfolio funds only updated_at INTEGER NOT NULL);-- The largest holdings of a portfolio fund, ranked by weight. Only the top-- slice is kept (a bond aggregate fund holds thousands of positions); rank 1-- is the largest. Replaced wholesale on each refresh.CREATE TABLE fund_holdings ( ticker TEXT NOT NULL REFERENCES symbols(ticker) ON DELETE CASCADE, rank INTEGER NOT NULL, -- 1 = largest weight name TEXT NOT NULL, pct REAL, -- percent of net assets, e.g. 8.4 value_usd REAL, PRIMARY KEY (ticker, rank));
@@ -14,6 +14,8 @@ pub struct SymbolRow { pub exchange: Option<String>, pub currency: String, pub cik: Option<String>, /// SEC fund series id; set for an ETF, NULL otherwise (see migration 0005). pub series_id: Option<String>, pub sector: Option<String>, pub industry: Option<String>, pub is_seeded: i64,
@@ -23,6 +25,8 @@ pub struct SymbolRow { pub history_last_date: Option<String>, pub fundamentals_synced_at: Option<i64>, pub filings_synced_at: Option<i64>, /// When this ETF's fund profile was last refreshed from SEC. pub fund_synced_at: Option<i64>, pub last_price: Option<f64>, pub prev_close: Option<f64>, pub last_quote_at: Option<i64>,
modified
src/providers/mod.rs
@@ -135,6 +135,76 @@ pub trait FundamentalsProvider: Send + Sync { async fn filings(&self, cik: &str) -> Result<Vec<FilingRecord>>;}// ── ETF fund profiles (Phase 18) ───────────────────────────────────────────//// ETFs file as registered funds: their portfolio comes from quarterly N-PORT// filings, not the XBRL companyfacts behind the stock fundamentals above. The// fund methods live on `SecProvider` as inherent methods (N-PORT is wholly// SEC-specific, with no second source to abstract over), but their data types// sit here next to `Fact` / `FilingRecord` for the scheduler and routes./// Identifies one ETF to the SEC's fund endpoints.#[derive(Debug, Clone)]pub struct FundId { /// 10-digit zero-padded registrant CIK. pub cik: String, /// SEC series id (e.g. `S000002839`), present when the registrant hosts /// more than one fund — then it, not the CIK, pins a lookup to this ETF. pub series_id: Option<String>,}/// One portfolio holding parsed from an N-PORT filing.#[derive(Debug, Clone)]pub struct FundHolding { /// Issuer / security name as the fund reported it. pub name: String, /// Percent of the fund's net assets, e.g. `8.4`. pub pct: Option<f64>, /// Market value of the position, USD. pub value_usd: Option<f64>, /// N-PORT asset-category code (`EC` equity, `DBT` debt, ...), for the mix. pub asset_cat: Option<String>,}/// What a fund's filing history reveals about how to read its portfolio.#[derive(Debug, Clone)]pub enum FundShape { /// A fund that files N-PORT: fetch this filing for its holdings. The value /// is the filing's EDGAR index-page URL, whose directory also holds the /// N-PORT XML — and which carries the registrant CIK even when a filing /// agent (not the fund) is the named filer on the accession number. Portfolio { nport_href: String }, /// A physical-commodity grantor trust (GLD, SLV): no N-PORT — it holds /// bullion, not a securities portfolio — so AUM comes from its 10-K. CommodityTrust, /// Neither pattern matched; the page can still show the filing list. Unknown,}/// The filing list for an ETF plus what it implies about the fund's shape.#[derive(Debug, Clone)]pub struct FundFilings { pub filings: Vec<FilingRecord>, pub shape: FundShape,}/// A fund's portfolio snapshot, parsed from one N-PORT filing.#[derive(Debug, Clone, Default)]pub struct PortfolioData { /// Total net assets (AUM), USD. pub net_assets: Option<f64>, /// Gross assets, USD. pub total_assets: Option<f64>, /// The date the holdings are reported as of, `YYYY-MM-DD`. pub report_date: Option<String>, /// Positions in the full portfolio (not just the top slice kept below). pub holdings_count: i64, /// The largest holdings by weight, largest first. pub top_holdings: Vec<FundHolding>, /// Asset-class mix as `(bucket, percent)` pairs, largest bucket first. pub asset_mix: Vec<(String, f64)>,}/// An upstream rejected a request with an explicit rate-limit signal (HTTP 429/// or 503). A provider returns this as the source of its `anyhow::Error` so the/// `EndpointGuard` (see `src/guard.rs`) can recognise it by downcast and trip
modified
src/providers/sec.rs
@@ -26,7 +26,13 @@ use chrono::{Datelike, NaiveDate};use reqwest::{header::RETRY_AFTER, StatusCode};use serde::Deserialize;use crate::providers::{Fact, FilingRecord, FundamentalsProvider, RateLimited};use quick_xml::events::{BytesStart, Event};use quick_xml::Reader;use crate::providers::{ Fact, FilingRecord, FundFilings, FundHolding, FundId, FundShape, FundamentalsProvider, PortfolioData, RateLimited,};/// Fundamentals and filings from SEC EDGAR.pub struct SecProvider {
@@ -474,3 +480,417 @@ impl FundamentalsProvider for SecProvider { Ok(out) }}// ── ETF fund profiles: N-PORT holdings, AUM, filing history (Phase 18) ─────//// An ETF files as a registered fund, so its portfolio is not in the XBRL// `companyfacts` above — it is in quarterly N-PORT filings, one large XML per// fund. These methods are inherent to `SecProvider` rather than behind a// trait: N-PORT is wholly SEC-specific, with no second source to abstract// over. Each method makes exactly one HTTP request so the scheduler can keep// wrapping every call in the endpoint guard, as it does for `facts`/`filings`./// Fund trusts that file with the SEC but are absent from/// `company_tickers_mf.json` (which is keyed on the series/class structure of/// open-end funds): the unit investment trusts (SPY, DIA) and the physical-/// commodity grantor trusts (GLD, SLV). Mapped straight to their registrant/// CIK, with no series id since each trust is a single fund.const FUND_FALLBACK: &[(&str, i64)] = &[ ("SPY", 884394), ("DIA", 1041130), ("GLD", 1222333), ("SLV", 1330568),];/// How many of a fund's holdings to keep — the largest by weight. A bond/// aggregate fund holds thousands of positions; the page shows only the top.const TOP_HOLDINGS: usize = 25;/// How many of a fund's filings to keep for the page's filing list.const MAX_FUND_FILINGS: usize = 40;/// `company_tickers_mf.json`: a `fields` header plus row tuples of/// `(cik, seriesId, classId, symbol)`.#[derive(Deserialize)]struct MfFile { data: Vec<(i64, String, String, String)>,}/// One filing parsed from a browse-edgar Atom feed.#[derive(Default)]struct AtomEntry { form: String, accession: String, filed: String, /// EDGAR filing-index page URL. href: String,}/// One holding accumulated while streaming through an `<invstOrSec>` block.#[derive(Default)]struct HoldingAcc { name: String, title: String, pct: Option<f64>, value: Option<f64>, asset_cat: Option<String>,}impl HoldingAcc { fn into_holding(self) -> FundHolding { // Prefer `title` (the issue title): it is clean mixed-case, where the // issuer `name` often arrives truncated and all-caps. Fall back to // `name` for the rare holding that carried no title. let name = if self.title.is_empty() { self.name } else { self.title }; FundHolding { name, pct: self.pct, value_usd: self.value, asset_cat: self.asset_cat, } }}impl SecProvider { /// Ticker -> fund identity, from the SEC mutual-fund ticker file plus the /// hardcoded fallback for fund trusts absent from it. Keys are normalised /// like `cik_map`'s. One bulk request, fetched while some ETF lacks a CIK. pub async fn fund_ticker_map(&self) -> Result<HashMap<String, FundId>> { let url = "https://www.sec.gov/files/company_tickers_mf.json"; let resp = self .get(url) .await? .ok_or_else(|| anyhow!("sec company_tickers_mf.json not found"))?; let body: MfFile = resp.json().await?; let mut map = HashMap::with_capacity(body.data.len() + FUND_FALLBACK.len()); for (cik, series_id, _class_id, symbol) in body.data { map.entry(normalize_ticker(&symbol)).or_insert(FundId { cik: format!("{cik:010}"), series_id: Some(series_id), }); } for (ticker, cik) in FUND_FALLBACK { map.entry(normalize_ticker(ticker)).or_insert(FundId { cik: format!("{cik:010}"), series_id: None, }); } Ok(map) } /// A fund's filing list, plus what the filing history says about its shape /// (whether to read an N-PORT for holdings, or treat it as a commodity /// trust). One browse-edgar request, keyed on the series id when the /// registrant hosts several funds so a sibling fund's filings never leak in. pub async fn fund_filings(&self, id: &FundId) -> Result<FundFilings> { let key = id.series_id.as_deref().unwrap_or(&id.cik); let url = format!( "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK={key}\ &type=&dateb=&owner=include&count=100&output=atom" ); let resp = self .get(&url) .await? .ok_or_else(|| anyhow!("edgar filing index for {key} not found"))?; let bytes = resp.bytes().await?; let entries = parse_edgar_atom(&bytes)?; // Entries arrive newest-first. Read the fund's shape from the whole set // before trimming the list to the material forms shown on the page. let mut nport: Option<String> = None; let mut has_ncen = false; let mut has_10k = false; for e in &entries { if nport.is_none() && e.form.starts_with("NPORT-P") && !e.href.is_empty() { nport = Some(e.href.clone()); } has_ncen |= e.form.starts_with("N-CEN"); has_10k |= e.form.starts_with("10-K"); } let shape = if let Some(nport_href) = nport { FundShape::Portfolio { nport_href } } else if has_10k && !has_ncen { // Files 10-Ks and no fund-census report: a grantor trust holding a // physical commodity rather than a securities portfolio. FundShape::CommodityTrust } else { FundShape::Unknown }; let filings = entries .into_iter() .filter(|e| is_material_fund_form(&e.form)) .take(MAX_FUND_FILINGS) .map(|e| FilingRecord { accession: e.accession, form: e.form, filed_at: e.filed, period_of_report: None, primary_doc: None, url: e.href, description: None, }) .collect(); Ok(FundFilings { filings, shape }) } /// Parse one N-PORT filing into a portfolio snapshot: net assets, the /// holdings (top slice by weight), the holding count, and the asset mix. /// `index_href` is the filing's EDGAR index-page URL; the N-PORT XML sits /// beside it in the same Archives directory. One request. pub async fn fund_portfolio(&self, index_href: &str) -> Result<PortfolioData> { // `.../data/{cik}/{nodash}/{accession}-index.htm` -> swap the index // page for `primary_doc.xml` in the same directory. let dir = index_href .rsplit_once('/') .map(|(d, _)| d) .ok_or_else(|| anyhow!("malformed filing href {index_href}"))?; let url = format!("{dir}/primary_doc.xml"); let resp = self .get(&url) .await? .ok_or_else(|| anyhow!("N-PORT not found at {url}"))?; let bytes = resp.bytes().await?; parse_nport(&bytes) } /// The latest total-assets figure a company has reported, USD. Gives a /// physical-commodity grantor trust (GLD, SLV) an AUM: those file 10-Ks, /// not N-PORT, so `Assets` from their XBRL companyfacts stands in for net /// assets. Unlike `facts`, this takes the single most recent value /// regardless of fiscal period, so a mid-year 10-Q figure beats a stale /// prior year-end. `None` when the company has no `Assets` concept. pub async fn fund_aum(&self, cik: &str) -> Result<Option<f64>> { let url = format!("https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json"); let Some(resp) = self.get(&url).await? else { return Ok(None); // 404: no XBRL facts }; let body: CompanyFacts = resp.json().await?; let Some(assets) = body.facts.us_gaap.get("Assets") else { return Ok(None); }; // Newest by period-end date, ties broken by the later filing. let mut best: Option<&UnitEntry> = None; for entries in assets.units.values() { for e in entries { let newer = best.map_or(true, |b| { (e.end.as_str(), e.filed.as_deref().unwrap_or("")) > (b.end.as_str(), b.filed.as_deref().unwrap_or("")) }); if newer { best = Some(e); } } } Ok(best.map(|e| e.val)) }}/// Filing forms worth showing on a fund's page: its portfolio reports/// (N-PORT), the annual fund census and shareholder reports, the prospectus,/// and — for a commodity trust — the 10-K family it files instead.fn is_material_fund_form(form: &str) -> bool { const PREFIXES: &[&str] = &[ "NPORT-P", "NPORT-EX", "N-CEN", "N-CSR", "485BPOS", "485APOS", "10-K", "10-Q", "8-K", ]; PREFIXES.iter().any(|p| form.starts_with(p))}/// Map an N-PORT `assetCat` code to a human asset-class bucket for the mix./// The codes are from the N-PORT technical schema; the long tail is "Other".fn asset_bucket(cat: &str) -> &'static str { match cat { "EC" | "EP" => "Equity", "DBT" | "SF" => "Bonds", "STIV" | "RA" => "Cash & equivalents", "COMM" => "Commodities", "RE" => "Real estate", "LON" => "Loans", c if c.starts_with("ABS") => "Bonds", // Every other derivative category code begins `D` (DBT is matched above). c if c.starts_with('D') => "Derivatives", _ => "Other", }}/// Read one attribute off a start tag as a `String`.fn attr_val(e: &BytesStart, key: &[u8]) -> Option<String> { e.attributes() .flatten() .find(|a| a.key.as_ref() == key) .and_then(|a| String::from_utf8(a.value.into_owned()).ok())}/// Set the current entry's form from a `<category term="..."/>` tag.fn set_form_from_category(e: &BytesStart, cur: &mut Option<AtomEntry>) { if let (Some(c), Some(term)) = (cur.as_mut(), attr_val(e, b"term")) { c.form = term; }}/// Parse a browse-edgar Atom feed into its filing entries, newest first.fn parse_edgar_atom(xml: &[u8]) -> Result<Vec<AtomEntry>> { let mut reader = Reader::from_reader(xml); let mut buf = Vec::new(); let mut path: Vec<Vec<u8>> = Vec::new(); let mut entries = Vec::new(); let mut cur: Option<AtomEntry> = None; loop { match reader.read_event_into(&mut buf)? { Event::Start(e) => { let name = e.local_name().as_ref().to_vec(); if name == b"entry" { cur = Some(AtomEntry::default()); } if name == b"category" { set_form_from_category(&e, &mut cur); } path.push(name); } // The form type usually rides a self-closing `<category term=".."/>`. Event::Empty(e) => { if e.local_name().as_ref() == b"category" { set_form_from_category(&e, &mut cur); } } Event::End(e) => { if e.local_name().as_ref() == b"entry" { if let Some(c) = cur.take() { if !c.accession.is_empty() { entries.push(c); } } } path.pop(); } Event::Text(t) => { let (Some(tag), Some(c)) = (path.last(), cur.as_mut()) else { continue; }; let raw = t.unescape().unwrap_or_default(); let txt = raw.trim(); if txt.is_empty() { continue; } match tag.as_slice() { b"accession-number" if c.accession.is_empty() => c.accession = txt.to_string(), b"filing-date" if c.filed.is_empty() => c.filed = txt.to_string(), b"filing-href" if c.href.is_empty() => c.href = txt.to_string(), b"filing-type" if c.form.is_empty() => c.form = txt.to_string(), _ => {} } } Event::Eof => break, _ => {} } buf.clear(); } Ok(entries)}/// Stream-parse an N-PORT `primary_doc.xml` into a portfolio snapshot. The file/// can run to many megabytes for a bond fund's thousands of positions, so this/// walks events rather than building a DOM, keeping only the running totals/// and, at the end, the largest holdings.fn parse_nport(xml: &[u8]) -> Result<PortfolioData> { let mut reader = Reader::from_reader(xml); let mut buf = Vec::new(); // Stack of open element local names, so a leaf is read in context. let mut path: Vec<Vec<u8>> = Vec::new(); let mut out = PortfolioData::default(); let mut all: Vec<FundHolding> = Vec::new(); // The holding currently being assembled, set while inside `<invstOrSec>`. let mut cur: Option<HoldingAcc> = None; loop { match reader.read_event_into(&mut buf)? { Event::Start(e) => { let name = e.local_name().as_ref().to_vec(); if name == b"invstOrSec" { cur = Some(HoldingAcc::default()); } path.push(name); } Event::End(e) => { if e.local_name().as_ref() == b"invstOrSec" { if let Some(h) = cur.take() { all.push(h.into_holding()); } } path.pop(); } Event::Text(t) => { let Some(tag) = path.last() else { continue; }; let raw = t.unescape().unwrap_or_default(); let txt = raw.trim(); if txt.is_empty() { continue; } let tag = tag.as_slice(); // Holding fields, captured only inside an `<invstOrSec>`. // First-wins: the issuer-level value precedes any nested block. if let Some(h) = cur.as_mut() { match tag { b"name" if h.name.is_empty() => h.name = txt.to_string(), b"title" if h.title.is_empty() => h.title = txt.to_string(), b"pctVal" if h.pct.is_none() => h.pct = txt.parse().ok(), b"valUSD" if h.value.is_none() => h.value = txt.parse().ok(), b"assetCat" if h.asset_cat.is_none() => { h.asset_cat = Some(txt.to_string()) } _ => {} } } // Fund-level fields, scoped by their parent element. let parent = path.iter().rev().nth(1).map(Vec::as_slice); match (parent, tag) { (Some(b"fundInfo"), b"netAssets") if out.net_assets.is_none() => { out.net_assets = txt.parse().ok(); } (Some(b"fundInfo"), b"totAssets") if out.total_assets.is_none() => { out.total_assets = txt.parse().ok(); } (Some(b"genInfo"), b"repPdDate") if out.report_date.is_none() => { out.report_date = Some(txt.to_string()); } _ => {} } } Event::Eof => break, _ => {} } buf.clear(); } out.holdings_count = all.len() as i64; // Asset-class mix: each holding's weight summed into its bucket. Tiny // residual buckets (rounding noise) are dropped. let mut mix: HashMap<&'static str, f64> = HashMap::new(); for h in &all { let bucket = h.asset_cat.as_deref().map_or("Other", asset_bucket); *mix.entry(bucket).or_insert(0.0) += h.pct.unwrap_or(0.0); } let mut mix: Vec<(String, f64)> = mix .into_iter() .filter(|(_, p)| *p >= 0.05) .map(|(b, p)| (b.to_string(), p)) .collect(); mix.sort_by(|a, b| b.1.total_cmp(&a.1)); out.asset_mix = mix; // Largest holdings first; keep only the top slice. all.sort_by(|a, b| b.pct.unwrap_or(0.0).total_cmp(&a.pct.unwrap_or(0.0))); all.truncate(TOP_HOLDINGS); out.top_holdings = all; Ok(out)}
modified
src/routes/symbols.rs
@@ -217,6 +217,14 @@ fn filing_title(form: &str) -> String { "Annual report" } else if form.starts_with("6-K") { "Interim report" } else if form.starts_with("NPORT") { "Portfolio holdings report" } else if form.starts_with("N-CEN") { "Annual fund census" } else if form.starts_with("N-CSR") { "Shareholder report" } else if form.starts_with("485") { "Prospectus" } else { "Filing" };
@@ -338,6 +346,111 @@ fn build_fundamentals(facts: &[FundFact], price: Option<f64>) -> Option<Fundamen })}// ── ETF fund profile (Phase 18) ────────────────────────────────────────────/// A `fund_profiles` row as stored.#[derive(sqlx::FromRow)]struct FundProfileRow { /// `portfolio` or `commodity_trust`. kind: String, net_assets: Option<f64>, holdings_count: Option<i64>, report_date: Option<String>, /// JSON `[[bucket, percent], ...]`. asset_mix: Option<String>,}/// A `fund_holdings` row as stored.#[derive(sqlx::FromRow)]struct HoldingRow { rank: i64, name: String, pct: Option<f64>, value_usd: Option<f64>,}/// One asset-class slice of an ETF's portfolio mix.#[derive(Serialize)]struct AssetSlice { label: String, /// Percent string, e.g. `99.8%`. pct: String, /// Segment width 0..100 for the mix bar. width: f64,}/// One holding row shaped for the page.#[derive(Serialize)]struct HoldingView { rank: i64, name: String, /// Weight as a percent string, e.g. `8.42%`. weight: String, /// Bar width 0..100, scaled so the largest holding shown fills the rail. bar_pct: f64, /// Position value, compact USD, e.g. `$3.3B`. value: String,}/// Everything the symbol page's ETF fund-profile section needs.#[derive(Serialize)]struct FundView { /// A physical-commodity grantor trust (GLD, SLV): holds bullion, not a /// securities portfolio, so it has no holdings and no asset mix. is_commodity: bool, /// Net assets / AUM, compact USD. `None` when the fund reported none. net_assets: Option<String>, holdings_count: Option<i64>, /// The N-PORT "as of" date, `YYYY-MM-DD`. report_date: Option<String>, asset_mix: Vec<AssetSlice>, holdings: Vec<HoldingView>,}/// Assemble the fund view from a stored profile row and its holdings.fn build_fund(profile: FundProfileRow, holdings: Vec<HoldingRow>) -> FundView { // Asset mix: the stored JSON `[[label, percent], ...]`. The percentages // already sum to ~100, so each is its own segment width directly. let asset_mix = profile .asset_mix .as_deref() .and_then(|j| serde_json::from_str::<Vec<(String, f64)>>(j).ok()) .unwrap_or_default() .into_iter() .map(|(label, pct)| AssetSlice { pct: format!("{pct:.1}%"), width: pct.clamp(0.0, 100.0), label, }) .collect(); // Holdings: each weight bar is scaled to the largest holding shown, so the // top position fills the rail and the rest read against it. let max_pct = holdings.iter().filter_map(|h| h.pct).fold(0.0_f64, f64::max); let holdings = holdings .into_iter() .map(|h| HoldingView { rank: h.rank, name: h.name, weight: h.pct.map_or_else(|| DASH.to_string(), |p| format!("{p:.2}%")), bar_pct: match h.pct { Some(p) if max_pct > 0.0 => (p / max_pct * 100.0).clamp(0.0, 100.0), _ => 0.0, }, value: h.value_usd.map_or_else(|| DASH.to_string(), fmt_usd_compact), }) .collect(); FundView { is_commodity: profile.kind == "commodity_trust", net_assets: profile.net_assets.map(fmt_usd_compact), holdings_count: profile.holdings_count, report_date: profile.report_date, asset_mix, holdings, }}async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) -> Response { let ticker = ticker.to_uppercase();
@@ -416,8 +529,10 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) } }); // Fundamentals and filings are stocks-only: ETFs and indexes do not file. // Fundamentals are stocks-only; an ETF gets a fund profile instead; an // index gets neither. Filings cover both stocks and ETFs. let is_stock = symbol.kind == "stock"; let is_etf = symbol.kind == "etf"; // Ratios price off the live quote, falling back to the last daily close. let price = quote .as_ref()
@@ -438,7 +553,7 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) None }; let filings: Vec<FilingView> = if is_stock { let filings: Vec<FilingView> = if is_stock || is_etf { sqlx::query_as::<_, FilingRow>( "SELECT form, filed_at, period_of_report, url FROM filings \ WHERE ticker = ? ORDER BY filed_at DESC, accession DESC LIMIT 18",
@@ -460,12 +575,42 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) Vec::new() }; // The ETF fund profile, when the SEC sweep has reached this symbol. let fund = if is_etf { let profile = sqlx::query_as::<_, FundProfileRow>( "SELECT kind, net_assets, holdings_count, report_date, asset_mix \ FROM fund_profiles WHERE ticker = ?", ) .bind(&ticker) .fetch_optional(&state.pool) .await .ok() .flatten(); match profile { Some(profile) => { let holdings = sqlx::query_as::<_, HoldingRow>( "SELECT rank, name, pct, value_usd FROM fund_holdings \ WHERE ticker = ? ORDER BY rank", ) .bind(&ticker) .fetch_all(&state.pool) .await .unwrap_or_default(); Some(build_fund(profile, holdings)) } None => None, } } else { None }; let extra = minijinja::context! { title => ticker, symbol => symbol, stats => stats, quote => quote, fundamentals => fundamentals, fund => fund, filings => filings, }; render(&state, "pages/symbol.html", &format!("/s/{ticker}"), extra)
modified
src/scheduler.rs
@@ -22,7 +22,7 @@//! hours apart, not seconds, and are inherently sequential (the pacing is the//! point), so they run inline in the loop task rather than via semaphores.use std::collections::HashMap;use std::collections::{HashMap, HashSet};use std::sync::Arc;use std::time::{Duration, Instant};
@@ -35,8 +35,8 @@ use crate::market;use crate::providers::sec::SecProvider;use crate::providers::yahoo::YahooProvider;use crate::providers::{ self, stooq::StooqProvider, Fact, FilingRecord, FundamentalsProvider, HistoryProvider, IntradayBar, Quote, QuoteProvider, self, stooq::StooqProvider, Fact, FilingRecord, FundId, FundShape, FundamentalsProvider, HistoryProvider, IntradayBar, PortfolioData, Quote, QuoteProvider,};use crate::stream::{Hub, QuoteUpdate, StreamEvent};use crate::{seed, Config};
@@ -159,12 +159,12 @@ pub fn spawn(pool: SqlitePool, config: Arc<Config>, hub: Arc<Hub>) -> JoinHandle } } // Intraday quotes: demand-driven (only symbols with a live viewer) // and gated to trading hours. Does no network work otherwise. if session.is_open() { if let Err(e) = run_intraday(&pool, &config, &hub).await { tracing::warn!("[scheduler] intraday: {e:#}"); } // Intraday quotes: demand-driven (only symbols a browser is // viewing). Inside a trading session every viewed symbol is // polled; outside it, only viewed futures, which trade nearly // around the clock. Does no network work when neither applies. if let Err(e) = run_intraday(&pool, &config, &hub, session).await { tracing::warn!("[scheduler] intraday: {e:#}"); } if let Err(e) = run_daily_close_if_due(&pool, &config, &hub).await {
@@ -368,19 +368,44 @@ async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::R////// Polls Yahoo only for the symbols a browser is currently viewing (the stream/// hub's interest registry). With nobody watching, `hub.viewed()` is empty and/// this returns at once having done no network work — that is the user's hard/// rule: poll only what is on screen. Each tick re-sweeps the viewed set, so/// one open symbol page refreshes about every minute, while an open dashboard/// (~144 symbols) refreshes more slowly as the guard's 1.5s pacing carries it./// this returns at once having done no network work: the user's hard rule is/// to poll only what is on screen.////// Which viewed symbols are polled depends on the session. Inside any trading/// session (pre, regular, post) every viewed symbol is fair game. Outside it,/// only viewed futures are polled: index futures and commodities trade nearly/// around the clock, while indexes, stocks and ETFs sit frozen until the next/// session, so polling them off-hours would only re-fetch a flat quote. This/// is what keeps the dashboard's commodity cards live overnight.////// A clean run is recorded only in `data_status` (plus each `quotes.fetched_at`/// row); a `fetch_log` row is written only for a notable run — an error or a/// guard stop — so the minute-cadence job does not bury the log.async fn run_intraday(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> {/// row); a `fetch_log` row is written only for a notable run, an error or a/// guard stop, so the minute-cadence job does not bury the log.async fn run_intraday( pool: &SqlitePool, config: &Config, hub: &Hub, session: market::Session,) -> anyhow::Result<()> { let viewed = hub.viewed(); if viewed.is_empty() { return Ok(()); } // Inside a session, poll every viewed symbol; outside it, only the futures. let targets: Vec<String> = if session.is_open() { viewed } else { let futures: HashSet<String> = sqlx::query_scalar("SELECT ticker FROM symbols WHERE kind = 'future'") .fetch_all(pool) .await? .into_iter() .collect(); viewed.into_iter().filter(|t| futures.contains(t)).collect() }; if targets.is_empty() { return Ok(()); } let started = now_ms(); mark_fetching(pool, "intraday").await?; notify_health(hub);
@@ -393,7 +418,7 @@ async fn run_intraday(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow:: let mut errors = 0i64; let mut stopped: Option<String> = None; for ticker in &viewed { for ticker in &targets { match guard.acquire().await? { Permit::Granted => {} Permit::Denied(why) => {
@@ -545,17 +570,22 @@ async fn run_daily_close_if_due( Ok(())}/// SEC fundamentals & filings sweep./// SEC fundamentals, filings & ETF fund-profile sweep.////// On the first run (and whenever new symbols appear) one bulk/// `company_tickers.json` fetch fills in each stock's CIK. Then every stock/// whose SEC data has gone stale is refreshed: its XBRL `companyfacts` into/// `fundamentals` and its submission history into `filings`. ETFs and indexes/// are skipped; they do not file with the SEC./// On the first run (and whenever new symbols appear) two bulk ticker-map/// fetches fill in CIKs — `company_tickers.json` for stocks,/// `company_tickers_mf.json` for ETFs. Then every symbol whose SEC data has/// gone stale is refreshed:/// - a stock's XBRL `companyfacts` into `fundamentals`, its submission/// history into `filings`;/// - an ETF's latest N-PORT into `fund_profiles` + `fund_holdings`, its/// filing history into `filings`. A physical-commodity grantor trust files/// no N-PORT, so its AUM comes from `companyfacts` instead./// Indexes are skipped; they do not file with the SEC.////// Resumable like the history job: each company's two timestamps/// (`fundamentals_synced_at`, `filings_synced_at`) are stamped only on a/// successful fetch, so a guard stop simply leaves the rest for the next cycle./// Resumable like the history job: each symbol's sync timestamps are stamped/// only on a successful fetch, so a guard stop simply leaves the rest for the/// next cycle.async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> { let started = now_ms(); let next = started + SEC_INTERVAL_SECS * 1000;
@@ -566,8 +596,8 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul let guard = EndpointGuard::with_budget(pool.clone(), sec.name(), SEC_BUDGET); let t0 = Instant::now(); // 1. CIK resolution. One bulk call maps the whole market; only needed // while some stock still lacks a CIK. // 1. Stock CIK resolution. One bulk call maps the whole market; only // needed while some stock still lacks a CIK. let missing: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM symbols WHERE kind = 'stock' AND cik IS NULL", )
@@ -603,10 +633,37 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul } } // 2. Stale sweep. A company is due when either of its SEC timestamps is // unset or older than the staleness window. // 1b. ETF fund-CIK resolution from the mutual-fund ticker map. Best-effort: // a guard stop or error here only leaves the ETFs for a later cycle, // rather than aborting the stock sweep below. let etfs_missing: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM symbols WHERE kind = 'etf' AND cik IS NULL", ) .fetch_one(pool) .await?; if etfs_missing > 0 { match guard.acquire().await? { Permit::Granted => match sec.fund_ticker_map().await { Ok(map) => { guard.record_success().await?; let resolved = resolve_fund_ciks(pool, &map).await?; tracing::info!("[scheduler] sec: resolved {resolved}/{etfs_missing} fund CIKs"); } Err(e) => { guard.record_failure(&e).await?; tracing::warn!("[scheduler] sec fund CIK map: {e:#}"); } }, Permit::Denied(why) => { tracing::info!("[scheduler] sec: fund CIK map skipped ({why})"); } } } // 2. Stale sweep. A symbol is due when one of its SEC timestamps is unset // or older than the staleness window. let cutoff = started - SEC_STALE_SECS * 1000; let stale: Vec<(String, String, Option<i64>, Option<i64>)> = sqlx::query_as( let stale_stocks: Vec<(String, String, Option<i64>, Option<i64>)> = sqlx::query_as( "SELECT ticker, cik, fundamentals_synced_at, filings_synced_at FROM symbols \ WHERE kind = 'stock' AND cik IS NOT NULL \ AND (fundamentals_synced_at IS NULL OR filings_synced_at IS NULL \
@@ -617,24 +674,38 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul .bind(cutoff) .fetch_all(pool) .await?; let stale_etfs: Vec<(String, String, Option<String>)> = sqlx::query_as( "SELECT ticker, cik, series_id FROM symbols \ WHERE kind = 'etf' AND cik IS NOT NULL \ AND (fund_synced_at IS NULL OR fund_synced_at < ?) \ ORDER BY ticker", ) .bind(cutoff) .fetch_all(pool) .await?; if stale.is_empty() { log_fetch(pool, "sec", "sec", "ok", Some("no stale companies"), Some(0), 0, started) if stale_stocks.is_empty() && stale_etfs.is_empty() { log_fetch(pool, "sec", "sec", "ok", Some("no stale companies or funds"), Some(0), 0, started) .await?; mark_ok(pool, "sec", Some(next)).await?; notify_health(hub); return Ok(()); } tracing::info!("[scheduler] sec: refreshing {} companies", stale.len()); tracing::info!( "[scheduler] sec: refreshing {} companies, {} funds", stale_stocks.len(), stale_etfs.len() ); // A company's metric is due when its timestamp is unset or past the cutoff. // A metric is due when its timestamp is unset or past the cutoff. let due = |at: Option<i64>| at.map_or(true, |t| t < cutoff); let mut funds_ok = 0i64; let mut filings_ok = 0i64; let mut etfs_ok = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None; 'sweep: for (ticker, cik, f_at, fl_at) in &stale { 'stocks: for (ticker, cik, f_at, fl_at) in &stale_stocks { if due(*f_at) { match guard.acquire().await? { Permit::Granted => match sec.facts(cik).await {
@@ -652,7 +723,7 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul }, Permit::Denied(why) => { stopped = Some(why); break 'sweep; break 'stocks; } } }
@@ -673,14 +744,94 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul }, Permit::Denied(why) => { stopped = Some(why); break 'sweep; break 'stocks; } } } } // 3. ETF fund-profile sweep, sharing the guard and the early-exit. Skipped // wholesale if the stock sweep above already hit a guard stop. if stopped.is_none() { 'funds: for (ticker, cik, series_id) in &stale_etfs { let id = FundId { cik: cik.clone(), series_id: series_id.clone(), }; // 3a. The filing list, and what it says about the fund's shape. let shape = match guard.acquire().await? { Permit::Granted => match sec.fund_filings(&id).await { Ok(ff) => { guard.record_success().await?; store_filings(pool, ticker, &ff.filings).await?; ff.shape } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_filings {ticker} failed: {e:#}"); continue 'funds; } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }; // 3b. Holdings from N-PORT, or AUM for a commodity trust. match shape { FundShape::Portfolio { nport_href } => match guard.acquire().await? { Permit::Granted => match sec.fund_portfolio(&nport_href).await { Ok(portfolio) => { guard.record_success().await?; store_fund_portfolio(pool, ticker, &portfolio).await?; mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_portfolio {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }, FundShape::CommodityTrust => match guard.acquire().await? { Permit::Granted => match sec.fund_aum(cik).await { Ok(aum) => { guard.record_success().await?; store_fund_commodity(pool, ticker, aum).await?; mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec fund_aum {ticker} failed: {e:#}"); } }, Permit::Denied(why) => { stopped = Some(why); break 'funds; } }, FundShape::Unknown => { // The filing list synced but there is no portfolio to // record. Stamp it so it is not retried until next stale. mark_fund_synced(pool, ticker).await?; etfs_ok += 1; } } } } let dur = t0.elapsed().as_millis() as i64; let counts = format!("{funds_ok} fundamentals, {filings_ok} filings, {errors} errors"); let counts = format!( "{funds_ok} fundamentals, {filings_ok} filings, {etfs_ok} fund profiles, {errors} errors" ); match stopped { Some(why) => { let detail = format!("stopped early ({why}); {counts}");
@@ -722,6 +873,35 @@ async fn resolve_ciks(pool: &SqlitePool, map: &HashMap<String, String>) -> sqlx: Ok(resolved)}/// Fill in `symbols.cik` and `symbols.series_id` for any ETF found in the bulk/// SEC mutual-fund ticker map. Returns how many were newly resolved.async fn resolve_fund_ciks( pool: &SqlitePool, map: &HashMap<String, FundId>,) -> sqlx::Result<i64> { let etfs: Vec<String> = sqlx::query_scalar("SELECT ticker FROM symbols WHERE kind = 'etf' AND cik IS NULL") .fetch_all(pool) .await?; let mut resolved = 0; for ticker in etfs { if let Some(id) = map.get(&crate::providers::sec::normalize_ticker(&ticker)) { let now = now_ms(); sqlx::query( "UPDATE symbols SET cik = ?, series_id = ?, updated_at = ? WHERE ticker = ?", ) .bind(&id.cik) .bind(&id.series_id) .bind(now) .bind(&ticker) .execute(pool) .await?; resolved += 1; } } Ok(resolved)}/// Stamp one of a symbol's SEC sync timestamps to now. `column` is one of two/// hardcoded literals (never user input), so interpolating it is safe.async fn mark_sec_synced(pool: &SqlitePool, ticker: &str, column: &str) -> sqlx::Result<()> {
@@ -802,6 +982,96 @@ async fn store_filings( Ok(())}/// Upsert one ETF's N-PORT fund profile and replace its stored holdings. The/// kept holdings are a small top slice, so they are deleted and re-inserted/// wholesale on each refresh.async fn store_fund_portfolio( pool: &SqlitePool, ticker: &str, p: &PortfolioData,) -> anyhow::Result<()> { // Asset mix is variable-length, so it rides in one JSON column rather than // its own table: [["Equity", 99.8], ["Cash & equivalents", 0.2], ...]. let asset_mix = serde_json::to_string(&p.asset_mix)?; let mut tx = pool.begin().await?; sqlx::query( "INSERT INTO fund_profiles \ (ticker, kind, net_assets, total_assets, holdings_count, report_date, \ asset_mix, updated_at) \ VALUES (?, 'portfolio', ?, ?, ?, ?, ?, ?) \ ON CONFLICT(ticker) DO UPDATE SET \ kind = excluded.kind, net_assets = excluded.net_assets, \ total_assets = excluded.total_assets, holdings_count = excluded.holdings_count, \ report_date = excluded.report_date, asset_mix = excluded.asset_mix, \ updated_at = excluded.updated_at", ) .bind(ticker) .bind(p.net_assets) .bind(p.total_assets) .bind(p.holdings_count) .bind(&p.report_date) .bind(&asset_mix) .bind(now_ms()) .execute(&mut *tx) .await?; sqlx::query("DELETE FROM fund_holdings WHERE ticker = ?") .bind(ticker) .execute(&mut *tx) .await?; for (i, h) in p.top_holdings.iter().enumerate() { sqlx::query( "INSERT INTO fund_holdings (ticker, rank, name, pct, value_usd) \ VALUES (?, ?, ?, ?, ?)", ) .bind(ticker) .bind(i as i64 + 1) .bind(&h.name) .bind(h.pct) .bind(h.value_usd) .execute(&mut *tx) .await?; } tx.commit().await?; Ok(())}/// Record a physical-commodity grantor trust's profile: just the AUM read from/// its 10-K. It holds bullion, not a securities portfolio, so there are no/// holdings and no asset mix.async fn store_fund_commodity( pool: &SqlitePool, ticker: &str, aum: Option<f64>,) -> sqlx::Result<()> { sqlx::query( "INSERT INTO fund_profiles \ (ticker, kind, net_assets, total_assets, holdings_count, report_date, \ asset_mix, updated_at) \ VALUES (?, 'commodity_trust', ?, NULL, NULL, NULL, NULL, ?) \ ON CONFLICT(ticker) DO UPDATE SET \ kind = excluded.kind, net_assets = excluded.net_assets, \ updated_at = excluded.updated_at", ) .bind(ticker) .bind(aum) .bind(now_ms()) .execute(pool) .await?; Ok(())}/// Stamp an ETF's fund profile as freshly synced.async fn mark_fund_synced(pool: &SqlitePool, ticker: &str) -> sqlx::Result<()> { let now = now_ms(); sqlx::query("UPDATE symbols SET fund_synced_at = ?, updated_at = ? WHERE ticker = ?") .bind(now) .bind(now) .bind(ticker) .execute(pool) .await?; Ok(())}/// Upsert one symbol's live quote into `quotes` and refresh the denormalized/// snapshot columns on `symbols` that the dashboard and SSE seeding read./// `pub(crate)`: the add-symbol route stores the quote its Yahoo lookup
modified
templates/pages/symbol.html
@@ -188,7 +188,7 @@ </section> {% endif %} {# --- fundamentals + filings: stocks only --- #} {# --- fundamentals + financials: stocks only --- #} {% if symbol.kind == 'stock' %} <h2 class="section-title">Fundamentals</h2>
@@ -228,6 +228,77 @@ </section> {% endif %} {% endif %} {# --- fund profile: ETFs only (Phase 18) --- #} {% if symbol.kind == 'etf' %} <h2 class="section-title">Fund profile</h2> {% if fund %} <section class="panel fund"> <dl class="fund__stats"> <div class="fund__stat"> <dt class="fund__stat-cap">Net assets</dt> <dd class="fund__stat-val num">{{ fund.net_assets if fund.net_assets else '·' }}</dd> </div> {% if not fund.is_commodity %} <div class="fund__stat"> <dt class="fund__stat-cap">Holdings</dt> <dd class="fund__stat-val num">{{ fund.holdings_count if fund.holdings_count is not none else '·' }}</dd> </div> {% endif %} {% if fund.report_date %} <div class="fund__stat"> <dt class="fund__stat-cap">As of</dt> <dd class="fund__stat-val num">{{ fund.report_date }}</dd> </div> {% endif %} </dl> {% if fund.is_commodity %} <p class="fund__note">{{ symbol.ticker }} is a grantor trust: it holds the physical commodity directly rather than a portfolio of securities, so it reports no holdings.</p> {% elif fund.asset_mix %} <div class="fund__mix"> <div class="mixbar"> {% for s in fund.asset_mix %} <span class="mixbar__seg" style="width:{{ s.width }}%" title="{{ s.label }} {{ s.pct }}"></span> {% endfor %} </div> <dl class="mixlegend"> {% for s in fund.asset_mix %} <div class="mixlegend__item"> <dt><i class="mixlegend__dot"></i>{{ s.label }}</dt> <dd class="num">{{ s.pct }}</dd> </div> {% endfor %} </dl> </div> {% endif %} </section> {% if fund.holdings %} <h2 class="section-title">Top holdings</h2> <section class="panel holdings"> <ol class="hold-list"> {% for h in fund.holdings %} <li class="hold"> <span class="hold__fill" style="width:{{ h.bar_pct }}%"></span> <span class="hold__rank num">{{ h.rank }}</span> <span class="hold__name">{{ h.name }}</span> <span class="hold__wt num">{{ h.weight }}</span> <span class="hold__val num">{{ h.value }}</span> </li> {% endfor %} </ol> </section> {% endif %} {% else %} <div class="fund-pending"> The fund profile for {{ symbol.ticker }} has not synced yet. It lands on the next SEC data refresh; see <a href="/health">data health</a>. </div> {% endif %} {% endif %} {# --- SEC filings: stocks and ETFs --- #} {% if filings %} <h2 class="section-title">Recent SEC filings</h2> <section class="panel filings">
@@ -249,8 +320,6 @@ </ul> </section> {% endif %} {% endif %}</div>{% endblock %}