@@ -52,7 +52,7 @@ There are no tests or linters configured.finance/├── Cargo.toml, Cargo.lock # rust deps├── Makefile, README.md, PLAN.md # top-level (PLAN.md is the living design doc)├── migrations/ # sqlx migrations 0001-0005, applied on boot├── migrations/ # sqlx migrations 0001-0006, applied on boot├── universe/starter.csv # curated seed list (~150 symbols)├── src/│ ├── main.rs # entry: env init, `seed` subcommand, server boot
@@ -84,7 +84,7 @@ The binary reads `templates/`, `dist/`, `migrations/`, and `universe/` from cwd## Key Routes- `/`: home dashboard — index/commodity sparkline cards over the day's top movers- `/s/{ticker}`: symbol page — candlestick chart with indicators, key stats; a stock also shows fundamentals, an ETF a fund profile (holdings, AUM), both show SEC filings- `/s/{ticker}`: symbol page — candlestick chart with indicators, key stats; a stock also shows fundamentals and a leadership roster, an ETF a fund profile (holdings, AUM), both show SEC filings- `/api/symbols/{ticker}/history`: candle + indicator series JSON for the chart- `/api/symbols` (POST): add a symbol not yet in the universe (validated against Yahoo)- `/search`: browse and search the whole universe (filter by kind, match ticker and company name)
@@ -99,7 +99,7 @@ All free, no account. See `PLAN.md` for the full anti-spam / caching policy.- **Historical daily OHLCV — Stooq.** One call returns a symbol's entire daily history. Gated behind a free apikey (`STOOQ_APIKEY`, in `.env`, gitignored).- **Intraday bars + live quotes — Yahoo Finance.** `v8/finance/chart`; no key, just a browser User-Agent.- **Fundamentals, filings + ETF profiles — SEC EDGAR.** `company_tickers.json` / `companyfacts` / `submissions` for stock fundamentals and filings; `company_tickers_mf.json` plus quarterly N-PORT filings for ETF fund profiles (holdings, AUM, asset mix). No key; a contact email (`SEC_CONTACT_EMAIL`) rides in the User-Agent. Indexes do not file with the SEC.- **Fundamentals, filings, leadership + ETF profiles — SEC EDGAR.** `company_tickers.json` / `companyfacts` / `submissions` for stock fundamentals and filings; Form 3/4/5 ownership XML for the officer/board roster (Phase 14); `company_tickers_mf.json` plus quarterly N-PORT filings for ETF fund profiles (holdings, AUM, asset mix). No key; a contact email (`SEC_CONTACT_EMAIL`) rides in the User-Agent. Indexes do not file with the SEC.## Tooling
@@ -32,11 +32,17 @@ and resume cleanly from this file alone, keeping token use low._Last updated: 2026-05-22_**Current phase:** Phases 0 through 12 (the MVP) plus Phase 18 (ETF profiles)and Phase 20 (strongest & weakest home panels) are complete, verified, and**live in production at https://finance.bythewood.me** (Phase 20 deployed2026-05-22). No phase is in progress. Remaining post-MVP work: theloose-ordered Phase 13-17 and 19 backlog.**Current phase: none in progress. Phase 14 (Company leadership) is completeand verified, not yet deployed.** Phases 0 through 12 (the MVP) plus Phase 18(ETF profiles) and Phase 20 (strongest & weakest home panels) are complete,verified, and **live in production at https://finance.bythewood.me**. Phase 14— a current officers-and-board roster from SEC Form 3/4/5 ownership XML plus aleadership-changes feed from 8-K item 5.02 — was built and verified on2026-05-22 (see the Done list and the decisions log) and ships on the next`git push server master`. Remaining post-MVP work: the loose-ordered Phase 13,15, 16, 17, 19 backlog plus the captured Phase 21 (home & search refinements),Phase 22 (show data age everywhere) and Phase 23 (Q4 in the quarterlyfinancials table).**Roadmap (restructured 2026-05-22, see decisions log):** the home-pageredesign and commodities are pre-ship MVP phases. Order: 9 Search +
@@ -563,12 +569,60 @@ schema, unused for now. ~225ms warm — the per-render standings scan, a fixed page-load snapshot as planned.- **Phase 14 company leadership.** Complete, verified, not yet deployed. - Migration `0006` adds the `leadership` table (one row per insider: director/officer flags, officer title, `last_seen`), `symbols.leadership_synced_at`, and an `items` column on `filings` for the 8-K item codes. - `providers/sec.rs`: two inherent `SecProvider` methods, one HTTP request each so the `sec` guard wraps every call (as with the Phase 18 fund methods) — `ownership_index` (a company's recent Form 3/4/5 filings, from the `submissions` JSON) and `ownership_doc` (one ownership XML, stream-parsed by `parse_ownership` into its reporting people and their `reportingOwnerRelationship` flags). The `submissions` parse also now reads the `items` array, so 8-K item 5.02 is stored on every filing. - The `submissions` feed names a Form 4's primary document as an xsl-styled viewer path (`xslF345X06/form4.xml`) that serves rendered HTML; the raw parseable XML is the bare filename, so `ownership_doc` strips to the basename. Caught in verification — rosters came back empty until the fix. - `scheduler.rs`: `run_sec` gained a 4th section — for each stock whose `leadership_synced_at` is stale (monthly cadence, `LEADERSHIP_STALE_SECS`), it parses up to `LEADERSHIP_MAX_FILINGS` (30) recent ownership filings (only those since the last sync, after the first sweep), filters to directors and officers, and upserts the roster (`store_leadership`, a `last_seen`-guarded conflict so a newer role always wins). Shares the `sec` guard and early-exit; resumable. - `routes/symbols.rs` + `symbol.html`: a Leadership section on the stock symbol page — the current officer/board roster (officers ahead of directors, chiefs first; names title-cased, e.g. `O'BRIEN` -> `O'Brien`) over a provenance note, plus a "Recent leadership changes" list of 8-K item-5.02 filings linking to EDGAR. Stocks only; an unsynced stock shows a pending note, ETFs and indexes show no section. The roster is filtered to insiders seen filing within ~18 months, so departed people age out (ownership filings carry no explicit departure signal). - The "industry insider vs outsider" read was dropped (not in SEC structured data) and there is no per-leader tenure track record — the scope the user chose (see the decisions log). - Verified: migration `0006` applied on the seeded DB; the `sec` job's leadership sweep parsed ownership XML for the curated stocks (AAPL's roster came back as Tim Cook CEO + 7 more officers and 7 directors with titles; 8-K item-5.02 changes detected back through 2024). `/s/AAPL` renders the roster and changes feed in the Paper Ledger look, CEO first; `/s/WMT` (not yet swept) shows the pending note; `/s/QQQ` and `/s/^SPX` show no Leadership section. Desktop (1280px) and phone (390px) render with no overflow and zero console errors. The sweep is paced and guarded — it backfills the universe over several daily `sec` cycles.**Resuming, next action****The MVP plus Phase 18 (ETF profiles) and Phase 20 (strongest & weakest homepanels) are live at https://finance.bythewood.me.** No phase is in progress.Phase 20 was deployed on 2026-05-22 via `git push server master`. Theremaining work is the loose-ordered Phase 13-17 and 19 backlog; the user pickswhich to take next. There is still no GitHub repo for finance: the user**Phase 14 (company leadership) is complete and verified** (2026-05-22), notyet deployed. The MVP plus Phase 18 and Phase 20 are live athttps://finance.bythewood.me; Phase 14 ships to production on the next`git push server master` (migration `0006` applies on the box and the `sec`job backfills the leadership rosters over its daily cycles, as Phase 18'sprofiles backfilled). No phase is in progress. Remaining post-MVP work: theloose-ordered Phase 13, 15, 16, 17, 19 backlog plus the captured Phase 21(home & search refinements), Phase 22 (show data age everywhere) and Phase 23(Q4 in the quarterly financials table); the user picks which to take next. There is still no GitHub repo for finance: the userdeferred that; if one is created later, add it as `origin` and the`overshard/finance` slug already in taproot's `projects.conf` lines up.
@@ -696,7 +750,10 @@ Timestamps are UTC epoch-ms; trading dates are `TEXT` `YYYY-MM-DD`.- `quotes` — latest live quote snapshot, one row per symbol.- `fundamentals` — long/narrow SEC XBRL facts, one row per metric/period; UNIQUE rekeyed to (ticker, metric, period) by migration `0004`.- `filings` — SEC filing history.- `filings` — SEC filing history; `items` (8-K item codes, e.g. `5.02`) added by migration `0006`.- `leadership` — a company's current officers and board, one row per insider, parsed from SEC Form 3/4/5 ownership XML (migration `0006`). Stocks only.- `watchlists` + `watchlist_items` — named lists of tickers. In the schema but unused: the watchlist feature is deferred to Phase 19 (see Status).- `fetch_log` — append-only history of background fetches.
@@ -805,13 +862,24 @@ depend on Phase 5 (live quotes) and Phase 7 (SEC data). (a treemap). Market cap is shares outstanding (from SEC, Phase 7) times the latest price. (The movers list and index sparklines once bundled here moved into the Phase 11 home redesign when it was promoted.)- [ ] **Phase 14: Company leadership.** Track each company's current executives and board, leadership changes over time, and a per-leader track record: how the company fared during their tenure, and whether they are an industry insider or an outsider. Data source needs research: SEC DEF 14A proxy statements and 8-K item 5.02 (officer / director changes) give the roster and the changes, but an objective "track record" is partly editorial and needs a deliberate sourcing decision.- [x] **Phase 14: Company leadership.** Complete and verified 2026-05-22 (not yet deployed) — see the Phase 14 entry in Status and the decisions log. (Picked as the next backlog phase and scoped 2026-05-22.) Two things on the symbol page, stocks only: a **current roster** of officers and board (name + title), built from SEC Form 3/4/5 ownership XML — each form carries a structured `reportingOwnerRelationship` (`isDirector` / `isOfficer` / `officerTitle`), so directors and Section-16 officers are identified and the >10%-owner filers filtered out; and a **leadership-changes feed**, a dated list of 8-K **item 5.02** filings (officer / director departures and appointments), read from the `items` array of the `submissions` JSON Phase 7 already fetches. The "industry insider vs outsider" read is **dropped** — it is not in SEC structured data (only DEF 14A prose), the same wall Phase 18 hit with expense ratios. **No per-leader tenure track record** (the user picked the roster-plus-changes scope over the fuller tenure-record one). Builds only on the SEC source already shipped; the ownership-XML sweep is network-heavier than Phase 7 (one request per filing) but paced and budgeted by the existing `sec` `EndpointGuard`.- [ ] **Phase 15: Industry trends.** Treat industries as first-class: aggregate symbols by industry (sector and industry come from SEC in Phase 7), show industry-level performance, seasonality (the months an industry
@@ -871,6 +939,48 @@ depend on Phase 5 (live quotes) and Phase 7 (SEC data). later layers leadership (Phase 14) and industry context (Phase 15) on top. Built in the Paper Ledger system.- [ ] **Phase 21: Home & search refinements.** (Captured 2026-05-22 from three vibe-coding "side notes" while Phase 14 was being scoped; budgeted here, not acted on mid-phase.) Three independent tweaks to already-shipped features: (1) **Home page — split commodities from indexes** into their own section, and during the pre-market and post-market sessions show the index *futures* in place of the cash indexes (e.g. the S&P 500 E-mini `ES=F` instead of `^SPX`), the way the commodity futures are already shown. Needs a small design pass: the index->future mapping (^SPX->ES=F, ^NDX->NQ=F, ^DJI->YM=F; ^RUT->RTY=F; ^NDQ and ^VIX have no clean index future — decide their treatment) and which sessions count as "show the future" (pre + post; decide the overnight Closed window). (2) **Add-symbol — pull all data immediately.** Today `POST /api/symbols` stores the lookup quote and brings the history job forward a tick; the user wants the full backfill (history + whatever else) pulled synchronously on add, not deferred to the next scheduler cycle. (3) **Search — auto-navigate on a single result.** When `/search?q=` matches exactly one symbol, redirect straight to that symbol's page instead of rendering a one-card result the user must then click.- [ ] **Phase 22: Show data age everywhere.** (Captured 2026-05-22 from a vibe-coding side note.) The user considers data freshness critically important and wants the age of displayed data surfaced consistently across the whole app, the home page included — not only on `/health`. Today freshness shows unevenly: the symbol header has a session-derived label and a last-close date, the ETF profile an "as of" date, `/health` the per-job last-ok times — but the home dashboard's sparkline cards and movers, the search cards, the new leadership roster and the fundamentals carry no visible age. The phase: a consistent, quiet "as of / N ago" treatment (quote time, daily-close date, last sync) wherever data is shown. Needs a small design pass on the wording and where it rides without cluttering the Paper Ledger look.- [ ] **Phase 23: Q4 in the quarterly financials table.** (Captured 2026-05-22 from a vibe-coding side note.) The symbol page's quarterly financials table shows only Q1-Q3. The cause is confirmed: SEC XBRL carries no discrete Q4 — there is no Q4 10-Q, the fourth quarter is reported only inside the 10-K's full-year `FY` figure (zero `fiscal_qtr = 4` rows exist in `fundamentals`). The fix is to derive it: Q4 = FY - (Q1 + Q2 + Q3) for the flow metrics shown in that table (revenue, net income, diluted EPS, dividend/share), computed in `routes/symbols.rs` for each fiscal year where the full-year figure and all three quarters are present. No schema change and no new data — a pure derivation from facts already stored.---## Key files
@@ -880,6 +990,7 @@ finance/ Cargo.toml Makefile migrations/ 0001_initial.sql 0002_endpoint_guard.sql 0003_guard_budget.sql 0004_fundamentals_unique.sql 0005_fund_profiles.sql 0006_leadership.sql universe/starter.csv curated seed list src/ main.rs entry + `seed` subcommand
@@ -1302,6 +1413,70 @@ finance/ the movers and the new panels, a fixed page-load snapshot as planned; `/` renders in ~225ms warm. No new data source and no new network calls. Deployed to production on 2026-05-22 via `git push server master`.- **2026-05-22: Phase 14 picked next and scoped.** Asked which loose-ordered backlog phase to take next, the user chose 14, company leadership. The plan had flagged it as needing a data-source decision; research settled it. SEC exposes two things cleanly and structured: a company's officer/board **roster** via Form 3/4/5 ownership XML (each carries `reportingOwnerRelationship` booleans + `officerTitle`), and a **leadership-changes** signal via 8-K **item 5.02**, readable from the `items` array of the `submissions` JSON Phase 7 already fetches (zero new network calls for the changes feed itself). The "industry insider vs outsider" read is not in SEC structured data — only DEF 14A prose — so it was dropped, the same call Phase 18 made on the expense ratio; the user confirmed drop over hand-curation. Scope chosen: the current roster + the changes feed, **not** the fuller variant that adds a per-leader return-during-tenure track record. Phase 14 builds only on the SEC source already shipped; its ownership-XML sweep is heavier than Phase 7 (one request per filing) but paced and budgeted by the existing `sec` `EndpointGuard`.- **2026-05-22: three home/search side notes captured as Phase 21.** While Phase 14 was being scoped the user floated three refinements to shipped features: split the home page's commodities out from the indexes and show index *futures* (the S&P E-mini, etc.) during pre/post-market; make the add-symbol flow pull a new ticker's full data immediately instead of deferring the backfill to the next scheduler tick; and auto-navigate to a symbol's page when a search yields exactly one result. Per the vibe-coding process they were budgeted into the plan rather than acted on mid-phase — added as Phase 21 (Home & search refinements), to be taken up after Phase 14. The home-page change carries an open design question (the index->future mapping and which sessions trigger it), noted in the Phase 21 entry.- **2026-05-22: Phase 14 company leadership shipped.** SEC Form 3/4/5 ownership XML is now a fifth use of the EDGAR source, behind two new inherent `SecProvider` methods (`ownership_index` / `ownership_doc`, one HTTP request each so the `sec` guard wraps every call, as with the Phase 18 fund methods). Design calls made during the build: (1) the roster is built incrementally — the first sweep parses the 30 most recent ownership filings, later monthly sweeps only the filings since, and `store_leadership` upserts with a `last_seen`-guarded conflict clause so a stale re-parse never overwrites a newer role; the symbol page filters the roster to insiders seen within ~18 months so departed people age out (ownership filings carry no explicit departure signal). (2) The leadership sweep runs on its own monthly cadence (`LEADERSHIP_STALE_SECS`), slower than the weekly fundamentals / filings sweep, because leadership changes slowly and the ownership-XML sweep is the heaviest SEC work — kept comfortably within the guard's pacing and budget. (3) A real bug surfaced in verification: the `submissions` feed names a Form 4's primary document as an xsl viewer path (`xslF345X06/form4.xml`) that serves rendered HTML; the raw parseable XML is the bare filename, so `ownership_doc` strips to the basename. (4) The 8-K item-5.02 changes feed reuses the `filings` table — a new `items` column (migration `0006`) populated from the `submissions` `items` array — so it cost no new request. Names are stored as filed (last-name-first, caps) and title-cased for display. Not yet deployed; ships on the next push.- **2026-05-22: two more side notes captured.** (1) The user wants data freshness — the age of displayed data — shown consistently across the whole app, the home page included, since it is critically important; budgeted as Phase 22. (2) The SEC User-Agent contact email should be the real `isaac@bythewood.me` on both local and the server, set via `.env`: the local `.env` already carries it, so no change there; the server's hand-written `.env` should be checked to match (a manual step — `.env` is not in the repo).- **2026-05-22: Q4-in-quarterly-financials side note captured as Phase 23.** The user noticed the symbol page's quarterly financials table stops at Q3. Confirmed the cause: SEC XBRL has no discrete Q4 (no Q4 10-Q; the quarter lives only in the 10-K's full-year figure — zero `fiscal_qtr = 4` rows in `fundamentals`). Budgeted as Phase 23 — derive Q4 as FY - (Q1+Q2+Q3) for the flow metrics.---
@@ -1326,4 +1501,8 @@ finance/- `/` carries a "Strongest & weakest" pair of panels below the movers, and a strong / fair / weak standing badge rides on the movers, the search result cards, and above the symbol page's ratio cards (Phase 20).- A stock page (`/s/AAPL`) shows a Leadership section: the current officer and board roster from SEC Form 3/4/5 ownership filings, plus a recent-changes list from 8-K item 5.02. An unsynced stock shows a pending note; ETFs and indexes show no Leadership section (Phase 14).- No automated test suite or linter, matching the sibling projects.
modified
frontend/static_src/symbol/styles/symbol.scss
@@ -679,6 +679,108 @@ color: var(--ink-faint);}/* ---------- leadership: roster + change feed ---------- */.leadership { padding: 14px 16px;}.roster { margin: 0; padding: 0; list-style: none;}.roster__row { display: flex; flex-wrap: wrap; align-items: baseline; justify-content: space-between; gap: 2px 16px; padding: 9px 0; border-top: 1px solid var(--rule);}.roster__row:first-child { border-top: none;}.roster__name { font-size: 0.9rem; font-weight: 600;}.roster__role { font-size: 0.8rem; color: var(--ink-dim); text-align: right;}/* the provenance note, set off below a hairline like .ratio__explain */.roster__src { margin-top: 12px; padding-top: 11px; border-top: 1px solid var(--rule); font-size: 0.74rem; color: var(--ink-faint);}/* recent officer/director changes, from 8-K item 5.02 */.lead-changes { margin-top: 16px; padding-top: 14px; border-top: 1px solid var(--rule-strong);}.lead-changes__title { @include eyebrow; color: var(--ink); margin-bottom: 2px;}.lead-changes__list { margin: 0; padding: 0; list-style: none;}.lead-change { border-top: 1px solid var(--rule);}.lead-change:first-child { border-top: none;}.lead-change__link { display: flex; align-items: center; gap: 12px; padding: 11px 0; color: var(--ink);}.lead-change__body { flex: 1; min-width: 0; display: flex; flex-direction: column; gap: 1px;}.lead-change__desc { font-size: 0.88rem; font-weight: 500;}.lead-change__link:hover .lead-change__desc { color: var(--ink-dim);}.lead-change__meta { font-size: 0.74rem; color: var(--ink-faint);}/* ---------- ETF fund profile ---------- */.fund { padding: 16px 18px;
added
migrations/0006_leadership.sql
@@ -0,0 +1,34 @@-- finance migration 0006: company leadership (Phase 14).---- A company's officers and board, parsed from SEC Form 3/4/5 ownership XML-- (each filing carries a structured reportingOwnerRelationship), plus the-- 8-K item-5.02 leadership-change events surfaced from the filing history.-- Stocks only: ETFs and indexes have no officers or board to track.-- When this stock's leadership roster was last refreshed from SEC. NULL = never.ALTER TABLE symbols ADD COLUMN leadership_synced_at INTEGER;-- The 8-K item codes a filing reported, comma-separated as EDGAR's submissions-- feed lists them (e.g. '5.02,9.01'). Set for 8-K rows; NULL for other forms-- and for filing rows stored before this migration (refilled on the next SEC-- sync). Item 5.02 is the officer/director departure-and-appointment event,-- which the symbol page reads as the leadership-changes feed.ALTER TABLE filings ADD COLUMN items TEXT;-- One row per current insider of a company: its directors and Section-16-- officers, identified by the reportingOwnerRelationship booleans on their-- ownership filings. Filers who are only >10% beneficial owners (institutions)-- are not stored — they are not leadership. Upserted incrementally as new-- ownership filings are parsed; `last_seen` is the most recent ownership-- filing date observed for the person, used to order the roster and to age out-- insiders who have gone quiet (a rough proxy for a departure, since ownership-- filings carry no clean "left the company" signal).CREATE TABLE leadership ( ticker TEXT NOT NULL REFERENCES symbols(ticker) ON DELETE CASCADE, name TEXT NOT NULL, -- as filed: last-name-first, upper-case is_director INTEGER NOT NULL DEFAULT 0, is_officer INTEGER NOT NULL DEFAULT 0, officer_title TEXT, -- when is_officer, e.g. 'Chief Executive Officer' last_seen TEXT NOT NULL, -- most recent ownership filing date, YYYY-MM-DD PRIMARY KEY (ticker, name));
@@ -29,6 +29,8 @@ pub struct SymbolRow { pub filings_synced_at: Option<i64>, /// When this ETF's fund profile was last refreshed from SEC. pub fund_synced_at: Option<i64>, /// When this stock's leadership roster was last refreshed from SEC. pub leadership_synced_at: Option<i64>, pub last_price: Option<f64>, pub prev_close: Option<f64>, pub last_quote_at: Option<i64>,
modified
src/providers/mod.rs
@@ -115,6 +115,10 @@ pub struct FilingRecord { /// Full URL to the filing's primary document (or index) on EDGAR. pub url: String, pub description: Option<String>, /// For an 8-K, the reported item codes, comma-separated as EDGAR lists them /// (e.g. `5.02,9.01`); `None` for other forms. Item 5.02 is the /// officer/director change the leadership-changes feed keys on (Phase 14). pub items: Option<String>,}/// Company fundamentals and filing history from SEC EDGAR. Implemented by
@@ -135,6 +139,36 @@ pub trait FundamentalsProvider: Send + Sync { async fn filings(&self, cik: &str) -> Result<Vec<FilingRecord>>;}// ── company leadership (Phase 14) ──────────────────────────────────────────//// A company's officers and board come from SEC Form 3/4/5 ownership filings:// every director and Section-16 officer must file these, and each carries a// structured `reportingOwnerRelationship`. Like the N-PORT fund methods, the// leadership methods are inherent to `SecProvider` (this is wholly EDGAR// territory), but their data types sit here beside `FilingRecord`./// One Form 3/4/5 ownership filing in a company's submission history — the/// pointer the scheduler needs to fetch and parse the ownership XML itself.#[derive(Debug, Clone)]pub struct OwnershipFiling { pub accession: String, /// Filing date, `YYYY-MM-DD`. pub filed_at: String, /// The ownership XML's file name within the filing's Archives directory. pub primary_doc: String,}/// One insider parsed from an ownership XML's `reportingOwnerRelationship`.#[derive(Debug, Clone)]pub struct OwnershipPerson { /// Name as filed: last-name-first and upper-case (`COOK TIMOTHY D`). pub name: String, pub is_director: bool, pub is_officer: bool, /// The officer title, present when `is_officer` and the filer gave one. pub officer_title: Option<String>,}// ── ETF fund profiles (Phase 18) ───────────────────────────────────────────//// ETFs file as registered funds: their portfolio comes from quarterly N-PORT
modified
src/providers/sec.rs
@@ -31,7 +31,7 @@ use quick_xml::Reader;use crate::providers::{ Fact, FilingRecord, FundFilings, FundHolding, FundId, FundShape, FundamentalsProvider, PortfolioData, RateLimited, OwnershipFiling, OwnershipPerson, PortfolioData, RateLimited,};/// Fundamentals and filings from SEC EDGAR.
@@ -321,6 +321,9 @@ struct RecentFilings { primary_document: Vec<String>, #[serde(default)] primary_doc_description: Vec<String>, /// 8-K item codes, one comma-separated string per filing (empty otherwise). #[serde(default)] items: Vec<String>,}/// The filing forms worth showing a market-watcher: periodic reports (10-K,
@@ -472,6 +475,7 @@ impl FundamentalsProvider for SecProvider { .get(i) .filter(|s| !s.is_empty()) .cloned(), items: r.items.get(i).filter(|s| !s.is_empty()).cloned(), }); if out.len() >= MAX_FILINGS { break;
@@ -633,6 +637,7 @@ impl SecProvider { primary_doc: None, url: e.href, description: None, items: None, }) .collect();
@@ -894,3 +899,161 @@ fn parse_nport(xml: &[u8]) -> Result<PortfolioData> { Ok(out)}// ── company leadership: officers & board from Form 3/4/5 (Phase 14) ────────//// Every director and Section-16 officer of a company must file Form 3 (on// becoming an insider) and Form 4 (on each trade); each ownership XML carries// a structured `reportingOwnerRelationship`. The roster is built by parsing a// window of a company's recent ownership filings. Like the fund methods above,// each of these makes exactly one HTTP request so the scheduler wraps every// call in the endpoint guard./// Whether `form` is an ownership form — Form 3, 4 or 5, or an amendment.fn is_ownership_form(form: &str) -> bool { matches!(form.trim_end_matches("/A"), "3" | "4" | "5")}impl SecProvider { /// A company's recent Form 3/4/5 ownership filings, newest first, read from /// the same `submissions` JSON that backs `filings`. One request. Only /// filings whose primary document is an `.xml` are returned: `ownership_doc` /// parses that XML, and the rare legacy filing with a non-XML primary /// document has nothing to parse. pub async fn ownership_index(&self, cik: &str) -> Result<Vec<OwnershipFiling>> { let url = format!("https://data.sec.gov/submissions/CIK{cik}.json"); let Some(resp) = self.get(&url).await? else { return Ok(Vec::new()); // 404: no submission history }; let body: Submissions = resp.json().await?; let r = body.filings.recent; let mut out = Vec::new(); for i in 0..r.accession_number.len() { let form = r.form.get(i).cloned().unwrap_or_default(); if !is_ownership_form(&form) { continue; } let accession = r.accession_number[i].clone(); let filed_at = r.filing_date.get(i).cloned().unwrap_or_default(); let primary_doc = r.primary_document.get(i).cloned().unwrap_or_default(); if accession.is_empty() || filed_at.is_empty() || !primary_doc.ends_with(".xml") { continue; } out.push(OwnershipFiling { accession, filed_at, primary_doc, }); } Ok(out) } /// Parse one ownership filing's XML into its reporting people. One request. /// A filing names more than one reporting person only for the rare joint /// filing; usually the vec holds one. `Ok(Vec::new())` for a 404 or an /// unparseable document — neither is a circuit-breaker failure. pub async fn ownership_doc( &self, cik: &str, accession: &str, primary_doc: &str, ) -> Result<Vec<OwnershipPerson>> { // EDGAR pads the CIK to 10 digits; the Archives path uses it unpadded. let cik_int = cik.trim_start_matches('0'); let nodash = accession.replace('-', ""); // The submissions feed sometimes names the primary document as an // xsl-styled viewer path (`xslF345X05/foo.xml`); the raw XML this parses // sits at the bare filename in the accession directory. let doc = primary_doc.rsplit('/').next().unwrap_or(primary_doc); let url = format!("https://www.sec.gov/Archives/edgar/data/{cik_int}/{nodash}/{doc}"); let Some(resp) = self.get(&url).await? else { return Ok(Vec::new()); // 404 }; let bytes = resp.bytes().await?; Ok(parse_ownership(&bytes)) }}/// Read an ownership-XML boolean. SEC writes these as `1`/`0`, very/// occasionally `true`.fn ownership_flag(txt: &str) -> bool { matches!(txt.trim(), "1" | "true" | "TRUE" | "Y")}/// One reporting person accumulated while streaming an ownership XML.#[derive(Default)]struct OwnerAcc { name: String, is_director: bool, is_officer: bool, officer_title: String,}/// Stream-parse an ownership Form 3/4/5 XML into its reporting people. Walks/// the `<reportingOwner>` blocks, reading each owner's name and the/// `reportingOwnerRelationship` flags. A malformed document yields whatever was/// parsed before the error rather than failing the whole leadership sweep.fn parse_ownership(xml: &[u8]) -> Vec<OwnershipPerson> { let mut reader = Reader::from_reader(xml); let mut buf = Vec::new(); // Stack of open element local names, so a leaf is read in context. let mut path: Vec<Vec<u8>> = Vec::new(); let mut owners: Vec<OwnerAcc> = Vec::new(); // The reporting person currently being assembled, set inside a // `<reportingOwner>` block. let mut cur: Option<OwnerAcc> = None; loop { match reader.read_event_into(&mut buf) { Ok(Event::Start(e)) => { let name = e.local_name().as_ref().to_vec(); if name == b"reportingOwner" { cur = Some(OwnerAcc::default()); } path.push(name); } Ok(Event::End(e)) => { if e.local_name().as_ref() == b"reportingOwner" { if let Some(o) = cur.take() { owners.push(o); } } path.pop(); } Ok(Event::Text(t)) => { let (Some(tag), Some(o)) = (path.last(), cur.as_mut()) else { continue; }; let raw = t.unescape().unwrap_or_default(); let txt = raw.trim(); if txt.is_empty() { continue; } match tag.as_slice() { b"rptOwnerName" if o.name.is_empty() => o.name = txt.to_string(), b"isDirector" => o.is_director |= ownership_flag(txt), b"isOfficer" => o.is_officer |= ownership_flag(txt), b"officerTitle" if o.officer_title.is_empty() => { o.officer_title = txt.to_string() } _ => {} } } Ok(Event::Eof) | Err(_) => break, _ => {} } buf.clear(); } owners .into_iter() .filter(|o| !o.name.is_empty()) .map(|o| OwnershipPerson { name: o.name, is_director: o.is_director, is_officer: o.is_officer, officer_title: (!o.officer_title.is_empty()).then_some(o.officer_title), }) .collect()}
modified
src/routes/symbols.rs
@@ -420,6 +420,161 @@ fn build_fund(profile: FundProfileRow, holdings: Vec<HoldingRow>) -> FundView { }}// ── company leadership (Phase 14) ──────────────────────────────────────────/// A `leadership` row as stored.#[derive(sqlx::FromRow)]struct LeadershipRow { name: String, is_director: i64, is_officer: i64, officer_title: Option<String>,}/// One person on the leadership roster, shaped for the page.#[derive(Serialize)]struct LeaderView { /// Display name, title-cased from the as-filed upper-case form. name: String, /// Role line, e.g. `Chief Executive Officer · Director`. role: String, /// Sort key only: officers ahead of directors, chiefs first. `serde(skip)` /// keeps it out of the template context. #[serde(skip)] rank: u8,}/// One 8-K item-5.02 leadership-change event, shaped for the page.#[derive(Serialize)]struct ChangeView { filed_at: String, url: String,}/// Everything the symbol page's Leadership section needs.#[derive(Serialize)]struct LeadershipView { /// Whether the SEC leadership sweep has reached this stock yet — picks the /// "not synced yet" pending note apart from a genuine empty roster. synced: bool, roster: Vec<LeaderView>, /// Recent officer/director changes, newest first. changes: Vec<ChangeView>,}/// Title-case a name filed in SEC's upper-case form: `COOK TIMOTHY D` ->/// `Cook Timothy D`. The first letter of each word, and of each part after an/// apostrophe or hyphen, is capitalized — so `O'BRIEN` reads `O'Brien`. The/// order is left as filed (last name first); reordering it is unreliable for/// compound surnames and generational suffixes.fn title_case(s: &str) -> String { let mut out = String::with_capacity(s.len()); let mut cap_next = true; for ch in s.chars() { if ch.is_whitespace() || ch == '\'' || ch == '-' { out.push(ch); cap_next = true; } else if cap_next { out.extend(ch.to_uppercase()); cap_next = false; } else { out.extend(ch.to_lowercase()); } } out}/// Sort rank for the roster: officers ahead of directors, with the chief/// executive / financial / operating officers ahead of the other officers./// Both the spelled-out titles and the abbreviations are matched, since filers/// use either (`Chief Executive Officer` or `CEO and Chairman`).fn role_rank(is_director: bool, is_officer: bool, title: Option<&str>) -> u8 { let t = title.unwrap_or("").to_lowercase(); let has = |needles: &[&str]| needles.iter().any(|n| t.contains(n)); if has(&["chief executive", "ceo"]) { 0 } else if has(&["chief financial", "cfo"]) { 1 } else if has(&["chief operating", "coo"]) { 2 } else if is_officer { 3 } else if is_director { 4 } else { 5 }}/// The role line for a roster row: the officer title (when the filer gave one)/// and `Director`, joined — e.g. `Chief Financial Officer · Director`.fn role_text(is_director: bool, is_officer: bool, title: Option<&str>) -> String { let mut parts: Vec<String> = Vec::new(); match title.map(str::trim).filter(|t| !t.is_empty()) { Some(t) => parts.push(t.to_string()), None if is_officer => parts.push("Officer".to_string()), None => {} } if is_director { parts.push("Director".to_string()); } if parts.is_empty() { parts.push("Insider".to_string()); } parts.join(" \u{00b7} ")}/// Load the Leadership section for a stock: the current officer/board roster/// and the recent 8-K item-5.02 change events. The roster is filtered to/// insiders seen filing within the recency window, so people who left long ago/// drop off (ownership filings carry no explicit departure signal).async fn build_leadership(pool: &sqlx::SqlitePool, ticker: &str, synced: bool) -> LeadershipView { // ~18 months: long enough that an annually-filing director still shows, // short enough that a departed insider ages out. let cutoff = (chrono::Utc::now().date_naive() - chrono::Duration::days(550)).to_string(); let rows: Vec<LeadershipRow> = sqlx::query_as( "SELECT name, is_director, is_officer, officer_title FROM leadership \ WHERE ticker = ? AND last_seen >= ?", ) .bind(ticker) .bind(&cutoff) .fetch_all(pool) .await .unwrap_or_default(); let mut roster: Vec<LeaderView> = rows .into_iter() .map(|r| { let (is_dir, is_off) = (r.is_director != 0, r.is_officer != 0); LeaderView { rank: role_rank(is_dir, is_off, r.officer_title.as_deref()), role: role_text(is_dir, is_off, r.officer_title.as_deref()), name: title_case(&r.name), } }) .collect(); roster.sort_by(|a, b| a.rank.cmp(&b.rank).then_with(|| a.name.cmp(&b.name))); let changes: Vec<ChangeView> = sqlx::query_as::<_, (String, String)>( "SELECT filed_at, url FROM filings \ WHERE ticker = ? AND form LIKE '8-K%' AND items LIKE '%5.02%' \ ORDER BY filed_at DESC, accession DESC LIMIT 8", ) .bind(ticker) .fetch_all(pool) .await .unwrap_or_default() .into_iter() .map(|(filed_at, url)| ChangeView { filed_at, url }) .collect(); LeadershipView { synced, roster, changes, }}async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) -> Response { let ticker = ticker.to_uppercase();
@@ -582,6 +737,14 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) None }; // The leadership roster + change feed (Phase 14): stocks only, like the // fundamentals above. let leadership = if is_stock { Some(build_leadership(&state.pool, &ticker, symbol.leadership_synced_at.is_some()).await) } else { None }; let extra = minijinja::context! { title => ticker, symbol => symbol,
@@ -590,6 +753,7 @@ async fn symbol_page(Path(ticker): Path<String>, State(state): State<AppState>) fundamentals => fundamentals, standing => standing, fund => fund, leadership => leadership, filings => filings, }; render(&state, "pages/symbol.html", &format!("/s/{ticker}"), extra)
modified
src/scheduler.rs
@@ -36,7 +36,7 @@ use crate::providers::sec::SecProvider;use crate::providers::yahoo::YahooProvider;use crate::providers::{ self, stooq::StooqProvider, Fact, FilingRecord, FundId, FundShape, FundamentalsProvider, HistoryProvider, IntradayBar, PortfolioData, Quote, QuoteProvider, HistoryProvider, IntradayBar, OwnershipPerson, PortfolioData, Quote, QuoteProvider,};use crate::stream::{Hub, QuoteUpdate, StreamEvent};use crate::{seed, Config};
@@ -80,6 +80,15 @@ const SEC_STALE_SECS: i64 = 7 * 24 * 3600;/// runaway loop well short of anything SEC's fair-access policy would refuse.const SEC_BUDGET: i64 = 600;/// Company-leadership refresh (Phase 14). Leadership changes slowly, so the/// roster is rebuilt monthly rather than on the weekly SEC cadence above. Each/// sweep parses at most this many of a company's most recent ownership filings/// (one HTTP request each): enough to capture the actively-filing officers and/// board on a first sweep, with later sweeps reading only the filings since,/// so the roster fills in and stays current at a small steady-state cost.const LEADERSHIP_STALE_SECS: i64 = 30 * 24 * 3600;const LEADERSHIP_MAX_FILINGS: usize = 30;/// Spawn the scheduler. The returned handle is normally dropped: dropping it/// detaches the task, which then runs for the lifetime of the process.pub fn spawn(pool: SqlitePool, config: Arc<Config>, hub: Arc<Hub>) -> JoinHandle<()> {
@@ -588,6 +597,8 @@ async fn run_daily_close_if_due(/// - an ETF's latest N-PORT into `fund_profiles` + `fund_holdings`, its/// filing history into `filings`. A physical-commodity grantor trust files/// no N-PORT, so its AUM comes from `companyfacts` instead./// - a stock's officer/board roster into `leadership`, parsed from its recent/// Form 3/4/5 ownership filings (Phase 14, on a slower monthly cadence)./// Indexes are skipped; they do not file with the SEC.////// Resumable like the history job: each symbol's sync timestamps are stamped
@@ -690,8 +701,19 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul .bind(cutoff) .fetch_all(pool) .await?; // Leadership has its own, longer staleness window (see LEADERSHIP_STALE_SECS). let leadership_cutoff = started - LEADERSHIP_STALE_SECS * 1000; let stale_leadership: Vec<(String, String, Option<i64>)> = sqlx::query_as( "SELECT ticker, cik, leadership_synced_at FROM symbols \ WHERE kind = 'stock' AND cik IS NOT NULL \ AND (leadership_synced_at IS NULL OR leadership_synced_at < ?) \ ORDER BY ticker", ) .bind(leadership_cutoff) .fetch_all(pool) .await?; if stale_stocks.is_empty() && stale_etfs.is_empty() { if stale_stocks.is_empty() && stale_etfs.is_empty() && stale_leadership.is_empty() { log_fetch(pool, "sec", "sec", "ok", Some("no stale companies or funds"), Some(0), 0, started) .await?; mark_ok(pool, "sec", Some(next)).await?;
@@ -699,9 +721,10 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul return Ok(()); } tracing::info!( "[scheduler] sec: refreshing {} companies, {} funds", "[scheduler] sec: refreshing {} companies, {} funds, {} rosters", stale_stocks.len(), stale_etfs.len() stale_etfs.len(), stale_leadership.len() ); // A metric is due when its timestamp is unset or past the cutoff.
@@ -709,6 +732,7 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul let mut funds_ok = 0i64; let mut filings_ok = 0i64; let mut etfs_ok = 0i64; let mut leaders_ok = 0i64; let mut errors = 0i64; let mut stopped: Option<String> = None;
@@ -835,9 +859,91 @@ async fn run_sec(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Resul } } // 4. Leadership sweep (Phase 14): for each stale stock, parse a window of // its recent Form 3/4/5 ownership filings into the officer/board roster. // Shares the guard and the early-exit; skipped wholesale once the sweeps // above have already hit a guard stop. if stopped.is_none() { 'leaders: for (ticker, cik, lead_at) in &stale_leadership { // The company's recent ownership filings, newest first. let index = match guard.acquire().await? { Permit::Granted => match sec.ownership_index(cik).await { Ok(idx) => { guard.record_success().await?; idx } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!("[scheduler] sec ownership_index {ticker} failed: {e:#}"); continue 'leaders; } }, Permit::Denied(why) => { stopped = Some(why); break 'leaders; } }; // First sweep (no prior sync): the most recent filings. Later // sweeps: only filings since the last sync, with a few days' slack, // so the steady-state cost is a handful of requests per company. let since: Option<String> = lead_at.and_then(|ms| { chrono::DateTime::from_timestamp_millis(ms - 5 * 86_400_000) .map(|dt| dt.format("%Y-%m-%d").to_string()) }); let to_parse: Vec<_> = index .into_iter() .filter(|f| since.as_deref().map_or(true, |s| f.filed_at.as_str() >= s)) .take(LEADERSHIP_MAX_FILINGS) .collect(); // Parse each filing's XML, keeping the directors and officers (a // filer who is only a >10% owner is not leadership and is dropped). let mut roster: Vec<(OwnershipPerson, String)> = Vec::new(); for f in &to_parse { match guard.acquire().await? { Permit::Granted => { match sec.ownership_doc(cik, &f.accession, &f.primary_doc).await { Ok(people) => { guard.record_success().await?; for p in people { if p.is_director || p.is_officer { roster.push((p, f.filed_at.clone())); } } } Err(e) => { guard.record_failure(&e).await?; errors += 1; tracing::warn!( "[scheduler] sec ownership_doc {ticker} failed: {e:#}" ); } } } Permit::Denied(why) => { stopped = Some(why); break; } } } // Upsert what was gathered. A guard stop mid-company still stores // the partial roster (the upsert is idempotent) but leaves // `leadership_synced_at` unset so the next cycle finishes the rest. store_leadership(pool, ticker, &roster).await?; if stopped.is_some() { break 'leaders; } mark_sec_synced(pool, ticker, "leadership_synced_at").await?; leaders_ok += 1; } } let dur = t0.elapsed().as_millis() as i64; let counts = format!( "{funds_ok} fundamentals, {filings_ok} filings, {etfs_ok} fund profiles, {errors} errors" "{funds_ok} fundamentals, {filings_ok} filings, {etfs_ok} fund profiles, \ {leaders_ok} rosters, {errors} errors" ); match stopped { Some(why) => {
@@ -909,8 +1015,8 @@ async fn resolve_fund_ciks( Ok(resolved)}/// Stamp one of a symbol's SEC sync timestamps to now. `column` is one of two/// hardcoded literals (never user input), so interpolating it is safe./// Stamp one of a symbol's SEC sync timestamps to now. `column` is one of a/// few hardcoded literals (never user input), so interpolating it is safe.async fn mark_sec_synced(pool: &SqlitePool, ticker: &str, column: &str) -> sqlx::Result<()> { let now = now_ms(); let sql = format!("UPDATE symbols SET {column} = ?, updated_at = ? WHERE ticker = ?");
@@ -966,13 +1072,13 @@ async fn store_filings( sqlx::query( "INSERT INTO filings \ (ticker, accession, form, filed_at, period_of_report, \ primary_doc, url, description) \ VALUES (?, ?, ?, ?, ?, ?, ?, ?) \ primary_doc, url, description, items) \ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) \ ON CONFLICT(ticker, accession) DO UPDATE SET \ form = excluded.form, filed_at = excluded.filed_at, \ period_of_report = excluded.period_of_report, \ primary_doc = excluded.primary_doc, url = excluded.url, \ description = excluded.description", description = excluded.description, items = excluded.items", ) .bind(ticker) .bind(&f.accession)
@@ -982,6 +1088,46 @@ async fn store_filings( .bind(&f.primary_doc) .bind(&f.url) .bind(&f.description) .bind(&f.items) .execute(&mut *tx) .await?; } tx.commit().await?; Ok(())}/// Upsert a company's leadership roster from parsed ownership filings (Phase/// 14). `roster` arrives newest-filing-first, so the first entry seen for a/// person is their most recent filing and any later duplicate is skipped. The/// upsert is guarded on `last_seen`, so a stale re-parse never overwrites a/// person's role with an older filing's; departed insiders simply stop being/// re-stamped and age out of the symbol page's recency window.async fn store_leadership( pool: &SqlitePool, ticker: &str, roster: &[(OwnershipPerson, String)],) -> sqlx::Result<()> { let mut seen: HashSet<&str> = HashSet::new(); let mut tx = pool.begin().await?; for (person, filed_at) in roster { if !seen.insert(person.name.as_str()) { continue; } sqlx::query( "INSERT INTO leadership \ (ticker, name, is_director, is_officer, officer_title, last_seen) \ VALUES (?, ?, ?, ?, ?, ?) \ ON CONFLICT(ticker, name) DO UPDATE SET \ is_director = excluded.is_director, is_officer = excluded.is_officer, \ officer_title = excluded.officer_title, last_seen = excluded.last_seen \ WHERE excluded.last_seen >= leadership.last_seen", ) .bind(ticker) .bind(&person.name) .bind(person.is_director as i64) .bind(person.is_officer as i64) .bind(&person.officer_title) .bind(filed_at) .execute(&mut *tx) .await?; }
modified
templates/pages/symbol.html
@@ -247,6 +247,49 @@ </section> {% endif %} {# --- leadership: officers, board & recent changes (Phase 14) --- #} <h2 class="section-title">Leadership</h2> {% if leadership and leadership.roster %} <section class="panel leadership"> <ul class="roster"> {% for p in leadership.roster %} <li class="roster__row"> <span class="roster__name">{{ p.name }}</span> <span class="roster__role">{{ p.role }}</span> </li> {% endfor %} </ul> <p class="roster__src">Officers and board are drawn from {{ symbol.ticker }}’s recent SEC ownership filings (Forms 3, 4 and 5).</p> {% if leadership.changes %} <div class="lead-changes"> <h3 class="lead-changes__title">Recent leadership changes</h3> <ul class="lead-changes__list"> {% for c in leadership.changes %} <li class="lead-change"> <a class="lead-change__link" href="{{ c.url }}" target="_blank" rel="noopener noreferrer"> <span class="lead-change__body"> <span class="lead-change__desc">Officer or director change</span> <span class="lead-change__meta">Reported in an 8-K filed {{ c.filed_at }}</span> </span> <svg class="filing__ext" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> <path d="M14 5h5v5M19 5l-9 9M11 5H6a1 1 0 0 0-1 1v12a1 1 0 0 0 1 1h12a1 1 0 0 0 1-1v-5"/> </svg> </a> </li> {% endfor %} </ul> </div> {% endif %} </section> {% elif leadership and leadership.synced %} <div class="fund-pending">No leadership data is available for {{ symbol.ticker }}.</div> {% else %} <div class="fund-pending"> Leadership data for {{ symbol.ticker }} has not synced yet. It lands on the next SEC data refresh; see <a href="/health">data health</a>. </div> {% endif %} {% endif %} {# --- fund profile: ETFs only (Phase 18) --- #}