repos

Phase 2: universe curation (S&P 500 reconcile + iShares/Vanguard ETFs)

b016b25d by Isaac Bythewood · 1 month ago

Phase 2: universe curation (S&P 500 reconcile + iShares/Vanguard ETFs)

Curate the universe to the current S&P 500 plus an iShares/Vanguard ETF set,
and fix a Yahoo history-granularity bug surfaced while verifying it.

Universe:
- Stocks already matched the current S&P 500 exactly (fetch + diff), no change.
- ETFs curated to 43: dropped other-issuer thematic/sector funds (SMH, ARKK,
  SCHD, XLK/XLF/XLE/XLV), kept the SPY/QQQ/DIA/GLD/SLV proxies, added the most
  common Vanguard + iShares funds including the core holdings IAU, IBIT, VBIL.
- seed::sync_universe (upsert + prune) now runs on every boot, so a starter.csv
  edit takes effect on deploy without a manual re-seed. Pruning keys on
  is_seeded=1 and cascades cleanly; user-added symbols are untouched.

Yahoo range=max downsampling fix:
- Yahoo silently returns weekly/monthly bars for interval=1d&range=max on
  long-history symbols (futures, ^RUT/^VIX, multi-year ETFs), and a single bar
  for some. The provider now detects a downsampled response (median timestamp
  gap > 4 days) and refetches a bounded range=10y window served at true daily
  granularity.
- run_history self-heals already-stored coarse or single-bar series: it flags a
  symbol with fewer than 30 bars in its trailing 90 days (recent density, so a
  deep symbol with a sparse pre-modern tail like ^SPX is not misread) and
  replaces them with a clean daily re-fetch. Prod self-heals on deploy.

Verified on dev: 43 ETFs, all 5 core holdings present, every seeded symbol
holds genuine daily bars (no 1-bar, no downsampled, zero false positives across
562 symbols), ^SPX deep daily history preserved, and the run_history heal
confirmed end to end.
modified PLAN.md
@@ -34,7 +34,7 @@ commit + auto-deploy (`git push server master`) and a clean breakpoint.## Status_Last updated: 2026-05-30__Last updated: 2026-05-30 (Phase 2 complete on dev)_**Major refactor in progress (the "distill + ETF-first" rewrite).** This planwas fully rewritten 2026-05-30 from a sprawling 3,700-line resume doc into this
@@ -54,13 +54,39 @@ focused roadmap. The decisions driving it are in the Decisions log under- **Everything gets distilled** into a fast-scannable, dual-first (mobile +  desktop) design while keeping the futuristic-clean "Paper Ledger" look.**Current work:** Phase 1 (Yahoo-only data layer) is **complete, verified, anddeployed to production** 2026-05-30 (commit `76f38f4`). Verified end to end onthe dev box: a fresh add-symbol backfilled DocuSign to 424 daily bars fromYahoo (`range=max`) with correct IPO-dated history (first bar 2018-04-23),`history_first_date`/`last_date` set correctly, charts render, `/api/health`shows only `yahoo` + `sec` (the `stooq` guard row is deleted on boot). Next:**Phase 2 (universe curation)**.**Current work:** Phase 2 (universe curation) is **complete and verified on thedev box**; commit + deploy pending. Next: **Phase 3 (drop short-horizonprediction → quality leaderboard)**.Phase 2 outcome:- **Stocks reconciled to the current S&P 500.** Fetched the live Wikipedia  constituent list (503) and diffed against ours: an exact match, zero changes  needed — the stock list was already current.- **ETF roster curated to iShares + Vanguard (43 ETFs).** Dropped the  other-issuer thematic/sector funds (SMH, ARKK, SCHD, XLK/XLF/XLE/XLV); kept  the SPY/QQQ/DIA/GLD/SLV proxies; added the most-common Vanguard + iShares  funds incl. the core holdings IAU, IBIT, VBIL. Issuer/category tags come from  the existing Yahoo `fund_metadata` (no new schema) per the user's choice.- **Seed now reconciles on every boot.** `seed::sync_universe` (upsert + prune)  runs each boot via `run_boot_seed`, so a CSV edit (symbols added or dropped)  takes effect on deploy without a manual re-seed. Pruning is `is_seeded = 1 AND  ticker NOT IN (csv)`, cascading cleanly; user-added symbols (`is_seeded = 0`)  are never touched. This also swept up 7 stale rows an older non-pruning seed  had left behind (MSTR, NET, RIVN, SHOP, SNOW, SOFI, MRVL — popular names from  a pre-S&P-narrowing universe; none are current S&P 500 members). **If the user  wants those back, that's a separate "popular non-S&P watchlist" decision (see  backlog).**- **Fixed a Phase-1 data-quality bug: Yahoo `range=max` downsampling.** Yahoo  silently returns weekly/monthly bars for `interval=1d&range=max` on symbols  with long histories (all the futures, ^RUT/^VIX, and every freshly-backfilled  multi-year ETF were monthly; some were 1-bar). Two-part fix: the provider  refetches a bounded `range=10y` window (which Yahoo serves at true daily  granularity) when it detects a downsampled response; and `run_history`  self-heals stored coarse/single-bar series (recent-density test) by replacing  them with a clean daily re-fetch — so **prod self-heals on deploy**. Verified:  every seeded symbol now holds genuine daily bars (median gap 1 day), no  symbol is 1-bar, ^SPX's deep daily history (back to 1789) is preserved, and  the density test flags zero false positives across all 562 symbols.**Dev server:** kept running in the background via `make` during sessions sothe user can review progress live.
@@ -186,14 +212,17 @@ Kill the rate-limit problem at the root.- Verify: guard never trips during a full backfill; `daily_prices` populates;  charts render; `/health` shows no stooq endpoint.### Phase 2 — Universe curation (S&P 500 + indexes + commodities + iShares/Vanguard ETFs)- Reconcile the ~503 stocks to the current S&P 500 constituents.- Curate ETFs to iShares + Vanguard families, guaranteeing the user's core  holdings (VTI, VXUS, IAU, IBIT, VBIL, …). Tag each ETF with issuer/family and  a category for the dashboard separation.- Confirm major indexes; decide index-futures (ES/NQ/YM) vs pure commodities  (CL/BZ/GC/SI/HG/NG) — user said "major commodities," lean to commodities.- Re-seed; confirm history backfills cleanly via the Phase 1 Yahoo path.### Phase 2 — Universe curation  ✅ DONE on dev (commit + deploy pending)- ✅ Reconciled the 503 stocks to the current S&P 500 (fetch + diff: exact match).- ✅ Curated ETFs to iShares + Vanguard (43 total): dropped other-issuer  thematic/sector funds, kept SPY/QQQ/DIA/GLD/SLV proxies, added the  most-common Vanguard + iShares funds incl. core holdings IAU/IBIT/VBIL.  Issuer/category tags reuse the Yahoo `fund_metadata` (no new schema).- ✅ Kept the 6 indexes and all 10 futures (4 index-futures + 6 commodities).- ✅ `seed::sync_universe` now reconciles (upsert + prune) on every boot, so CSV  edits take effect on deploy. Pruned 7 stale non-S&P leftovers in passing.- ✅ Fixed Yahoo `range=max` downsampling (provider 10y fallback + `run_history`  self-heal); every seeded symbol now holds true daily bars.### Phase 3 — Drop short-horizon prediction → quality leaderboard + home de-dup- Remove Day/Week picks and the short-horizon backtest machinery.
@@ -233,13 +262,34 @@ Kill the rate-limit problem at the root.### Backlog / parked- Watchlists (tables exist, unused — user wants an opinionated no-customization  view for now).- **Popular non-S&P watchlist.** Phase 2 narrowed stocks to exactly the S&P 500,  which dropped some popular names the universe used to carry (MSTR, NET, RIVN,  SHOP, SNOW, SOFI, MRVL). Add them back as an opt-in "popular / most-watched"  band if the user wants them.- Issuer-direct ETF feeds (iShares/Vanguard) if Yahoo/SEC prove thin.- Deep pre-2000 history (lost with Stooq; revisit only if charts feel thin).  Note: index/futures daily history via Yahoo caps at ~10y (the `range=10y`  fallback); ^SPX/^DJI/^NDX still carry deep daily history from before.---## Decisions log**2026-05-30 — Phase 2 (universe curation).** Answered 4 clarifying questions:1. **ETF roster:** keep iShares + Vanguard + the SPY/QQQ/DIA/GLD/SLV proxies;   drop other-issuer thematic/sector funds; add the most-common Vanguard +   iShares funds. Landed at 43 ETFs.2. **Core holdings:** "get the most common Vanguard and iShares funds" (not just   the named IAU/IBIT/VBIL) — drove the broadened ETF set above.3. **ETF tags:** reuse Yahoo `fund_metadata.category` / `fund_family` for the   dashboard band separation; no new schema/migration.4. **S&P 500 recon:** fetch the live constituent list and diff to match exactly   (result: already an exact match).Plus, surfaced and fixed during verification: the seed never reconciled onre-boot (now does, via `sync_universe`), and Yahoo `range=max` silentlydownsamples `interval=1d` to weekly/monthly for long-history symbols (fixed inthe provider + a `run_history` self-heal). See Status for the full outcome.**2026-05-30 — The "distill + ETF-first" refactor kickoff.** User steered abroad refactor; answered 10 clarifying questions. Decisions:1. **Drop next-day/next-week prediction.** Confirmed it's a gamble and the main
@@ -283,6 +333,21 @@ from this doc to keep it scannable.## Hard-won lessons (don't relearn these)- **Yahoo `range=max` silently downsamples `interval=1d`.** For symbols with  long histories (futures, ^RUT/^VIX, multi-year ETFs) the v8 chart endpoint  returns weekly/monthly bars even when a daily interval is asked for — and for  some it returned a single bar. A *bounded* window (`range=10y`, or explicit  `period1`/`period2`) is served at true daily granularity. The provider detects  a downsampled `range=max` response (median timestamp gap > 4 days) and refetches  10y; `run_history` self-heals already-stored coarse/single-bar series (it flags  a symbol with < 30 bars in its trailing 90 days, replacing them with a daily  re-fetch). Detect coarseness from *recent* spacing, not whole-span density —  ^SPX has daily data for decades but a sparse pre-1900 tail, and a whole-span  test wrongly flags it.- **The seed must reconcile on every boot, not only first-run.** `seed_completed`  gates the history *backfill*, but `sync_universe` (upsert + prune) runs each  boot so a `starter.csv` edit (symbols added or dropped) takes effect on deploy.  Pruning keys on `is_seeded = 1` so user-added symbols are safe.- **Yahoo `quoteSummary` is crumb-gated.** Must prime `fc.yahoo.com` + fetch  `/v1/test/getcrumb` with cookies, cache the crumb, rotate on 401/403. Already  implemented in `src/providers/yahoo.rs`.
modified src/providers/yahoo.rs
@@ -260,7 +260,10 @@ impl YahooProvider {    /// Error / "unknown symbol" semantics match [`Self::fetch_chart`].    async fn fetch_daily(&self, ticker: &str, since: Option<&str>) -> Result<Option<ChartResult>> {        let sym = urlencoding::encode(&yahoo_symbol(ticker)).into_owned();        let window = match since {        let chart_url = |window: &str| {            format!("https://query1.finance.yahoo.com/v8/finance/chart/{sym}?interval=1d&{window}")        };        match since {            Some(d) => {                // Re-fetch from the day before `since` to "now + 1 day" (epoch                // seconds). The day-before back-step and the day-ahead end
@@ -273,14 +276,29 @@ impl YahooProvider {                    .map(|dt| dt.and_utc().timestamp())                    .unwrap_or(0);                let p2 = chrono::Utc::now().timestamp() + 86_400;                format!("period1={p1}&period2={p2}")                self.request_chart(&chart_url(&format!("period1={p1}&period2={p2}")))                    .await            }            None => "range=max".to_string(),        };        let url = format!(            "https://query1.finance.yahoo.com/v8/finance/chart/{sym}?interval=1d&{window}"        );        self.request_chart(&url).await            None => {                // Deep backfill: ask for the full history. Yahoo honours                // `interval=1d` at `range=max` for most symbols (full daily                // history, e.g. ^SPX back to the 1700s), but silently                // downsamples it to monthly / quarterly bars for some index and                // futures symbols (^RUT, ^VIX, the `=F` futures) even though a                // daily interval was requested. When that happens, refetch a                // bounded 10-year window, which Yahoo *does* serve at daily                // granularity — far better for a daily chart than coarse                // max-range bars. (Costs one extra request for those few                // symbols, once per deep backfill.)                let res = self.request_chart(&chart_url("range=max")).await?;                match &res {                    Some(r) if is_downsampled(r) => {                        self.request_chart(&chart_url("range=10y")).await                    }                    _ => Ok(res),                }            }        }    }    /// GET a v8 chart URL and parse it into at most one [`ChartResult`].
@@ -864,6 +882,29 @@ fn chart_to_quote_data(ticker: &str, result: ChartResult) -> Result<QuoteData> {    Ok(QuoteData { quote, bars })}/// True when a supposedly-daily chart response is actually coarser than daily.////// Yahoo silently downsamples `interval=1d&range=max` to monthly / quarterly/// bars for some index and futures symbols (^RUT, ^VIX, the `=F` futures),/// ignoring the requested interval. Detect it from the *median* spacing of the/// returned timestamps: the *median* gap of a genuine daily series is 1 day/// (consecutive trading days; only weekends and holidays stretch it to 3-4),/// while a weekly series medians ~7 days and a monthly one ~30. A median over/// four days therefore means the series is coarser than daily. Fewer than three/// bars carries no usable spacing, so it is treated as fine.fn is_downsampled(result: &ChartResult) -> bool {    let Some(ts) = result.timestamp.as_ref() else {        return false;    };    if ts.len() < 3 {        return false;    }    let mut gaps: Vec<i64> = ts.windows(2).map(|w| w[1] - w[0]).collect();    gaps.sort_unstable();    let median = gaps[gaps.len() / 2];    median > 4 * 86_400}/// Turn a parsed `interval=1d` chart result into daily bars, oldest first.////// Yahoo timestamps each daily bar at the start of the trading day in UTC
modified src/scheduler.rs
@@ -367,6 +367,18 @@ async fn reset_states_on_boot(pool: &SqlitePool) -> sqlx::Result<()> {/// boot, the handful of symbols the seed just touched.async fn run_boot_seed(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> {    if get_meta(pool, "seed_completed").await?.as_deref() == Some("1") {        // The full backfill is done, but still reconcile the curated list each        // boot so a CSV edit (symbols added to or dropped from the universe)        // takes effect on deploy without a manual re-seed. This is local-only        // and cheap; any newly-added symbol's history is backfilled by the        // incremental `run_history` job (it picks up `history_synced_at IS NULL`).        match seed::sync_universe(pool, config).await {            Ok(r) if r.pruned > 0 => {                tracing::info!("[scheduler] universe sync: {} symbols, {} pruned", r.total, r.pruned)            }            Ok(_) => {}            Err(e) => tracing::warn!("[scheduler] universe sync: {e:#}"),        }        return Ok(());    }    tracing::info!("[scheduler] seed_completed unset: running first-run seed");
@@ -416,6 +428,29 @@ async fn run_boot_seed(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow:/// appends each symbol's bar (and stamps `history_synced_at`), so nothing is/// stale here. It still earns its keep for symbols missed while the server was/// down at the close, weekends, and freshly added tickers./// True when a stored daily series is actually coarser than daily, judged from/// its *recent* density: `recent_bars` is the number of bars in the trailing 90/// days of the series, which is ~63 for a genuine daily series (trading days),/// ~13 for weekly and ~3 for monthly — so under 30 means coarser than daily./// Only judged once the series spans more than 180 days, so a young-but-daily/// symbol is not flagged for merely having a short history. Mirrors the SQL test/// in [`run_history`]'s selection query.fn is_coarser_than_daily(first: &Option<String>, last: &Option<String>, recent_bars: i64) -> bool {    let (Some(f), Some(l)) = (first, last) else {        return false;    };    if f == l {        return false;    }    let (Ok(fd), Ok(ld)) = (        chrono::NaiveDate::parse_from_str(f, "%Y-%m-%d"),        chrono::NaiveDate::parse_from_str(l, "%Y-%m-%d"),    ) else {        return false;    };    (ld - fd).num_days() > 180 && recent_bars < 30}async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::Result<()> {    let started = now_ms();    let next = started + HISTORY_INTERVAL_SECS * 1000;
@@ -427,11 +462,40 @@ async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::R    // exactly as a seeded one does. (The seed itself stays curated-list-only;    // it is the only job that keys on `is_seeded`.) Futures are included now —    // Yahoo serves `=F` daily history, unlike the Stooq source this replaced.    // A symbol is refreshed when any of these hold:    //   - its history has gone stale (the normal incremental case);    //   - it holds a single stored bar (`history_first_date = history_last_date`)    //     — a symbol added after the initial seed whose `history_last_date` was    //     stamped by a `daily_close` snapshot before any range=max backfill ran,    //     so the incremental window can never reach its deep history;    //   - its stored history is coarser than daily. Yahoo silently downsamples    //     `interval=1d` to weekly / monthly bars when a range spans many years    //     (see `YahooProvider::is_downsampled`), so a symbol first backfilled via    //     range=max can hold monthly bars. Detect it by *recent* density — the    //     bar count in the trailing 90 days of the series: a daily series carries    //     ~63 (trading days), a weekly one ~13, a monthly one ~3, so fewer than    //     30 means coarser than daily. The trailing window (rather than whole-    //     span density) is what keeps a deep symbol with a sparse pre-modern tail    //     — e.g. ^SPX, daily for decades but with century-old gaps — from being    //     misread as coarse. Only judged once a symbol spans over 180 days, so a    //     young-but-daily symbol is not flagged for its short history.    // The last two are selected regardless of staleness so the heal is not gated    // behind the stale cutoff; both are repaired with a full daily re-fetch (the    // `deep` branch below). `recent_bars` rides along so the same test can pick    // the fetch window without a second query.    let cutoff = started - HISTORY_STALE_SECS * 1000;    let stale: Vec<(String, Option<String>)> = sqlx::query_as(        "SELECT ticker, history_last_date FROM symbols \         WHERE history_synced_at IS NULL OR history_synced_at < ? \         ORDER BY ticker",    let stale: Vec<(String, Option<String>, Option<String>, i64)> = sqlx::query_as(        "SELECT s.ticker, s.history_first_date, s.history_last_date, \                (SELECT COUNT(*) FROM daily_prices d WHERE d.ticker = s.ticker \                   AND d.d >= date(s.history_last_date, '-90 days')) AS recent_bars \         FROM symbols s \         WHERE s.history_synced_at IS NULL OR s.history_synced_at < ? \            OR s.history_first_date = s.history_last_date \            OR (s.history_last_date IS NOT NULL \                AND julianday(s.history_last_date) - julianday(s.history_first_date) > 180 \                AND (SELECT COUNT(*) FROM daily_prices d WHERE d.ticker = s.ticker \                       AND d.d >= date(s.history_last_date, '-90 days')) < 30) \         ORDER BY s.ticker",    )    .bind(cutoff)    .fetch_all(pool)
@@ -459,7 +523,7 @@ async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::R    let mut total_bars = 0i64;    let mut stopped: Option<String> = None;    for (ticker, last_date) in &stale {    for (ticker, first_date, last_date, recent_bars) in &stale {        match guard.acquire().await? {            Permit::Granted => {}            Permit::Denied(why) => {
@@ -467,9 +531,28 @@ async fn run_history(pool: &SqlitePool, config: &Config, hub: &Hub) -> anyhow::R                break;            }        }        match yahoo.daily(ticker, last_date.as_deref()).await {        // Decide the fetch window. A symbol with no full daily history yet —        // none stored, a single bar, or a coarser-than-daily series — needs a        // deep re-fetch, so ask for range=max (`None`); the provider re-asks a        // bounded window when Yahoo downsamples it. Everything else just wants        // the incremental window since its last stored bar.        let single_bar = first_date.is_some() && first_date == last_date;        let coarse = is_coarser_than_daily(first_date, last_date, *recent_bars);        let deep = last_date.is_none() || single_bar || coarse;        let since = if deep { None } else { last_date.as_deref() };        match yahoo.daily(ticker, since).await {            Ok(bars) if !bars.is_empty() => {                guard.record_success().await?;                // A coarse series is replaced, not merged: drop its weekly /                // monthly bars so the fresh daily ones do not interleave with                // leftover old-granularity bars. Done only once we have new                // bars in hand, so the symbol is never left empty.                if coarse {                    sqlx::query("DELETE FROM daily_prices WHERE ticker = ?")                        .bind(ticker)                        .execute(pool)                        .await?;                }                total_bars += bars.len() as i64;                seed::store_daily(pool, ticker, &bars).await?;                ok += 1;
modified src/seed.rs
@@ -55,18 +55,29 @@ fn parse_universe(path: &Path) -> Result<Vec<SeedSymbol>> {    Ok(out)}/// Run the seed: upsert symbols, then backfill daily history for any that/// still lack it.pub async fn run(pool: &SqlitePool, config: &Config, history: &dyn HistoryProvider) -> Result<()> {    let started = Instant::now();/// Outcome of a universe sync, for logging.pub struct SyncReport {    /// Symbols in the curated CSV (all upserted).    pub total: usize,    /// Seeded symbols deleted because they were dropped from the CSV.    pub pruned: u64,}/// Reconcile the `symbols` table to the curated CSV. Local only, no network:/// upsert every listed symbol and prune the curated ones that were dropped from/// the list. Idempotent, so it runs on every boot (see `scheduler::run_boot_seed`)/// and a CSV edit (added or removed symbols) takes effect on the next deploy/// without a manual re-seed. The history backfill for any newly-added symbol is/// handled separately: the first-run seed loop below, or — once the seed has/// completed — the incremental `run_history` job, which picks up any symbol with/// a `NULL history_synced_at`.pub async fn sync_universe(pool: &SqlitePool, config: &Config) -> Result<SyncReport> {    let path = config.root.join("universe/starter.csv");    let symbols = parse_universe(&path)?;    tracing::info!("seed: {} symbols in {}", symbols.len(), path.display());    // Upsert every symbol. Local only, no network. Phase 28: the curated    // benchmark column is set from the CSV on each seed pass, so a re-run    // picks up any newly-curated mapping. A `NULL` benchmark stays `NULL`    // (don't overwrite a user-edited one — they all live in the CSV).    // Upsert every symbol. Phase 28: the curated benchmark column is set from    // the CSV on each pass, so a re-run picks up any newly-curated mapping. A    // `NULL` benchmark stays `NULL` (they all live in the CSV).    for s in &symbols {        let now = now_ms();        sqlx::query(
@@ -89,12 +100,52 @@ pub async fn run(pool: &SqlitePool, config: &Config, history: &dyn HistoryProvid        .await?;    }    // Only symbols with no history yet need a fetch. This keeps re-runs cheap    // and lets a quota-limited run resume later. Futures are included now:    // Yahoo serves `=F` daily history, unlike the Stooq source this replaced.    // Prune curated symbols dropped from the CSV. Only `is_seeded = 1` rows are    // eligible, so a user-added symbol (`is_seeded = 0`) is never touched. The    // delete cascades to every child table (all `REFERENCES symbols(ticker) ON    // DELETE CASCADE`, foreign keys enabled in db::init), so no orphan rows are    // left behind. The IN-list is bounded by the curated list size (~560), well    // under SQLite's bound-parameter limit.    let placeholders = vec!["?"; symbols.len()].join(",");    let sql = format!(        "DELETE FROM symbols WHERE is_seeded = 1 AND ticker NOT IN ({placeholders})"    );    let mut q = sqlx::query(&sql);    for s in &symbols {        q = q.bind(&s.ticker);    }    let pruned = q.execute(pool).await?.rows_affected();    Ok(SyncReport { total: symbols.len(), pruned })}/// Run the seed: reconcile the universe, then backfill daily history for any/// symbol that still lacks it.pub async fn run(pool: &SqlitePool, config: &Config, history: &dyn HistoryProvider) -> Result<()> {    let started = Instant::now();    let report = sync_universe(pool, config).await?;    tracing::info!(        "seed: {} symbols synced, {} pruned",        report.total,        report.pruned    );    // Symbols that still need a deep backfill. Two cases:    //  - `history_last_date IS NULL`: never fetched (the normal first-run case).    //  - `history_first_date = history_last_date`: only a single stored bar,    //    which means the symbol was added after the initial seed and a    //    `daily_close` snapshot stamped its `history_last_date` before any    //    range=max backfill ran — so the incremental path, which only asks for    //    the window since the last bar, can never reach its deep history. These    //    get re-fetched with range=max (the loop always passes `None`). A symbol    //    Yahoo genuinely has one bar for simply stays a one-bar re-fetch; cheap.    // This keeps re-runs cheap and lets a quota-limited run resume later.    // Futures are included now: Yahoo serves `=F` daily history, unlike the    // Stooq source this replaced.    let pending: Vec<String> = sqlx::query_scalar(        "SELECT ticker FROM symbols \         WHERE is_seeded = 1 AND history_last_date IS NULL \         WHERE is_seeded = 1 \           AND (history_last_date IS NULL OR history_first_date = history_last_date) \         ORDER BY ticker",    )    .fetch_all(pool)
modified universe/starter.csv
@@ -1,548 +1,563 @@ticker,name,kind,exchange,benchmark^SPX,S&P 500,index,,^DJI,Dow Jones Industrial Average,index,,^NDX,Nasdaq 100,index,,^NDQ,Nasdaq Composite,index,,^RUT,Russell 2000,index,,^VIX,CBOE Volatility Index,index,,SPY,SPDR S&P 500 ETF Trust,etf,NYSEARCA,^SPXVOO,Vanguard S&P 500 ETF,etf,NYSEARCA,^SPXIVV,iShares Core S&P 500 ETF,etf,NYSEARCA,^SPXVTI,Vanguard Total Stock Market ETF,etf,NYSEARCA,^SPXQQQ,Invesco QQQ Trust,etf,NASDAQ,^NDXDIA,SPDR Dow Jones Industrial Average ETF,etf,NYSEARCA,^DJIIWM,iShares Russell 2000 ETF,etf,NYSEARCA,^RUTVEA,Vanguard FTSE Developed Markets ETF,etf,NYSEARCA,VWO,Vanguard FTSE Emerging Markets ETF,etf,NYSEARCA,VUG,Vanguard Growth ETF,etf,NYSEARCA,^SPXVTV,Vanguard Value ETF,etf,NYSEARCA,^SPXSCHD,Schwab US Dividend Equity ETF,etf,NYSEARCA,^SPXVIG,Vanguard Dividend Appreciation ETF,etf,NYSEARCA,^SPXVYM,Vanguard High Dividend Yield ETF,etf,NYSEARCA,^SPXVXUS,Vanguard Total International Stock ETF,etf,NASDAQ,BND,Vanguard Total Bond Market ETF,etf,NASDAQ,AGG,iShares Core US Aggregate Bond ETF,etf,NYSEARCA,TLT,iShares 20+ Year Treasury Bond ETF,etf,NASDAQ,GLD,SPDR Gold Shares,etf,NYSEARCA,SLV,iShares Silver Trust,etf,NYSEARCA,XLK,Technology Select Sector SPDR Fund,etf,NYSEARCA,^SPXXLF,Financial Select Sector SPDR Fund,etf,NYSEARCA,^SPXXLE,Energy Select Sector SPDR Fund,etf,NYSEARCA,^SPXXLV,Health Care Select Sector SPDR Fund,etf,NYSEARCA,^SPXSMH,VanEck Semiconductor ETF,etf,NASDAQ,^NDXARKK,ARK Innovation ETF,etf,NYSEARCA,^NDXEFA,iShares MSCI EAFE ETF,etf,NYSEARCA,EEM,iShares MSCI Emerging Markets ETF,etf,NYSEARCA,MMM,3M,stock,,AOS,A. O. Smith,stock,,ABT,Abbott Laboratories,stock,,ABBV,AbbVie,stock,,ACN,Accenture,stock,,ADBE,Adobe Inc.,stock,,AMD,Advanced Micro Devices,stock,,AES,AES Corporation,stock,,AFL,Aflac,stock,,A,Agilent Technologies,stock,,APD,Air Products,stock,,ABNB,Airbnb,stock,,AKAM,Akamai Technologies,stock,,ALB,Albemarle Corporation,stock,,ARE,Alexandria Real Estate Equities,stock,,ALGN,Align Technology,stock,,ALLE,Allegion,stock,,LNT,Alliant Energy,stock,,ALL,Allstate,stock,,GOOGL,Alphabet Inc. (Class A),stock,,GOOG,Alphabet Inc. (Class C),stock,,MO,Altria,stock,,AMZN,Amazon,stock,,AMCR,Amcor,stock,,AEE,Ameren,stock,,AEP,American Electric Power,stock,,AXP,American Express,stock,,AIG,American International Group,stock,,AMT,American Tower,stock,,AWK,American Water Works,stock,,AMP,Ameriprise Financial,stock,,AME,Ametek,stock,,AMGN,Amgen,stock,,APH,Amphenol,stock,,ADI,Analog Devices,stock,,AON,Aon plc,stock,,APA,APA Corporation,stock,,APO,Apollo Global Management,stock,,AAPL,Apple Inc.,stock,,AMAT,Applied Materials,stock,,APP,AppLovin,stock,,APTV,Aptiv,stock,,ACGL,Arch Capital Group,stock,,ADM,Archer Daniels Midland,stock,,ARES,Ares Management,stock,,ANET,Arista Networks,stock,,AJG,Arthur J. Gallagher & Co.,stock,,AIZ,Assurant,stock,,T,AT&T,stock,,ATO,Atmos Energy,stock,,ADSK,Autodesk,stock,,ADP,Automatic Data Processing,stock,,AZO,AutoZone,stock,,AVB,AvalonBay Communities,stock,,AVY,Avery Dennison,stock,,AXON,Axon Enterprise,stock,,BKR,Baker Hughes,stock,,BALL,Ball Corporation,stock,,BAC,Bank of America,stock,,BAX,Baxter International,stock,,BDX,Becton Dickinson,stock,,BRK.B,Berkshire Hathaway,stock,,BBY,Best Buy,stock,,TECH,Bio-Techne,stock,,BIIB,Biogen,stock,,BLK,BlackRock,stock,,BX,Blackstone Inc.,stock,,XYZ,"Block, Inc.",stock,,BNY,BNY Mellon,stock,,BA,Boeing,stock,,BKNG,Booking Holdings,stock,,BSX,Boston Scientific,stock,,BMY,Bristol Myers Squibb,stock,,AVGO,Broadcom,stock,,BR,Broadridge Financial Solutions,stock,,BRO,Brown & Brown,stock,,BF.B,Brown–Forman,stock,,BLDR,Builders FirstSource,stock,,BG,Bunge Global,stock,,BXP,"BXP, Inc.",stock,,CHRW,C.H. Robinson,stock,,CDNS,Cadence Design Systems,stock,,CPT,Camden Property Trust,stock,,CPB,Campbell's Company (The),stock,,COF,Capital One,stock,,CAH,Cardinal Health,stock,,CCL,Carnival Corporation,stock,,CARR,Carrier Global,stock,,CVNA,Carvana,stock,,CASY,Casey's,stock,,CAT,Caterpillar Inc.,stock,,CBOE,Cboe Global Markets,stock,,CBRE,CBRE Group,stock,,CDW,CDW Corporation,stock,,COR,Cencora,stock,,CNC,Centene Corporation,stock,,CNP,CenterPoint Energy,stock,,CF,CF Industries,stock,,CRL,Charles River Laboratories,stock,,SCHW,Charles Schwab Corporation,stock,,CHTR,Charter Communications,stock,,CVX,Chevron Corporation,stock,,CMG,Chipotle Mexican Grill,stock,,CB,Chubb Limited,stock,,CHD,Church & Dwight,stock,,CIEN,Ciena,stock,,CI,Cigna,stock,,CINF,Cincinnati Financial,stock,,CTAS,Cintas,stock,,CSCO,Cisco,stock,,C,Citigroup,stock,,CFG,Citizens Financial Group,stock,,CLX,Clorox,stock,,CME,CME Group,stock,,CMS,CMS Energy,stock,,KO,Coca-Cola Company (The),stock,,CTSH,Cognizant,stock,,COHR,Coherent Corp.,stock,,COIN,Coinbase,stock,,CL,Colgate-Palmolive,stock,,CMCSA,Comcast,stock,,FIX,Comfort Systems USA,stock,,CAG,Conagra Brands,stock,,COP,ConocoPhillips,stock,,ED,Consolidated Edison,stock,,STZ,Constellation Brands,stock,,CEG,Constellation Energy,stock,,COO,Cooper Companies (The),stock,,CPRT,Copart,stock,,GLW,Corning Inc.,stock,,CPAY,Corpay,stock,,CTVA,Corteva,stock,,CSGP,CoStar Group,stock,,COST,Costco,stock,,CRH,CRH plc,stock,,CRWD,CrowdStrike,stock,,CCI,Crown Castle,stock,,CSX,CSX Corporation,stock,,CMI,Cummins,stock,,CVS,CVS Health,stock,,DHR,Danaher Corporation,stock,,DRI,Darden Restaurants,stock,,DDOG,Datadog,stock,,DVA,DaVita,stock,,DECK,Deckers Brands,stock,,DE,Deere & Company,stock,,DELL,Dell Technologies,stock,,DAL,Delta Air Lines,stock,,DVN,Devon Energy,stock,,DXCM,Dexcom,stock,,FANG,Diamondback Energy,stock,,DLR,Digital Realty,stock,,DG,Dollar General,stock,,DLTR,Dollar Tree,stock,,D,Dominion Energy,stock,,DPZ,Domino's,stock,,DASH,DoorDash,stock,,DOV,Dover Corporation,stock,,DOW,Dow Inc.,stock,,DHI,D. R. Horton,stock,,DTE,DTE Energy,stock,,DUK,Duke Energy,stock,,DD,DuPont,stock,,ETN,Eaton Corporation,stock,,EBAY,eBay Inc.,stock,,SATS,EchoStar,stock,,ECL,Ecolab,stock,,EIX,Edison International,stock,,EW,Edwards Lifesciences,stock,,EA,Electronic Arts,stock,,ELV,Elevance Health,stock,,EME,Emcor,stock,,EMR,Emerson Electric,stock,,ETR,Entergy,stock,,EOG,EOG Resources,stock,,EPAM,EPAM Systems,stock,,EQT,EQT Corporation,stock,,EFX,Equifax,stock,,EQIX,Equinix,stock,,EQR,Equity Residential,stock,,ERIE,Erie Indemnity,stock,,ESS,Essex Property Trust,stock,,EL,Estée Lauder Companies (The),stock,,EG,Everest Group,stock,,EVRG,Evergy,stock,,ES,Eversource Energy,stock,,EXC,Exelon,stock,,EXE,Expand Energy,stock,,EXPE,Expedia Group,stock,,EXPD,Expeditors International,stock,,EXR,Extra Space Storage,stock,,XOM,ExxonMobil,stock,,FFIV,"F5, Inc.",stock,,FDS,FactSet,stock,,FICO,Fair Isaac,stock,,FAST,Fastenal,stock,,FRT,Federal Realty Investment Trust,stock,,FDX,FedEx,stock,,FIS,Fidelity National Information Services,stock,,FITB,Fifth Third Bancorp,stock,,FSLR,First Solar,stock,,FE,FirstEnergy,stock,,FISV,Fiserv,stock,,F,Ford Motor Company,stock,,FTNT,Fortinet,stock,,FTV,Fortive,stock,,FOXA,Fox Corporation (Class A),stock,,FOX,Fox Corporation (Class B),stock,,BEN,Franklin Resources,stock,,FCX,Freeport-McMoRan,stock,,GRMN,Garmin,stock,,IT,Gartner,stock,,GE,GE Aerospace,stock,,GEHC,GE HealthCare,stock,,GEV,GE Vernova,stock,,GEN,Gen Digital,stock,,GNRC,Generac,stock,,GD,General Dynamics,stock,,GIS,General Mills,stock,,GM,General Motors,stock,,GPC,Genuine Parts Company,stock,,GILD,Gilead Sciences,stock,,GPN,Global Payments,stock,,GL,Globe Life,stock,,GDDY,GoDaddy,stock,,GS,Goldman Sachs,stock,,HAL,Halliburton,stock,,HIG,Hartford (The),stock,,HAS,Hasbro,stock,,HCA,HCA Healthcare,stock,,DOC,Healthpeak Properties,stock,,HSIC,Henry Schein,stock,,HSY,Hershey Company (The),stock,,HPE,Hewlett Packard Enterprise,stock,,HLT,Hilton Worldwide,stock,,HD,Home Depot (The),stock,,HON,Honeywell,stock,,HRL,Hormel Foods,stock,,HST,Host Hotels & Resorts,stock,,HWM,Howmet Aerospace,stock,,HPQ,HP Inc.,stock,,HUBB,Hubbell Incorporated,stock,,HUM,Humana,stock,,HBAN,Huntington Bancshares,stock,,HII,Huntington Ingalls Industries,stock,,IBM,IBM,stock,,IEX,IDEX Corporation,stock,,IDXX,Idexx Laboratories,stock,,ITW,Illinois Tool Works,stock,,INCY,Incyte,stock,,IR,Ingersoll Rand,stock,,PODD,Insulet Corporation,stock,,INTC,Intel,stock,,IBKR,Interactive Brokers,stock,,ICE,Intercontinental Exchange,stock,,IFF,International Flavors & Fragrances,stock,,IP,International Paper,stock,,INTU,Intuit,stock,,ISRG,Intuitive Surgical,stock,,IVZ,Invesco,stock,,INVH,Invitation Homes,stock,,IQV,IQVIA,stock,,IRM,Iron Mountain,stock,,JBHT,J.B. Hunt,stock,,JBL,Jabil,stock,,JKHY,Jack Henry & Associates,stock,,J,Jacobs Solutions,stock,,JNJ,Johnson & Johnson,stock,,JCI,Johnson Controls,stock,,JPM,JPMorgan Chase,stock,,KVUE,Kenvue,stock,,KDP,Keurig Dr Pepper,stock,,KEY,KeyCorp,stock,,KEYS,Keysight Technologies,stock,,KMB,Kimberly-Clark,stock,,KIM,Kimco Realty,stock,,KMI,Kinder Morgan,stock,,KKR,KKR & Co.,stock,,KLAC,KLA Corporation,stock,,KHC,Kraft Heinz,stock,,KR,Kroger,stock,,LHX,L3Harris,stock,,LH,Labcorp,stock,,LRCX,Lam Research,stock,,LVS,Las Vegas Sands,stock,,LDOS,Leidos,stock,,LEN,Lennar,stock,,LII,Lennox International,stock,,LLY,Lilly (Eli),stock,,LIN,Linde plc,stock,,LYV,Live Nation Entertainment,stock,,LMT,Lockheed Martin,stock,,L,Loews Corporation,stock,,LOW,Lowe's,stock,,LULU,Lululemon Athletica,stock,,LITE,Lumentum,stock,,LYB,LyondellBasell,stock,,MTB,M&T Bank,stock,,MPC,Marathon Petroleum,stock,,MAR,Marriott International,stock,,MRSH,Marsh McLennan,stock,,MLM,Martin Marietta Materials,stock,,MAS,Masco,stock,,MA,Mastercard,stock,,MKC,McCormick & Company,stock,,MCD,McDonald's,stock,,MCK,McKesson Corporation,stock,,MDT,Medtronic,stock,,MRK,Merck & Co.,stock,,META,Meta Platforms,stock,,MET,MetLife,stock,,MTD,Mettler Toledo,stock,,MGM,MGM Resorts,stock,,MCHP,Microchip Technology,stock,,MU,Micron Technology,stock,,MSFT,Microsoft,stock,,MAA,Mid-America Apartment Communities,stock,,MRNA,Moderna,stock,,TAP,Molson Coors Beverage Company,stock,,MDLZ,Mondelez International,stock,,MPWR,Monolithic Power Systems,stock,,MNST,Monster Beverage,stock,,MCO,Moody's Corporation,stock,,MS,Morgan Stanley,stock,,MOS,Mosaic Company (The),stock,,MSI,Motorola Solutions,stock,,MSCI,MSCI Inc.,stock,,NDAQ,"Nasdaq, Inc.",stock,,NTAP,NetApp,stock,,NFLX,Netflix,stock,,NEM,Newmont,stock,,NWSA,News Corp (Class A),stock,,NWS,News Corp (Class B),stock,,NEE,NextEra Energy,stock,,NKE,"Nike, Inc.",stock,,NI,NiSource,stock,,NDSN,Nordson Corporation,stock,,NSC,Norfolk Southern,stock,,NTRS,Northern Trust,stock,,NOC,Northrop Grumman,stock,,NCLH,Norwegian Cruise Line Holdings,stock,,NRG,NRG Energy,stock,,NUE,Nucor,stock,,NVDA,Nvidia,stock,,NVR,"NVR, Inc.",stock,,NXPI,NXP Semiconductors,stock,,ORLY,O’Reilly Automotive,stock,,OXY,Occidental Petroleum,stock,,ODFL,Old Dominion,stock,,OMC,Omnicom Group,stock,,ON,ON Semiconductor,stock,,OKE,Oneok,stock,,ORCL,Oracle Corporation,stock,,OTIS,Otis Worldwide,stock,,PCAR,Paccar,stock,,PKG,Packaging Corporation of America,stock,,PLTR,Palantir Technologies,stock,,PANW,Palo Alto Networks,stock,,PSKY,Paramount Skydance Corporation,stock,,PH,Parker Hannifin,stock,,PAYX,Paychex,stock,,PYPL,PayPal,stock,,PNR,Pentair,stock,,PEP,PepsiCo,stock,,PFE,Pfizer,stock,,PCG,PG&E Corporation,stock,,PM,Philip Morris International,stock,,PSX,Phillips 66,stock,,PNW,Pinnacle West Capital,stock,,PNC,PNC Financial Services,stock,,POOL,Pool Corporation,stock,,PPG,PPG Industries,stock,,PPL,PPL Corporation,stock,,PFG,Principal Financial Group,stock,,PG,Procter & Gamble,stock,,PGR,Progressive Corporation,stock,,PLD,Prologis,stock,,PRU,Prudential Financial,stock,,PEG,Public Service Enterprise Group,stock,,PTC,PTC Inc.,stock,,PSA,Public Storage,stock,,PHM,PulteGroup,stock,,PWR,Quanta Services,stock,,QCOM,Qualcomm,stock,,DGX,Quest Diagnostics,stock,,Q,Qnity Electronics,stock,,RL,Ralph Lauren Corporation,stock,,RJF,Raymond James Financial,stock,,RTX,RTX Corporation,stock,,O,Realty Income,stock,,REG,Regency Centers,stock,,REGN,Regeneron Pharmaceuticals,stock,,RF,Regions Financial Corporation,stock,,RSG,Republic Services,stock,,RMD,ResMed,stock,,RVTY,Revvity,stock,,HOOD,Robinhood Markets,stock,,ROK,Rockwell Automation,stock,,ROL,"Rollins, Inc.",stock,,ROP,Roper Technologies,stock,,ROST,Ross Stores,stock,,RCL,Royal Caribbean Group,stock,,SPGI,S&P Global,stock,,CRM,Salesforce,stock,,SNDK,Sandisk,stock,,SBAC,SBA Communications,stock,,SLB,Schlumberger,stock,,STX,Seagate Technology,stock,,SRE,Sempra,stock,,NOW,ServiceNow,stock,,SHW,Sherwin-Williams,stock,,SPG,Simon Property Group,stock,,SWKS,Skyworks Solutions,stock,,SJM,J.M. Smucker Company (The),stock,,SW,Smurfit Westrock,stock,,SNA,Snap-on,stock,,SOLV,Solventum,stock,,SO,Southern Company,stock,,LUV,Southwest Airlines,stock,,SWK,Stanley Black & Decker,stock,,SBUX,Starbucks,stock,,STT,State Street Corporation,stock,,STLD,Steel Dynamics,stock,,STE,Steris,stock,,SYK,Stryker Corporation,stock,,SMCI,Supermicro,stock,,SYF,Synchrony Financial,stock,,SNPS,Synopsys,stock,,SYY,Sysco,stock,,TMUS,T-Mobile US,stock,,TROW,T. Rowe Price,stock,,TTWO,Take-Two Interactive,stock,,TPR,"Tapestry, Inc.",stock,,TRGP,Targa Resources,stock,,TGT,Target Corporation,stock,,TEL,TE Connectivity,stock,,TDY,Teledyne Technologies,stock,,TER,Teradyne,stock,,TSLA,"Tesla, Inc.",stock,,TXN,Texas Instruments,stock,,TPL,Texas Pacific Land Corporation,stock,,TXT,Textron,stock,,TMO,Thermo Fisher Scientific,stock,,TJX,TJX Companies,stock,,TKO,TKO Group Holdings,stock,,TTD,Trade Desk (The),stock,,TSCO,Tractor Supply,stock,,TT,Trane Technologies,stock,,TDG,TransDigm Group,stock,,TRV,Travelers Companies (The),stock,,TRMB,Trimble Inc.,stock,,TFC,Truist Financial,stock,,TYL,Tyler Technologies,stock,,TSN,Tyson Foods,stock,,USB,U.S. Bancorp,stock,,UBER,Uber,stock,,UDR,"UDR, Inc.",stock,,ULTA,Ulta Beauty,stock,,UNP,Union Pacific Corporation,stock,,UAL,United Airlines Holdings,stock,,UPS,United Parcel Service,stock,,URI,United Rentals,stock,,UNH,UnitedHealth Group,stock,,UHS,Universal Health Services,stock,,VLO,Valero Energy,stock,,VEEV,Veeva Systems,stock,,VTR,Ventas,stock,,VLTO,Veralto,stock,,VRSN,Verisign,stock,,VRSK,Verisk Analytics,stock,,VZ,Verizon,stock,,VRTX,Vertex Pharmaceuticals,stock,,VRT,Vertiv,stock,,VTRS,Viatris,stock,,VICI,Vici Properties,stock,,V,Visa Inc.,stock,,VST,Vistra Corp.,stock,,VMC,Vulcan Materials Company,stock,,WRB,W. R. Berkley Corporation,stock,,GWW,W. W. Grainger,stock,,WAB,Wabtec,stock,,WMT,Walmart,stock,,DIS,Walt Disney Company (The),stock,,WBD,Warner Bros. Discovery,stock,,WM,Waste Management,stock,,WAT,Waters Corporation,stock,,WEC,WEC Energy Group,stock,,WFC,Wells Fargo,stock,,WELL,Welltower,stock,,WST,West Pharmaceutical Services,stock,,WDC,Western Digital,stock,,WY,Weyerhaeuser,stock,,WSM,"Williams-Sonoma, Inc.",stock,,WMB,Williams Companies,stock,,WTW,Willis Towers Watson,stock,,WDAY,"Workday, Inc.",stock,,WYNN,Wynn Resorts,stock,,XEL,Xcel Energy,stock,,XYL,Xylem Inc.,stock,,YUM,Yum! Brands,stock,,ZBRA,Zebra Technologies,stock,,ZBH,Zimmer Biomet,stock,,ZTS,Zoetis,stock,,ES=F,S&P 500 E-Mini Futures,future,CME,NQ=F,Nasdaq 100 E-Mini Futures,future,CME,YM=F,Dow E-Mini Futures,future,CME,RTY=F,Russell 2000 E-Mini Futures,future,CME,CL=F,Crude Oil Futures (WTI),future,NYMEX,BZ=F,Brent Crude Oil Futures,future,NYMEX,GC=F,Gold Futures,future,COMEX,SI=F,Silver Futures,future,COMEX,HG=F,Copper Futures,future,COMEX,NG=F,Natural Gas Futures,future,NYMEX,ticker,name,kind,exchange,benchmark^SPX,S&P 500,index,,^DJI,Dow Jones Industrial Average,index,,^NDX,Nasdaq 100,index,,^NDQ,Nasdaq Composite,index,,^RUT,Russell 2000,index,,^VIX,CBOE Volatility Index,index,,SPY,SPDR S&P 500 ETF Trust,etf,,^SPXVOO,Vanguard S&P 500 ETF,etf,,^SPXIVV,iShares Core S&P 500 ETF,etf,,^SPXVTI,Vanguard Total Stock Market ETF,etf,,^SPXITOT,iShares Core S&P Total U.S. Stock Market ETF,etf,,^SPXQQQ,Invesco QQQ Trust,etf,,^NDXDIA,SPDR Dow Jones Industrial Average ETF,etf,,^DJIVUG,Vanguard Growth ETF,etf,,^SPXVTV,Vanguard Value ETF,etf,,^SPXIWF,iShares Russell 1000 Growth ETF,etf,,^SPXIWD,iShares Russell 1000 Value ETF,etf,,^SPXVIG,Vanguard Dividend Appreciation ETF,etf,,^SPXVYM,Vanguard High Dividend Yield ETF,etf,,^SPXVO,Vanguard Mid-Cap ETF,etf,,^SPXIJH,iShares Core S&P Mid-Cap ETF,etf,,^SPXVB,Vanguard Small-Cap ETF,etf,,^RUTIJR,iShares Core S&P Small-Cap ETF,etf,,^RUTIWM,iShares Russell 2000 ETF,etf,,^RUTVGT,Vanguard Information Technology ETF,etf,,^SPXVNQ,Vanguard Real Estate ETF,etf,,^SPXVXUS,Vanguard Total International Stock ETF,etf,,IXUS,iShares Core MSCI Total International Stock ETF,etf,,VEA,Vanguard FTSE Developed Markets ETF,etf,,IEFA,iShares Core MSCI EAFE ETF,etf,,EFA,iShares MSCI EAFE ETF,etf,,VWO,Vanguard FTSE Emerging Markets ETF,etf,,IEMG,iShares Core MSCI Emerging Markets ETF,etf,,EEM,iShares MSCI Emerging Markets ETF,etf,,VT,Vanguard Total World Stock ETF,etf,,BND,Vanguard Total Bond Market ETF,etf,,AGG,iShares Core US Aggregate Bond ETF,etf,,BNDX,Vanguard Total International Bond ETF,etf,,LQD,iShares iBoxx $ Investment Grade Corporate Bond ETF,etf,,HYG,iShares iBoxx $ High Yield Corporate Bond ETF,etf,,TIP,iShares TIPS Bond ETF,etf,,IEF,iShares 7-10 Year Treasury Bond ETF,etf,,TLT,iShares 20+ Year Treasury Bond ETF,etf,,SHY,iShares 1-3 Year Treasury Bond ETF,etf,,VBIL,Vanguard 0-3 Month Treasury Bill ETF,etf,,GLD,SPDR Gold Shares,etf,,IAU,iShares Gold Trust,etf,,SLV,iShares Silver Trust,etf,,IBIT,iShares Bitcoin Trust,etf,,MMM,3M,stock,,AOS,A. O. Smith,stock,,ABT,Abbott Laboratories,stock,,ABBV,AbbVie,stock,,ACN,Accenture,stock,,ADBE,Adobe Inc.,stock,,AMD,Advanced Micro Devices,stock,,AES,AES Corporation,stock,,AFL,Aflac,stock,,A,Agilent Technologies,stock,,APD,Air Products,stock,,ABNB,Airbnb,stock,,AKAM,Akamai Technologies,stock,,ALB,Albemarle Corporation,stock,,ARE,Alexandria Real Estate Equities,stock,,ALGN,Align Technology,stock,,ALLE,Allegion,stock,,LNT,Alliant Energy,stock,,ALL,Allstate,stock,,GOOGL,Alphabet Inc. (Class A),stock,,GOOG,Alphabet Inc. (Class C),stock,,MO,Altria,stock,,AMZN,Amazon,stock,,AMCR,Amcor,stock,,AEE,Ameren,stock,,AEP,American Electric Power,stock,,AXP,American Express,stock,,AIG,American International Group,stock,,AMT,American Tower,stock,,AWK,American Water Works,stock,,AMP,Ameriprise Financial,stock,,AME,Ametek,stock,,AMGN,Amgen,stock,,APH,Amphenol,stock,,ADI,Analog Devices,stock,,AON,Aon plc,stock,,APA,APA Corporation,stock,,APO,Apollo Global Management,stock,,AAPL,Apple Inc.,stock,,AMAT,Applied Materials,stock,,APP,AppLovin,stock,,APTV,Aptiv,stock,,ACGL,Arch Capital Group,stock,,ADM,Archer Daniels Midland,stock,,ARES,Ares Management,stock,,ANET,Arista Networks,stock,,AJG,Arthur J. Gallagher & Co.,stock,,AIZ,Assurant,stock,,T,AT&T,stock,,ATO,Atmos Energy,stock,,ADSK,Autodesk,stock,,ADP,Automatic Data Processing,stock,,AZO,AutoZone,stock,,AVB,AvalonBay Communities,stock,,AVY,Avery Dennison,stock,,AXON,Axon Enterprise,stock,,BKR,Baker Hughes,stock,,BALL,Ball Corporation,stock,,BAC,Bank of America,stock,,BAX,Baxter International,stock,,BDX,Becton Dickinson,stock,,BRK.B,Berkshire Hathaway,stock,,BBY,Best Buy,stock,,TECH,Bio-Techne,stock,,BIIB,Biogen,stock,,BLK,BlackRock,stock,,BX,Blackstone Inc.,stock,,XYZ,"Block, Inc.",stock,,BNY,BNY Mellon,stock,,BA,Boeing,stock,,BKNG,Booking Holdings,stock,,BSX,Boston Scientific,stock,,BMY,Bristol Myers Squibb,stock,,AVGO,Broadcom,stock,,BR,Broadridge Financial Solutions,stock,,BRO,Brown & Brown,stock,,BF.B,Brown–Forman,stock,,BLDR,Builders FirstSource,stock,,BG,Bunge Global,stock,,BXP,"BXP, Inc.",stock,,CHRW,C.H. Robinson,stock,,CDNS,Cadence Design Systems,stock,,CPT,Camden Property Trust,stock,,CPB,Campbell's Company (The),stock,,COF,Capital One,stock,,CAH,Cardinal Health,stock,,CCL,Carnival Corporation,stock,,CARR,Carrier Global,stock,,CVNA,Carvana,stock,,CASY,Casey's,stock,,CAT,Caterpillar Inc.,stock,,CBOE,Cboe Global Markets,stock,,CBRE,CBRE Group,stock,,CDW,CDW Corporation,stock,,COR,Cencora,stock,,CNC,Centene Corporation,stock,,CNP,CenterPoint Energy,stock,,CF,CF Industries,stock,,CRL,Charles River Laboratories,stock,,SCHW,Charles Schwab Corporation,stock,,CHTR,Charter Communications,stock,,CVX,Chevron Corporation,stock,,CMG,Chipotle Mexican Grill,stock,,CB,Chubb Limited,stock,,CHD,Church & Dwight,stock,,CIEN,Ciena,stock,,CI,Cigna,stock,,CINF,Cincinnati Financial,stock,,CTAS,Cintas,stock,,CSCO,Cisco,stock,,C,Citigroup,stock,,CFG,Citizens Financial Group,stock,,CLX,Clorox,stock,,CME,CME Group,stock,,CMS,CMS Energy,stock,,KO,Coca-Cola Company (The),stock,,CTSH,Cognizant,stock,,COHR,Coherent Corp.,stock,,COIN,Coinbase,stock,,CL,Colgate-Palmolive,stock,,CMCSA,Comcast,stock,,FIX,Comfort Systems USA,stock,,CAG,Conagra Brands,stock,,COP,ConocoPhillips,stock,,ED,Consolidated Edison,stock,,STZ,Constellation Brands,stock,,CEG,Constellation Energy,stock,,COO,Cooper Companies (The),stock,,CPRT,Copart,stock,,GLW,Corning Inc.,stock,,CPAY,Corpay,stock,,CTVA,Corteva,stock,,CSGP,CoStar Group,stock,,COST,Costco,stock,,CRH,CRH plc,stock,,CRWD,CrowdStrike,stock,,CCI,Crown Castle,stock,,CSX,CSX Corporation,stock,,CMI,Cummins,stock,,CVS,CVS Health,stock,,DHR,Danaher Corporation,stock,,DRI,Darden Restaurants,stock,,DDOG,Datadog,stock,,DVA,DaVita,stock,,DECK,Deckers Brands,stock,,DE,Deere & Company,stock,,DELL,Dell Technologies,stock,,DAL,Delta Air Lines,stock,,DVN,Devon Energy,stock,,DXCM,Dexcom,stock,,FANG,Diamondback Energy,stock,,DLR,Digital Realty,stock,,DG,Dollar General,stock,,DLTR,Dollar Tree,stock,,D,Dominion Energy,stock,,DPZ,Domino's,stock,,DASH,DoorDash,stock,,DOV,Dover Corporation,stock,,DOW,Dow Inc.,stock,,DHI,D. R. Horton,stock,,DTE,DTE Energy,stock,,DUK,Duke Energy,stock,,DD,DuPont,stock,,ETN,Eaton Corporation,stock,,EBAY,eBay Inc.,stock,,SATS,EchoStar,stock,,ECL,Ecolab,stock,,EIX,Edison International,stock,,EW,Edwards Lifesciences,stock,,EA,Electronic Arts,stock,,ELV,Elevance Health,stock,,EME,Emcor,stock,,EMR,Emerson Electric,stock,,ETR,Entergy,stock,,EOG,EOG Resources,stock,,EPAM,EPAM Systems,stock,,EQT,EQT Corporation,stock,,EFX,Equifax,stock,,EQIX,Equinix,stock,,EQR,Equity Residential,stock,,ERIE,Erie Indemnity,stock,,ESS,Essex Property Trust,stock,,EL,Estée Lauder Companies (The),stock,,EG,Everest Group,stock,,EVRG,Evergy,stock,,ES,Eversource Energy,stock,,EXC,Exelon,stock,,EXE,Expand Energy,stock,,EXPE,Expedia Group,stock,,EXPD,Expeditors International,stock,,EXR,Extra Space Storage,stock,,XOM,ExxonMobil,stock,,FFIV,"F5, Inc.",stock,,FDS,FactSet,stock,,FICO,Fair Isaac,stock,,FAST,Fastenal,stock,,FRT,Federal Realty Investment Trust,stock,,FDX,FedEx,stock,,FIS,Fidelity National Information Services,stock,,FITB,Fifth Third Bancorp,stock,,FSLR,First Solar,stock,,FE,FirstEnergy,stock,,FISV,Fiserv,stock,,F,Ford Motor Company,stock,,FTNT,Fortinet,stock,,FTV,Fortive,stock,,FOXA,Fox Corporation (Class A),stock,,FOX,Fox Corporation (Class B),stock,,BEN,Franklin Resources,stock,,FCX,Freeport-McMoRan,stock,,GRMN,Garmin,stock,,IT,Gartner,stock,,GE,GE Aerospace,stock,,GEHC,GE HealthCare,stock,,GEV,GE Vernova,stock,,GEN,Gen Digital,stock,,GNRC,Generac,stock,,GD,General Dynamics,stock,,GIS,General Mills,stock,,GM,General Motors,stock,,GPC,Genuine Parts Company,stock,,GILD,Gilead Sciences,stock,,GPN,Global Payments,stock,,GL,Globe Life,stock,,GDDY,GoDaddy,stock,,GS,Goldman Sachs,stock,,HAL,Halliburton,stock,,HIG,Hartford (The),stock,,HAS,Hasbro,stock,,HCA,HCA Healthcare,stock,,DOC,Healthpeak Properties,stock,,HSIC,Henry Schein,stock,,HSY,Hershey Company (The),stock,,HPE,Hewlett Packard Enterprise,stock,,HLT,Hilton Worldwide,stock,,HD,Home Depot (The),stock,,HON,Honeywell,stock,,HRL,Hormel Foods,stock,,HST,Host Hotels & Resorts,stock,,HWM,Howmet Aerospace,stock,,HPQ,HP Inc.,stock,,HUBB,Hubbell Incorporated,stock,,HUM,Humana,stock,,HBAN,Huntington Bancshares,stock,,HII,Huntington Ingalls Industries,stock,,IBM,IBM,stock,,IEX,IDEX Corporation,stock,,IDXX,Idexx Laboratories,stock,,ITW,Illinois Tool Works,stock,,INCY,Incyte,stock,,IR,Ingersoll Rand,stock,,PODD,Insulet Corporation,stock,,INTC,Intel,stock,,IBKR,Interactive Brokers,stock,,ICE,Intercontinental Exchange,stock,,IFF,International Flavors & Fragrances,stock,,IP,International Paper,stock,,INTU,Intuit,stock,,ISRG,Intuitive Surgical,stock,,IVZ,Invesco,stock,,INVH,Invitation Homes,stock,,IQV,IQVIA,stock,,IRM,Iron Mountain,stock,,JBHT,J.B. Hunt,stock,,JBL,Jabil,stock,,JKHY,Jack Henry & Associates,stock,,J,Jacobs Solutions,stock,,JNJ,Johnson & Johnson,stock,,JCI,Johnson Controls,stock,,JPM,JPMorgan Chase,stock,,KVUE,Kenvue,stock,,KDP,Keurig Dr Pepper,stock,,KEY,KeyCorp,stock,,KEYS,Keysight Technologies,stock,,KMB,Kimberly-Clark,stock,,KIM,Kimco Realty,stock,,KMI,Kinder Morgan,stock,,KKR,KKR & Co.,stock,,KLAC,KLA Corporation,stock,,KHC,Kraft Heinz,stock,,KR,Kroger,stock,,LHX,L3Harris,stock,,LH,Labcorp,stock,,LRCX,Lam Research,stock,,LVS,Las Vegas Sands,stock,,LDOS,Leidos,stock,,LEN,Lennar,stock,,LII,Lennox International,stock,,LLY,Lilly (Eli),stock,,LIN,Linde plc,stock,,LYV,Live Nation Entertainment,stock,,LMT,Lockheed Martin,stock,,L,Loews Corporation,stock,,LOW,Lowe's,stock,,LULU,Lululemon Athletica,stock,,LITE,Lumentum,stock,,LYB,LyondellBasell,stock,,MTB,M&T Bank,stock,,MPC,Marathon Petroleum,stock,,MAR,Marriott International,stock,,MRSH,Marsh McLennan,stock,,MLM,Martin Marietta Materials,stock,,MAS,Masco,stock,,MA,Mastercard,stock,,MKC,McCormick & Company,stock,,MCD,McDonald's,stock,,MCK,McKesson Corporation,stock,,MDT,Medtronic,stock,,MRK,Merck & Co.,stock,,META,Meta Platforms,stock,,MET,MetLife,stock,,MTD,Mettler Toledo,stock,,MGM,MGM Resorts,stock,,MCHP,Microchip Technology,stock,,MU,Micron Technology,stock,,MSFT,Microsoft,stock,,MAA,Mid-America Apartment Communities,stock,,MRNA,Moderna,stock,,TAP,Molson Coors Beverage Company,stock,,MDLZ,Mondelez International,stock,,MPWR,Monolithic Power Systems,stock,,MNST,Monster Beverage,stock,,MCO,Moody's Corporation,stock,,MS,Morgan Stanley,stock,,MOS,Mosaic Company (The),stock,,MSI,Motorola Solutions,stock,,MSCI,MSCI Inc.,stock,,NDAQ,"Nasdaq, Inc.",stock,,NTAP,NetApp,stock,,NFLX,Netflix,stock,,NEM,Newmont,stock,,NWSA,News Corp (Class A),stock,,NWS,News Corp (Class B),stock,,NEE,NextEra Energy,stock,,NKE,"Nike, Inc.",stock,,NI,NiSource,stock,,NDSN,Nordson Corporation,stock,,NSC,Norfolk Southern,stock,,NTRS,Northern Trust,stock,,NOC,Northrop Grumman,stock,,NCLH,Norwegian Cruise Line Holdings,stock,,NRG,NRG Energy,stock,,NUE,Nucor,stock,,NVDA,Nvidia,stock,,NVR,"NVR, Inc.",stock,,NXPI,NXP Semiconductors,stock,,ORLY,O’Reilly Automotive,stock,,OXY,Occidental Petroleum,stock,,ODFL,Old Dominion,stock,,OMC,Omnicom Group,stock,,ON,ON Semiconductor,stock,,OKE,Oneok,stock,,ORCL,Oracle Corporation,stock,,OTIS,Otis Worldwide,stock,,PCAR,Paccar,stock,,PKG,Packaging Corporation of America,stock,,PLTR,Palantir Technologies,stock,,PANW,Palo Alto Networks,stock,,PSKY,Paramount Skydance Corporation,stock,,PH,Parker Hannifin,stock,,PAYX,Paychex,stock,,PYPL,PayPal,stock,,PNR,Pentair,stock,,PEP,PepsiCo,stock,,PFE,Pfizer,stock,,PCG,PG&E Corporation,stock,,PM,Philip Morris International,stock,,PSX,Phillips 66,stock,,PNW,Pinnacle West Capital,stock,,PNC,PNC Financial Services,stock,,POOL,Pool Corporation,stock,,PPG,PPG Industries,stock,,PPL,PPL Corporation,stock,,PFG,Principal Financial Group,stock,,PG,Procter & Gamble,stock,,PGR,Progressive Corporation,stock,,PLD,Prologis,stock,,PRU,Prudential Financial,stock,,PEG,Public Service Enterprise Group,stock,,PTC,PTC Inc.,stock,,PSA,Public Storage,stock,,PHM,PulteGroup,stock,,PWR,Quanta Services,stock,,QCOM,Qualcomm,stock,,DGX,Quest Diagnostics,stock,,Q,Qnity Electronics,stock,,RL,Ralph Lauren Corporation,stock,,RJF,Raymond James Financial,stock,,RTX,RTX Corporation,stock,,O,Realty Income,stock,,REG,Regency Centers,stock,,REGN,Regeneron Pharmaceuticals,stock,,RF,Regions Financial Corporation,stock,,RSG,Republic Services,stock,,RMD,ResMed,stock,,RVTY,Revvity,stock,,HOOD,Robinhood Markets,stock,,ROK,Rockwell Automation,stock,,ROL,"Rollins, Inc.",stock,,ROP,Roper Technologies,stock,,ROST,Ross Stores,stock,,RCL,Royal Caribbean Group,stock,,SPGI,S&P Global,stock,,CRM,Salesforce,stock,,SNDK,Sandisk,stock,,SBAC,SBA Communications,stock,,SLB,Schlumberger,stock,,STX,Seagate Technology,stock,,SRE,Sempra,stock,,NOW,ServiceNow,stock,,SHW,Sherwin-Williams,stock,,SPG,Simon Property Group,stock,,SWKS,Skyworks Solutions,stock,,SJM,J.M. Smucker Company (The),stock,,SW,Smurfit Westrock,stock,,SNA,Snap-on,stock,,SOLV,Solventum,stock,,SO,Southern Company,stock,,LUV,Southwest Airlines,stock,,SWK,Stanley Black & Decker,stock,,SBUX,Starbucks,stock,,STT,State Street Corporation,stock,,STLD,Steel Dynamics,stock,,STE,Steris,stock,,SYK,Stryker Corporation,stock,,SMCI,Supermicro,stock,,SYF,Synchrony Financial,stock,,SNPS,Synopsys,stock,,SYY,Sysco,stock,,TMUS,T-Mobile US,stock,,TROW,T. Rowe Price,stock,,TTWO,Take-Two Interactive,stock,,TPR,"Tapestry, Inc.",stock,,TRGP,Targa Resources,stock,,TGT,Target Corporation,stock,,TEL,TE Connectivity,stock,,TDY,Teledyne Technologies,stock,,TER,Teradyne,stock,,TSLA,"Tesla, Inc.",stock,,TXN,Texas Instruments,stock,,TPL,Texas Pacific Land Corporation,stock,,TXT,Textron,stock,,TMO,Thermo Fisher Scientific,stock,,TJX,TJX Companies,stock,,TKO,TKO Group Holdings,stock,,TTD,Trade Desk (The),stock,,TSCO,Tractor Supply,stock,,TT,Trane Technologies,stock,,TDG,TransDigm Group,stock,,TRV,Travelers Companies (The),stock,,TRMB,Trimble Inc.,stock,,TFC,Truist Financial,stock,,TYL,Tyler Technologies,stock,,TSN,Tyson Foods,stock,,USB,U.S. Bancorp,stock,,UBER,Uber,stock,,UDR,"UDR, Inc.",stock,,ULTA,Ulta Beauty,stock,,UNP,Union Pacific Corporation,stock,,UAL,United Airlines Holdings,stock,,UPS,United Parcel Service,stock,,URI,United Rentals,stock,,UNH,UnitedHealth Group,stock,,UHS,Universal Health Services,stock,,VLO,Valero Energy,stock,,VEEV,Veeva Systems,stock,,VTR,Ventas,stock,,VLTO,Veralto,stock,,VRSN,Verisign,stock,,VRSK,Verisk Analytics,stock,,VZ,Verizon,stock,,VRTX,Vertex Pharmaceuticals,stock,,VRT,Vertiv,stock,,VTRS,Viatris,stock,,VICI,Vici Properties,stock,,V,Visa Inc.,stock,,VST,Vistra Corp.,stock,,VMC,Vulcan Materials Company,stock,,WRB,W. R. Berkley Corporation,stock,,GWW,W. W. Grainger,stock,,WAB,Wabtec,stock,,WMT,Walmart,stock,,DIS,Walt Disney Company (The),stock,,WBD,Warner Bros. Discovery,stock,,WM,Waste Management,stock,,WAT,Waters Corporation,stock,,WEC,WEC Energy Group,stock,,WFC,Wells Fargo,stock,,WELL,Welltower,stock,,WST,West Pharmaceutical Services,stock,,WDC,Western Digital,stock,,WY,Weyerhaeuser,stock,,WSM,"Williams-Sonoma, Inc.",stock,,WMB,Williams Companies,stock,,WTW,Willis Towers Watson,stock,,WDAY,"Workday, Inc.",stock,,WYNN,Wynn Resorts,stock,,XEL,Xcel Energy,stock,,XYL,Xylem Inc.,stock,,YUM,Yum! Brands,stock,,ZBRA,Zebra Technologies,stock,,ZBH,Zimmer Biomet,stock,,ZTS,Zoetis,stock,,ES=F,S&P 500 E-Mini Futures,future,CME,NQ=F,Nasdaq 100 E-Mini Futures,future,CME,YM=F,Dow E-Mini Futures,future,CME,RTY=F,Russell 2000 E-Mini Futures,future,CME,CL=F,Crude Oil Futures (WTI),future,NYMEX,BZ=F,Brent Crude Oil Futures,future,NYMEX,GC=F,Gold Futures,future,COMEX,SI=F,Silver Futures,future,COMEX,HG=F,Copper Futures,future,COMEX,NG=F,Natural Gas Futures,future,NYMEX,