repos

Dashboard: full-week chart frame after Friday's close

5ccf9479 by Isaac Bythewood · 1 month ago

Dashboard: full-week chart frame after Friday's close

From Friday 4pm ET through the weekend until Monday 7am ET, every
overview and watchlist chart spans the whole trading week (Mon 7am to
Fri 8pm ET) instead of just the last single Schwab day, and the headline
% becomes the full-week move (prior Friday's close to Friday's close)
rather than the one-day move. The x-axis labels weekdays in this mode.

The routine intraday poll only stores one day of 15-minute bars at a
time, so any weekday the dashboard was not open is missing from the week
view. A one-off guarded range=5d backfill (backfill_intraday_week),
fired detached from /api/dashboard/refresh, fills those gaps in the
background and skips symbols already covering the week.
modified CLAUDE.md
@@ -132,7 +132,8 @@ All free, no account, no API key. Every outbound call goes through the `Endpoint- **Guard state is shared across server + `seed` subcommand** via SQLite, and survives restarts. A boot-time breaker trip is normal after a deploy; it recovers via the half-open probe.- **Supertrend is drawn as a single line series coloured per data point** (one value/one colour per bar), so the two trend colours can never render at the same x; the band jumps sides at a flip. **Do not** use two whitespace-gapped series — lightweight-charts connects the line straight across the gaps and draws both colours at once (the "constantly green" bug).- **Yahoo NAV is only as fresh as you keep it.** Comparing a live price to a weeks-old NAV yields a meaningless premium/discount. The ETF quality read's tracking factor gates on `nav_synced_at` freshness and drops to `—` rather than assert a bogus premium. NAV is fetched on demand when an ETF page is viewed and stale.- **The index overview slots are a hybrid by design.** The chart *line* is the E-mini future (so pre/regular/after-hours all move), while the headline value + % are the **cash index** quote (`regularMarketPrice` vs `chartPreviousClose`) so the number matches everyone (e.g. S&P 500 +0.41% = `^GSPC`). On big-basis days the line (futures) can diverge from the headline (cash) after 4pm — that's expected, not a bug. Each chart frames exactly one Schwab day (regular + extended hours, 7:00 AM–8:00 PM ET) of the most recent session.- **The index overview slots are a hybrid by design.** The chart *line* is the E-mini future (so pre/regular/after-hours all move), while the headline value + % are the **cash index** quote (`regularMarketPrice` vs `chartPreviousClose`) so the number matches everyone (e.g. S&P 500 +0.41% = `^GSPC`). On big-basis days the line (futures) can diverge from the headline (cash) after 4pm — that's expected, not a bug. Each chart normally frames exactly one Schwab day (regular + extended hours, 7:00 AM–8:00 PM ET) of the most recent session.- **The dashboard switches to a full-week frame after Friday's close.** From Friday 4:00 PM ET through the weekend until Monday 7:00 AM ET (`week_window` in `routes/home.rs`), every overview + watchlist chart spans the whole trading week (Mon 7 AM → Fri 8 PM ET) and the headline % is the full-week move (prior Friday's daily close → Friday's close), not the one-day move. Because the routine intraday poll only stores one day of 15-minute bars at a time, a one-off guarded `range=5d` backfill (`scheduler::backfill_intraday_week`, fired detached from `/api/dashboard/refresh`) fills any week-days the dashboard was not open for; it skips symbols already covering the week, so it is a no-op once filled.- **`cargo` isn't on PATH in this dev container** — use `~/.cargo/bin/cargo`, and run the dev binary with `FINANCE_ROOT=/home/dev/code/finance` (or from the project dir so paths resolve from cwd).## Tooling
modified frontend/static_src/home/scripts/hero.js
@@ -67,6 +67,18 @@ function fmtAxisTime(tSec) {    hour12: true,  });}function fmtWeekday(tSec) {  return new Date(tSec * 1000).toLocaleDateString("en-US", {    timeZone: "America/New_York",    weekday: "short",  });}// Axis tick formatter for the end-of-week full-week frame: label day-boundary// ticks (DayOfMonth and coarser) with the weekday, intraday ticks with the time,// so a Mon→Fri frame reads "Mon … 12 PM … Tue …" instead of repeating times.function fmtWeekTick(tSec, tickMarkType) {  return tickMarkType <= 2 ? fmtWeekday(tSec) : fmtAxisTime(tSec);}function fmtCrosshairTime(tSec) {  return new Date(tSec * 1000).toLocaleString("en-US", {    timeZone: "America/New_York",
@@ -369,6 +381,8 @@ function drawSeries(entry, s) {    crosshairMarkerBackgroundColor: color,  });  entry.chart.applyOptions({    // A full-week frame labels the axis by weekday; a single day, by time.    timeScale: { tickMarkFormatter: s.week ? fmtWeekTick : (t) => fmtAxisTime(t) },    localization: { priceFormatter: (v) => fmtValue(v, s.unit), timeFormatter: (t) => fmtCrosshairTime(t) },  });
modified src/providers/yahoo.rs
@@ -241,11 +241,19 @@ impl YahooProvider {    /// explicit rate-limit signal, surfaced as the typed [`RateLimited`] so the    /// endpoint guard trips its breaker at once.    async fn fetch_chart(&self, ticker: &str) -> Result<Option<ChartResult>> {        self.fetch_chart_range(ticker, "1d").await    }    /// Fetch the v8 chart payload for `ticker` over an explicit intraday `range`    /// (e.g. `1d` for the routine quote, `5d` to backfill the whole trading week    /// for the end-of-week dashboard view). `interval=15m` is held constant, so    /// `5d` returns five trading days of 15-minute bars in one request.    async fn fetch_chart_range(&self, ticker: &str, range: &str) -> Result<Option<ChartResult>> {        // `^` is not a bare path character; percent-encode the symbol.        let sym = urlencoding::encode(&yahoo_symbol(ticker)).into_owned();        let url = format!(            "https://query1.finance.yahoo.com/v8/finance/chart/{sym}\             ?interval=15m&range=1d&includePrePost=true"             ?interval=15m&range={range}&includePrePost=true"        );        self.request_chart(&url).await    }
@@ -969,6 +977,19 @@ impl QuoteProvider for YahooProvider {    }}impl YahooProvider {    /// Fetch a wider intraday window (e.g. `range=5d`) of 15-minute bars in one    /// request, used to backfill the whole trading week for the end-of-week    /// dashboard view. Same shape as [`Self::quote`] (live quote + bars); callers    /// store only the bars.    pub async fn intraday_window(&self, ticker: &str, range: &str) -> Result<QuoteData> {        match self.fetch_chart_range(ticker, range).await? {            Some(result) => chart_to_quote_data(ticker, result),            None => Err(anyhow!("yahoo returned no chart result for {ticker}")),        }    }}#[async_trait]impl HistoryProvider for YahooProvider {    fn name(&self) -> &'static str {
modified src/routes/home.rs
@@ -97,6 +97,24 @@ fn overview_tickers() -> Vec<&'static str> {const SCHWAB_OPEN_MIN: u32 = 7 * 60; // 7:00 AM ETconst SCHWAB_CLOSE_MIN: u32 = 20 * 60; // 8:00 PM ET/// Once the regular session closes on Friday (4:00 PM ET) the dashboard switches/// from the single-day frame to the whole trading week (Mon 7 AM → Fri 8 PM ET),/// so the weekend read shows how the full week went, not just where Friday landed./// It reverts to the single-day frame at Monday's extended-hours open (7:00 AM ET).const FRIDAY_CLOSE_MIN: u32 = 16 * 60; // 4:00 PM ET/// Epoch-ms for `min` minutes-of-day on the ET calendar `date`. Picks the earlier/// instant on a fall-back DST repeat; both are fine for these window bounds.fn et_ms(date: chrono::NaiveDate, min: u32) -> Option<i64> {    use chrono::TimeZone as _;    use chrono_tz::America::New_York;    let naive = date.and_hms_opt(min / 60, min % 60, 0)?;    New_York        .from_local_datetime(&naive)        .earliest()        .map(|dt| dt.timestamp_millis())}/// The Schwab trading-day window [open, close] in epoch-ms for the ET calendar/// day that `latest_ms` falls in (so a Friday-evening view frames Friday, a/// weekend view still frames Friday's last session, etc.).
@@ -104,15 +122,36 @@ fn schwab_day_window(latest_ms: i64) -> Option<(i64, i64)> {    use chrono::TimeZone as _;    use chrono_tz::America::New_York;    let date = New_York.timestamp_millis_opt(latest_ms).single()?.date_naive();    let at = |min: u32| {        let naive = date.and_hms_opt(min / 60, min % 60, 0)?;        // Pick the earlier instant on a fall-back DST repeat; both are fine here.        New_York            .from_local_datetime(&naive)            .earliest()            .map(|dt| dt.timestamp_millis())    Some((et_ms(date, SCHWAB_OPEN_MIN)?, et_ms(date, SCHWAB_CLOSE_MIN)?))}/// The full-week window when the end-of-week view is active, else `None`.////// Active from Friday's regular close (4:00 PM ET) through the weekend until/// Monday's extended-hours open (7:00 AM ET). When active it frames the trading/// week that just ended: Monday 7:00 AM → Friday 8:00 PM ET. Returns that window/// plus the Monday `NaiveDate` (the caller reads the prior Friday's close — the/// last daily close strictly before Monday — as the week's % base).fn week_window(now_ms: i64) -> Option<(i64, i64, chrono::NaiveDate)> {    use chrono::{Datelike as _, Duration, TimeZone as _, Timelike as _, Weekday};    use chrono_tz::America::New_York;    let now = New_York.timestamp_millis_opt(now_ms).single()?;    let minutes = now.hour() * 60 + now.minute();    // How many days back the just-closed Friday sits from `now`'s ET date.    let days_back = match now.weekday() {        Weekday::Fri if minutes >= FRIDAY_CLOSE_MIN => 0,        Weekday::Sat => 1,        Weekday::Sun => 2,        Weekday::Mon if minutes < SCHWAB_OPEN_MIN => 3,        _ => return None,    };    Some((at(SCHWAB_OPEN_MIN)?, at(SCHWAB_CLOSE_MIN)?))    let friday = now.date_naive() - Duration::days(days_back);    let monday = friday - Duration::days(4);    Some((        et_ms(monday, SCHWAB_OPEN_MIN)?,        et_ms(friday, SCHWAB_CLOSE_MIN)?,        monday,    ))}/// The value unit for a symbol: index points for equity indexes, dollars for
@@ -186,11 +225,15 @@ struct Series {    change_pct: f64,    /// True when the day move is not negative — drives the green/red line colour.    up: bool,    /// UNIX seconds bounding the Schwab trading-day frame (extended-hours open    /// and close). The chart always spans exactly this one day, so a partial day    /// plots from the left rather than stretching across the width.    /// UNIX seconds bounding the chart frame (extended-hours open and close).    /// Normally a single Schwab day; in end-of-week mode the whole trading week    /// (Mon 7 AM → Fri 8 PM ET). A partial frame plots from the left rather than    /// stretching across the width.    start_t: i64,    end_t: i64,    /// True when the frame spans the whole week (Fri 4 PM → Mon 7 AM ET), so the    /// chart axis labels days instead of just times.    week: bool,    points: Vec<SeriesPoint>,}
@@ -319,7 +362,8 @@ async fn dashboard_refresh(State(state): State<AppState>, headers: HeaderMap) ->    let session = watchlist::resolve(&state.pool, &headers).await;    // The watchlist cards, the session's overview symbols, and the VIX / volume    // reads — everything the open dashboard shows gets a fresh quote.    let mut tickers = watchlist::list(&state.pool, &session.sid).await;    let wl = watchlist::list(&state.pool, &session.sid).await;    let mut tickers = wl.clone();    for t in overview_tickers() {        tickers.push(t.to_string());    }
@@ -329,6 +373,33 @@ async fn dashboard_refresh(State(state): State<AppState>, headers: HeaderMap) ->    let refreshed =        crate::scheduler::refresh_quotes(&state.pool, &state.config, &state.hub, &tickers).await;    // In the end-of-week view the charts span Mon–Fri, but the routine poll only    // ever stores one day of 15-minute bars at a time, so any day the dashboard    // wasn't open is missing. Backfill the whole week (one guarded range=5d pull    // per still-incomplete symbol) for the symbols actually drawn as charts: the    // overview's chart tickers (the futures lines) and the watchlist. It runs    // detached — the paced, guarded pulls take many seconds and only need to    // happen once per weekend, so we don't hold the refresh request open for    // them; the filled bars land on the next ~60s dashboard poll. Already-covered    // symbols are skipped, so this is a no-op once the week is complete.    if let Some((start_ms, end_ms, _monday)) =        week_window(chrono::Utc::now().timestamp_millis())    {        let mut charted: Vec<String> = OVERVIEW.iter().map(|s| s.chart.to_string()).collect();        charted.extend(wl.iter().cloned());        let bg = state.clone();        tokio::spawn(async move {            crate::scheduler::backfill_intraday_week(                &bg.pool,                &bg.config,                &charted,                start_ms,                end_ms,            )            .await;        });    }    let mut resp = Json(serde_json::json!({ "refreshed": refreshed })).into_response();    if let Some(c) = session.set_cookie {        if let Ok(v) = header::HeaderValue::from_str(&c) {
@@ -361,17 +432,25 @@ async fn overview_series(    name: &str,    dollar: bool,) -> Option<Series> {    // Anchor to the Schwab day that the chart ticker's most recent bar falls in,    // so the chart shows just that one day and never the previous one.    let latest_ms: i64 =        sqlx::query_scalar("SELECT MAX(ts) FROM intraday_bars WHERE ticker = ?")            .bind(chart_ticker)            .fetch_optional(&state.pool)            .await            .ok()            .flatten()            .flatten()?;    let (start_ms, end_ms) = schwab_day_window(latest_ms)?;    // After Friday's close the chart frames the whole trading week; the rest of    // the time it anchors to the Schwab day the chart ticker's most recent bar    // falls in, so it shows just that one day and never the previous one.    let now_ms = chrono::Utc::now().timestamp_millis();    let (start_ms, end_ms, week_monday) = match week_window(now_ms) {        Some((s, e, mon)) => (s, e, Some(mon)),        None => {            let latest_ms: i64 =                sqlx::query_scalar("SELECT MAX(ts) FROM intraday_bars WHERE ticker = ?")                    .bind(chart_ticker)                    .fetch_optional(&state.pool)                    .await                    .ok()                    .flatten()                    .flatten()?;            let (s, e) = schwab_day_window(latest_ms)?;            (s, e, None)        }    };    let rows: Vec<(i64, f64, f64)> = sqlx::query_as(        "SELECT ts, open, close FROM intraday_bars \
@@ -406,11 +485,29 @@ async fn overview_series(    .flatten();    let (last_price, prev_close) = quote.unwrap_or((None, None));    // Last value: the quote price, else the last drawn bar's close.    // Last value: the quote price, else the last drawn bar's close. Over the    // weekend the quote price is Yahoo's frozen Friday close — exactly the week's    // end value we want.    let last = last_price.or_else(|| rows.last().map(|r| r.2))?;    // Reference: the previous close, else the session's first-bar open.    // Reference: the session's first-bar open (also the % base's last-resort).    let open = rows.first().map(|r| if r.1 > 0.0 { r.1 } else { r.2 });    let base = prev_close.filter(|p| *p > 0.0).or(open)?;    // The % base. Single-day mode: the previous close (the universally-quoted    // day move). Week mode: the prior Friday's close — the last daily close    // strictly before this week's Monday — so the move is the full-week change.    let base = if let Some(monday) = week_monday {        let prior_close: Option<f64> = sqlx::query_scalar(            "SELECT close FROM daily_prices WHERE ticker = ? AND d < ? ORDER BY d DESC LIMIT 1",        )        .bind(quote_ticker)        .bind(monday.to_string())        .fetch_optional(&state.pool)        .await        .ok()        .flatten();        prior_close.filter(|p| *p > 0.0).or(open)?    } else {        prev_close.filter(|p| *p > 0.0).or(open)?    };    if base <= 0.0 {        return None;    }
@@ -429,6 +526,7 @@ async fn overview_series(        up: change_pct >= 0.0,        start_t: start_ms / 1000,        end_t: end_ms / 1000,        week: week_monday.is_some(),        points,    })}
modified src/scheduler.rs
@@ -1529,6 +1529,59 @@ async fn refresh_quote(pool: &SqlitePool, config: &Config, hub: &Hub, ticker: &s    }}/// Intraday range that covers a whole trading week (Mon–Fri) of 15-minute bars/// in one request — enough for the end-of-week dashboard view.const INTRADAY_WEEK_RANGE: &str = "5d";/// Backfill the whole trading week's 15-minute bars for `tickers` when the/// stored bars don't already cover the early week. The routine intraday poll/// only ever stores one day at a time (`range=1d`), so the end-of-week view is/// missing any day the dashboard wasn't open. One guarded `range=5d` request per/// still-incomplete symbol fills the gap; symbols whose stored bars already/// reach the week's start are skipped, so a reload doesn't re-hit Yahoo.pub(crate) async fn backfill_intraday_week(    pool: &SqlitePool,    config: &Config,    tickers: &[String],    week_start_ms: i64,    week_end_ms: i64,) -> usize {    // "Already covered": the earliest in-window bar sits within ~36h of the    // week's open. A normally-polled week starts at Monday's open; a    // holiday-Monday week at Tuesday's — both inside this margin, so they're not    // refetched. Only a week missing its first two days (the gap the user sees)    // falls outside it.    let covered_before = week_start_ms + 36 * 3_600 * 1000;    let mut filled = 0;    for t in tickers {        let earliest: Option<i64> = sqlx::query_scalar(            "SELECT MIN(ts) FROM intraday_bars WHERE ticker = ? AND ts >= ? AND ts <= ?",        )        .bind(t)        .bind(week_start_ms)        .bind(week_end_ms)        .fetch_optional(pool)        .await        .ok()        .flatten()        .flatten();        if matches!(earliest, Some(ms) if ms <= covered_before) {            continue;        }        let yahoo = YahooProvider::new(providers::http::build_client(config));        let guard = EndpointGuard::with_budget(pool.clone(), "yahoo", YAHOO_BUDGET);        match guarded(&guard, yahoo.intraday_window(t, INTRADAY_WEEK_RANGE)).await {            Some(Ok(data)) if !data.bars.is_empty() => {                let _ = store_intraday(pool, t, &data.bars).await;                filled += 1;            }            Some(Err(e)) => tracing::warn!("[week] intraday {t}: {e:#}"),            _ => {}        }    }    filled}/// Pull the daily history a viewed symbol is missing: the window since its last/// stored bar (incremental) when it already has history, else a full/// `range=max` backfill. Cheaper than the deep re-fetch on a routine load.