Power Ratings
| # | Team |
Net Rating
|
Offense | Defense | SOS | Playoff Odds |
|---|
Week Board
My Picks Log
Season Accuracy
Team Deep-Dive
Model Lab
| Experiment | Result | Decision |
|---|---|---|
| QB feature (trailing EPA/dropback) | Accuracy 60.0% to 62.6%, biggest single gain of the project | ACCEPT |
| Ridge alpha tuning (200 -> 15) | Real backtest sweep across 1-1600, found interior optimum | ACCEPT |
| Recency half-life tuning (6/10/16) | 16 games won on log loss | ACCEPT |
| Rest days, weather, travel distance, altitude | All flat or worse across every framing tested | REJECT |
| Divisional games, trap/look-ahead games, blowout letdown | No real signal beyond existing team-strength features | REJECT |
| Gradient boosting vs. logistic regression | +0.83pt accuracy but AUC drops 0.670→0.654; bootstrap CI included zero | REJECT |
| QB Shrinkage Re-tune (k=96, then k=128) | Investigation, not a clean win: the original k=8->96 decision didn't reproduce once OL removal + weekly refit changed the model underneath it (a real shrinkage-target leak was also found and fixed along the way, see ratings_engine.py). A fresh retune under the fixed, current config picked k=128 on validation, but on confirmatory data k=8 won 3 of 4 metrics (accuracy, Brier, AUC) against k=128, which only edged log loss -- bootstrap CI included zero either way. Reverted to k=8, the current live default. | INCONCLUSIVE |
| QB quality-differential change signal | Identical to no-signal-at-all; binary flag clearly better | REJECT |
| Team-specific home-field advantage | Split-half reliability 0.078 -- mostly noise, not a real trait | REJECT |
| Pass/rush-split matchup features | Worse on every metric; 0.947 correlation with existing aggregate | REJECT |
| All 9 rejected features combined, regularized | No C value beat baseline even with proper scaling | REJECT |
| O-line continuity | Bootstrap CI included zero; later beaten by "no OL feature" on confirmatory test | REMOVED (was live) |
| Coaching changes (5 framings) | Every variant tied or underperformed baseline | REJECT |
| Early-season extra shrinkage | Real gap confirmed (61.7% vs 65.9%) but fix didn't close it | REJECT |
| Weekly-refitting backtest methodology | Tied-or-better on every metric, free, more methodologically correct | ACCEPT |
| Platt scaling / isotonic calibration | Platt worse on every metric; isotonic's log loss broke (0.64→1.00) from overconfident extremes | REJECT |
| Player availability / injury model | Re-confirmed 2026-08-25 via nflverse's own status page: "no 2025 data, no ETA." Still genuinely blocked, not just stale. | DEFERRED |
| Dynamic Model A/B weighting by disagreement magnitude | High-disagreement bucket's B-edge is real (bootstrap CI excludes zero), but B already wins in every bucket -- no dynamic scheme beat simply always using Model B on confirmatory data | REJECT |
| Margin-of-victory modeling (regression -> sign -> winner) | -1.56pt vs. current classifier; CI [-3.13, +0.09] nearly significant negative. Regression's MSE objective isn't aligned with getting the sign right on close games -- broke 50 previously-correct calls to fix only 33 | REJECT |
| Snap-share "key player disruption" signal (Objective 3 follow-up) | INVALIDATED, not rejected -- used this-week's own snap percentage as a feature, which is only knowable after that week's game is played. Confirmed via direct correlation with the very outcome it claimed to predict. A real mistake, caught before shipping. Depends on Objective 2 (blocked) to do legitimately. | LEAKAGE -- INVALIDATED |
| CPOE-based QB rating (alone and blended with EPA) | CPOE-only clearly worse on every metric; EPA+CPOE blend just flat/redundant. CPOE is stable but a weaker predictor of wins than EPA specifically | REJECT |
| Rookie-specific QB shrinkage (hierarchical refinement of k=96) | +0.92pt on validation looked promising, reversed to -0.74pt on true confirmatory test (CI includes zero). A real example of why validation-then-confirm matters | REJECT |
| Nonlinear architectures (GBM, Random Forest) | RF's +0.55pt accuracy edge not significant (CI [-2.57,+3.68]) and comes with worse log loss/Brier/AUC -- same pattern GBM showed in the first cycle. XGBoost/LightGBM untestable, no network access to install | REJECT |
| Early-season weakness diagnosis | Not a broad phenomenon -- concentrated in early-season QB changes specifically (48.0% accuracy, n=25) vs. 62.8% for early-season games without a QB change. Real mechanism found, targeted fix not yet built | DIAGNOSED |
| Early-season QB-change interaction term | +0.37pt confirmatory, CI [-0.37,+1.10] includes zero. Real mechanism behind it but only 25 games -- genuinely can't distinguish from noise yet | INCONCLUSIVE |
| Schedule-strength trajectory (next-3-games opponent difficulty) | +0.18pt, CI [-1.65,+2.02], validation negative. Different mechanism from the earlier-rejected single-game trap-game test, same null result | REJECT |
| Referee/officiating crew tendencies | Split-half reliability 0.252 (real, better than team home-field's 0.078) but translates to zero predictive value once shrunk properly -- flat across every metric | REJECT |
| weekly_rosters as an injury-data alternative | Confirmed NOT gameday active/inactive status via real player history checks (static ACT/INA over many weeks, not game-to-game). No inactives-list loader exists anywhere in nflreadpy -- Objective 2/3 blocker confirmed structural, not just stale data | DEFERRED |
| FTN charting QB-fault-rate (sacks/INTs attributable to the QB) | Completely flat -- identical to baseline on every metric. EPA already captures the cost of a bad play regardless of whose fault it was | REJECT |
| Total (over/under) line as a Model B input | Small, mixed-direction result -- accuracy nudges up slightly but log loss/Brier/AUC move marginally worse. CI [-0.37,+0.74], includes zero | INCONCLUSIVE |
| QB-change x spread interaction (re-tested under current leak-fixed code) | Ties baseline exactly on confirmatory accuracy -- re-confirms the earlier rejection still holds after the leak fix and shrinkage revert | REJECT |
| Off_matchup x spread interaction | Clean null -- identical confirmatory accuracy to baseline | REJECT |
| Stacking ensemble (Model A probability + market probability as a 2-input meta-model) | Collapses to being statistically identical to market alone -- the meta-model discounts Model A's weaker signal almost entirely. Current flat 5-feature Model B beats it on every confirmatory metric (CI [-1.10,+0.55], not significant, but consistently negative) | REJECT |
| Market efficiency trend across seasons (2022-2025) | No real trend -- market's edge over Model A bounces from +1.1pt to +9.9pt with no consistent direction, tracking Model A's own year-to-year variance rather than any change in the market itself | NO PATTERN |
| First-half vs. second-half season market efficiency | Market's edge narrows slightly (+5.5 to +4.2) as the season progresses, but this tracks Model A's own known early-season weakness rather than revealing anything new about the market's own behavior | NOT A NEW FINDING |
| Market accuracy vs. travel distance | No pattern across distance buckets (64-70% range, no trend), and the largest-distance bucket has only 23 games -- too small to trust on its own | REJECT |
| Season-boundary-specific recency half-life | Tested extra discounting (0.85, 0.7, 0.5) for plays crossing a season boundary, on top of normal recency decay. Every level was flat-or-worse than baseline on every metric, both splits, with a clean monotonic pattern -- stronger discount, worse performance. A team's true strength doesn't reset at a season boundary the way this assumed | REJECT |
| Referee crew x team-penalty-rate interaction | A narrower framing than the earlier referee-tendency test: does a specific crew's penalty-calling rate interact with how penalty-prone the teams are? CI [-1.10,+0.74], includes zero, worse on validation too | REJECT |
| Systematic narrow-edge search (12 partitions) | Scanned disagreement magnitude, spread size, QB-change presence, home favorite/underdog, and model confidence for ANY subset where Model A beats the market. Best candidate was +0.74pt with CI [-2.60,+4.09] -- and with 12 comparisons tested, one weak positive is exactly what noise produces. No defensible narrow edge exists | REJECT |
| Model A's low-confidence games are genuinely unreliable | Re-derived from scratch 2026-09-06 and it holds. Across the 343 of 1,087 backtested games where Model A's own probability sits within 0.05 of a coin flip, Model A hits 52.48% while the market hits 64.14% on those same games — a −11.66pt gap, CI [−18.37, −5.25], which excludes zero. This row previously carried 52.5% / 64.4% / −11.95pt with no script anywhere in the repository that produced them; Booth flagged that as UNVERIFIABLE while auditing PR #18, and it was right to. src/verify_low_confidence_finding.py now regenerates the whole thing, and the original figures agreed to within 0.3pt — a gap consistent with the cross-platform accuracy sensitivity documented below, since this claim is stated in accuracy. Model A's uncertainty is trustworthy: when it says it doesn't know, it really doesn't | CONFIRMED FINDING |
| Line-movement archive (infrastructure check) | Confirmed working end-to-end -- real movement captured (CLE@TB moved a full point, JAX@DEN a half point). But only 2 of 32 games moved at all across consecutive days, so a meaningful predictive test needs many more weeks of accumulation, not a forced test on n=2 | DEFERRED |
| Against-the-spread evaluation (would this have made money?) | The betting question, asked for the first time on 2026-09-05. Model B outputs a win probability, not a margin, so it cannot make an ATS pick directly — and margin-of-victory modelling is already rejected above. The well-posed version is a different target on the same features: fit the weekly-refit walk-forward on home covered instead of home won. Result over 1,058 games: 51.61% against the spread, 95% CI [48.58%, 54.63%]. The interval contains 50%, and it does not reach the 52.38% needed to break even at −110. Three controls (always home, always away, always the favourite) all landed at exactly 50.00%, confirming the cover arithmetic, and the spread-to-margin correlation (+0.455) is asserted at runtime so an inverted sign convention cannot silently flip the result. Bootstrap and Wilson intervals agree to 0.02pt. Deliberately no subgroup search: the earlier 12-partition narrow-edge hunt already showed what that produces on noise | REJECT — NO ATS EDGE |
| Model B vs. market, paired bootstrap on the calibration gaps | The reliability diagram showed Model B with more resolution than the market (0.039321 vs 0.038180) and worse reliability (0.001697 vs 0.001334). A 5,000-resample paired bootstrap over the same 1,087 games says neither gap is real: Brier −0.000622 CI [−0.002395, +0.001141], log loss −0.001189 CI [−0.005156, +0.002832], reliability +0.000362 CI [−0.002010, +0.003052], resolution +0.001141 CI [−0.002196, +0.004597]. All four include zero. The chart's apparent split between "sharper" and "worse calibrated" was noise, and the panel has been rewritten to say so. Cross-checked against a closed-form paired standard error on Brier, which agrees to 3e-5 | INCONCLUSIVE |
| Model B beats Model A on proper scoring rules (not just accuracy) | Same paired bootstrap, Model B vs Model A: Brier −0.019133 CI [−0.025838, −0.012687], log loss −0.043603 CI [−0.058621, −0.029231], resolution +0.016814 CI [+0.008990, +0.024819] — three intervals clearly excluding zero. The project's existing claim for the spread's value rested on 68.2% vs 62.8% accuracy, and accuracy has since been shown to move by a full game between platforms on identical code. This is the same conclusion resting on metrics that survived that finding. Reliability alone is inconclusive (CI [−0.005963, +0.002452]): the spread makes Model B sharper, not better calibrated | CONFIRMED FINDING |
| The market genuinely beats Model A (now with intervals) | Model A vs market, signed so a positive number means Model A is worse: Brier +0.018511 CI [+0.011496, +0.025553], log loss +0.042414 CI [+0.026992, +0.058291], resolution −0.015674 CI [−0.024072, −0.007423]. "We do not beat the market" has been this project's headline honest finding from the start; this is the first time it has been stated with confidence intervals rather than a raw accuracy gap. The market's edge is resolution — it separates games better — not calibration, where the two are indistinguishable | CONFIRMED FINDING |
| Backtest accuracy is not reproducible across platforms | The same commit and the same 1599-game dataset, run on a GitHub Actions Linux runner and on a local Windows machine, disagree: Model A 0.6283 vs 0.6274, Model B 0.6817 vs 0.6826 -- gaps of exactly one to two games out of ~1087. Each machine is perfectly deterministic when re-run (byte-identical across every metric), so this is a difference between platforms, not run-to-run noise; almost certainly floating-point/BLAS. Log loss, Brier and AUC agree to four decimals across those same two runs. Accuracy is a threshold metric -- it only records which side of 0.5 a probability lands on -- so perturbations around 1e-6 flip games sitting near a coin flip while every calibration metric holds steady. Practical consequence: "Model B adds zero accuracy over the raw spread" is true on Linux and false on Windows, by one game. That claim has to be stated as indistinguishable, and accuracy alone cannot carry a result at this sample size | CONFIRMED FINDING |
Version History
Initial 3-feature model: opponent-adjusted offense, defense, QB rating.
Added O-line continuity and QB-change detection (5 features).
QB shrinkage constant retuned (k=8 → k=96) via proper validation/test split.
O-line continuity removed after two independent negative tests (4 features: off, def, qb, qb-change).
Backtest methodology switched to weekly refitting, matching what the live pipeline already did in production.
Fixed a real leak in the QB rating's shrinkage target (was a global average, not cutoff-scoped -- see ratings_engine.py); re-tuned QB_SHRINK_K 96 -> 128 via a real, committed, reproducible script (tune_qb_shrink_k.py), superseding the prior retune whose own numbers couldn't be reproduced from this repo.
QB shrinkage investigation: the fresh k=128 retune turned out inconsistent on its own confirmatory data (k=8 won 3 of 4 metrics). Reverted to k=8 as the defensible default. Current.
Methodology
How the model works
The whole method, in enough detail that a sceptical reader could rebuild it.
Model A vs. Model B — What Actually Separates Them
Model A (football-only) is built entirely from what we can compute ourselves: opponent-adjusted team ratings, per-QB performance, and QB-change detection — all derived from real play-by-play data, with zero market information involved. Its job is to answer "what does the football itself say," independent of what oddsmakers think. This is the model we've actually validated feature-by-feature, backtested, and can fully explain the reasoning behind (every game on the Week Board says in a sentence which of these is driving it, with the underlying figures one tap away).
Model B (market-blended) takes everything Model A knows and adds the current Vegas spread as an additional input. The spread reflects information Model A structurally can't see — injuries our data doesn't capture, weather forecasts, sharp money, line moves in response to breaking news. In our backtest, Model B consistently outperforms Model A (68.2% vs. 62.8% accuracy) precisely because the market prices in real-world information our football-only features are blind to. The honest tradeoff: Model B's edge comes mostly from the market itself, not from our own modeling — so it's the stronger predictor, but leaning on it means leaning on Vegas, not on what's unique about this project.
Data
Real nflverse play-by-play, filtered to pass/rush attempts with valid EPA, used to fit opponent-adjusted team ratings and per-QB ratings.
Data pipeline migrated from nfl_data_py to nflreadpy (2026-08) after nfl_data_py was officially deprecated. Verified column-schema match and exact value match (scores, spreads, EPA, snap counts) on a real completed week before merging -- see What's Changed.
Rating System — Opponent-Adjusted, Recency-Weighted
A two-way fixed-effects ridge regression: play-level EPA regressed on offense-team dummies + defense-team dummies, using only plays strictly before the point being rated. Each play is recency-weighted. A team's offensive rating already accounts for the strength of every defense it faced.
Backtest Results — Chronological, Weekly-Refitting Walk-Forward
Trained only on strictly earlier data, refit every week within the test season (not just once per season) -- validated 2026-08 (Stage 9) to match what the live model actually does: weekly_update.py always trains on all real completed games available up to "now," every single run. Evaluated on 2022-2025 (1,087 games):
| Model | Accuracy | Log Loss | Brier | AUC |
|---|---|---|---|---|
| Coin flip | 50.0% | 0.693 | 0.250 | 0.500 |
| Home team always wins | 54.6% | - | - | - |
| 3-feature (off+def+QB) only | 62.4% | 0.654 | 0.231 | 0.662 |
| Model A -- live, 4 features, weekly refit | 62.8% | 0.650 | 0.229 | 0.670 |
| Vegas market alone | 68.2% | 0.607 | 0.210 | 0.725 |
| Model B -- live, 4 features + market, weekly refit | 68.2% | 0.606 | 0.209 | 0.727 |
Update (2026-08-31): these numbers reflect two real changes, not a silent swap. First, a leak in the QB rating's shrinkage target was found and fixed (it was averaging in future weeks instead of only strictly-earlier ones -- see ratings_engine.py). Second, we revisited our own earlier QB-shrinkage tuning decision (k=8 -> 96) after it turned out not to reproduce, re-ran the retune under the fixed, current config, and found the fresh result (k=128) didn't hold up consistently on confirmatory data either -- so we reverted to the original k=8. See Model Lab, "QB Shrinkage Re-tune," for the full investigation.
Update (2026-08, research Stage 6): O-line continuity was removed from the live model after two independent negative results -- a paired bootstrap found its individual contribution statistically indistinguishable from zero (95% CI included zero), and a later confirmatory test found "no OL feature at all" outperformed both the original and a smoothed version of it on held-out 2024-2025 data. It's still computed and logged as informational data, just no longer fed into the model, in case future evidence changes this. QB-change detection has held up across every test run against it and remains live.
Separately: the market barely changes when our full feature set is blended in (68.2% either way on raw accuracy, marginally better on log loss/Brier/AUC). The Vegas line still captures almost everything we can see, plus real-world information we structurally can't.
How We Know a Result Is Real (Not Just Luck)
Every experiment on Model Lab reports a real accuracy number, but a small difference between two numbers doesn't automatically mean one is genuinely better -- with only a few hundred games in a season, some of that difference is just random luck in which games happened to go a certain way.
To check, we use a technique called bootstrap resampling: take the real games, randomly reshuffle which ones get counted (with repeats allowed) thousands of times, and see how much the accuracy difference bounces around. If it consistently stays positive across nearly all of those reshuffles, that's real signal. If it swings between positive and negative depending on which games got picked, that's a sign that we can't confidently tell the difference from noise yet -- even if the original number looked good.
This is why you'll see "CI [-1.2, +3.4]" type notation on Model Lab -- that's the range the true effect most likely falls in. When that range includes zero, we mark it INCONCLUSIVE rather than pretending we know for sure.
Which Features Actually Do the Work
Most of this project's research has tested whether adding something helps. A formal ablation pass asks the opposite question: remove each existing feature one at a time, fully retrain, and measure what actually breaks. That's how you find out where a model's predictive power really lives, rather than assuming every feature is pulling its weight.
| Feature removed | Accuracy change | 95% CI |
|---|---|---|
| QB matchup | -2.39pt | [-5.15, +0.37] |
| QB change | -0.55pt | [-1.93, +0.83] |
| Offense matchup | -0.37pt | [-1.20, +0.46] |
| Defense matchup | +0.37pt | [-0.83, +1.56] |
QB quality is doing most of the work -- removing it costs more than four times what any other feature costs. That matches football intuition, and it's why so much of this project's research effort has gone into getting the QB rating right.
Two honest caveats. First, none of these individually reach statistical significance -- every confidence interval includes zero. That's expected with correlated features: the model partially compensates for a missing one using the others, so no single removal produces a clean, sharp drop. Second, defense matchup's point estimate is actually positive when removed, though well within noise. It isn't clearly earning its place, but there's no real evidence it's hurting either -- so it stays, and this is noted rather than acted on.
Why the Confidence Ranking Is the Right Strategy, Not Just a Guess
For a standard confidence pool -- rank your picks, assign points 1 through N, earn those points if correct -- the strategy of assigning the highest point value to your highest-confidence pick, and so on down the list, isn't just a reasonable habit. It's mathematically provable as the strategy that maximizes your expected total points, assuming the underlying probabilities are well-calibrated.
This follows from the rearrangement inequality: when you're pairing two ordered lists (win probabilities and point values) to maximize their summed product, pairing largest-with-largest and smallest-with-smallest always beats any other pairing. We checked this isn't just theory -- a real simulation compared the rank-order strategy against 20,000 random point assignments for a realistic 16-game week, and rank-ordering beat every single one.
What this means practically: the confidence ranking you see on the Week Board isn't a stylistic choice or a rough heuristic -- it's the actual optimal strategy for this pool format, given whatever our probabilities say. The only way to do better is to have better probabilities, not a cleverer ranking scheme.
Trying to Beat the Market: An Honest Accounting
A recurring question for Model B has been whether it can be made meaningfully more sophisticated than "our features plus the raw spread in one regression." Across two research cycles, we've tried roughly a dozen distinct approaches: a stacking ensemble combining Model A and market probabilities, feature-by-spread interaction terms, the total (over/under) line as an added input, dynamic weighting by model disagreement, market efficiency trends across and within seasons, and market accuracy conditional on travel distance.
None of them improved on the current flat regression. Most came back as clean rejections; a couple were inconclusive with confidence intervals straddling zero. Not one showed a real, defensible edge.
We think this negative result is worth stating plainly rather than quietly moving past: the betting market for NFL point spreads is genuinely difficult to improve on with the data and methods available here. That's not a failure of effort -- it's a real, repeatedly-tested finding about market efficiency, and arguably more honest than claiming a marginal edge that wouldn't hold up under scrutiny.
Known Gaps
Weekly Automated Process
- Pull fresh play-by-play + schedule data
- Refit team + QB ratings at the current "as of" cutoff
- Refit Model A and Model B themselves (not just the underlying ratings) on all real completed games available -- confirmed 2026-08 to already match the weekly-refitting approach validated as the correct backtest methodology
- Web-search for QB/injury news that might contradict the assumed starter
- Generate both models' predictions, save before kickoff (never overwritten)
- Grade the previous week's saved predictions against actual results
- Regenerate this dashboard
How this compares to other ways of picking games
The question a sceptical reader asks first.
The Short Version
On raw predictive accuracy, this model does not beat the Vegas betting market, and neither do most public NFL models. Across two full research cycles, roughly a dozen distinct approaches were tested to try to add something the market doesn't already price in. None of them worked. That's documented in detail on Model Lab and in the Methodology page's "Trying to Beat the Market" section.
Added 2026-09-05, because it is the question a sceptical reader asks first and this project had never actually asked it: would any of this have beaten the spread? Picking winners and beating the spread are not the same test, and the second one is the real claim. A model fitted on the same features against a cover target, over 1,058 backtested games, hit 51.61% against the spread — 95% CI [48.58%, 54.63%]. That interval contains 50%, so it is not distinguishable from a coin flip; and at standard −110 juice you need 52.38% just to break even, which it does not reach either. Three trivial controls — always home, always away, always the favourite — all came back at exactly 50.00%, which is what a spread is for and what confirms the arithmetic. This model would not have made money. See src/ats_evaluation.py.
What this project offers instead is transparency about how it actually works and what it actually gets wrong -- which is genuinely rare among prediction sites, most of which publish a number and no accounting of how it was arrived at or how often it fails.
Where This Model Stands
| Dimension | How this project does |
|---|---|
| Raw accuracy vs. market | Behind. The market is a genuinely hard benchmark and this doesn't clear it |
| Football-only accuracy (no market data) | Reasonable -- roughly 63% on held-out data, well above the ~54% you'd get by always picking home teams |
| Methodology transparency | Every constant is backtested and documented with its real justification; the tuning scripts are committed and re-runnable |
| Honest failure reporting | Model Lab documents every rejected experiment with real numbers, including a data leak that was caught and fixed |
| Leak-free discipline | A real, executable test suite guards the walk-forward properties -- and it caught a genuine production leak |
| Live, ongoing track record | Predictions are locked in before kickoff and graded automatically -- no post-hoc editing |
What Would Actually Close the Gap
Being specific rather than vague about this: the single biggest missing input is injury and player-availability data, which the market prices in heavily and this model has no access to (the upstream data source has been broken with no ETA -- see Data Sources). Beyond that, the market aggregates information this project structurally can't reach: sharp money, insider knowledge, and real-time news. Closing that gap isn't a matter of a better algorithm.
Where the numbers come from
Every dataset this project touches, honestly labelled.
In Production (used every single weekly run)
| Dataset | Used For |
|---|---|
| Play-by-play | Team ratings (ridge regression), per-QB trailing EPA |
| Schedules | Game results, spread/total lines, home/away, current-week matchups |
| Snap counts | Computed and logged each run for O-line continuity, though that feature was removed from the live model in v2.1 -- data collection continues in case future evidence changes that |
All three migrated from the now-deprecated nfl_data_py to nflreadpy in 2026-08, verified with real column-schema and exact-value checks before merging -- see What's Changed.
Research-Only (fetched to test a specific idea, never part of live predictions)
| Dataset | What happened |
|---|---|
| Officials | Rejected |
| FTN charting | Rejected |
| Trades | Rejected |
| Rosters (weekly) | Not suitable -- confirmed not gameday active/inactive status |
| Participation | Not suitable -- same limitation as snap counts, also doesn't update during the season |
| ESPN QBR | Alternative QB quality metric -- Source confirmed abandoned, stops at 2023 |
This section exists specifically so a claim like "we tried X" is checkable, not just asserted -- every row here has a real check behind it, documented on Model Lab or What's Changed.
Words used on this page
Look anything up here rather than guessing from context.
Model & Prediction Terms
| Term | What it means |
|---|---|
| EPA (Expected Points Added) | How much a single play changed a team's expected points on that drive. A 40-yard gain has high positive EPA; a sack has negative EPA. Summed up, it's a better measure of team quality than raw yards |
| Model A | The football-only model -- team ratings, QB performance, QB changes. Uses no betting market data at all |
| Model B | Model A's features plus the current Vegas point spread |
| Opponent-adjusted rating | A team's performance adjusted for who they played. Beating a strong defense counts more than beating a weak one |
| Playoff Odds | We play out the rest of the season times on a computer and count how often each team makes the playoffs. Each run assumes every team stays exactly as good as it is today, which is never quite true — teams get better, get worse, and get injured. So treat these as a rough guide, not a prediction, and trust them less the further away a game is |
| Recency weighting | Recent games count more than old ones. This model uses a 16-game half-life -- a game 16 games ago counts half as much as this week's |
| Shrinkage | Pulling a small-sample estimate toward the league average. A QB with 20 career dropbacks shouldn't be rated as confidently as one with 2,000 |
Evaluation Terms
| Term | What it means |
|---|---|
| Log loss | Penalizes confident wrong predictions heavily. Lower is better. A model that says 95% and is wrong is punished far more than one that said 55% |
| Brier score | Average squared difference between predicted probability and what actually happened. Lower is better. Similar spirit to log loss, less harsh on confident misses |
| AUC | How well the model ranks games from most to least likely home win. 0.5 is random, 1.0 is perfect. Measures ordering ability, not calibration |
| Calibration | Whether stated confidence matches reality -- do games predicted at 70% actually win about 70% of the time? |
| Bootstrap / confidence interval | Reshuffling the real results thousands of times to see how much a result bounces around. If the range includes zero, the difference can't be distinguished from luck |
| Walk-forward backtest | Testing a model using only data that existed before each game. Prevents the model from "knowing" future results it couldn't have known |
| Data leakage | When information from the future accidentally gets used to predict the past, making results look better than they really are. This project found and fixed a real one -- see Checking the AI's work |
Betting Terms
| Term | What it means |
|---|---|
| Point spread | The margin the favorite is expected to win by. A -7 favorite is expected to win by 7 |
| Over/under (total) | The combined score both teams are expected to reach |
| Market-implied probability | Converting a point spread into a win percentage, so it can be compared directly against a model's prediction |
| Confidence pool | A pick'em format where you rank games and assign point values -- most confident pick gets the most points. See Methodology for why ranking by probability is provably optimal here |
What's Changed
config.py rather than written out by hand, and reversals are kept: a history that quietly deletes the decisions that were later undone is just marketing.Model versions
What got built
The model is only half of it. This is the rest — the pages, the weekly routine, and the record-keeping that turns a script into something you can actually use on a Sunday.
Opponent-Adjusted Ratings
Two-way fixed-effects ridge regression on real play-by-play, replacing flat trailing averages.
Recency Weighting
Backtested half-life values; 16 games won on log loss.
Market Blend Model
Model B combines football-only features with the Vegas spread, evaluated honestly against market alone.
Tuned Ridge Alpha
Backtested 1-1600; alpha=200 was over-regularized, alpha=15 wins.
QB-Specific Features
Leak-free per-starter trailing EPA/dropback. Biggest single accuracy gain: 60.0% to 62.6%.
Automated Weekly Routine
Scheduled, self-updating: auto-detects the current week, waits until 7 days out to lock in predictions, walks straight through the playoffs with no manual intervention.
My Picks Log
Tap your actual pick per game, tracked locally with a real win/loss record against graded results.
Season Accuracy + Calibration
Real running record for Model A, Model B, and market, plus a calibration table checking whether confidence buckets actually hit their rate.
Confidence Ranking + Why Breakdown
Games auto-ranked by pick confidence for confidence-pool scoring; every pick shows its real feature-level math, not a black box.
Past-Week Browsing
Week selector on the Board and Picks pages instead of only ever showing the latest week.
Per-Game Context Notes
Generalized from a single QB-only flag into a real multi-note list the routine's web-search step can populate with injuries, coaching changes, or anything else worth flagging — shown as a real bulleted list per game.
Line Movement Tracking
Every routine run now logs the current spread for the upcoming week's games to a running archive -- even weeks that are too early to lock in real predictions. Building real, first-party movement data starting this season.
Formal Backup-QB Detection
Tested honestly first: a leak-free "did this team change starting QB since last week" flag improved every metric (accuracy +0.46pt, log loss/Brier/AUC all better too) and shows a clean, correctly-signed coefficient. Real signal -- now a live 5th model feature.
Team Deep-Dive Page
New tab: pick any team, see real season-end net ratings for 2020-2025, with live week-by-week 2026 data appending automatically as the season is actually played.
Weekly-Refitting Backtest
Backtest methodology now refits every week within a season (not just once), matching what the live weekly routine already does. Not statistically significant on its own (+0.37pt, CI included zero) but tied-or-better on every metric and free -- adopted as the more methodologically correct practice.
Migrated to nflreadpy
nfl_data_py (our data source since day one) was officially deprecated by nflverse. Migrated to nflreadpy, the actively-maintained successor, with a safety toggle to instantly revert if needed. Verified two ways before merging: every required column present, and real values from a completed 2025 week -- scores, spread lines, EPA and snap counts -- matched exactly between the old and new sources.
What we tried that did not work
Ideas that sounded good, got a real backtest, and lost. They are here because a list of only the wins tells you nothing about whether the wins were earned.
Rest, Weather & Travel Distance
Real backtest on all three: rest days, dome/cold/wind, and haversine travel distance all made the model flat or slightly worse (62.6% down to ~61.5-62.2%). Honest negative results, none adopted.
Time-Decay QB-Change Signal
Real backtest: the decay version actually scored slightly worse on accuracy (62.6-62.9% vs the binary flag's 63.1%) while only marginally improving log loss/Brier/AUC. Keeping the simpler binary flag -- it's not just simpler, it's genuinely no worse.
Re-test Ensemble/Gradient Boosting
With 5 features, GBM does edge out logistic regression on raw accuracy (63.4% vs 62.6%) -- but AUC drops notably (0.654 vs 0.670), meaning worse discrimination between confidence levels. Since confidence-ranked picks depend on that discrimination, staying with logistic regression.
O-Line Continuity
Was live as a Model A/B feature; pulled after two independent tests (bootstrap CI included zero; a later confirmatory test found no-OL-feature beat both raw and smoothed versions on held-out data). Still computed and logged as data, not fed into the model.
Coaching Changes
Tested 5 framings (season-level flag, in-season change, 3 decay half-lives) using real per-game coach data from nflverse schedules. Every variant tied or underperformed the baseline. Likely redundant with what opponent-adjusted team ratings already absorb.
Early-Season Prior Blending
Confirmed a real gap (61.7% weeks 1-4 vs. 65.9% weeks 15+), but explicit extra shrinkage didn't close it -- the existing continuous recency-weighted ratings already blend across the season boundary correctly. Gap remains open; import_win_totals() (real preseason market data) flagged as a genuinely different angle for later.
Preseason EPA as Early-Season Prior
Checked directly: nflreadpy's play-by-play loader has zero preseason rows, for any season 2020-2025. Not a data-quality problem to work around -- the play-level detail simply isn't in this pipeline. Ruled out, not deferred.
Checking the AI's work
Why This Page Exists
An AI agent confidently stating something false -- about a result, about the state of a codebase, about whether a test actually passed -- is a real, well-documented failure mode, not a hypothetical. The honest mitigation isn't "use a smarter model" or "ask it to double-check itself" (the same failure mode that produced the wrong claim can just as easily produce a confident re-confirmation of it). The mitigation that actually worked here, every time, was external, mechanical verification -- demanding the real command output, the real screenshot, the real re-run, rather than accepting a description of one.
Below are the three real incidents from this project where that discipline caught something that would otherwise have shipped wrong.
Incident 1 -- False Claim About Repository State
Claim made: Scout stated that a specific pull request "hasn't actually been merged yet," contradicting an earlier confirmation that it had been.
How it was caught: Rather than accept either statement, a direct screenshot of the actual PR page was requested. It showed a purple "Merged" badge -- the PR genuinely was merged. Scout's later claim was the false one.
Why it matters: This was a claim about objective, checkable repo state -- not a judgment call or a probabilistic model output. It still came out wrong. Ground truth (a real screenshot) resolved it in seconds; trusting the more recent-sounding claim would not have.
Incident 2 -- An Unreproducible Result
Claim made: An earlier tuning decision (a model hyperparameter, retuned from 8 to 96) was recorded on this dashboard as a resolved ACCEPT, with specific supporting accuracy numbers.
How it was caught: When the same comparison was re-run from the actual current codebase, months later, the original numbers could not be reproduced under any configuration tried -- not with the original methodology, not with the current one. The result had been evaluated against a version of the model that no longer existed, and nothing had gone back to re-check it after later, independently-correct changes altered what it was really being compared against.
Why it matters: Nobody fabricated this -- each individual step was reasonable at the time. The failure was in never re-verifying an old claim against the current state of the system. See Model Lab's "QB Shrinkage Re-tune" entry for the full resolution (re-tested fresh, found genuinely inconclusive, reverted to the simpler original default).
Incident 3 -- A Test Suite That Wasn't Actually Testing Anything
Claim made: A leak-free test suite existed on the main branch, implying the codebase's leak-free properties were being actively verified.
How it was caught: When actually executed (not just confirmed to exist), 4 of its 5 tests failed immediately with an import error -- the file's own path configuration pointed at a directory that didn't exist -- and the fifth silently skipped. The suite had never successfully run a single real check.
Why it matters: A test file's presence in the repo is not evidence it works. This is arguably the most important lesson on this page: a verification system that isn't itself verified provides false confidence, which is worse than no verification system at all.
The Actual Rule
Formalized in VERIFICATION.md: any claim about test results, repo state, or reproducibility must be backed by real, re-executed command output -- not a summary of one. See also Booth, a dedicated verification role built specifically to audit Scout's work against this rule.