The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting

Tara Sowrirajan, Yanqiao WangarXiv 2026 · arXiv

In the world of machine learning, the standard recipe for better performance is usually "more": more data, more parameters, or a more complex architecture. However, a new paper from researchers at Amazon suggests we might be overlooking the most powerful lever of all: the observation window.

In their study, "The Window Lever," authors Tara Sowrirajan and Yanqiao Wang demonstrate that simply giving a model a longer look at a customer's history is an order of magnitude more effective than scaling up the model or the dataset size.

TL;DR

For forecasting customer behavior, increasing the length of historical data (the observation window) provides a massive boost in accuracy that dwarfs the gains from adding more training examples, increasing model parameters, or switching to state-of-the-art architectures. If you're building a forecasting system, start by lengthening your window before you reach for a bigger GPU cluster.

What Problem Does This Paper Address?

Predicting how a customer will behave over the next year is a classic industrial challenge. Usually, engineers try to improve these forecasts by scaling along three axes:

  1. Data Scale: Training on millions more customers.
  2. Model Capacity: Making the neural network wider or deeper.
  3. Architecture: Moving from simple linear models to LSTMs, GRUs, or Transformers.

The authors argue that the "history length"—how many weeks of past behavior the model sees—is often treated as a fixed hyperparameter rather than a primary lever for improvement. They set out to run a controlled comparison to see which of these four axes actually moves the needle the most.

Methods

The researchers conducted a massive sweep across eight model families, three population scales (1M to 10M customers), and various model capacities.

To ensure their findings were scientifically rigorous, they employed two key techniques:

  • Matched Cohorts: A common trap in window studies is "composition bias." Longer windows naturally exclude "new" customers who don't have enough history, making the test set easier. The authors solved this by scoring all models on the exact same set of "matched" customer-weeks across all window lengths.
  • Internal Registration: To avoid p-hacking, they pre-committed to four falsifiable predictions before running their evaluations. This "blind" approach is rare in ML research and adds significant weight to their findings.
Table 1: Of the four internally registered commitments, only the fully blind one, the mixture tail (iii), missed. The other three are hits under the qualifications in the notes below. Each prediction and its falsifier were committed before the deciding evaluation ran. The last column states what was already known at commit time, since the commitments are not equally blind. Two of the predictions, (ii) and (iv), are mechanical curve fits to already-measured points with no free parameter choices. Of the other two, (i) is a threshold and (iii) is a per-decile extrapolation with a calibration window pinned in advance. What each registration adds is the discriminating criterion, fixed in advance (§4.5). †Commitment (ii) is a hit on levels, scored across a cohort shift. Scored on the gain instead, the observation sits between the two predictions, nearer the saturating one; we report both scorings (observed gain +0.015 vs. saturating −0.008 / log-linear +0.053; each predicted gain is that form’s R2(24) less the fit cohort’s k=16 level, 0.4426, so the negative saturating value is the fit sitting below that point, not a predicted decline; §4.5). The two forms were also nearly tied on the data they were fit to, and the losing one was already strained by the curvature within the fit range, so this row tests whether the better in-sample form extrapolates rather than adjudicating two live hypotheses (§4.5). ‡Commitment (iv) is scored as committed; the deciding measurement is single-seed and carries a training-extract confound: an unmeasured transformer-side offset like the −0.033 observed on the ridge sub-ladder would put the observation near the log-linear prediction (§2.5). §The observed intervals in this table are row-sampling uncertainty at the three test anchors, not intervals over calendar anchors: the anchors sit in one eight-week band with almost entirely coinciding forward label windows, so the calendar direction carries close to a single effective draw and the bootstrap does not resample it. Measured anchor by anchor on the Huber objective, the same eight-week extension is large at the earliest anchor and worth approximately nothing at the latest. That spread is wider than this row’s interval, which does not cover it. Those per-anchor figures come from a control that is void under its own criterion and are reported only as diagnostics (§1).

Datasets

The study primarily uses a large-scale internal Amazon dataset. To prove the "Window Lever" isn't just an internal quirk, they replicated the direction of their findings on five public datasets:

  • Alibaba Taobao (User behavior)
  • Online Retail II
  • The Complete Journey (Grocery purchases)
  • CDNOW (Music sales)
  • H&M (Fashion transactions)

Key Insights and Ideas

  • The Order-of-Magnitude Gap: Doubling the observation window from 8 to 16 weeks improved the log-space $R^2$ by roughly 0.07 to 0.10. In contrast, the effects of data scale, model capacity, and architecture were all bounded at roughly 0.006—ten times smaller.
  • Architecture Convergence: Interestingly, as the observation window gets longer, the gap between different architectures narrows. A simple ridge regression model with a long window can often rival a sophisticated Transformer with a short window.
  • Activity-Based Saturation: Not every customer needs the same amount of history. High-activity customers "saturate" quickly (the model learns their pattern in ~4 weeks), while low-activity customers continue to benefit from more history even out to 16 weeks and beyond.
  • The 6-Month Ceiling: For this specific task, the gains from history eventually plateau around 22–24 weeks. Adding history beyond six months yielded diminishing returns.

Key Results

The most striking result is visualized in the comparison of the different "levers." While scaling data or parameters results in nearly flat lines, the window lever shows a steep, consistent climb.

The Window Lever vs. Others: Extending the window (left) provides a massive gain compared to the relatively flat improvements from scaling data (center) or model capacity (right).

Even when comparing different model families, the "climb" provided by the window is the dominant feature of the graph.

Performance across families: Every model family tested—from simple Ridge to complex Transformers—showed a similar, massive improvement as the input window increased.

Limitations and Future Work

The authors are careful to note a significant confound: Calendar Coverage. Because their test anchors were fixed in time, lengthening the window backward also meant the model started seeing "holiday season" data (November/December). While they ran controls suggesting the history effect persists outside of holidays, they couldn't perfectly separate "more history" from "better seasonal coverage" in their main Transformer runs.

Future work would require a "calendar-balanced" re-extraction of data to isolate the pure effect of history length from the specific months being observed. Additionally, the study focused on a fixed 364-day target; it remains to be seen if the window lever is as dominant for shorter-term forecasts.

Summary updated

Abstract

In our experiments on long-horizon customer-behavior forecasting, a longer observation window outweighed the observed effects of more data, more capacity and newer architectures. The window is the weeks of history a model sees before the week it forecasts from. Going from an 8- to a 16-week window adds a raw +0.070 to 0.107 log-space $R^2$ in every family at 1M. Matched test cohorts, which score the same customer-weeks at every window, show this raw comparison understates the gain in every family they cover. The direction holds on five public datasets, four spanning over a year (ridge and our transformer on all five). On our internal data, longer windows admit up to 10% more and colder training customers, which matching does not fix. Our design does not separate window length from calendar coverage, because every test anchor (the week a forecast is made from) falls in one eight-week band. On training-period anchors, the weekly ridge probe's gain persists outside the holiday season, keeping about 72% of it. We therefore report a robust positive association with longer windows and do not identify the magnitude of a calendar-independent history effect. Of four internally registered predictions, the only fully blind one failed. An activity-mixture model predicted +0.014 from 24 to 32 weeks; the weekly-aggregate probe returned -0.0004. For the sequence families, the other three levers are about an order of magnitude smaller, bounded and unresolved. Their observed effects are at most 0.006 between the nominal 1M and 10M tiers and 0.0052 among five sequence families at 16 weeks with clocks as built, both single-seed on the full test split, and 0.006 for capacity on three-seed means. These are observed intervals, not equivalence bounds. Our claim is the order-of-magnitude contrast between the window and those bounds. A team building on this class of data should start by lengthening the window.
BibTeX
@misc{sowrirajan2026window,
  author = {Tara Sowrirajan and Yanqiao Wang},
  title = {{The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting}},
  year = {2026},
  eprint = {2611.26521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26521},
}
Figure