In the world of machine learning, the standard recipe for better performance is usually "more": more data, more parameters, or a more complex architecture. However, a new paper from researchers at Amazon suggests we might be overlooking the most powerful lever of all: the observation window.
In their study, "The Window Lever," authors Tara Sowrirajan and Yanqiao Wang demonstrate that simply giving a model a longer look at a customer's history is an order of magnitude more effective than scaling up the model or the dataset size.
TL;DR
For forecasting customer behavior, increasing the length of historical data (the observation window) provides a massive boost in accuracy that dwarfs the gains from adding more training examples, increasing model parameters, or switching to state-of-the-art architectures. If you're building a forecasting system, start by lengthening your window before you reach for a bigger GPU cluster.
What Problem Does This Paper Address?
Predicting how a customer will behave over the next year is a classic industrial challenge. Usually, engineers try to improve these forecasts by scaling along three axes:
- Data Scale: Training on millions more customers.
- Model Capacity: Making the neural network wider or deeper.
- Architecture: Moving from simple linear models to LSTMs, GRUs, or Transformers.
The authors argue that the "history length"—how many weeks of past behavior the model sees—is often treated as a fixed hyperparameter rather than a primary lever for improvement. They set out to run a controlled comparison to see which of these four axes actually moves the needle the most.
Methods
The researchers conducted a massive sweep across eight model families, three population scales (1M to 10M customers), and various model capacities.
To ensure their findings were scientifically rigorous, they employed two key techniques:
- Matched Cohorts: A common trap in window studies is "composition bias." Longer windows naturally exclude "new" customers who don't have enough history, making the test set easier. The authors solved this by scoring all models on the exact same set of "matched" customer-weeks across all window lengths.
- Internal Registration: To avoid p-hacking, they pre-committed to four falsifiable predictions before running their evaluations. This "blind" approach is rare in ML research and adds significant weight to their findings.
Datasets
The study primarily uses a large-scale internal Amazon dataset. To prove the "Window Lever" isn't just an internal quirk, they replicated the direction of their findings on five public datasets:
- Alibaba Taobao (User behavior)
- Online Retail II
- The Complete Journey (Grocery purchases)
- CDNOW (Music sales)
- H&M (Fashion transactions)
Key Insights and Ideas
- The Order-of-Magnitude Gap: Doubling the observation window from 8 to 16 weeks improved the log-space $R^2$ by roughly 0.07 to 0.10. In contrast, the effects of data scale, model capacity, and architecture were all bounded at roughly 0.006—ten times smaller.
- Architecture Convergence: Interestingly, as the observation window gets longer, the gap between different architectures narrows. A simple ridge regression model with a long window can often rival a sophisticated Transformer with a short window.
- Activity-Based Saturation: Not every customer needs the same amount of history. High-activity customers "saturate" quickly (the model learns their pattern in ~4 weeks), while low-activity customers continue to benefit from more history even out to 16 weeks and beyond.
- The 6-Month Ceiling: For this specific task, the gains from history eventually plateau around 22–24 weeks. Adding history beyond six months yielded diminishing returns.
Key Results
The most striking result is visualized in the comparison of the different "levers." While scaling data or parameters results in nearly flat lines, the window lever shows a steep, consistent climb.
Even when comparing different model families, the "climb" provided by the window is the dominant feature of the graph.
Limitations and Future Work
The authors are careful to note a significant confound: Calendar Coverage. Because their test anchors were fixed in time, lengthening the window backward also meant the model started seeing "holiday season" data (November/December). While they ran controls suggesting the history effect persists outside of holidays, they couldn't perfectly separate "more history" from "better seasonal coverage" in their main Transformer runs.
Future work would require a "calendar-balanced" re-extraction of data to isolate the pure effect of history length from the specific months being observed. Additionally, the study focused on a fixed 364-day target; it remains to be seen if the window lever is as dominant for shorter-term forecasts.
Summary updated