Machine learning models deployed in online applications frequently encounter data with strict cyclical patterns. For instance, recommender systems on platforms like YouTube or TikTok experience distinct user behavior shifts depending on the time of day, while financial markets exhibit yearly seasonality. Despite the prevalence of these patterns, standard machine learning architectures often struggle to natively adapt to them, either lagging behind the trend or requiring massive storage overhead.
This paper introduces Fourier Learning and the Fourier-MLP (F-MLP), a novel neural network component designed to encode time periodicity directly into the frequency domain. By training via streaming-SGD, this approach seamlessly boosts predictive performance on periodic data, yielding significant gains in massive real-world recommender systems.
The Challenge of Cyclical Data
When training models on streaming, cyclical data, standard online learning algorithms constantly play a game of catch-up. Because they only learn from past data, they perpetually lag behind the shifting distribution—a problem known as linear dynamic regret.
To circumvent this, engineers sometimes deploy a pluralistic approach: splitting the day into discrete chunks (e.g., morning, afternoon, night) and training a separate model for each. However, this is computationally expensive, scales poorly, and suffers from crude, artificial boundaries between time segments. Alternatively, simply feeding "time of day" into a neural network as an additional feature rarely works out-of-the-box, as standard neural networks struggle to learn periodic functions natively.
The authors aim to build a system that naturally understands and exploits this cyclicality without exploding the model's size or complexity.
Methods: The Fourier-MLP
To tackle this, the authors propose mapping the time variable into the frequency domain using a partial Fourier expansion. Instead of trying to learn a generic function $f(x, t)$ where $x$ represents features and $t$ represents time, they restructure the model to learn coefficient functions:
$$f_N(x, t) = \sum_{n=0}^N \left[ a_n(x) \sin\left(\frac{2\pi nt}{T}\right) + b_n(x) \cos\left(\frac{2\pi nt}{T}\right) \right]$$
Here, $T$ is the known period (e.g., 24 hours). The innovation lies in how this mathematical concept is implemented for deep learning:
- Fourier-MLP (F-MLP): The authors design a neural network architecture where the final hidden layer outputs the coefficients $a_n(x)$ and $b_n(x)$. These outputs are then element-wise multiplied by sine and cosine basis vectors of time $t$.
- Streaming-SGD: The model is updated in real-time as data arrives sequentially. The authors provide rigorous mathematical proofs that this training method converges in both strongly convex and non-convex settings.
- Logit Mixing (Practical Trick): For production deployment, the output of the new Fourier model is mixed with the output of a standard baseline model. This acts as a safeguard: if the data's periodicity is weak on a given day, the baseline anchors the prediction; if it's strong, the Fourier features provide a performance boost.
Key Insights
- Separation of Concerns: By forcing the network to handle features ($x$) and time ($t$) differently, the model doesn't have to "learn" how to be periodic. The periodicity is hard-wired into the architecture via the sine/cosine bases, freeing the neural network to focus entirely on learning complex feature mappings.
- Strict Capacity Increase: Replacing a standard output layer with an F-MLP strictly increases the model's capacity. If the higher-frequency Fourier coefficients naturally drop to zero during training, the F-MLP elegantly degrades back into the original, non-periodic model.
- Bigger Isn't Always Better: The authors proved that simply increasing the size of a standard online learning model to match the parameter count of the Fourier model does not close the performance gap. The architecture's inductive bias toward periodicity is the true driver of success.
Results and Real-World Impact
The authors tested their approach across multiple datasets, including a synthetic time-varying system, a modified public Twitter dataset (Sentiment140), and a massive real-world recommender system.
On synthetic and public data, Fourier learning drastically outperformed standard online learning, which showed almost no error reduction over time due to its inability to anticipate cyclical shifts. It also easily beat the pluralistic multi-model approach.
Most notably, when deployed on a production-level Conversion Rate (CVR) prediction system—processing 30 to 60 million training samples per day across 3,600 CPU cores—the Fourier model achieved a consistent Area under the Curve (AuC) improvement of > 0.1% over the baseline. In the context of large-scale ad recommender systems, a 0.1% AuC bump is highly significant and can translate to millions of dollars in monthly revenue growth.
Limitations and Future Work
Currently, the framework assumes that the period $T$ (e.g., 24 hours) is known in advance. Additionally, the authors note a theoretical quirk: under their current definition, even perfectly static data could technically be deemed "periodic" with an arbitrary period. Future work will need to establish a mathematical way to measure the degree to which a data distribution oscillates, as Fourier learning provides its tangible advantages primarily when the data exhibits meaningful fluctuations.
Summary updated