Abstract
In this book, the authors provide a systematic and rigorous introduction to supervised, unsupervised and reinforcement learning by establishing essential definitions and theorems.
Introduction
Supervised learning serves as the cornerstone of predictive modeling in quantitative finance. At its core, it is a process of inference: learning a function $f_\theta$ that maps input features to target outputs based on historical data. This chapter establishes a rigorous framework for this process, navigating the complexities of model selection, loss quantification, and numerical optimization. The goal is to capture non-linear patterns inherent in financial markets while maintaining a balance between the twin traps of underfitting and overfitting.
Methodology: The Six Pillars of Supervised Learning
The authors propose a standardized framework for regression and classification tasks, structured around six fundamental pillars:
- Dataset: The collection of raw input-output pairs used for training.
- Model ($f_\theta$): The functional form, ranging from simple linear regressions to complex non-linear structures like splines, decision trees, and neural networks.
- Empirical Loss: The metric used to quantify error, typically quadratic loss for regression and cross-entropy for classification.
- Optimization: The process of finding optimal parameters ($\theta^*$). The primary focus is on Gradient Descent (GD), where the model parameters are updated in the direction of the steepest descent—the negative gradient.
- Prediction: Applying the trained model to unseen data ($x_*$).
- Validation: Assessing performance using metrics such as $R^2$ for regression or accuracy for classification.
A significant portion of the methodology is dedicated to Ensemble Methods. These techniques combine multiple "weak learners" to create a "strong learner." The authors categorize these into parallel methods (Bagging/Pasting), sequential methods where each model corrects its predecessor (Boosting), and hierarchical methods that use a meta-model (Stacking).
Numerical Example: The MNIST Dataset
To demonstrate these concepts, the authors utilize the MNIST dataset of handwritten digits. They construct a binary classifier specifically to identify the digit "8" among 60,000 images. This task highlights the challenge of class imbalance—since "8s" are a minority, simple accuracy can be a deceptive metric. A model that always predicts "Not an 8" would achieve 94.65% accuracy but fail entirely in its objective. This underscores the necessity of using Precision, Recall, and the Area Under the ROC Curve (AUC) in financial contexts where rare events, such as market crashes, are of primary interest.
Key Insights and Optimization
The authors provide a "Mountain Fog" intuition for Gradient Descent: navigating a descent in zero visibility by feeling the local slope. This is mathematically backed by a formal proof using Taylor expansion. The choice of Learning Rate ($\eta$) is critical; too small, and convergence is inefficient; too large, and the optimizer may overshoot the minimum or get stuck in local minima.
Furthermore, the "Wisdom of the Crowd" effect in ensembles is demonstrated. If 1,000 independent weak learners each have an accuracy of only 55%, their combined majority vote can achieve nearly 100% accuracy, provided they are mutually independent.
Results and Performance
- Optimization Efficiency: Stochastic Gradient Descent (SGD) and Mini-batch GD are shown to be more computationally efficient and better at escaping local minima than standard Batch Gradient Descent (BGD).
- Classification Metrics: In the MNIST "digit 8" task, the model achieved a Precision of 0.7520, a Recall of 0.6725, and an AUC of 0.9423, indicating a strong classifier despite the class imbalance.
- Ensemble Scaling: Performance gains in ensemble models increase monotonically with the number of base learners, eventually reaching a theoretical plateau as the system reaches its maximum potential.
Limitations and Future Directions
While powerful, these methods have limitations. Gradient Descent is sensitive to initialization and does not guarantee a global minimum. Hyperparameter tuning via Grid Search or Cross-Validation is computationally intensive and often requires pre-specifying search ranges. Additionally, the theoretical success of ensembles assumes independence among learners, a condition often difficult to achieve when training on the same financial datasets. This foundational work sets the stage for more complex topics like Deep Learning and Reinforcement Learning.
Summary updated