Overview
This paper introduces Target-Embedding Autoencoders (TEA), a novel framework designed to improve supervised prediction by learning latent representations of high-dimensional targets rather than input features. By jointly optimizing for target reconstruction and feature-based prediction, TEA acts as a powerful regularizer that significantly boosts generalization in complex tasks, such as multivariate sequence forecasting.
The Problem: High-Dimensional Targets
In traditional representation learning, autoencoders are typically used to find structure in the input feature space ($\mathcal{X}$). This approach is highly effective when features are high-dimensional and targets ($\mathcal{Y}$) are relatively simple, such as a single class label.
However, many real-world problems involve high-dimensional target spaces. Examples include multi-label classification, image segmentation, and multivariate time-series forecasting. The authors argue that in these "inverse" settings, the standard strategy of reconstructing features—known as Feature-Embedding Autoencoders (FEA)—is sub-optimal. Learning to encapsulate the variation in $\mathcal{X}$ does not necessarily help the model map to a complex, high-dimensional $\mathcal{Y}$. Instead, models require intermediate representations that are both predictable from features and predictive of targets, leveraging the prior that variations in complex targets are often driven by a compact set of underlying latent factors.
Methodology: Target-Embedding Autoencoders
To address this, the authors formalize the Target-Embedding Autoencoder (TEA). Unlike a standard autoencoder that maps $\mathcal{X} \to \mathcal{Z} \to \mathcal{X}$, the TEA architecture focuses on the target space.
The TEA framework consists of three main components:
- An Encoder ($e: \mathcal{Y} \to \mathcal{Z}$): Maps high-dimensional targets to a lower-dimensional latent space.
- An Upstream Predictor ($u: \mathcal{X} \to \mathcal{Z}$): Learns to map input features to that same latent space.
- A Shared Forward Model ($\theta: \mathcal{Z} \to \mathcal{Y}$): Maps the latent representation back to the target space.
Training Pipeline Training is executed in three distinct stages to ensure stability and performance:
- Stage 1: The autoencoder components ($e$ and $\theta$) are trained to reconstruct the targets.
- Stage 2: The predictor ($u$) is trained to regress the learned embeddings while the autoencoder is frozen.
- Stage 3 (Joint): All components are trained together, balancing the prediction loss and the reconstruction loss.
The paper provides a theoretical guarantee of generalization for linear TEAs by proving uniform stability. They demonstrate that the auxiliary reconstruction task acts as a form of regularization. It provides stability similar to norm-based methods (like Tikhonov regularization) but without the explicit bias of weight shrinkage.
Datasets and Evaluation
The authors evaluate TEA on three challenging real-world medical datasets focused on multivariate disease trajectory forecasting.
These datasets include:
- UKCF (UK Cystic Fibrosis Registry): Tracking 11 infections and 23 comorbidities over time for over 10,000 patients.
- ADNI (Alzheimer’s Disease Neuroimaging Initiative): Forecasting 8 biomarkers and 16 cognitive tests for over 1,700 patients.
- MIMIC (Medical Information Mart for Intensive Care): Predicting 361 physiological variables (vital signs and labs) for over 22,000 ICU patients.
Key Insights
- Flipping the Script: The core insight of the paper is that when the target space is the primary source of complexity, models should autoencode the labels, not the features.
- Regularization via Reconstruction: The joint reconstruction task forces the model to find a latent space that captures the "essence" of the targets, preventing the model from over-fitting to noise in the high-dimensional output.
- Orthogonal Bias: Empirical tests show that TEA provides a benefit that is "orthogonal" to standard $L_2$ regularization. While they can be used together, TEA's benefit stems from its structural assumption of compact latent factors rather than simply shrinking weights.
- Generality: While the theoretical proofs focus on linear models, empirical results confirm the framework works exceptionally well with non-linear recurrent architectures like GRUs.
Key Results
TEA consistently outperformed the "Base" (direct prediction), "REG" ($L_2$ regularization), and "FEA" (feature-reconstruction) models across all evaluated datasets:
- UKCF: Using a linear model, TEA achieved a Precision-Recall Curve (PRC) score of 0.450, a significant improvement over the Base model's 0.322.
- ADNI: In the non-linear (RNN) setting, TEA reduced the Mean Squared Error (MSE) for biomarkers to 0.063, compared to 0.105 for the Base model.
- MIMIC: For highly complex ICU data, TEA improved the PRC from 0.142 (Base) to 0.239.
Crucially, the results demonstrate that the "staged + joint" training approach is necessary; neither unsupervised pre-training nor joint training alone performed as well as the full TEA pipeline.
Limitations and Future Work
The primary limitation of TEA is its reliance on the "Strength of Prior" assumption. The framework assumes that targets are driven by a compact, predictable set of factors. The authors note in a contrived negative example that if targets are essentially random or independent, forcing them through a bottleneck latent space will actually degrade performance.
Future work may involve extending TEAs to semi-supervised settings where some target data is missing, or exploring more sophisticated domain-specific architectures, such as Graph Neural Networks, within the TEA framework.
Summary updated