Abstract
In-Context Learning (ICL) has revolutionized how Large Language Models (LLMs) perform tasks without explicit weight updates. However, standard ICL typically ignores the structural information present in unlabeled data points within a prompt. This paper introduces In-Context Semi-Supervised Learning (IC-SSL), a framework that enables Transformers to utilize both a small set of labeled examples and a larger set of unlabeled points to improve classification accuracy. By employing a modular two-stage architecture designed for manifold representation learning and categorical inference, the proposed model demonstrates significant performance gains in low-label regimes and exhibits remarkable zero-shot transfer across different data modalities.
1. Introduction
Modern LLMs excel at ICL by identifying patterns in a few provided $(input, label)$ pairs. Yet, in many real-world scenarios, labeled data is scarce while unlabeled data is abundant. Traditional ICL treats queries in isolation, failing to exploit the global structure of the data distribution. IC-SSL addresses this gap by asking whether a Transformer can "self-organize" unlabeled data within its context window to learn a more robust task representation on the fly.
2. Methodology: The IC-SSL Framework
The authors propose a two-stage Transformer architecture that mimics classical semi-supervised manifold learning algorithms. This modular design ensures that the model explicitly learns the geometry of the data before attempting classification.
2.1 Stage 1: Representation Learning ($TF_{rep}$)
This module processes all input points (labeled and unlabeled) to compute context-dependent features. It mechanistically implements a discrete Laplacian Eigenmap algorithm. By using RBF-based attention to construct an affinity graph and linear attention for power-iteration, the model extracts eigenvectors that characterize the manifold's shape.
2.2 Stage 2: Categorical ICL ($TF_{sup}$)
The second stage performs functional gradient descent in a Reproducing Kernel Hilbert Space (RKHS). It uses the learned features and the available labels to propagate class information across the manifold, effectively "filling in" the labels for the unlabeled points.
3. Experimental Setup and Datasets
To evaluate the model's ability to learn geometric structures, the study utilizes a variety of datasets ranging from synthetic shapes to complex image features.
- Synthetic Manifolds: Basic 3D shapes like the Swiss Roll and Torus serve as the primary training ground.
- Image Manifolds: A novel benchmark where images are generated via Stable Diffusion v1.5 by interpolating between latent vectors, creating a continuous "image manifold."
- Real-World Data: ImageNet100 features are used to test the model's scalability to standard computer vision tasks.
4. Key Insights and Results
The research reveals several critical properties of IC-SSL:
- Phase Transitions in Learning: The model exhibits a sharp jump in accuracy during training. This transition correlates with a "separation score" increase, indicating the moment the internal embeddings successfully group classes together.
- Sample Efficiency: In low-label regimes, the IC-SSL model is significantly more efficient than standard Transformers. For instance, on the Cylinder manifold, the proposed architecture outperformed baselines by 5–7 percentage points.
- Zero-Shot Transfer: A model trained exclusively on simple 3D synthetic manifolds can successfully perform semi-supervised learning on high-dimensional Stable Diffusion images. This suggests that the model learns a modality-agnostic geometric algorithm.
On the Stable Diffusion image task, the model achieved approximately 77% accuracy with only 15% of the data labeled. Even with as few as 3 labeled images out of 100, it maintained performance above 62%, highlighting the power of leveraging unlabeled context.
5. Conclusion and Future Work
IC-SSL represents a significant step toward more efficient in-context learning. By incorporating spectral inductive biases, Transformers can effectively "read between the labels" to understand data geometry. Future research will focus on scaling this approach to larger context windows, extending it to regression tasks, and integrating these biases into standard monolithic decoder architectures.
Summary updated