In standard machine learning, we assume a symmetry between training and testing: the features available to the model during training are the same ones it will see in the real world. However, this symmetry is often a bottleneck. In many domains, we have access to "superpowers" during training—expensive sensors, high-fidelity simulations, or expert annotations—that are impossible to use during deployment due to cost, privacy, or latency.
The research direction of Learning Using Privileged Information (LUPI), unified with Knowledge Distillation, provides a way to break this symmetry. It allows a model to "learn with a teacher" who sees more than the student, enabling the student to internalize complex patterns and maintain high performance even when the extra information is removed.
The Scaffolding Intuition
To understand why this works, imagine learning to navigate a dark room. If you only ever practice in the dark, you will struggle to learn the layout. But if you first explore the room with the lights on, you can build a mental map. Later, when the lights are off, you can navigate effectively because you have internalized the structural information that was only visible when the lights were on.
In this analogy, the "lights" are privileged information. As formalized by Vladimir Vapnik, LUPI involves using an auxiliary data modality ($X^*$) available only during training to guide the learning of a primary modality ($X$).
David Lopez-Paz, Leon Bottou, and Bernhard Schölkopf bridged this concept with Hinton’s distillation in Unifying distillation and privileged information. They showed that a "teacher" model trained on privileged data can pass its "wisdom" to a "student" model via soft labels. These soft labels contain more information than binary ground-truth targets because they reveal the teacher's uncertainty and the structural relationships between classes.
Cross-Modal Guidance
The most common application of this idea is transferring knowledge from a rich, high-fidelity modality to a sparse or low-cost one.
- Visual Scaffolding: In Face Verification from Depth using Privileged Information, Guido Borghi and colleagues use RGB images as privileged information to train a depth-based network. The RGB data helps the model identify discriminative facial features that are subtle in depth maps, allowing the depth-only model to perform better at inference.
- Sensor Hallucination: Eric Granger demonstrates in HalluciDet how an infrared detector can "hallucinate" RGB-like features. By using a pre-trained RGB detector as a teacher, the infrared model learns to ignore thermal clutter and focus on pedestrian shapes.
- Privacy-Preserving Monitoring: Sizhe An and Ümit Y. Ogras developed the mRI dataset to train mmWave radar models using RGB-D cameras as privileged teachers. This allows for accurate 3D pose estimation in homes using only privacy-preserving radar sensors.
Internalizing Expert Knowledge
Beyond sensor modalities, privileged information can take the form of expert-derived data or complex simulations that are too slow for real-time use.
In Tabular Insights, Visual Impacts: Transferring Expertise from Tables to Images, Jun-Peng Jiang and Han-Jia Ye use expert-curated tables to guide image models. The model learns to "see" the structured expertise hidden in the tables, improving its visual recognition of complex patterns. Similarly, Yiming Wang uses high-fidelity structural simulations as privileged information in Kinome-AI to predict protein mutations, allowing the final model to generalize to new protein families without needing new simulations.
Even human biology can serve as a teacher. Jacopo Cavazza and Vittorio Murino have explored using brain activity (EEG/fMRI) as privileged information to help models understand abstract action concepts in Understanding action concepts from videos and brain activity through subjects’ consensus. The neural consensus of human observers acts as a semantic scaffold for the computer vision model.
Why It Works: Inductive Bias and Regularization
The fundamental reason this framework succeeds is that privileged information acts as a powerful regularizer.
When a student model like the one in Auxiliary Modality Learning with Generalized Curriculum Distillation by Yu Shen is forced to mimic a teacher with access to $X^*$, it is essentially being told: "Find a representation of $X$ that is consistent with the insights found in $X^*$."
This narrows the search space of the student model. Instead of finding any arbitrary mapping from $X$ to $Y$, the student is pushed toward a mapping that captures the underlying physical or semantic structure of the problem. As Ben Day shows in Meta-learning using privileged information for dynamics, incorporating physical constants as privileged information allows models to internalize complex dynamics that they simply cannot "see" in raw observational data alone.
By the time the model reaches inference, the "scaffolding" of the privileged data is gone, but the "building"—the optimized weights and feature representations—remains, now capable of standing on its own.