Abstract
Human activity recognition often suffers from a discrepancy between the rich data available during model training and the limited data available during real-world deployment. This paper introduces LUPI-HCRF, a Hidden Conditional Random Field framework designed for Learning Using Privileged Information (LUPI). By incorporating extra modalities like audio or 3D poses as "privileged information" during training and employing a robust Student’s $t$-distribution to handle noise, the proposed model significantly improves classification accuracy. Evaluations across multiple benchmarks show that LUPI-HCRF outperforms standard models and effectively complements deep learning features.
Introduction: The Challenge of Missing Data
In many machine learning scenarios, there is a "structure imbalance" between training and testing environments. During training, researchers might have access to high-fidelity auxiliary data—such as clean audio tracks, precise 3D skeletal joint positions, or detailed semantic attributes. However, at test time, the system must rely solely on raw video input. This paradigm is known as Learning Using Privileged Information (LUPI).
Standard models struggle to effectively utilize this "bonus" training data because it is absent during inference. Furthermore, real-world data is inherently messy; standard Gaussian distributions used in many probabilistic models are highly sensitive to outliers, which can skew the model’s learning process when the auxiliary data is noisy.
Methodology: Robust Privileged Learning
The authors propose the LUPI-HCRF model to bridge this gap. This framework integrates the LUPI paradigm into a Hidden Conditional Random Field (HCRF), a discriminative graphical model that represents activities as sequences of hidden states.
Key Technical Innovations
- Privileged Feature Function: The model's energy formulation includes a specific function to account for privileged features.
- Student’s $t$-distribution for Robustness: Instead of the common Gaussian distribution, the model uses a Student’s $t$-distribution to relate regular visual features to privileged attributes. The "heavier tails" of the $t$-distribution make the model mathematically less sensitive to outliers and noisy data points.
- Feature Mapping and Marginalization: During training, the model learns a mapping from visual features to privileged information. At test time, it marginalizes over the missing privileged data to make a robust prediction based on the learned relationships.
Experimental Setup and Datasets
The LUPI-HCRF framework was validated on three distinct datasets, each providing different types of privileged information:
- TV Human Interaction (TVHI): Focuses on common interactions like handshakes and kisses. Privileged info: Audio (MFCC features).
- SBU Kinect Interaction: Involves complex two-person interactions. Privileged info: 3D Poses (joint positions).
- Unstructured Social Activity Attribute (USAA): Contains diverse social scenes. Privileged info: Semantic attributes (e.g., "indoor," "dancing music").
Results and Key Insights
The LUPI-HCRF consistently outperformed state-of-the-art baselines, including SVM+ and standard HCRFs.
- TVHI: Achieved 84.9% accuracy, representing a ~10% jump over SVM+.
- SBU: Achieved 85.4% accuracy.
- USAA: Achieved 58.1% accuracy, which is significant given the high intra-class variability of this dataset.
A significant finding was the model's synergy with deep learning. When using features from a 3D Convolutional Neural Network (CNN), the accuracy on the SBU dataset reached 94.9%. This suggests that LUPI-HCRF is a valuable structural layer that can enhance modern neural architectures.
Conclusion
The core philosophy of this work is treating privileged information as a "teacher" that provides high-level guidance during training. This guidance helps the model discover superior visual representations that persist even when the auxiliary data is no longer available. By prioritizing robustness through the Student’s $t$-distribution, the framework effectively handles the messiness of real-world data across various modalities. Future work may involve developing "deep" probabilistic versions of this framework to further integrate structured graphical modeling with end-to-end learning.
Summary updated