TL;DR
This paper introduces an Adversarial Autoencoder (AAE) framework to tackle the challenge of missing sensory data in human activity and context recognition. By learning to reconstruct missing features and synthesize realistic new data, their method significantly improves classification performance, even when a substantial portion of sensor modalities are absent.
What Problem Does This Paper Address?
Recognizing human activities and their associated context is crucial for applications like assisted living and personalized support. This task relies heavily on multimodal sensor data (e.g., from smartphones and smartwatches). However, real-world scenarios often present challenges: sensor data can be noisy, imbalanced, and, critically, missing. Users might not wear their devices, sensors can malfunction, or privacy concerns might lead to data opt-outs. Traditional methods struggle with missing data, often leading to a significant drop in recognition accuracy. This paper addresses the need for robust context recognition systems that can effectively handle missing sensory modalities.
Methods
The core of this work is an Adversarial Autoencoder (AAE). An autoencoder is a neural network trained to reconstruct its input, learning a compressed representation (latent code) in the process. The "adversarial" aspect comes from incorporating a discriminator network, which helps the autoencoder learn to generate data that mimics the real data distribution.
Here's how they leverage it:
- Reconstruction: The AAE is trained to reconstruct missing sensory features. This is achieved by corrupting the input data with structured noise (masking, adding noise, setting features to a specific value) and training the autoencoder to recover the original, clean data. This allows the model to learn how to infer missing information from available modalities.
- Synthesis: By conditioning the decoder part of the AAE on class labels and a latent code, the model can generate synthetic, class-conditional multimodal data. This is crucial for augmenting datasets, especially when dealing with imbalanced classes or privacy concerns that limit data sharing.
- Classification: The learned latent representations from the encoder can be used for downstream tasks. They extend the AAE by adding classification layers on top of the encoder, allowing for multi-label context recognition. This approach leverages both clean and reconstructed data, and can even incorporate unlabeled instances.
Datasets
The primary dataset used in this research is the ExtraSensory Dataset. This dataset is notable for being collected in a real-world environment, with participants engaged in their daily routines, making it more realistic than lab-collected data. It comprises over 300,000 labeled and unlabeled measurements from sixty users, utilizing various sensors from smartphones and smartwatches. The dataset naturally exhibits missing data due to device usage patterns and sensor availability.
Key Insights and Ideas
- AAE for Imputation is Superior: The AAE framework significantly outperforms traditional methods like PCA, mean/median imputation, and simple value filling (-1) in reconstructing missing sensory features. This suggests that the non-linear, learned representations are more effective at capturing complex relationships between modalities.
- Synthetic Data for Robustness: Generating synthetic data with the AAE allows for training classifiers that are more robust to missing modalities. The performance of classifiers trained on synthetic data is comparable to those trained on real data, even when evaluated on real test sets.
- Privacy-Preserving Embeddings: The latent representations learned by the AAE can serve as privacy-preserving embeddings of sensor data. This is a significant advantage for applications where raw sensor data is sensitive, as these compressed representations can be shared while mitigating privacy risks.
- Unified Framework: The AAE provides a unified approach to both reconstructing missing data and generating synthetic data, streamlining the process for building robust context recognition systems.
Key Results
- Reconstruction Performance: The AAE achieved a significantly lower Root Mean Square Error (RMSE) for modality reconstruction (average RMSE of 0.227) compared to PCA (0.937) on a clean subset of the test folds. This improvement was observed across most modalities, with phone state and gyroscope showing some exceptions due to their binary nature or specific data distribution.
- Classification Performance with Missing Modalities: In scenarios where multiple modalities were missing (e.g., Gyro, WAcc, Loc, Aud), the AAE-based classifier consistently achieved higher Balanced Accuracy (BA) compared to other methods. For instance, in one scenario with four missing modalities, AAE achieved a BA of 0.713, while PCA and mean imputation were around 0.526 and 0.669, respectively.
- Synthetic Data Performance: Classifiers trained on synthetic data generated by the AAE showed comparable performance to those trained on real data. For example, a classifier trained on synthetic data and tested on real data achieved a BA of 0.715, close to the 0.752 BA achieved by a model trained and tested on real data.
Limitations and Future Work
- Imbalance Handling: While the paper uses instance weighting and synthetic data generation to address class imbalance, the authors note that results were not entirely satisfactory without combining these with instance weighting. They suggest further exploration in this area.
- Binary Features: The reconstruction of binary features (like phone state) was less accurate compared to continuous features, indicating a potential area for improvement.
- Semi-Supervised Learning: The authors suggest extending the AAE to semi-supervised learning to leverage unlabeled examples more effectively.
- Real-time Deployment: Future work could focus on optimizing the AAE for real-time detection on edge devices, potentially through ensemble methods with multi-layer neural networks.
- Privacy Enhancement: Further enhancing privacy guarantees by combining deep learning with differential privacy techniques is also proposed.
Summary updated