TL;DR
This paper tackles the challenge of applying deep learning to sensor data, which often lacks the large, labeled datasets required for training. The authors propose transforming non-visual sensor data (specifically, pressure sensor data from footsteps) into image-like representations. This "modality transformation" allows them to leverage pre-trained Convolutional Neural Networks (CNNs) for person identification, achieving state-of-the-art results.
What Problem Does This Paper Address?
Convolutional Neural Networks (CNNs) have revolutionized computer vision, achieving remarkable success in tasks like image recognition and object detection. However, their effectiveness hinges on vast amounts of labeled training data. Many real-world applications, particularly those involving sensor data from pervasive or wearable computing environments, suffer from a scarcity of such labeled datasets. This is especially true for sensor modalities that are not inherently visual, making it difficult to directly apply standard CNN architectures. The authors identify this gap: how can we harness the power of deep learning for sensor data when the data itself isn't in a format that deep learning models readily understand, and when obtaining large labeled datasets is impractical?
Methods
The core innovation of this paper is the modality transformation technique. The authors propose converting non-visual sensor data into a visual domain that CNNs can process. For their specific application of footstep detection using pressure sensor data, they transform the 2D pressure mappings into image-like representations. They explore three strategies for this transformation:
- Max-Frame: Capturing the frame with the highest pixel sum within a step sequence.
- Average-Frame: Averaging all frames in a step sequence to create a single image.
- All Frames (Sequential Analysis): Using all frames in a step sequence, potentially as a temporal sequence.
Once the sensor data is transformed into images, they employ transfer learning. This involves using a pre-trained CNN (specifically, Inception-v3, chosen for its performance and efficiency) as a fixed feature extractor. The final classification layer of the pre-trained CNN is removed, and the activations from a preceding layer are used as feature descriptors. These descriptors are then fed into a new classifier (a fully-connected layer with a softmax activation) to perform the person identification task. They also explore using Recurrent Neural Networks (RNNs) in conjunction with CNNs to capture the temporal information within the step sequences.
Datasets
The research utilizes a dataset collected from their previous work, consisting of pressure sensor data from footsteps. This dataset comprises 529 step sequences from 13 individuals. Each sequence is labeled with a specific person ID, serving as the class label for the CNN. The dataset captures variations in participant height, weight, and shoe size to account for real-world variance.
Key Insights and Ideas
- Bridging the Domain Gap: The most significant insight is that even data that is not "easily visually interpretable" can be transformed into a visual domain suitable for CNNs. This opens up deep learning applications to a much wider range of sensor modalities.
- Pre-trained CNNs as Powerful Feature Extractors: The paper demonstrates that CNNs pre-trained on large, generic image datasets (like ImageNet) possess powerful feature extraction capabilities that can be effectively transferred to novel, visually transformed sensor data.
- Modality Transformation is Key: The success hinges on the chosen modality transformation. While the paper explores several methods, the "average frame" and "all frames with RNN" approaches show promise in capturing crucial temporal and spatial information.
- Transfer Learning Reduces Data Requirements: By leveraging pre-trained models, the need for massive, task-specific labeled datasets is significantly reduced, making deep learning more accessible for niche applications.
Key Results
The paper reports impressive results for person identification based on footstep pressure data:
- Max-Frame Strategy: Achieved an accuracy of 71.99%.
- Average-Frame Strategy: Improved accuracy to 78.41%.
- All Frames with RNN Strategy: Achieved the highest accuracy of 87.66%, significantly outperforming their previous work using wavelet transformations (76.9%). This result also surpasses the state-of-the-art for this specific task by over 10%.
Limitations and Future Work
- Generalizability of Transformation: The authors acknowledge that finding the optimal modality transformation for any sensor data can be challenging, especially for data that is not easily visualized. They suggest that even if the visual representations aren't immediately intuitive, trying these transformation approaches is still worthwhile.
- Exploration of Other Sensors: They propose extending this method to other sensor types, such as accelerometers and gyroscopes, which might not have an obvious visual transformation.
- Learning the Transformation: A future direction is to investigate learning the transformation function from the source to the target mode, rather than relying on predefined strategies.
- Real-time Applications: While Inception-v3 was chosen for its efficiency, further research into real-time processing of high-velocity sensor data is implied.
Summary updated