In computer vision, multi-view data—such as multi-camera security setups or complex Light Field data—offers a wealth of information. However, most deep learning models treat these perspectives as independent streams, only combining them at the final stage. This research introduces the Multi-Perspective LSTM (MP-LSTM), a novel recurrent cell designed to learn the relationships between different views simultaneously with the patterns within each view.
The Challenge of Multi-View Fusion
When observing a subject from multiple angles, human perception integrates these views to understand 3D structure and movement. Standard Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, typically rely on two suboptimal fusion strategies:
- Early Fusion: Concatenating all views into a single input vector, which ignores the distinct physical perspectives.
- Late Fusion: Training separate LSTMs for each view and averaging scores, which fails to capture "inter-perspective" dynamics where movement in one angle clarifies an occlusion in another.
Methodology: The MP-LSTM Cell
The core innovation is the MP-LSTM cell, redesigned to handle $m$ perspectives simultaneously. Unlike a standard LSTM that updates its state once per time step, the MP-LSTM iterates through each perspective. The cell state for a subsequent perspective is updated based on the "jointly learned" state of the preceding one. This allows the network to identify complementary or contradictory information across views in real-time.
The architecture incorporates additional gates and "peephole connections" to control the flow of information from different perspectives into a shared long-term memory. These cells are typically deployed in a bi-directional network to capture the full temporal context of visual events.
Experimental Results and Insights
The model was validated on two distinct data types: the OuluVS2 dataset for multi-perspective lip reading and the LFFW/LFFC datasets for Light Field face recognition.
Performance and Convergence
On the OuluVS2 dataset, the MP-LSTM achieved an accuracy of 87.22% using three perspectives, outperforming the best benchmark (SLF-LSTM) by approximately 5%. Beyond accuracy, a key finding was the model's efficiency. The MP-LSTM demonstrates significantly faster convergence than standard multi-input variants. By resolving ambiguities through cross-perspective analysis, the training loss drops more rapidly.
Key Takeaways
- Joint Learning vs. Fusion: Integrating inter-perspective relationships directly into the LSTM cell architecture is superior to external feature or score fusion.
- Baseline Effect: The model provides the most significant performance boost when camera angles are far apart (a "wide baseline"), though it maintains an edge even in high-density Light Field captures.
- Efficiency: Despite its architectural complexity, the MP-LSTM was the fastest to train among tested variants, requiring only 1.24 seconds per iteration due to its rapid convergence rate.
Conclusion
The MP-LSTM represents a significant step forward in processing multi-perspective visual data. By treating multiple views as a single, integrated stream of information at the cell level, it achieves state-of-the-art results in lip reading and face recognition. Future work may extend this joint learning strategy to other multi-modal data types, such as synchronized video and depth sensor streams.
Summary updated