Multi-Perspective LSTM for Joint Visual Representation Learning

Alireza Sepas-Moghaddam, Fernando Pereira, Paulo Lobato Correia, Ali Etemad

In computer vision, multi-view data—such as multi-camera security setups or complex Light Field data—offers a wealth of information. However, most deep learning models treat these perspectives as independent streams, only combining them at the final stage. This research introduces the Multi-Perspective LSTM (MP-LSTM), a novel recurrent cell designed to learn the relationships between different views simultaneously with the patterns within each view.

The Challenge of Multi-View Fusion

When observing a subject from multiple angles, human perception integrates these views to understand 3D structure and movement. Standard Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, typically rely on two suboptimal fusion strategies:

  1. Early Fusion: Concatenating all views into a single input vector, which ignores the distinct physical perspectives.
  2. Late Fusion: Training separate LSTMs for each view and averaging scores, which fails to capture "inter-perspective" dynamics where movement in one angle clarifies an occlusion in another.
Conceptual overview of inter-perspective and intra-perspective relationships in multi-view sequences.

Methodology: The MP-LSTM Cell

The core innovation is the MP-LSTM cell, redesigned to handle $m$ perspectives simultaneously. Unlike a standard LSTM that updates its state once per time step, the MP-LSTM iterates through each perspective. The cell state for a subsequent perspective is updated based on the "jointly learned" state of the preceding one. This allows the network to identify complementary or contradictory information across views in real-time.

The architecture incorporates additional gates and "peephole connections" to control the flow of information from different perspectives into a shared long-term memory. These cells are typically deployed in a bi-directional network to capture the full temporal context of visual events.

Experimental Results and Insights

The model was validated on two distinct data types: the OuluVS2 dataset for multi-perspective lip reading and the LFFW/LFFC datasets for Light Field face recognition.

Performance and Convergence

On the OuluVS2 dataset, the MP-LSTM achieved an accuracy of 87.22% using three perspectives, outperforming the best benchmark (SLF-LSTM) by approximately 5%. Beyond accuracy, a key finding was the model's efficiency. The MP-LSTM demonstrates significantly faster convergence than standard multi-input variants. By resolving ambiguities through cross-perspective analysis, the training loss drops more rapidly.

Training loss comparison (a and b) showing faster convergence of MP-LSTM against standard fusion baselines.

Key Takeaways

  • Joint Learning vs. Fusion: Integrating inter-perspective relationships directly into the LSTM cell architecture is superior to external feature or score fusion.
  • Baseline Effect: The model provides the most significant performance boost when camera angles are far apart (a "wide baseline"), though it maintains an edge even in high-density Light Field captures.
  • Efficiency: Despite its architectural complexity, the MP-LSTM was the fastest to train among tested variants, requiring only 1.24 seconds per iteration due to its rapid convergence rate.

Conclusion

The MP-LSTM represents a significant step forward in processing multi-perspective visual data. By treating multiple views as a single, integrated stream of information at the cell level, it achieves state-of-the-art results in lip reading and face recognition. Future work may extend this joint learning strategy to other multi-modal data types, such as synchronized video and depth sensor streams.

Summary updated

Abstract

We present a novel LSTM cell architecture capable of learning both intra- and inter-perspective relationships available in visual sequences captured from multiple perspectives. Our architecture adopts a novel recurrent joint learning strategy that uses additional gates and memories at the cell level. We demonstrate that by using the proposed cell to create a network, more effective and richer visual representations are learned for recognition tasks. We validate the performance of our proposed architecture in the context of two multi-perspective visual recognition tasks namely lip reading and face recognition. Three relevant datasets are considered and the results are compared against fusion strategies, other existing multi-input LSTM architectures, and alternative recognition solutions. The experiments show the superior performance of our solution over the considered benchmarks, both in terms of recognition accuracy and complexity. We make our code publicly available at https://github.com/arsm/MPLSTM.
BibTeX
@inproceedings{sepasmoghaddam2021multi,
  author = {Alireza Sepas-Moghaddam and Fernando Pereira and Paulo Lobato Correia and Ali Etemad},
  title = {{Multi-Perspective LSTM for Joint Visual Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01627},
}
Figure