Introduction
Light Field (LF) imaging captures the spatial and angular distribution of light rays, providing a rich multi-view representation of a scene from a single capture. In the context of face recognition, this translates to a dense array of sub-aperture images (SAIs), each offering a slightly different perspective. While this redundancy provides robustness against occlusions and pose variations, effectively modeling the spatio-angular dependencies between these views remains a challenge.
Standard Long Short-Term Memory (LSTM) networks are typically restricted to processing a single sequence. To handle multi-view LF data, researchers often process horizontal and vertical view sequences independently, fusing them only at the decision level. This "late fusion" approach fails to capture the intricate intra-sequence correlations during the feature learning process.
Proposed Fusion Architectures
To address these limitations, this work introduces two novel LSTM cell architectures designed for deep, intra-cell fusion: Gate-Level Fusion (GLF-LSTM) and State-Level Fusion (SLF-LSTM).
Gate-Level Fusion (GLF-LSTM)
In the GLF-LSTM architecture, the cell independently calculates the forget, input, and output gates for both the horizontal and vertical sequences. These gates are then merged to produce a single set of fused gates that govern the cell's memory and hidden states. This allows the network to learn which parts of the combined parallax information are most relevant for the task.
State-Level Fusion (SLF-LSTM)
Alternatively, SLF-LSTM maintains independent cell and hidden states for each input sequence throughout the cell's internal operations. These parallel states are fused only at the final output stage of the cell, creating a joint embedding that represents the combined information from all perspectives.
Methodology
The complete recognition pipeline utilizes a pre-trained ResNet-50 CNN to extract spatial features from each SAI. These features are then fed into a Bi-directional version of the proposed fusion LSTMs. A subsequent attention mechanism learns to assign higher weights to the most discriminative viewpoints, effectively filtering out less informative or obscured perspectives before final classification.
Experimental Results
The models were evaluated on the IST-EURECOM Light Field Face Database (LFFD), which includes 4,000 images with significant variations in pose, expression, and illumination. The proposed architectures demonstrated substantial improvements over existing state-of-the-art methods across three distinct evaluation protocols:
- Protocol 1 (One-Shot): GLF-LSTM achieved a Rank-1 accuracy of 97.75%, significantly outperforming the previous benchmark of 88.55%.
- Protocol 2 (Low Complexity): Accuracy reached 98.00%, compared to the previous best of 91.95%.
- Protocol 3 (Complex Enrollment): The model achieved near-perfect accuracy at 99.80%.
Key Insights
- Deep Fusion Superiority: Fusing information within the LSTM cell (intra-cell fusion) is more effective than late fusion, as it allows the model to learn complex parallax relationships directly.
- Gate vs. State Fusion: GLF-LSTM slightly outperformed SLF-LSTM, suggesting that establishing learning interactions at the gate level provides a richer representation than fusing only at the state level.
- The Role of Attention: The attention mechanism is vital for multi-view data, as it allows the network to focus on the most reliable facial features across different angles.
Conclusion
By redesigning the LSTM cell to natively support multi-sequence fusion, this research provides a powerful framework for exploiting the rich information in Light Field data. The proposed GLF and SLF architectures are generic and hold potential for other domains involving synchronized multi-modal or multi-view sequences, such as action recognition or audio-visual processing.
Summary updated