Let the Gallery Learn Itself: Training-Free Iterative Side-View Face Recognition

Zijun Huang, Suncheng XiangarXiv 2026 · arXiv

TL;DR

Gallery Self-Evolution (GSE) is a training-free framework that allows face recognition systems to "learn" from every encounter without a single gradient update. By iteratively archiving new viewpoints of a person into the gallery after successful matches, GSE closes the performance gap for side-view faces by over 20 points—all while running on a standard CPU.

What Problem Does This Paper Address?

Most face recognition systems are built on a shaky assumption: that the gallery (the database of known faces) is static and consists of high-quality, frontal images. In the real world—think traffic enforcement or smart-city surveillance—this assumption falls apart. Cameras often capture subjects at oblique angles (side profiles), where self-occlusion hides half the face and causes standard embeddings to "drift" away from the frontal enrollment template.

Existing solutions usually try to fix the model—either by using GANs to "frontalize" the face, building complex 3D reconstructions, or retraining the backbone for pose invariance. These approaches are computationally expensive, often requiring heavy GPUs, and don't take advantage of a key fact: in many deployments, the system sees the same people repeatedly.

Fig. 1. Research motivation. A static model fails on hard samples (pose, light, occlusion) due to poor generalization. Iterative learning continuously mines hard samples from deployment, updates the gallery, and refines recognition to achieve better performance.

Methods

The authors propose Gallery Self-Evolution (GSE), an engineering-oriented approach that leaves the feature extractor untouched and focuses on making the gallery more diverse over time.

The GSE pipeline follows a five-stage process: detection, pose estimation, embedding extraction, similarity matching, and the novel Iterative Learning stage.

The "Dedup-FIFO" Mechanism

The core of GSE is a clever curation strategy:

  1. High-Threshold Deduplication: When a person is successfully recognized, the system checks if the new capture offers a "novel" viewpoint. If the similarity to existing gallery samples is above 0.985, it's considered a redundant near-duplicate and discarded.
  2. Enrollment Preservation: The original frontal enrollment template is "locked" and never deleted. This ensures the system doesn't lose its baseline performance.
  3. FIFO Eviction: To keep the gallery lightweight, additional samples are capped (e.g., $K_{max}=30$). When the limit is reached, the oldest archived sample is evicted to make room for new ones.

This entire process requires no backpropagation or GPU memory overhead, making it "drop-in" ready for edge devices like a Raspberry Pi.

System overview showing the iterative learning module (green box) archiving new samples.

Datasets

The researchers evaluated GSE on three major benchmarks:

  • CFP-FP: 500 identities with frontal-profile pairs, used to test extreme yaw angles.
  • IJB-C: A large-scale, unconstrained dataset with over 3,500 identities, representing real-world "in the wild" conditions.
  • CelebA: Over 10,000 identities, used primarily for testing how the system scales with gallery size.

Key Insights and Ideas

  • Gallery Diversity > Model Complexity: You don't necessarily need a more complex model to handle side views; you just need a gallery that "knows" what that person looks like from the side.
  • The "Aha" Moment on Eviction: The authors found that "pure" FIFO (First-In-First-Out) is dangerous because it eventually evicts the high-quality frontal enrollment. By specifically protecting the original template, they ensured that performance "degrades gracefully rather than catastrophically."
  • Natural Boundaries: They discovered a natural similarity boundary around 0.985. Same-pose samples of the same person usually exceed 0.99, while cross-pose samples drop below 0.97. This makes simple thresholding an effective way to capture diverse viewpoints without "gallery bloat."
Fig. 4. Supplementary analyses. (a) θdup sensitivity. (b) Enrollment preser- vation vs. pure FIFO.

Key Results

GSE delivers massive gains for very little computational cost:

  • Side-View Accuracy: On the CFP-FP benchmark, GSE improved Rank-1 accuracy at a 45° yaw angle from 71.2% to 94.5% (+23.3 points).
  • Noise Robustness: In noisy conditions at 30° yaw, the performance gap between the baseline and GSE widened to as much as +31 points.
  • Speed: The system runs at approximately 42 FPS on a CPU, proving it is truly edge-deployable.
  • Comparison: GSE outperformed GAN-based and continual learning methods (which require retraining) while remaining training-free.
Main results showing accuracy gains across different yaw angles and noise levels.

Limitations and Future Work

While impressive, the authors note a few areas for further exploration:

  • Simulated Data: Much of the yaw and noise testing was done via embedding-space perturbation rather than real-world image rendering. Validation on raw, unconstrained roadside surveillance footage is still needed.
  • Identity Assumptions: The system assumes recurring encounters of the same identity. In scenarios where people are only seen once, GSE offers no benefit.
  • Curation Strategies: While FIFO is efficient, future work could explore more sophisticated curation, such as Least-Recently-Used (LRU) or diversity-based selection, to further optimize the 30-sample limit.

Summary updated

Abstract

What if a face recognition system could learn from every encounter without a single gradient update? We present Gallery Self-Evolution (GSE), a training-free framework where the gallery itself adapts at test time. The core mechanism---dedup-FIFO curation---archives novel appearance samples after successful recognition while permanently locking the enrollment template, bounded at 30 embeddings per identity. On CFP-FP and IJB-C benchmarks, GSE closes the side-view gap by +23 points at 45° yaw and boosts TAR@FAR=0.001 by +9.4 points, all at CPU-level cost. No retraining. No GPU. Just a gallery that grows smarter every time it meets you.
BibTeX
@misc{huang2026let,
  author = {Zijun Huang and Suncheng Xiang},
  title = {{Let the Gallery Learn Itself: Training-Free Iterative Side-View Face Recognition}},
  year = {2026},
  eprint = {2611.11391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11391},
}
Figure