Abstract
The recent explosive growth in convolutional neural network (CNN) research has produced a variety of new architectures for deep learning. One intriguing new architecture is the bilinear CNN (B-CNN), which has shown dramatic performance gains on certain fine-grained recognition problems. We apply this new CNN to the challenging new face recognition benchmark, the IARPA Janus Benchmark A (IJB-A). It features faces from a large number of identities in challenging real-world conditions. Because the face images were not identified automatically using a computerized face detection system, it does not have the bias inherent in such a database. We demonstrate the performance of the B-CNN model beginning from an AlexNet-style network pre-trained on ImageNet. We then show results for fine-tuning using a moderate-sized and public external database, FaceScrub. We also present results with additional fine-tuning on the limited training data provided by the protocol. In each case, the fine-tuned bilinear model shows substantial improvements over the standard CNN. Finally, we demonstrate how a standard CNN pre-trained on a large face database, the recently released VGG-Face model, can be converted into a B-CNN without any additional feature training. This B-CNN improves upon the CNN performance on the IJB-A benchmark, achieving 89.5% rank-1 recall.
Introduction
While 1:1 face verification has reached near-perfect accuracy on older datasets like Labeled Faces in the Wild (LFW), real-world applications often require 1:N face identification—finding a probe face within a large gallery of known identities. The IARPA Janus Benchmark A (IJB-A) pushes the boundaries of this problem with extreme pose variations and multi-image "templates" for single subjects. This paper explores the application of Bilinear Convolutional Neural Networks (B-CNNs), originally designed for fine-grained image classification, to this rigorous identification task.
Methods
The authors frame face identification as a fine-grained classification problem. A B-CNN consists of two CNNs whose convolutional outputs are multiplied via an outer product at each spatial location and then pooled. This creates a rich, orderless descriptor capturing localized feature interactions.
The study utilizes an ImageNet-pretrained VGG "M-net," fine-tuned on the FaceScrub dataset and then on IJB-A training splits. For multi-image templates, two pooling strategies were tested: score pooling (max SVM score) and feature pooling (max feature vector). Additionally, a pre-trained VGG-Face network was converted into a symmetric B-CNN without further feature training to evaluate the architecture's inherent benefits.
Key Insights: Faces as Fine-Grained Categories
Treating face recognition as fine-grained classification proves highly effective. The B-CNN naturally learns to act as a part-detector, capturing subtle identity differences without explicit landmark detection.
Furthermore, feature pooling outperforms score pooling for B-CNNs. By taking the maximum feature response across a template, the network effectively combines the "best views" of different facial parts from varying poses into a single robust representation.
Results
On the IJB-A benchmark, the fully fine-tuned B-CNN achieved 58.8% Rank-1 accuracy, significantly outperforming the government off-the-shelf (GOTS) baseline (44.3%) and a standard CNN (53.6%). At Rank-5, the B-CNN reached 79.6% accuracy.
Converting the pre-trained VGG-Face network into a B-CNN without additional fine-tuning resulted in a Rank-1 accuracy of 89.5%, surpassing the standard VGG-Face model (89.2%) and other deep CNN methods (86%).
Discussion and Conclusion
While B-CNNs offer substantial performance gains, the resulting feature vectors are large (e.g., 262,144 dimensions), which may impact memory for massive galleries. Future work includes end-to-end training of the VGG-Face B-CNN on larger datasets and developing "Multi-view CNN" objectives to explicitly optimize for max-pooling across template images during training.
Summary updated