Dual Directed Capsule Network for Very Low Resolution Image Recognition

Maneet Singh, Shruti Nagpal, Richa Singh, Mayank VatsaICCV 2019 · OpenReview

Abstract

Very low resolution (VLR) image recognition corresponds to classifying images with resolution 16x16 or less. Though it has widespread applicability when objects are captured at a very large stand-off distance (e.g. surveillance scenario) or from wide angle mobile cameras, it has received limited attention. This research presents a novel Dual Directed Capsule Network model, termed as DirectCapsNet, for addressing VLR digit and face recognition. The proposed architecture utilizes a combination of capsule and convolutional layers for learning an effective VLR recognition model. The architecture also incorporates two novel loss functions: (i) the proposed HR-anchor loss and (ii) the proposed targeted reconstruction loss, in order to overcome the challenges of limited information content in VLR images. The proposed losses use high resolution images as auxiliary data during training to "direct" discriminative feature learning. Multiple experiments for VLR digit classification and VLR face recognition are performed along with comparisons with state-of-the-art algorithms. The proposed DirectCapsNet consistently showcases state-of-the-art results; for example, on the UCCS face database, it shows over 95% face recognition accuracy when 16x16 images are matched with 80x80 images.

1. Introduction

In real-world surveillance and long-range mobile photography, the "region of interest" (like a face or a license plate) is often captured at a Very Low Resolution (VLR), typically $16 \times 16$ pixels or less. Standard deep learning models struggle in these scenarios due to two primary factors: information scarcity and the domain gap. Information scarcity refers to the lack of texture or detail for traditional Convolutional Neural Networks (CNNs) to extract meaningful features. The domain gap arises because models trained on high-quality images often fail when applied to VLR data, and simple up-sampling techniques often introduce noise rather than useful signal.

2. Proposed Method: DirectCapsNet

The proposed DirectCapsNet architecture departs from standard CNNs by utilizing Capsule Networks. Unlike standard neurons that output a single scalar, capsules output a vector. The length of this vector represents the probability of an entity's existence, while its orientation encodes "instantiation parameters" such as pose, lighting, and deformation. These parameters are hypothesized to be more invariant to resolution changes than standard CNN features.

DirectCapsNet architecture featuring the HR-Anchor loss and targeted reconstruction module.

The "Dual Directed" nature of the network stems from two novel loss functions used during training:

  • HR-anchor Loss: During the feature extraction phase, the network calculates an "anchor"—the average feature vector for a specific class in high resolution. The loss function then "pushes" the features extracted from VLR images to be closer to these HR anchors.
  • Targeted Reconstruction Loss: Traditional Capsule Networks often reconstruct the input image to ensure the model learns relevant features. DirectCapsNet forces the decoder to reconstruct the High-Resolution version of the object, regardless of whether the input is VLR or HR. This forces the capsules to capture the underlying "essence" of the object that persists across resolutions.

3. Datasets and Experimental Setup

The model was evaluated on three distinct challenges:

  • SVHN (Street View House Numbers): A classic digit recognition dataset. VLR versions were created at $8 \times 8$ resolution.
  • CMU Multi-PIE: A constrained face recognition dataset used to test the model against variations in pose and illumination.
  • UCCS (Unconstrained College Students): A challenging dataset featuring faces captured from 100-150 meters away in real-world surveillance conditions.
Examples of Very Low Resolution (VLR) and High Resolution (HR) images for digits (a) and faces (b, c).

4. Results and Discussion

DirectCapsNet demonstrated significant improvements over existing state-of-the-art (SOTA) methods:

  • Digit Recognition (SVHN): The model achieved 84.51% top-1 accuracy on $8 \times 8$ images, representing a nearly 27% improvement over previous SOTA CNN-based methods.
  • Face Recognition (UCCS): On the unconstrained UCCS dataset, it achieved 95.81% accuracy when matching $16 \times 16$ probes against $80 \times 80$ gallery images.
  • Robustness: Tests on the CMU Multi-PIE dataset showed that the model's performance remained remarkably stable even as resolution dropped from $18 \times 15$ down to $10 \times 9$ pixels.

5. Conclusion

DirectCapsNet effectively addresses the challenges of VLR recognition by using high-resolution data as a "teacher" during training. By forcing VLR features to mimic HR features and requiring the network to reconstruct HR versions of blurry inputs, the model learns robust, discriminative representations. Future work will explore handling other covariates such as disguise, aging, and spectral variations.

Summary updated

Abstract

Very low resolution (VLR) image recognition corresponds to classifying images with resolution 16×16 or less. Though it has widespread applicability when objects are captured at a very large stand-off distance (e.g. surveillance scenario) or from wide angle mobile cameras, it has received limited attention. This research presents a novel Dual Directed Capsule Network model, termed as DirectCapsNet, for addressing VLR digit and face recognition. The proposed architecture utilizes a combination of capsule and convolutional layers for learning an effective VLR recognition model. The architecture also incorporates two novel loss functions: (i) the proposed HR-anchor loss and (ii) the proposed targeted reconstruction loss, in order to overcome the challenges of limited information content in VLR images. The proposed losses use high resolution images as auxiliary data during training to "direct" discriminative feature learning. Multiple experiments for VLR digit classification and VLR face recognition are performed along with comparisons with state-of-the-art algorithms. The proposed DirectCapsNet consistently showcases state-of-the-art results; for example, on the UCCS face database, it shows over 95% face recognition accuracy when 16×16 images are matched with 80×80 images.
BibTeX
@inproceedings{singh2019dual,
  author = {Maneet Singh and Shruti Nagpal and Richa Singh and Mayank Vatsa},
  title = {{Dual Directed Capsule Network for Very Low Resolution Image Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=IQG4LBw7pPT},
}
Figure