How deep should be the depth of convolutional neural networks: a backyard dog case study

Alexander N. Gorban, Evgeny M. Mirkes, I. Y. Tukin

Introduction: The "Backyard Dog" Problem

State-of-the-art face recognition systems are typically designed for "universality"—the ability to distinguish between millions of different identities across diverse environmental conditions. To achieve this, researchers build increasingly deep networks with dozens or hundreds of layers and millions of parameters.

However, many real-world applications do not require universal recognition. This paper defines the "Backyard Dog" problem: a specialized system that only needs to recognize a small group of "family members" (FM) and distinguish them from "strangers." For a home security camera or a smart lock, the model does not need to recognize every celebrity on Earth; it only needs to identify a handful of specific individuals.

The Challenge of Universal Models

Massive "legacy" models like VGG16 or FaceNet are too slow and memory-intensive for edge devices like a Raspberry Pi. As shown in the architectural comparison below, these models carry a significant computational burden that is often unnecessary for localized tasks.

Architectural complexity of state-of-the-art face recognition models.

Methodology: Network Shallowing via ASPC

The authors propose a Network Shallowing approach. Instead of training a small network from scratch or using expensive iterative fine-tuning, they "prune" a pre-trained legacy network (specifically VGG16) through a non-iterative process:

  1. Truncation: The network is cut at various layers, removing the computationally heavy fully connected layers.
  2. Feature Extraction: Images are passed through the truncated network to generate raw feature vectors.
  3. Advanced Supervised PCA (ASPC): This core innovation finds a subspace that minimizes the distance between images of the same person while maximizing the distance between different people.
  4. Non-Iterative Learning: The final classification layer is constructed mathematically in a single step, avoiding thousands of rounds of backpropagation.

Performance on Edge Hardware

The primary goal was to fit within a 300 MiB RAM limit and achieve sub-second processing on a Raspberry Pi 3B. Without shallowing, legacy models are virtually unusable on such hardware. For instance, VGG16 requires over 65 seconds for a single inference on a Raspberry Pi core.

Inference times for legacy models across different hardware platforms.

By applying the shallowing technique, the authors demonstrated that a 5-layer version of VGG16 could process a 64x64 image in under 1 second on a single core of a Raspberry Pi. Despite this drastic reduction in depth, the 5-layer model maintained a respectable 5.78% error rate when identifying a "family" of 10 people against a pool of strangers. A slightly deeper 17-layer shallowed model achieved an even lower error rate of 3.27%.

Conclusion

This research demonstrates that by narrowing the scope of a task from "universal" to "specialized," the required depth of a neural network collapses. The proposed ASPC-based shallowing method allows for the rapid generation of lean, hardware-aware networks from massive legacy models without the need for extensive re-training. Future work will explore applying this method to modern mobile-first architectures like MobileNet to push the boundaries of edge efficiency even further.

Summary updated

Abstract

The work concerns the problem of reducing a pre-trained deep neuronal network to a smaller network, with just few layers, whilst retaining the network's functionality on a given task The proposed approach is motivated by the observation that the aim to deliver the highest accuracy possible in the broadest range of operational conditions, which many deep neural networks models strive to achieve, may not necessarily be always needed, desired, or even achievable due to the lack of data or technical constraints. In relation to the face recognition problem, we formulated an example of such a usecase, the `backyard dog' problem. The `backyard dog', implemented by a lean network, should correctly identify members from a limited group of individuals, a `family', and should distinguish between them. At the same time, the network must produce an alarm to an image of an individual who is not in a member of the family. To produce such a network, we propose a shallowing algorithm. The algorithm takes an existing deep learning model on its input and outputs a shallowed version of it. The algorithm is non-iterative and is based on the Advanced Supervised Principal Component Analysis. Performance of the algorithm is assessed in exhaustive numerical experiments. In the above usecase, the `backyard dog' problem, the method is capable of drastically reducing the depth of deep learning neural networks, albeit at the cost of mild performance deterioration. We developed a simple non-iterative method for shallowing down pre-trained deep networks. The method is generic in the sense that it applies to a broad class of feed-forward networks, and is based on the Advanced Supervise Principal Component Analysis. The method enables generation of families of smaller-size shallower specialized networks tuned for specific operational conditions and tasks from a single larger and more universal legacy network.
BibTeX
@article{gorban2020how,
  author = {Alexander N. Gorban and Evgeny M. Mirkes and I. Y. Tukin},
  title = {{How deep should be the depth of convolutional neural networks: a backyard dog case study}},
  journal = {Cogn. Comput.},
  year = {2020},
  doi = {10.1007/s12559-019-09667-7},
}
Figure