Nested Learning For Multi-Granular Tasks

Raphaël Achddou, J. Matías Di Martino, Guillermo SapiroarXiv 2020 · arXiv

Introduction

In contemporary deep learning, models are typically designed as specialists. We train a network to identify a specific breed of dog or a particular handwritten digit with high precision. However, this "all-or-nothing" approach often falters when faced with inputs that are blurry, occluded, or otherwise ambiguous. In such cases, standard models frequently fail—often with dangerously high confidence in an incorrect label.

A recent paper by Achddou et al., "Nested Learning for Multi-Granular Tasks," proposes a fundamental shift in neural network architecture. Instead of forcing a single fine-grained prediction, the authors introduce a framework that learns to perceive the world in layers of detail, mirroring human cognitive processes.

The Limitations of Standard DNNs

Standard Deep Neural Networks (DNNs) suffer from three primary weaknesses that the nested learning framework seeks to address:

  1. Overconfidence: Models often assign high probabilities to incorrect labels when encountering "out-of-distribution" data.
  2. Rigidity: Training is usually locked into one level of granularity. Models struggle to integrate datasets with mixed label levels (e.g., "Animal" vs. "Golden Retriever").
  3. Fragility: Fine-grained classifiers are easily misled by minor noise or adversarial perturbations that would not fool a human observer.

The authors argue that classification is inherently hierarchical. A robust system should maintain confidence in a broad category (e.g., "human") even when specific details (e.g., a person's name) are obscured.

The Nested Learning Framework

The core of this approach is a network topology grounded in Information Theory, utilizing sequential information bottlenecks and skip connections.

Architecture and Training

  • Nested Information Bottlenecks: The network is structured as a sequence of stages. The initial stage compresses input into a low-dimensional bottleneck to predict a coarse label. Subsequent stages incrementally add information to refine the representation for finer labels.
  • Skip Connections: To counteract the "data-processing inequality"—where information is lost at each successive layer—the authors employ skip connections. These allow finer layers to access raw input details discarded by coarse layers, ensuring that each level adds information to the previous one.
  • Cascaded Training: The model is trained using a cascaded protocol, starting with coarse layers and gradually "unfreezing" finer layers. This stabilizes the learning of the hierarchy.
  • Calibration and Rejection: To mitigate overconfidence, the framework incorporates temperature scaling and a "rejection class." If a sample is unrecognizable at a certain level, the model assigns it to a "none of the above" category rather than guessing.

Key Insights: Multi-Granular Intelligence

One of the most significant findings is that adding coarsely annotated data actually improves fine-grained task performance. This acts as a form of regularization, helping the model learn more generalizable features.

Figure 1: Nested learning allows for confident coarse predictions even when fine-grained details are missing due to occlusion or blur.

As shown in the figure above, the nested model provides a hierarchy of predictions. When an image is clear, it identifies the specific individual ("John"). As information is removed via occlusion or pixelation, the model's confidence in the specific identity drops, but it remains highly confident in broader categories like "Person" or "Under 50." In contrast, standard DNNs often maintain high confidence in a specific (and often incorrect) identity.

Experimental Results and Robustness

The researchers evaluated their approach across six diverse datasets, including MNIST, CIFAR-100, and DBPEDIA.

Performance Gains

On the DBPEDIA dataset, the nested model achieved 84.7% fine-grained accuracy, significantly outperforming the 76.4% achieved by a standard end-to-end model. Furthermore, t-SNE visualizations revealed that nested learning forces the model to cluster similar sub-classes together (e.g., grouping different types of trucks), creating a more logical feature geometry.

Graceful Degradation and Adversarial Defense

A standout feature of nested models is their resilience. When subjected to "turbulence" noise or distortion, their fine-grained accuracy may drop, but their coarse-grained accuracy remains stable.

Figure 2: Comparison of nested learning vs. standard DNNs under distortion. The nested model maintains coarse accuracy while the standard model fails completely.

In the vehicle classification example, a distorted truck image causes a standard DNN to incorrectly predict "Cat" with low confidence. The nested model, however, correctly identifies it as a "Vehicle" and "Car/Truck" with high confidence, even as its confidence in the specific "Truck" label decreases.

The framework also provides inherent adversarial robustness. To achieve a 30% error rate via Fast Gradient Sign Method (FGSM) attacks, an attacker had to use more than double the noise magnitude ($\epsilon$) against the nested model compared to a standard one.

Conclusion and Future Directions

Nested learning offers a path toward more reliable and interpretable AI by moving away from binary "all-or-nothing" classification. While the current framework relies on pre-defined taxonomies and adds some training complexity, its benefits in accuracy, robustness, and graceful degradation are compelling. Future work will likely focus on automating the discovery of optimal nested structures and scaling the approach to massive hierarchies like ImageNet.

Reference: Achddou, R., Di Martino, J. M., & Sapiro, G. (2020). Nested Learning for Multi-Granular Tasks. arXiv:2007.06402v1.

Summary updated

Abstract

Standard deep neural networks (DNNs) are commonly trained in an end-to-end fashion for specific tasks such as object recognition, face identification, or character recognition, among many examples. This specificity often leads to overconfident models that generalize poorly to samples that are not from the original training distribution. Moreover, such standard DNNs do not allow to leverage information from heterogeneously annotated training data, where for example, labels may be provided with different levels of granularity. Furthermore, DNNs do not produce results with simultaneous different levels of confidence for different levels of detail, they are most commonly an all or nothing approach. To address these challenges, we introduce the concept of nested learning: how to obtain a hierarchical representation of the input such that a coarse label can be extracted first, and sequentially refine this representation, if the sample permits, to obtain successively refined predictions, all of them with the corresponding confidence. We explicitly enforce this behavior by creating a sequence of nested information bottlenecks. Looking at the problem of nested learning from an information theory perspective, we design a network topology with two important properties. First, a sequence of low dimensional (nested) feature embeddings are enforced. Then we show how the explicit combination of nested outputs can improve both the robustness and the accuracy of finer predictions. Experimental results on Cifar-10, Cifar-100, MNIST, Fashion-MNIST, Dbpedia, and Plantvillage demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion.
BibTeX
@misc{achddou2020nested,
  author = {Raphaël Achddou and J. Matías Di Martino and Guillermo Sapiro},
  title = {{Nested Learning For Multi-Granular Tasks}},
  year = {2020},
  eprint = {2007.06402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.06402},
}
Figure