Variational Feature Pyramid Networks

Panagiotis Dimitrakopoulos, Giorgos Sfikas, Christophoros NikouICML 2022 · OpenReview

This paper introduces Variational Feature Pyramid Networks (VFPN), a Bayesian approach to automatically designing efficient and high-performing architectures for object detection and instance segmentation.

The Challenge with Standard FPNs

In modern computer vision, detecting objects at vastly different scales—such as a tiny bird in the distance versus a large car in the foreground—is a major challenge. To solve this, researchers use Feature Pyramid Networks (FPNs), which fuse high-resolution features from early layers of a neural network with semantically rich features from deeper layers.

However, most FPNs are either hand-designed (like BiFPN or PANet) or discovered via Neural Architecture Search (NAS), which requires massive computational resources. Furthermore, these architectures are usually "one size fits all"; they do not adapt to the specific scale characteristics of a particular dataset. The authors ask: Can we learn an optimal, sparse fusion architecture specifically for the task at hand without the overhead of NAS?

To avoid the computational burden of NAS, the authors propose starting with a "fully connected" initial network—a complex web of skip connections between all levels of the feature pyramid.

Initial dense feature pyramid network

Instead of treating the weights of these connections as fixed numbers, VFPN treats them as stochastic random variables. The architecture is then refined through a probabilistic pruning process:

  1. Variational Inference (VI): The model uses VI to estimate the posterior distribution of the connection weights. By optimizing the Variational Lower Bound (VLB), the network learns which connections are essential for performance.
  2. Automatic Relevance Determination (ARD): The core mechanism for sparsity is the ARD prior. During training, this prior naturally pushes the weights of redundant or useless connections toward zero.
  3. Correlated Weights (FullARD): Standard ARD assumes weights are independent. The authors extend this to a multivariate Gaussian prior that accounts for correlations between connections, allowing for more sophisticated and stable pruning.
  4. Pruning: After training, any connection with a mean weight below a certain threshold is dropped. If an entire building block loses all its inputs, it is removed entirely, resulting in a highly efficient inference-time model.
Training iterations showing the reduction of active connections and weight values converging to zero

Key Insights and Results

The models were evaluated on three distinct datasets to demonstrate how the architecture adapts to different object scales: COCO (wide variety of sizes), PlantDoc (medium/large objects), and Cards (small objects).

  • Dataset-Specific Architectures: The most striking finding is that the pruning process results in entirely different architectures depending on the dataset. For the "Cards" dataset, the model retains connections that emphasize high-resolution features, whereas for "PlantDoc," it favors different pathways suited for larger objects.
  • Superior Accuracy: On the COCO benchmark using a Faster R-CNN backbone, the VFPN (FullARD) achieved an Average Precision (AP) of 0.322, outperforming standard BiFPN (0.293) and even the computationally expensive NAS-FPN (0.307).
  • Extreme Sparsity: The method successfully pruned the initial complex network down to just 25% of its original connections while actually improving accuracy.
  • Uncertainty for Free: Because the weights are probabilistic, the model can perform multiple forward passes (Monte Carlo sampling) at test time. This allows the model to quantify its own uncertainty. By using "variance voting" to refine bounding box predictions, the authors saw a significant boost in AP across all datasets (e.g., COCO AP jumped from 0.313 to 0.341).

Limitations and Future Work

While VFPN is highly efficient, the authors acknowledge that the initial "super-network" still requires some manual definition; the model can only prune what is already present in the search space.

Future work includes exploring non-Gaussian priors (like the Laplace or Student’s-t distributions), which might offer even better sparsity or handle "heavy-tailed" data more effectively. They also suggest expanding the initial architecture to include more hidden layers and modules, giving the variational "sculptor" more material to work with.

Summary updated

Abstract

Recent architectures for object detection adopt a Feature Pyramid Network as a backbone for deep feature extraction. Many works focus on the design of pyramid networks which produce richer feature representations. In this work, we opt to learn a dataset-specific architecture for Feature Pyramid Networks. With the proposed method, the network fuses features at multiple scales, it is efficient in terms of parameters and operations, and yields better results across a variety of tasks and datasets. Starting by a complex network, we adopt Variational Inference to prune redundant connections. Our model, integrated with standard detectors, outperforms the state-of-the-art feature fusion networks.
BibTeX
@inproceedings{dimitrakopoulos2022variational,
  author = {Panagiotis Dimitrakopoulos and Giorgos Sfikas and Christophoros Nikou},
  title = {{Variational Feature Pyramid Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=iDq6jYQ1lS},
}
Figure