This paper introduces Variational Feature Pyramid Networks (VFPN), a Bayesian approach to automatically designing efficient and high-performing architectures for object detection and instance segmentation.
The Challenge with Standard FPNs
In modern computer vision, detecting objects at vastly different scales—such as a tiny bird in the distance versus a large car in the foreground—is a major challenge. To solve this, researchers use Feature Pyramid Networks (FPNs), which fuse high-resolution features from early layers of a neural network with semantically rich features from deeper layers.
However, most FPNs are either hand-designed (like BiFPN or PANet) or discovered via Neural Architecture Search (NAS), which requires massive computational resources. Furthermore, these architectures are usually "one size fits all"; they do not adapt to the specific scale characteristics of a particular dataset. The authors ask: Can we learn an optimal, sparse fusion architecture specifically for the task at hand without the overhead of NAS?
A Bayesian Approach to Architecture Search
To avoid the computational burden of NAS, the authors propose starting with a "fully connected" initial network—a complex web of skip connections between all levels of the feature pyramid.
Instead of treating the weights of these connections as fixed numbers, VFPN treats them as stochastic random variables. The architecture is then refined through a probabilistic pruning process:
- Variational Inference (VI): The model uses VI to estimate the posterior distribution of the connection weights. By optimizing the Variational Lower Bound (VLB), the network learns which connections are essential for performance.
- Automatic Relevance Determination (ARD): The core mechanism for sparsity is the ARD prior. During training, this prior naturally pushes the weights of redundant or useless connections toward zero.
- Correlated Weights (FullARD): Standard ARD assumes weights are independent. The authors extend this to a multivariate Gaussian prior that accounts for correlations between connections, allowing for more sophisticated and stable pruning.
- Pruning: After training, any connection with a mean weight below a certain threshold is dropped. If an entire building block loses all its inputs, it is removed entirely, resulting in a highly efficient inference-time model.
Key Insights and Results
The models were evaluated on three distinct datasets to demonstrate how the architecture adapts to different object scales: COCO (wide variety of sizes), PlantDoc (medium/large objects), and Cards (small objects).
- Dataset-Specific Architectures: The most striking finding is that the pruning process results in entirely different architectures depending on the dataset. For the "Cards" dataset, the model retains connections that emphasize high-resolution features, whereas for "PlantDoc," it favors different pathways suited for larger objects.
- Superior Accuracy: On the COCO benchmark using a Faster R-CNN backbone, the VFPN (FullARD) achieved an Average Precision (AP) of 0.322, outperforming standard BiFPN (0.293) and even the computationally expensive NAS-FPN (0.307).
- Extreme Sparsity: The method successfully pruned the initial complex network down to just 25% of its original connections while actually improving accuracy.
- Uncertainty for Free: Because the weights are probabilistic, the model can perform multiple forward passes (Monte Carlo sampling) at test time. This allows the model to quantify its own uncertainty. By using "variance voting" to refine bounding box predictions, the authors saw a significant boost in AP across all datasets (e.g., COCO AP jumped from 0.313 to 0.341).
Limitations and Future Work
While VFPN is highly efficient, the authors acknowledge that the initial "super-network" still requires some manual definition; the model can only prune what is already present in the search space.
Future work includes exploring non-Gaussian priors (like the Laplace or Student’s-t distributions), which might offer even better sparsity or handle "heavy-tailed" data more effectively. They also suggest expanding the initial architecture to include more hidden layers and modules, giving the variational "sculptor" more material to work with.
Summary updated