Abstract
In this paper, we propose a novel method to incorporate partial evidence in the inference of deep convolutional neural networks. Contrary to the existing, top performing methods, which either iteratively modify the input of the network or exploit external label taxonomy to take the partial evidence into account, we add separate network modules ("Plugin Networks") to the intermediate layers of a pre-trained convolutional network. The goal of these modules is to incorporate additional signal, ie information about known labels, into the inference procedure and adjust the predicted output accordingly. Since the attached plugins have a simple structure, consisting of only fully connected layers, we drastically reduced the computational cost of training and inference. At the same time, the proposed architecture allows to propagate information about known labels directly to the intermediate layers to improve the final representation. Extensive evaluation of the proposed method confirms that our Plugin Networks outperform the state-of-the-art in a variety of tasks, including scene categorization, multi-label image annotation, and semantic segmentation.
Introduction
Standard deep learning models are typically designed as "black boxes" optimized for specific tasks. However, in many real-world scenarios, additional information—referred to as "partial evidence"—is available at inference time. For instance, knowing a photograph was taken outdoors significantly reduces the likelihood of it containing indoor-specific objects like a "refrigerator." Incorporating such hints can boost accuracy, but traditional approaches often fall short. Retraining models is computationally prohibitive and risks catastrophic forgetting, while iterative optimization methods like FeedbackProp are too slow for real-time applications. This paper introduces Plugin Networks, a modular solution that allows pre-trained models to integrate extra information with minimal overhead.
Methods
The proposed architecture utilizes small, fully connected sub-networks, termed Plugin Networks ($F_p$), which are attached to the intermediate layers of a frozen, pre-trained base Convolutional Neural Network (CNN).
Architecture and Fusion
The plugins are simple multi-layer perceptrons with ReLU activations. They accept a vector representing partial evidence (e.g., known labels) and produce an output $r$. This output is then fused with the activations $z$ of a specific layer in the base network to produce modified activations $\tilde{z}$. Several fusion operators were evaluated, including additive ($\tilde{z} = z + r$), multiplicative, and more complex affine or residual combinations. Experimental results consistently showed that the simple additive fusion operator outperformed more complex alternatives.
Training Strategy
A key advantage of this approach is that the base network (e.g., ResNet or AlexNet) remains frozen throughout the process. Only the parameters of the lightweight plugin modules are trained. This ensures that the base model's general feature extraction capabilities are preserved while significantly reducing training time and computational requirements.
Results and Discussion
The effectiveness of Plugin Networks was validated across three tasks: hierarchical scene categorization (SUN397), multi-label image annotation (COCO 2014), and semantic segmentation (Pascal VOC 2011).
Fusion Performance
As shown in Figure 4, the additive method achieves higher multi-class accuracy on the SUN397 dataset compared to affine, multiplicative, or residual approaches. The authors argue that since they use ReLU activations, a simple translation in the feature space is enough to "silence" or "activate" specific channels.
Efficiency and Accuracy
Plugin Networks demonstrate state-of-the-art performance with negligible computational cost. On the COCO dataset, the inference time for the entire test set was 25.72 seconds, nearly identical to the base model (25.64s) and significantly faster than FeedbackProp (over 100s). Furthermore, on SUN397, Plugin Networks achieved 57.59% accuracy, surpassing both the base model (53.30%) and iterative methods (55.01%). The model also proved highly data-efficient, outperforming competitors even when trained on only 20% of the available data.
Conclusion
Plugin Networks provide a robust and efficient framework for incorporating partial evidence into deep learning models. By using lightweight, modular attachments to frozen base networks, the method achieves superior accuracy and speed compared to existing iterative optimization techniques. Future research may explore extending this framework to handle multi-modal evidence, such as audio or sensor data, and developing more generalized plugin architectures.
Paper Citation: Koperski, M., Konopczynski, T., Nowak, R., Semberecki, P., & Trzcinski, T. (2020). Plugin Networks for Inference under Partial Evidence. arXiv:1901.00326v3 [cs.CV].
Summary updated