Abstract
A family of recent successful approaches to few-shot learning relies on learning an embedding space in which predictions are made by computing similarities between examples. This corresponds to combining information between support and query examples at a very late stage of the prediction pipeline. Inspired by this observation, we hypothesize that there may be benefits to combining the information at various levels of abstraction along the pipeline. We present an architecture called Cross-Modulation Networks which allows support and query examples to interact throughout the feature extraction process via a feature-wise modulation mechanism. We adapt the Matching Networks architecture to take advantage of these interactions and show encouraging initial results on miniImageNet in the 5-way, 1-shot setting, where we close the gap with state-of-the-art.
Introduction
Deep learning models typically require vast amounts of data to learn new concepts. In contrast, humans exhibit the ability to generalize from just a few examples, a capability known as few-shot learning. A dominant paradigm in this field is metric learning, where a network learns an embedding space to compare a "query" image against a small set of labeled "support" images using similarity metrics.
However, most existing architectures extract features for support and query images independently, only allowing them to interact at the final stage of the pipeline. This paper argues that such a late-stage interaction is restrictive. By allowing the network to adjust its internal feature extraction based on the specific relationship between support and query images during processing, the model can produce more effective task-specific representations.
Methodology
The authors propose Cross-Modulation Networks, building upon the Matching Networks architecture by introducing a multi-level interaction mechanism.
FiLM Layers and Interaction
The core of this approach is Feature-wise Linear Modulation (FiLM). A FiLM layer applies an affine transformation to convolutional feature maps: $$FiLM(x) = \gamma \odot x + \beta$$ where $\gamma$ (scale) and $\beta$ (shift) are parameters that modulate the features. In this architecture, these parameters are not static; they are dynamically generated by a FiLM generator sub-network that takes features from both the support ($s$) and query ($q$) images as input.
By feeding both images into the generator, the network enables the query image to influence the support image's processing and vice versa. These cross-modulation blocks are inserted after the second, third, and fourth convolutional blocks to facilitate interaction at multiple levels of abstraction. To prevent overfitting, $L_1$ regularization is applied to the modulation parameters, encouraging the model to focus on the most relevant features.
Analysis and Discussion
Self-Modulation vs. Cross-Modulation
Analysis of the FiLM generator's weight matrices reveals that the model performs both "self-modulation" (adjusting its own features) and "cross-modulation" (adjusting based on the paired image). While self-modulation typically has a higher average norm, cross-modulation provides the critical performance boost necessary for state-of-the-art results.
Block Sensitivity
Ablation studies indicate that modulation is most effective in the second and fourth blocks of the CNN. This suggests that interaction is particularly beneficial for both mid-level textures and high-level semantic components. The importance of these learned interactions was further validated by replacing modulation parameters with random noise, which resulted in a significant accuracy drop (from ~51% to ~43%).
Experimental Results
The model was evaluated on the miniImageNet benchmark in 5-way, 1-shot and 5-shot settings. On the 1-shot task, Cross-Modulation Networks achieved 50.94% accuracy, outperforming the baseline Matching Networks (49.39%) and reaching parity with other state-of-the-art methods like Meta-SGD and Relation Nets.
Conclusion
Cross-Modulation Networks demonstrate that early and mid-stage interactions between support and query features can significantly improve few-shot learning performance. While the current approach increases computational complexity due to the Cartesian product of image pairs, it establishes a general principle that could be integrated into other frameworks like Prototypical Networks or scaled to deeper ResNet architectures in future work.
Summary updated