Abstract
This paper focuses on visual counting, which aims to predict the number of occurrences given a natural image and a query (e.g. a question or a category). Unlike most prior works that use explicit, symbolic models which can be computationally expensive and limited in generalization, we propose a simple and effective alternative by revisiting modulated convolutions that fuse the query and the image locally. Following the design of residual bottleneck, we call our method MoVie, short for Modulated conVolutional bottlenecks. Notably, MoVie reasons implicitly and holistically and only needs a single forward-pass during inference. Nevertheless, MoVie showcases strong performance for counting: 1) advancing the state-of-the-art on counting-specific VQA tasks while being more efficient; 2) outperforming prior-art on difficult benchmarks like COCO for common object counting; 3) helped us secure the first place of 2020 VQA challenge when integrated as a module for 'number' related questions in generic VQA models. Finally, we show evidence that modulated convolutions such as MoVie can serve as a general mechanism for reasoning tasks beyond counting.
Introduction
Visual counting—answering questions like "how many kids are on the sofa?"—is a complex multi-modal task requiring open-ended query grounding and multi-step visual reasoning. Historically, researchers have tackled this by mimicking human counting: explicitly mapping image regions to symbols and counting them iteratively based on their relationships. While interpretable, this explicit, region-based approach is computationally expensive and requires manually designing specialized modules for every possible reasoning task.
The authors propose a simple, implicit alternative that operates directly on convolutional feature maps. By moving away from explicit object proposals, the proposed method, MoVie, achieves a more holistic understanding of the scene. This shift allows the model to reason implicitly and facilitates better generalization to tasks beyond simple counting, such as spatial reasoning.
The MoVie Architecture
The core of the proposed approach is MoVie (Modulated conVolutional bottlenecks), a module built on stacked residual bottlenecks. Unlike standard VQA models that pool visual features into a single global vector before fusion, MoVie fuses multi-modal information locally.
Modulated Convolutions
MoVie builds upon Feature-wise Linear Modulation (FiLM). Instead of just scaling and shifting feature maps based on a query, MoVie introduces a learnable weight matrix to compute inner products between the visual feature vector and the query's modulation vector, followed by a residual connection. This allows the model to evaluate the image densely and holistically in a single feed-forward pass, rather than relying on sparse, bottom-up region proposals.
Three-Branch Training
To use MoVie as a "plug-and-play" module for generic Visual Question Answering (VQA) models, the authors use a three-branch training scheme. They train the base VQA model, the MoVie module, and a joint representation simultaneously. This prevents "feature co-adaptation" (where one module relies entirely on the other) and ensures both learn strong, independent representations. During inference, only the joint branch is used, adding minimal computational overhead.
Experimental Results
The authors evaluated MoVie across a wide range of datasets, including counting-specific tasks (HowMany-QA, TallyQA), common object counting (COCO, Pascal VOC), and general reasoning (CLEVR, GQA).
- State-of-the-Art in Counting: On the HowMany-QA dataset, MoVie achieved 64.0% accuracy, a ~4% absolute improvement over prior state-of-the-art while using fewer parameters and FLOPs.
- Object Counting Without Instance Supervision: On the COCO dataset, MoVie outperformed the state-of-the-art (CountSeg) and Faster R-CNN baselines across multiple RMSE metrics, notably doing so without requiring instance-level supervision (bounding boxes).
- Winning the VQA Challenge: When plugged into the MCAN-Large VQA architecture, MoVie boosted accuracy specifically on "number" questions. This integration helped the team win 1st place in the 2020 VQA Challenge.
- Beyond Counting: MoVie achieved a near-perfect 97.42% accuracy on the CLEVR reasoning dataset, proving the architecture is highly capable of general logical and spatial reasoning.
Conclusion
MoVie demonstrates that dense convolutional features, when properly modulated by a query, can implicitly learn to isolate and count objects more effectively than complex symbolic counters. By utilizing local fusion and multi-scale training, MoVie overcomes traditional CNN limitations in scale invariance. The results on CLEVR and GQA suggest that MoVie's architecture can serve as a general-purpose mechanism for a wide spectrum of visual reasoning tasks beyond counting.
Summary updated