Abstract
Fine-grained scene classification presents a unique challenge in computer vision, as categories often share nearly identical global visual layouts. Distinguishing between a "Bakery" and a "Cafe," for instance, frequently requires identifying specific textual cues or localized objects rather than relying on general environmental structure. This paper introduces the Modality-Agnostic Adapter (MAA), a framework designed to treat visual and textual information with equal priority. By utilizing a Transformer-based fusion strategy that eliminates "vision-first" architectural biases, the MAA adaptively prioritizes the most informative cues for a given image. Experimental results on the Con-Text and Crowd Activity benchmarks demonstrate that the MAA achieves state-of-the-art performance, highlighting the effectiveness of modality-agnostic feature integration.
Introduction
In standard image classification, models often succeed by identifying broad visual patterns. However, fine-grained scene classification requires a more nuanced approach. In these scenarios, the deciding factor for a classification might be a small piece of text on a storefront sign or a specific object in a window display.
Existing multi-modal models typically employ a hierarchical fusion approach where global visual features serve as the primary "query," and other modalities—such as Optical Character Recognition (OCR) text or local object crops—are treated as auxiliary data. This "vision-first" preference is a fundamental limitation; in many fine-grained contexts, the text is actually the most informative feature. The Modality-Agnostic Adapter (MAA) addresses this by allowing the model to dynamically determine which modality should take precedence on a case-by-case basis.
Methodology
The MAA framework breaks the multi-modal fusion process into two distinct stages: Modal Difference Elimination and Modality-Agnostic Transformer Fusion.
Modal Difference Elimination
Features derived from different specialist models—such as Vision Transformers (ViT) for global images and KnowBert for textual data—possess varying dimensions and statistical distributions. To reconcile these differences, the MAA utilizes independent Multi-Layer Perceptrons (MLPs) to project all diverse features into a unified, shared feature space. This step ensures that the subsequent fusion mechanism can process all inputs as equivalent tokens.
Modality-Agnostic Transformer Fusion
Once projected into the shared space, the features are concatenated and processed by a standard Transformer encoder. A key innovation in this architecture is the intentional omission of position embeddings. By removing positional information, the Transformer is forced to treat every input token—whether it represents an image patch or a word—with equal priority.
Instead of position embeddings, the authors use modality embeddings to identify the source of the feature without implying a hierarchy. This allows the self-attention mechanism to naturally discover semantic relationships across modalities. For example, if a sign in an image reads "Bakery," the model can attend heavily to that specific text token while downplaying generic architectural features.
Experimental Results
The MAA was evaluated on two primary datasets that require high-precision recognition:
- Con-Text: A subset of ImageNet featuring 24,255 images across 28 categories (e.g., pizzeria, pharmacy). This dataset is specifically designed to test the integration of visual and textual cues.
- Crowd Activity: Consisting of 8,785 images across 21 categories of human gatherings (e.g., Graduation Ceremony, Republican Rally), this dataset relies on both environmental context and textual signs.
Performance Metrics
The MAA achieved state-of-the-art (SOTA) results on both benchmarks:
- Con-Text: Reached 90.87% mAP, surpassing the previous best of 89.53%.
- Crowd Activity: Reached 88.29% mAP, an improvement over the previous SOTA of 87.45%.
Qualitative analysis indicated that the inclusion of local visual features (crops of specific objects) significantly boosted model confidence. In instances where a global view appeared ambiguous, local features—such as baskets of bread in a bakery—provided the necessary evidence for correct classification.
Discussion and Insights
The success of the MAA suggests that "Prior Equality" is a powerful concept in multi-modal learning. Because the importance of different modalities is inconsistent across different images, a fixed fusion strategy is inherently sub-optimal. The MAA’s "plug-and-play" nature also allows for easy expansion; adding new modalities like audio or depth data would simply require adding an additional MLP branch without redesigning the core fusion logic.
However, the model remains dependent on the quality of its underlying specialist models. If an OCR system fails to detect text due to complex backgrounds or lighting, the MAA cannot recover that lost information. Furthermore, the researchers observed that increasing the number of Transformer layers beyond two led to overfitting, suggesting that shallow fusion is currently more effective for these specific fine-grained tasks.
Conclusion
The Modality-Agnostic Adapter represents a shift away from vision-centric architectures toward a more balanced, flexible approach to multi-modal fusion. By treating all data sources as equal contributors, the MAA effectively navigates the complexities of fine-grained scene classification, setting new performance standards for the field.
Summary updated