Abstract
In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.1% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation.
Introduction
In standard image classification, the network "head" typically employs Global Average Pooling (GAP) to compress spatial feature maps into a single vector. While computationally efficient, GAP is sub-optimal for complex scenes containing multiple objects of varying scales and positions, as it discards critical spatial context.
Recent advancements have explored Transformer-based decoders to "query" spatial data for specific classes. However, these architectures face a significant scalability hurdle: their computational complexity grows quadratically ($O(N^2)$) with the number of classes ($N$). This limitation makes them impractical for "extreme classification" tasks involving thousands of labels, such as the Open Images dataset. ML-Decoder addresses this by providing the performance benefits of attention mechanisms with the efficiency required for large-scale deployment.
Architectural Innovations
ML-Decoder streamlines the standard Transformer-decoder through two primary modifications:
1. Self-Attention Removal
The authors identified that the self-attention block within the decoder is largely redundant for classification tasks. By removing it, the computational complexity is reduced from quadratic to linear ($O(N)$). This optimization significantly accelerates the model without compromising accuracy, enabling attention-based heads to operate on resource-constrained devices.
2. Group-Decoding
To achieve even greater scalability, ML-Decoder decouples the number of attention queries from the total number of classes. Instead of maintaining one query per class, it utilizes a fixed set of $K$ "group queries." These queries are subsequently expanded to the final $N$ class logits via a novel Group Fully-Connected layer. This ensures that the computationally intensive spatial-pooling phase remains independent of the class count, allowing the model to scale to datasets with tens of thousands of labels.
Versatility and Generalization
Beyond standard classification, ML-Decoder is designed for versatility. It can utilize learnable latent queries for fixed-label tasks or NLP-based word embeddings for Zero-Shot Learning (ZSL). To enhance generalization to unseen labels, the authors introduced "Query Augmentations," which involve adding noise or random queries during the training phase.
Experimental Results
ML-Decoder was evaluated across diverse benchmarks, demonstrating consistent improvements over traditional GAP heads and specialized architectures:
- Multi-label Classification: On MS-COCO, ML-Decoder achieved 91.1% mAP, setting a new state-of-the-art.
- Zero-Shot Learning: On NUS-WIDE, it reached 31.1% mAP, representing a 4.8% absolute improvement over previous methods.
- Single-label Classification: A vanilla ResNet50 backbone equipped with ML-Decoder reached 80.7% top-1 accuracy on ImageNet, a significant boost achieved without additional data or distillation.
- Extreme Scalability: On the Open Images dataset (~9,600 classes), ML-Decoder maintained high accuracy and efficiency where standard Transformer decoders failed due to memory constraints.
Conclusion
ML-Decoder represents a significant step forward in classification head design. By treating classification as a sophisticated pooling problem and optimizing the underlying attention mechanism, it offers a universal, scalable, and high-performance alternative to Global Average Pooling. Future work may explore semantic clustering within the group-decoding scheme to further refine the mapping between visual features and class labels.
Summary updated