Abstract: In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.4% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation.
Introduction
In computer vision, the "classification head" is the final part of a network that turns internal features into class predictions. For years, the industry standard has been Global Average Pooling (GAP), which averages all spatial information into a single vector. While efficient, GAP is "spatial-blind"—it struggles when an image contains multiple objects of different sizes and locations, particularly in multi-label classification.
ML-Decoder is a novel, attention-based classification head designed to replace GAP. It achieves state-of-the-art accuracy across multi-label, single-label, and zero-shot learning tasks while remaining computationally efficient even when dealing with thousands of classes. Unlike standard Transformer-decoders, which suffer from a "quadratic bottleneck" ($O(N^2)$) as the number of classes ($N$) increases, ML-Decoder provides the accuracy of an attention mechanism with the scalability of a simple linear layer.
Methods
The authors introduce two primary architectural innovations to the standard transformer-decoder to achieve this scalability:
- Self-Attention Removal: Standard decoders perform self-attention between class queries. The authors discovered this is redundant for classification because the queries are already processed by a projection layer. Removing this step drops the complexity from quadratic ($O(N^2)$) to linear ($O(N)$) relative to the number of classes.
- Group-Decoding: To further decouple the cost from the class count, ML-Decoder uses a fixed number of "group queries" ($K$) rather than one query per class ($N$). These $K$ queries (where $K \ll N$) interact with the image features via cross-attention. A novel "group fully-connected" layer then expands these group features into the final $N$ class predictions. This makes the heavy attention math independent of the total number of classes.
For Zero-Shot Learning (ZSL), the model uses fixed NLP word embeddings as queries, allowing it to generalize to unseen classes by leveraging semantic relationships between words. Novel query augmentations, such as "random-query" and "query-noise" during training, further improve its generalization ability by forcing the model to become more robust at recognizing novel query vectors.
Key Results and Insights
The model was rigorously tested on benchmarks including MS-COCO (multi-label), NUS-WIDE (zero-shot), and ImageNet-1K (single-label).
- State-of-the-Art Performance: ML-Decoder achieved 91.4% mAP on MS-COCO and 31.1% ZSL mAP on NUS-WIDE, outperforming previous solutions significantly. On ImageNet, it improved a vanilla ResNet50 to a top-1 score of 80.7% without extra data.
- Efficiency and Scalability: On the "extreme" Open Images dataset (9600 classes), ML-Decoder outperformed GAP-based models where standard transformer-decoders failed due to memory limits. The authors demonstrate that ML-Decoder provides a better speed-accuracy trade-off than simply using a larger backbone; it is more efficient to have a "smarter" head than a "heavier" body.
- Versatility: ML-Decoder acts as a "drop-in" replacement for GAP across various tasks. Its design proves that class queries do not need to interact via self-attention to produce high-quality predictions, which is the key to its scalability.
Limitations and Future Work
While highly efficient, ML-Decoder still introduces more FLOPs than a basic GAP layer. Future research could extend the group-decoding concept to other spatial tasks like object detection, segmentation, and video recognition, or apply the architecture to non-visual fields involving large-scale label sets in NLP.
Summary updated