ML-Decoder: Scalable and Versatile Classification Head

Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben Baruch, Asaf Noy

Abstract

In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.1% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation.

Introduction

In standard image classification, the network "head" typically employs Global Average Pooling (GAP) to compress spatial feature maps into a single vector. While computationally efficient, GAP is sub-optimal for complex scenes containing multiple objects of varying scales and positions, as it discards critical spatial context.

Recent advancements have explored Transformer-based decoders to "query" spatial data for specific classes. However, these architectures face a significant scalability hurdle: their computational complexity grows quadratically ($O(N^2)$) with the number of classes ($N$). This limitation makes them impractical for "extreme classification" tasks involving thousands of labels, such as the Open Images dataset. ML-Decoder addresses this by providing the performance benefits of attention mechanisms with the efficiency required for large-scale deployment.

ML-Decoder overview showing versatility across tasks and superior computational scaling compared to standard Transformer decoders.

Architectural Innovations

ML-Decoder streamlines the standard Transformer-decoder through two primary modifications:

1. Self-Attention Removal

The authors identified that the self-attention block within the decoder is largely redundant for classification tasks. By removing it, the computational complexity is reduced from quadratic to linear ($O(N)$). This optimization significantly accelerates the model without compromising accuracy, enabling attention-based heads to operate on resource-constrained devices.

2. Group-Decoding

To achieve even greater scalability, ML-Decoder decouples the number of attention queries from the total number of classes. Instead of maintaining one query per class, it utilizes a fixed set of $K$ "group queries." These queries are subsequently expanded to the final $N$ class logits via a novel Group Fully-Connected layer. This ensures that the computationally intensive spatial-pooling phase remains independent of the class count, allowing the model to scale to datasets with tens of thousands of labels.

Comparison of the standard Transformer-Decoder and the optimized ML-Decoder architecture, highlighting self-attention removal and group decoding for improved scalability.

Versatility and Generalization

Beyond standard classification, ML-Decoder is designed for versatility. It can utilize learnable latent queries for fixed-label tasks or NLP-based word embeddings for Zero-Shot Learning (ZSL). To enhance generalization to unseen labels, the authors introduced "Query Augmentations," which involve adding noise or random queries during the training phase.

Experimental Results

ML-Decoder was evaluated across diverse benchmarks, demonstrating consistent improvements over traditional GAP heads and specialized architectures:

  • Multi-label Classification: On MS-COCO, ML-Decoder achieved 91.1% mAP, setting a new state-of-the-art.
  • Zero-Shot Learning: On NUS-WIDE, it reached 31.1% mAP, representing a 4.8% absolute improvement over previous methods.
  • Single-label Classification: A vanilla ResNet50 backbone equipped with ML-Decoder reached 80.7% top-1 accuracy on ImageNet, a significant boost achieved without additional data or distillation.
  • Extreme Scalability: On the Open Images dataset (~9,600 classes), ML-Decoder maintained high accuracy and efficiency where standard Transformer decoders failed due to memory constraints.

Conclusion

ML-Decoder represents a significant step forward in classification head design. By treating classification as a sophisticated pooling problem and optimizing the underlying attention mechanism, it offers a universal, scalable, and high-performance alternative to Global Average Pooling. Future work may explore semantic clustering within the group-decoding scheme to further refine the mapping between visual features and class labels.

Summary updated

Abstract

In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.1% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation. Public code will be available.
BibTeX
@inproceedings{ridnik2023ml,
  author = {Tal Ridnik and Gilad Sharir and Avi Ben-Cohen and Emanuel Ben Baruch and Asaf Noy},
  title = {{ML-Decoder: Scalable and Versatile Classification Head}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=jw9rCDpufaI},
}
Figure