Abstract
Recent multi-modal detectors based on transformers and modality encoders have successfully achieved impressive results on end-to-end visual object detection conditioned on a raw text query. However, they require a large model size and an enormous amount of computations to achieve high performance, which makes it difficult to deploy mobile applications that are limited by tight hardware resources. In this paper, we present a Lightweight modulated detector, Lite-MDETR, to facilitate efficient end-to-end multi-modal understanding on mobile devices. The key primitive is that Dictionary-Lookup-Transformations (DLT) is proposed to replace Linear Transformation (LT) in multi-modal detectors where each weight in Linear Transformation (LT) is approximately factorized into a smaller dictionary, index, and coefficient. This way, the enormous linear projection with weights is converted into efficient linear projection with dictionaries, a few lookups and scalings with indices and coefficients. DLT can be applied to any pretrained multi-modal detectors, removing the need to perform expensive training from scratch. To tackle the challenging training of DLT due to non-differentiable index, we convert the index and coefficient into a sparse matrix, train this sparse matrix during the fine-tuning phase, and recover it back to index and coefficient during the inference phase. Our experiments on phrase grounding, referring expression comprehension and segmentation, and VQA show that our Lite-MDETR achieves similar accuracy as the prior multi-modal detectors with up to ~ 4.1 × model size reduction.
Introduction
Multi-modal detectors like MDETR (Modulated Detection TRansformer) have set new standards for tasks requiring a deep understanding of the interplay between visual and textual data. By processing raw text queries alongside images, these models can identify and segment objects with remarkable precision. However, their massive size—often exceeding 450MB—poses a significant barrier to deployment on resource-constrained mobile and edge devices.
The primary bottleneck in these architectures is the Linear Transformation (LT) layer, which accounts for approximately 90% of the total model parameters. While traditional compression methods like pruning or knowledge distillation are available, they often necessitate expensive re-training from scratch on massive datasets. Lite-MDETR addresses this by introducing a novel Dictionary-Lookup-Transformation (DLT) primitive that allows for significant compression by "recycling" pre-trained weights, achieving up to a 4.1x reduction in model size without sacrificing performance.
Methodology: Dictionary-Lookup-Transformation (DLT)
The core innovation of Lite-MDETR is the replacement of standard Linear Transformation (LT) layers with the DLT primitive. In a standard LT layer, an input is multiplied by a large, dense weight matrix. DLT factorizes this weight matrix into three more efficient components: a small Dictionary ($D$) of basis vectors, an Index ($I$) pointing to specific vectors, and a Coefficient ($C$) for scaling.
During inference, the model avoids massive matrix multiplications. Instead, it performs a small projection to generate a temporary dictionary, followed by efficient lookups and scalings. This shift from heavy arithmetic to memory-efficient operations is particularly beneficial for mobile hardware.
Training and Optimization
A significant challenge in implementing DLT is that index lookups are non-differentiable, preventing standard backpropagation. To overcome this, the authors employ a multi-step training strategy:
- Initialization: Pre-trained weights are decomposed using Singular Value Decomposition (SVD) to initialize the dictionary.
- Sparse Matrix Proxy: During fine-tuning, indices and coefficients are represented as a single sparse matrix ($B$), which is differentiable and can be optimized using standard gradients.
- Deployment Conversion: Post-training, the sparse matrix is converted back into discrete indices and coefficients for maximum inference efficiency.
Experimental Evaluation
Lite-MDETR was evaluated across a diverse set of vision-language benchmarks:
- Referring Expression Comprehension (REC): RefCOCO, RefCOCO+, and RefCOCOg.
- Phrase Grounding (PG): Flickr30k.
- Referring Expression Segmentation (RES): PhraseCut.
- Visual Question Answering (VQA): GQA dataset.
The results demonstrate that Lite-MDETR maintains high accuracy while significantly reducing the memory footprint. For instance, on the RefCOCO benchmark, a 261MB Lite-MDETR-T model achieved 85.4% accuracy, outperforming much larger models like UNITER-L and VILLA-L (both 1212MB).
Key Results and Discussion
The "Lite-MDETR-TTQ" variant, which incorporates 16-bit quantization, reduced the original MDETR model from 450MB to just 110MB—a 4.1x compression. In VQA tasks, the model size was reduced from 490MB to 130MB with only a marginal 3% drop in accuracy.
A key insight from the study is the varying sensitivity of model components. The multi-modal transformer, which fuses text and image features, proved more sensitive to compression than the text encoder. By adjusting the dictionary size for these specific components, the authors were able to fine-tune the balance between efficiency and accuracy.
Conclusion
Lite-MDETR presents a robust framework for compressing heavy multi-modal models for mobile deployment. By leveraging the DLT primitive, it bypasses the need for expensive pre-training while delivering state-of-the-art performance in a lightweight package. Future work may explore even more aggressive quantization (e.g., 8-bit or 4-bit) and automated tuning of dictionary ratios to further optimize the trade-off between model size and intelligence.
Summary updated