Towards Efficient and Elastic Visual Question Answering with Doubly Slimmable Transformer

Zhou Yu, Zitian Jin, Jun Yu, Mingliang Xu, Jianping Fan

Abstract

Transformer-based models have revolutionized Visual Question Answering (VQA) but remain computationally expensive and difficult to deploy on resource-constrained devices. We present the Bilaterally Slimmable Transformer (BST), a framework that enables a single model to be dynamically adjusted in both width and depth at runtime. By training one elastic model, developers can extract various submodels optimized for different hardware profiles without retraining. Our results show that BST-enhanced models achieve superior efficiency-accuracy trade-offs across multiple VQA architectures and benchmarks.

Introduction

Transformer-based architectures have set new benchmarks in VQA, yet their deployment is often hindered by their massive parameter counts and high computational requirements (FLOPs). This "heaviness" makes them impractical for capacity-restricted platforms like smartphones or IoT devices. Existing solutions, such as pruning or knowledge distillation, typically yield a single static compact model. Supporting multiple devices with varying power profiles usually necessitates designing and training separate models for each.

The Bilaterally Slimmable Transformer (BST) addresses this lack of elasticity. It provides a "one-size-fits-all" architecture capable of adapting its complexity on the fly based on available resources, allowing a single trained model to serve a wide range of hardware constraints.

The Bilaterally Slimmable Transformer (BST) Framework

The BST framework introduces several innovations to make Transformers elastic:

The BST framework allows training one elastic model and deploying multiple submodels to various devices without retraining.
  1. Bilateral Slimming: Unlike previous methods that only adjust model width (hidden dimensions), BST adjusts both width (dimensionality of layers) and depth (number of layers).
  2. Triangle Selection Strategy: To avoid training inefficient architectures, the authors employ a "triangle" filtering method. This strategy prioritizes "deep-and-narrow" configurations, which are generally more efficient than shallow-and-wide ones, thereby reducing training overhead.
  3. Slim-Middle Depth Strategy: Recognizing that the initial and final layers are most critical for input processing and output generation, BST prioritizes pruning middle layers first.
  4. One-Stage Distillation Training: The framework uses a weight-sharing approach where a full-sized "teacher" model guides the simultaneous training of various smaller "student" submodels. This synergy often results in submodels performing better than if they were trained in isolation.

Experimental Evaluation

The researchers integrated BST into three popular VQA architectures: MCAN, UNITER, and CLIP-ViL. These models were evaluated on standard benchmarks including VQA-v2 and GQA, with Visual Genome used for data augmentation during from-scratch training.

Key Results and Insights

The BST framework demonstrates significant improvements in efficiency and flexibility across different training paradigms.

Accuracy vs. number of parameters for BST-enhanced models (red lines) compared to various static baselines in both from-scratch and pre-trained settings.
  • Efficiency and Accuracy: A slimmed version of the MCAN model achieved accuracy comparable to the original while being 0.38× smaller and requiring 0.27× fewer FLOPs.
  • Mobile Readiness: The smallest MCAN-BST submodel, with only 9M parameters, achieved an inference latency of less than 60ms on a Snapdragon 888, enabling real-time mobile VQA.
  • The "Deep-and-Narrow" Advantage: The study confirms that for a fixed computational budget, increasing depth is more economical than increasing width for Transformers.
  • Weight-Sharing Synergy: Training multiple submodels within a single weight-sharing framework improves the performance of smaller models, as they inherit high-quality features from larger configurations.
  • U-Shaped Layer Importance: Visualization of layer importance justified the "slim-middle" strategy, showing that input and output layers carry more weight than middle layers.

Conclusion and Future Work

While BST significantly enhances VQA elasticity, some limitations remain. The initial embedding layers and final classifier currently represent a fixed computational cost. Future work may involve making these components adjustable or developing more dynamic importance scoring for layers. Additionally, applying bilateral slimming to visual backbones (e.g., ResNets or ViTs) could further optimize total system efficiency.

Summary updated

Abstract

Recent Transformer architectures (Vaswani et al., 2017) have brought remarkable improvements to visual question answering (VQA). Nevertheless, Transformer-based VQA models are usually deep and wide to guarantee good performance, so they can only run on powerful GPU servers and cannot run on capacity-restricted platforms such as mobile phones. Therefore, it is desirable to learn an elastic VQA model that supports adaptive pruning at runtime to meet the efficiency constraints of different platforms. To this end, we present the bilaterally slimmable Transformer (BST), a general framework that can be seamlessly integrated into arbitrary Transformer-based VQA models to train a single model once and obtain various slimmed submodels of different widths and depths. To verify the effectiveness and generality of this method, we integrate the proposed BST framework with three typical Transformer-based VQA approaches, namely MCAN (Yu et al., 2019), UNITER (Chen et al., 2020), and CLIP-ViL (Shen et al., 2021), and conduct extensive experiments on two commonly-used benchmark datasets. In particular, one slimmed MCAN$_\mathsf {BST}$ submodel achieves comparable accuracy on VQA-v2, while being 0.38× smaller in model size and having 0.27× fewer FLOPs than the reference MCAN model. The smallest MCAN$_\mathsf {BST}$ submodel only has 9 M parameters and 0.16 G FLOPs during inference, making it possible to deploy it on a mobile device with less than 60 ms latency.
BibTeX
@article{yu2023towards,
  author = {Zhou Yu and Zitian Jin and Jun Yu and Mingliang Xu and Jianping Fan},
  title = {{Towards Efficient and Elastic Visual Question Answering with Doubly Slimmable Transformer}},
  journal = {IEEE Transactions on Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=8Gyd8IbZfL},
}
Figure