Abstract
Transformer-based models have revolutionized Visual Question Answering (VQA) but remain computationally expensive and difficult to deploy on resource-constrained devices. We present the Bilaterally Slimmable Transformer (BST), a framework that enables a single model to be dynamically adjusted in both width and depth at runtime. By training one elastic model, developers can extract various submodels optimized for different hardware profiles without retraining. Our results show that BST-enhanced models achieve superior efficiency-accuracy trade-offs across multiple VQA architectures and benchmarks.
Introduction
Transformer-based architectures have set new benchmarks in VQA, yet their deployment is often hindered by their massive parameter counts and high computational requirements (FLOPs). This "heaviness" makes them impractical for capacity-restricted platforms like smartphones or IoT devices. Existing solutions, such as pruning or knowledge distillation, typically yield a single static compact model. Supporting multiple devices with varying power profiles usually necessitates designing and training separate models for each.
The Bilaterally Slimmable Transformer (BST) addresses this lack of elasticity. It provides a "one-size-fits-all" architecture capable of adapting its complexity on the fly based on available resources, allowing a single trained model to serve a wide range of hardware constraints.
The Bilaterally Slimmable Transformer (BST) Framework
The BST framework introduces several innovations to make Transformers elastic:
- Bilateral Slimming: Unlike previous methods that only adjust model width (hidden dimensions), BST adjusts both width (dimensionality of layers) and depth (number of layers).
- Triangle Selection Strategy: To avoid training inefficient architectures, the authors employ a "triangle" filtering method. This strategy prioritizes "deep-and-narrow" configurations, which are generally more efficient than shallow-and-wide ones, thereby reducing training overhead.
- Slim-Middle Depth Strategy: Recognizing that the initial and final layers are most critical for input processing and output generation, BST prioritizes pruning middle layers first.
- One-Stage Distillation Training: The framework uses a weight-sharing approach where a full-sized "teacher" model guides the simultaneous training of various smaller "student" submodels. This synergy often results in submodels performing better than if they were trained in isolation.
Experimental Evaluation
The researchers integrated BST into three popular VQA architectures: MCAN, UNITER, and CLIP-ViL. These models were evaluated on standard benchmarks including VQA-v2 and GQA, with Visual Genome used for data augmentation during from-scratch training.
Key Results and Insights
The BST framework demonstrates significant improvements in efficiency and flexibility across different training paradigms.
- Efficiency and Accuracy: A slimmed version of the MCAN model achieved accuracy comparable to the original while being 0.38× smaller and requiring 0.27× fewer FLOPs.
- Mobile Readiness: The smallest MCAN-BST submodel, with only 9M parameters, achieved an inference latency of less than 60ms on a Snapdragon 888, enabling real-time mobile VQA.
- The "Deep-and-Narrow" Advantage: The study confirms that for a fixed computational budget, increasing depth is more economical than increasing width for Transformers.
- Weight-Sharing Synergy: Training multiple submodels within a single weight-sharing framework improves the performance of smaller models, as they inherit high-quality features from larger configurations.
- U-Shaped Layer Importance: Visualization of layer importance justified the "slim-middle" strategy, showing that input and output layers carry more weight than middle layers.
Conclusion and Future Work
While BST significantly enhances VQA elasticity, some limitations remain. The initial embedding layers and final classifier currently represent a fixed computational cost. Future work may involve making these components adjustable or developing more dynamic importance scoring for layers. Additionally, applying bilateral slimming to visual backbones (e.g., ResNets or ViTs) could further optimize total system efficiency.
Summary updated