TL;DR
This paper introduces TransCODE, a framework for co-designing transformer models and specialized hardware accelerators to achieve efficient training and inference. It tackles the challenge of deploying large transformer models on resource-constrained devices by proposing DynaProp, a dynamic pruning technique for activations and gradients, and ELECTOR, a flexible accelerator simulator.
What Problem Does This Paper Address?
Transformer models, like BERT and GPT, have revolutionized AI but are computationally demanding, making them difficult to deploy on resource-constrained hardware, especially for training. Existing accelerators often focus only on inference or are limited to specific models and hardware. This paper addresses the need for a holistic approach that optimizes both the transformer architecture and its accompanying hardware for efficient training and inference, considering factors like latency, energy consumption, and chip area.
Methods
The core of TransCODE lies in its hardware-software co-design approach. Key components include:
- DynaProp: A novel, low-overhead dynamic pruning technique that reduces memory and computation by skipping ineffectual operations during both training (pruning activations, weights, and gradients) and inference. This is achieved by dynamically pruning values below a certain threshold.
- ELECTOR Framework: A flexible simulator that models a diverse design space of accelerators. It supports various transformer architectures from the FlexiBERT 2.0 design space and can simulate their training and inference on different hardware configurations. ELECTOR includes hardware modules for key operations like matrix multiplication, softmax, and layer normalization.
- TransCODE Optimizer: This framework leverages a Bayesian optimization technique (BOSHCODE) to search for the optimal transformer-accelerator pair. It uses a surrogate model to predict performance metrics (accuracy, latency, energy, area) and iteratively queries for new design points.
Datasets
The models were evaluated on tasks from the GLUE benchmark, including SST-2, MNLI, QQP, QNLI, MRPC, COLA, STS-B, RTE, and WNLI. The paper uses these datasets to evaluate the accuracy of the proposed transformer models.
Key Insights and Ideas
- Training is significantly more resource-intensive than inference: The paper highlights that transformer training requires substantially more memory and computation due to gradients and additional activation operations.
- Dynamic pruning is crucial for efficiency: DynaProp's ability to prune activations, weights, and gradients at runtime significantly reduces memory footprint and computational overhead, making on-device training feasible.
- Co-design unlocks significant gains: Optimizing both the model architecture and the hardware simultaneously, rather than independently, leads to superior performance compared to using pre-designed accelerators or models.
- Flexibility in accelerators is key: The ELECTOR framework's ability to simulate a vast design space of accelerators allows for tailoring hardware to specific transformer architectures and task requirements.
Key Results
The TransCODE framework achieved state-of-the-art results:
- The converged transformer-accelerator pair achieved 0.3% higher accuracy than the state-of-the-art.
- It incurred 5.2× lower latency and 3.0× lower energy consumption.
- DynaProp training showed a 2.3× reduction in training time on the ELECTOR accelerator compared to traditional training on a GPU.
- DynaProp enabled significant memory savings, requiring 1.5× to 1.9× less memory for training BERT-Tiny and BERT-Base models.
- The ELECTOR framework supports a design space of 14,850,000 accelerators.
Limitations and Future Work
- Reconfigurability: While ELECTOR supports a wide range of accelerators, the hardware architectures themselves are not dynamically reconfigurable at runtime, which could be beneficial for highly dynamic workloads.
- Multi-objective Optimization: The current co-design pipeline uses a linear combination of objectives. Extending this to true multi-objective optimization could yield a richer set of Pareto-optimal solutions.
- Co-design for Groups of Transformers: The current framework focuses on co-designing for a single transformer. Extending this to optimize for a group of transformers could be a valuable future direction.
- In-memory and Reconfigurable Processors: The paper notes that the framework optimizes for specific accelerators, and future work could explore integration with in-memory or reconfigurable processors.
Summary updated