Introduction
Large language models (LLMs) like GPT-3 and ATHENA have demonstrated remarkable capabilities in solving complex mathematical word problems. However, their deployment on edge devices—such as smartphones or embedded systems—is hindered by their massive parameter counts and high computational costs. The primary bottleneck is the Multi-Head Attention (MHA) mechanism, where every head in every layer performs intensive calculations. Research indicates that many of these heads are redundant or introduce noise, particularly in tasks requiring strict symbolic logic and numerical precision.
Methodology
To address these efficiency challenges, we propose a framework combining dynamic attention head pruning with recursive knowledge distillation.
Dynamic Attention Head Pruning
Unlike static pruning, our approach evaluates the importance of each attention head in real-time during training. We calculate a joint importance score based on two primary factors:
- Weight Norm: The physical magnitude of the parameters.
- Attention Entropy: A measure of focus. Low entropy indicates a head is attending to specific, critical tokens, while high entropy suggests redundancy.
The pruning ratio is increased gradually to minimize performance shocks.
Recursive Knowledge Distillation
To recover accuracy lost during pruning, we employ a recursive distillation process. An unpruned "teacher" model trains a slightly pruned "student." This student then serves as the teacher for a subsequent, more aggressively pruned student. This iterative "relay race" allows the model to adapt progressively to smaller architectures.
Experimental Setup
The framework was evaluated on two benchmarks:
- Math23k: A large-scale Chinese dataset containing over 23,000 primary school math problems.
- ASDiv-A: An English dataset of 2,305 math word problems testing multi-step logical reasoning.
Results and Analysis
Our results indicate that significant efficiency gains can be achieved with minimal impact on reasoning accuracy.
Efficiency and Accuracy Trade-offs
As shown in the table below, increasing the pruning ratio leads to substantial improvements in speed and parameter reduction. At a 30% pruning ratio on the Math23k dataset, the model achieved an 18.7% reduction in parameters and a 27.5% increase in inference speed, with only a 0.7% drop in accuracy.
Key Insights
- Entropy vs. Weight Magnitude: Attention entropy proved to be a superior indicator of head importance. Pruning based solely on weight magnitude caused accuracy to plummet from 84.4% to 71.2%.
- Model Scaling: Larger models (e.g., ATHENA-large) exhibited higher redundancy, making them even more suitable for this pruning strategy.
- Complexity Resilience: The pruned models remained robust on high-complexity problems involving five or more operators or nested logic, maintaining stable performance.
Conclusion
The combination of dynamic pruning and recursive knowledge distillation offers a viable path for deploying high-performance mathematical reasoning models on resource-constrained hardware. By leveraging attention entropy to identify redundant components, we can significantly reduce model size and latency while preserving the logical integrity required for mathematical tasks.
Summary updated