Abstract
Large vision models (LMs) offer powerful feature extraction for downstream tasks but often suffer from significant redundancy and computational bloat. This paper presents E3RP (Enhancing Low-Rank Adaptation with Recoverability-Based Reinforcement Pruning), a framework designed to optimize large models for object counting. By employing a Deep Q-Network (DQN) agent to adaptively prune up to 63% of redundant parameters and introducing a Patch Query Flip Attention (PQFA) mechanism, E3RP achieves state-of-the-art accuracy on drone-view and surveillance datasets while maintaining a lightweight footprint.
Introduction
Object counting—the task of estimating the number of instances (e.g., pedestrians, vehicles) in an image—is a fundamental challenge in computer vision. Two primary obstacles hinder current approaches: scale variation, where objects appear at vastly different sizes due to perspective, and model bloat, where large-scale foundational models like the Segment Anything Model (SAM) contain excessive parameters for specific counting tasks. While Low-Rank Adaptation (LoRA) provides an efficient fine-tuning path, it leaves the underlying redundant weights of the large model intact. E3RP addresses these issues by integrating reinforcement learning-based pruning with a scale-aware attention mechanism.
Methodology
The E3RP framework consists of three core components designed to balance efficiency with representational power.
1. Reinforcement Pruning (DQN-P)
Rather than relying on manual heuristics for model compression, E3RP utilizes a Deep Q-Network (DQN) agent to navigate the pruning search space. The agent iteratively determines the optimal pruning ratio for different layers, effectively "testing" the impact of removing weights on the model's performance. This automated search ensures that the pruning process is tailored to the specific data distribution of the counting task.
2. Recoverability-Based Reward
To guide the DQN agent, the authors developed a "recoverability" metric. This reward function evaluates how well the pruned model preserves the essential features of the original large model. It combines cosine similarity with a specialized "action classification" network. This dual-signal approach provides a more robust feedback mechanism than simple similarity metrics, ensuring the agent retains critical knowledge while discarding "dead weight."
3. Patch Query Flip Attention (PQFA)
To mitigate the scale variation problem, E3RP introduces the Patch Query Flip Attention (PQFA) mechanism within the decoder. By flipping query tokens, the model facilitates bidirectional interactions between patches of varying sizes. This allows the network to better align features across different scales, improving counting accuracy for both large foreground objects and small background instances.
Experimental Results
The researchers evaluated E3RP across four benchmark datasets covering remote sensing (CARPK, PUCPR) and crowd counting (Shanghai Tech Part A and B).
Performance and Efficiency
- Parameter Reduction: The method successfully reduced total parameters by up to 63% without sacrificing performance.
- State-of-the-Art Accuracy: E3RP achieved the lowest Mean Absolute Error (MAE) on drone-view datasets. On the CARPK dataset, it reached an MAE of 4.18, outperforming specialized models like HKINet (4.89).
- Comparison to LoRA: In head-to-head tests on the PUCPR dataset, E3RP (utilizing SAM-B + DQN-P + LoRA) achieved an MAE of 1.40, significantly better than standard LoRA's 1.90, while using nearly 40% fewer parameters (56.41M vs 93.83M).
Conclusion
E3RP demonstrates that pruning is not merely a post-processing step for efficiency but a vital component of the adaptation process for large models. By treating pruning as a reinforcement learning problem guided by feature recoverability, the framework effectively removes noise and focuses the model on task-specific features. Future work will explore extending this reinforcement pruning logic to multimodal models and temporal video data for real-time surveillance applications.
Summary updated