Enhancing Low-Rank Adaptation with Recoverability-Based Reinforcement Pruning for Object Counting

Haojie Guo, Junyu Gao, Yuan YuanAAAI 2025 · DOI

Abstract

Large vision models (LMs) offer powerful feature extraction for downstream tasks but often suffer from significant redundancy and computational bloat. This paper presents E3RP (Enhancing Low-Rank Adaptation with Recoverability-Based Reinforcement Pruning), a framework designed to optimize large models for object counting. By employing a Deep Q-Network (DQN) agent to adaptively prune up to 63% of redundant parameters and introducing a Patch Query Flip Attention (PQFA) mechanism, E3RP achieves state-of-the-art accuracy on drone-view and surveillance datasets while maintaining a lightweight footprint.

Introduction

Object counting—the task of estimating the number of instances (e.g., pedestrians, vehicles) in an image—is a fundamental challenge in computer vision. Two primary obstacles hinder current approaches: scale variation, where objects appear at vastly different sizes due to perspective, and model bloat, where large-scale foundational models like the Segment Anything Model (SAM) contain excessive parameters for specific counting tasks. While Low-Rank Adaptation (LoRA) provides an efficient fine-tuning path, it leaves the underlying redundant weights of the large model intact. E3RP addresses these issues by integrating reinforcement learning-based pruning with a scale-aware attention mechanism.

Methodology

The E3RP framework consists of three core components designed to balance efficiency with representational power.

1. Reinforcement Pruning (DQN-P)

Rather than relying on manual heuristics for model compression, E3RP utilizes a Deep Q-Network (DQN) agent to navigate the pruning search space. The agent iteratively determines the optimal pruning ratio for different layers, effectively "testing" the impact of removing weights on the model's performance. This automated search ensures that the pruning process is tailored to the specific data distribution of the counting task.

2. Recoverability-Based Reward

To guide the DQN agent, the authors developed a "recoverability" metric. This reward function evaluates how well the pruned model preserves the essential features of the original large model. It combines cosine similarity with a specialized "action classification" network. This dual-signal approach provides a more robust feedback mechanism than simple similarity metrics, ensuring the agent retains critical knowledge while discarding "dead weight."

The E3RP architecture featuring the DQN Pruning Agent and the Scale-Aware Counting Head.

3. Patch Query Flip Attention (PQFA)

To mitigate the scale variation problem, E3RP introduces the Patch Query Flip Attention (PQFA) mechanism within the decoder. By flipping query tokens, the model facilitates bidirectional interactions between patches of varying sizes. This allows the network to better align features across different scales, improving counting accuracy for both large foreground objects and small background instances.

The Patch Query Flip Attention (PQFA) mechanism facilitates bidirectional interaction for scale-aware feature alignment.

Experimental Results

The researchers evaluated E3RP across four benchmark datasets covering remote sensing (CARPK, PUCPR) and crowd counting (Shanghai Tech Part A and B).

Performance and Efficiency

  • Parameter Reduction: The method successfully reduced total parameters by up to 63% without sacrificing performance.
  • State-of-the-Art Accuracy: E3RP achieved the lowest Mean Absolute Error (MAE) on drone-view datasets. On the CARPK dataset, it reached an MAE of 4.18, outperforming specialized models like HKINet (4.89).
  • Comparison to LoRA: In head-to-head tests on the PUCPR dataset, E3RP (utilizing SAM-B + DQN-P + LoRA) achieved an MAE of 1.40, significantly better than standard LoRA's 1.90, while using nearly 40% fewer parameters (56.41M vs 93.83M).

Conclusion

E3RP demonstrates that pruning is not merely a post-processing step for efficiency but a vital component of the adaptation process for large models. By treating pruning as a reinforcement learning problem guided by feature recoverability, the framework effectively removes noise and focuses the model on task-specific features. Future work will explore extending this reinforcement pruning logic to multimodal models and temporal video data for real-time surveillance applications.

Summary updated

BibTeX
@inproceedings{guo2025enhancing,
  author = {Haojie Guo and Junyu Gao and Yuan Yuan},
  title = {{Enhancing Low-Rank Adaptation with Recoverability-Based Reinforcement Pruning for Object Counting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32334},
}
Figure