Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding

Zhaokai Wang, Xizhou Zhu, Xue Yang, Gen Luo, Hao Li, Changyao Tian, Wenhan Dou, Junqi Ge, Lewei Lu, Yu Qiao, Jifeng Dai

Abstract

Traditional image pyramids enhance multi-scale feature capture but incur prohibitive computational costs by applying large models across all resolutions. We present Parameter-Inverted Image Pyramid Networks (PIIP), an architecture that reverses this convention by processing high-resolution inputs with lightweight models and low-resolution inputs with heavy semantic models. By integrating cross-branch feature interactions, PIIP achieves up to a 60% reduction in FLOPs while improving performance in object detection, segmentation, and multimodal large language models (MLLMs).

Introduction

Capturing both global context and fine-grained detail is a fundamental challenge in computer vision. Standard "image pyramid" approaches address this by feeding multiple image resolutions into a shared neural network. However, because computational complexity (FLOPs) scales quadratically with resolution, running a massive Vision Transformer (ViT) on high-resolution images is inefficient. This bottleneck often forces a compromise on input resolution, sacrificing the fine details necessary for tasks like small object detection or OCR in multimodal understanding.

The PIIP Architecture

The core innovation of PIIP is the Parameter-Inverted design. Instead of a uniform model across scales, PIIP utilizes a multi-branch system where model capacity is inversely proportional to image resolution.

Comparison of visual perception and multimodal understanding paradigms, highlighting the proposed Parameter-Inverted Image Pyramid (h).

Key Components:

  1. Inverted Branches: A large, semantically rich model (e.g., ViT-Large) processes low-resolution images to extract global context. Simultaneously, a lightweight model (e.g., ViT-Tiny or ConvNeXt) processes high-resolution images to capture local details.
  2. Cross-Branch Interactions: To prevent isolated processing, "interaction units" using deformable cross-attention are placed between branches. This allows high-resolution spatial details to be informed by low-resolution semantic depth.
  3. Heterogeneous Fusion: PIIP supports mixing architectures, such as using CNNs for high-res branches (leveraging their local inductive bias) and ViTs for low-res branches (leveraging global attention).

Experimental Results

The authors evaluated PIIP across standard benchmarks for perception and multimodal understanding:

  • Efficiency and Accuracy: When integrated with the InternViT-6B foundation model, PIIP reduced FLOPs by 43% for object detection and 58% for segmentation. Despite the lower compute, performance increased by 1.9% Box AP on COCO and 1.3% mIoU on ADE20K.
  • Multimodal Performance: In the PIIP-LLaVA configuration, the model outperformed existing high-resolution baselines like LLaVA-HR by 1.4% on average. With expanded training data, it achieved 73.0% on TextVQA and 74.5% on MMBench, demonstrating superior handling of text-rich, high-resolution imagery.

Key Insights

  • Semantic vs. Detail Requirements: Large models do not require high-resolution inputs to grasp the "gist" of a scene. Conversely, capturing fine edges and textures does not require the deep, complex reasoning of a multi-billion parameter model.
  • The Necessity of Interaction: Simple ensembling of different models is insufficient. Dynamic communication via cross-attention is critical for the branches to effectively fuse multi-scale information.

Conclusion

PIIP challenges the conventional "one-size-fits-all" approach to multi-scale processing. By inverting the relationship between model size and image resolution, it provides a scalable path for high-resolution vision without the typical computational penalty. Future work may extend this paradigm to video and 3D vision tasks.

Summary updated

Abstract

Image pyramids are widely adopted in top-performing methods to obtain multi-scale features for precise visual perception and understanding. However, current image pyramids use the same large-scale model to process multiple resolutions of images, leading to significant computational cost. To address this challenge, we propose a novel network architecture, called Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses pretrained models (ViTs or CNNs) as branches to process multi-scale images, where images of higher resolutions are processed by smaller network branches to balance computational cost and performance. To integrate information from different spatial scales, we further propose a novel cross-branch feature interaction mechanism. To validate PIIP, we apply it to various perception models and a representative multimodal large language model called LLaVA, and conduct extensive experiments on various tasks such as object detection, segmentation, image classification and multimodal understanding. PIIP achieves superior performance compared to single-branch and existing multi-resolution approaches with lower computational cost. When applied to InternViT-6B, a large-scale vision foundation model, PIIP can improve its performance by 1%-2% on detection and segmentation with only 40%-60% of the original computation, finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and 74.5% on MMBench with only 2.8M training data. Our code is released at https://github.com/OpenGVLab/PIIP.
BibTeX
@article{wang2025parameter,
  author = {Zhaokai Wang and Xizhou Zhu and Xue Yang and Gen Luo and Hao Li and Changyao Tian and Wenhan Dou and Junqi Ge and Lewei Lu and Yu Qiao and Jifeng Dai},
  title = {{Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3593283},
}
Figure