Abstract
Traditional image pyramids enhance multi-scale feature capture but incur prohibitive computational costs by applying large models across all resolutions. We present Parameter-Inverted Image Pyramid Networks (PIIP), an architecture that reverses this convention by processing high-resolution inputs with lightweight models and low-resolution inputs with heavy semantic models. By integrating cross-branch feature interactions, PIIP achieves up to a 60% reduction in FLOPs while improving performance in object detection, segmentation, and multimodal large language models (MLLMs).
Introduction
Capturing both global context and fine-grained detail is a fundamental challenge in computer vision. Standard "image pyramid" approaches address this by feeding multiple image resolutions into a shared neural network. However, because computational complexity (FLOPs) scales quadratically with resolution, running a massive Vision Transformer (ViT) on high-resolution images is inefficient. This bottleneck often forces a compromise on input resolution, sacrificing the fine details necessary for tasks like small object detection or OCR in multimodal understanding.
The PIIP Architecture
The core innovation of PIIP is the Parameter-Inverted design. Instead of a uniform model across scales, PIIP utilizes a multi-branch system where model capacity is inversely proportional to image resolution.
Key Components:
- Inverted Branches: A large, semantically rich model (e.g., ViT-Large) processes low-resolution images to extract global context. Simultaneously, a lightweight model (e.g., ViT-Tiny or ConvNeXt) processes high-resolution images to capture local details.
- Cross-Branch Interactions: To prevent isolated processing, "interaction units" using deformable cross-attention are placed between branches. This allows high-resolution spatial details to be informed by low-resolution semantic depth.
- Heterogeneous Fusion: PIIP supports mixing architectures, such as using CNNs for high-res branches (leveraging their local inductive bias) and ViTs for low-res branches (leveraging global attention).
Experimental Results
The authors evaluated PIIP across standard benchmarks for perception and multimodal understanding:
- Efficiency and Accuracy: When integrated with the InternViT-6B foundation model, PIIP reduced FLOPs by 43% for object detection and 58% for segmentation. Despite the lower compute, performance increased by 1.9% Box AP on COCO and 1.3% mIoU on ADE20K.
- Multimodal Performance: In the PIIP-LLaVA configuration, the model outperformed existing high-resolution baselines like LLaVA-HR by 1.4% on average. With expanded training data, it achieved 73.0% on TextVQA and 74.5% on MMBench, demonstrating superior handling of text-rich, high-resolution imagery.
Key Insights
- Semantic vs. Detail Requirements: Large models do not require high-resolution inputs to grasp the "gist" of a scene. Conversely, capturing fine edges and textures does not require the deep, complex reasoning of a multi-billion parameter model.
- The Necessity of Interaction: Simple ensembling of different models is insufficient. Dynamic communication via cross-attention is critical for the branches to effectively fuse multi-scale information.
Conclusion
PIIP challenges the conventional "one-size-fits-all" approach to multi-scale processing. By inverting the relationship between model size and image resolution, it provides a scalable path for high-resolution vision without the typical computational penalty. Future work may extend this paradigm to video and 3D vision tasks.
Summary updated