PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization

Hao Jiang, Xin Li, Annan Wang, Zhi Yang, Haoxiang Zhang, Yichi Zhang, Weisi LinarXiv 2026 · arXiv

TL;DR

Generative listwise ranking with Large Multimodal Models (LMMs) breaks down over long contexts due to "parse collapse," where models silently omit candidates and terminate early. PRISMR solves this by using a lightweight hypernetwork to encode multimodal candidates in parallel into instance-specific LoRA weights, moving candidate information from fragile prompt tokens into structured parameter space. This approach eliminates parse collapse, boosts ranking accuracy, and cuts inference latency compared to traditional constrained decoding.

What Problem Does This Paper Address?

Generative listwise ranking uses Large Multimodal Models (LMMs) to evaluate an entire list of candidates in a single forward pass, capturing global context and item-to-item interactions. However, when candidate lists grow long and contain interleaved text and dense visual features, LMMs suffer from severe attention dilution and the "lost in the middle" phenomenon.

The authors identify a catastrophic failure mode called parse collapse: the autoregressive decoder produces fluent text but fails to adhere to structural constraints, silently omitting candidates or terminating early. This isn't just a simple formatting error that prompt engineering or constrained decoding can easily fix; it is a fundamental context-utilization bottleneck. Furthermore, as shown in the paper's analysis, adding more images to the context directly exacerbates this collapse.

Figure 1: Conventional generative listwise ranking feeds all multimodal candidates into the decoder context, causing attention dilution and parse collapse as the list grows.

Methods

The authors introduce PRISMR (Parameterized Representation Internalization for Semantic Multimodal Ranking). Instead of forcing the LMM to digest a massive, complex multimodal prompt, PRISMR offloads candidate processing to a lightweight, shared hypernetwork ($H_\phi$).

Figure 2: Overview of PRISMR. A shared hypernetwork Hϕ maps each multimodal candidate di to an item-specific LoRA adapter (Ai, Bi). The N adapters are combined by one of two zero-cost synthesis modes of the same checkpoint—rank-dimension concatenation (α-mode) or mean pooling (β-mode)—yielding a composite increment ∆W mounted on the frozen LMM for listwise decoding. PRISMR uses α-mode for N ≤50 and β-mode for N >50 by default.
  1. Parallel Encoding: The hypernetwork processes each multimodal candidate (text + images) independently and in parallel, predicting low-rank LoRA factors ($A_i, B_i$) for the base LMM.
  2. Adapter Synthesis: The per-item LoRA weights are combined into a single composite weight increment ($\Delta W$) mounted onto the frozen base LMM. PRISMR introduces two zero-cost test-time synthesis modes:
  3. $\alpha$-mode (Rank-Dimension Concatenation): Sums the updates ($\sum B_i A_i$), allowing the effective rank to scale with list size. This provides high capacity for in-distribution list lengths ($N \le 50$).
  4. $\beta$-mode (Mean Pooling): Averages the updates ($\frac{1}{N} \sum B_i A_i$), keeping the update scale bounded. This ensures stability during length extrapolation ($N > 50$).
  5. Optimization: The model is trained using a joint objective: token-level distillation (Negative Log-Likelihood) to mirror a teacher model's scoring format, combined with a listwise LambdaRank-style NDCG loss to enforce correct relative ordering.

Datasets

The paper introduces a large-scale, domain-specific multimodal review-ranking benchmark derived from Amazon Reviews 2023.

  • Structure: Contains 9,962 products and 880,554 user reviews across three categories: Baby Products, Amazon Fashion, and Software. Each example consists of 10 to 100 reviews containing titles, text, and user-uploaded images.
  • Ground Truth: Listwise supervision is provided by a frontier teacher model, validated to have a 90.6% pairwise agreement with human annotators.

For zero-shot and cross-domain transfer evaluation, the authors also test on two off-the-shelf multimodal Information Retrieval benchmarks: INQUIRE-Rerank (nature photos) and MMDocIR (PDF document screenshots).

Key Insights and Ideas

  • Parametric vs. In-Context Conditioning: Moving candidate data out of the prompt context window and into the model's weight space (via dynamically generated LoRAs) fundamentally bypasses the attention limits that cause parse collapse.
  • The Capacity-Robustness Trade-off: How you aggregate dynamically generated weights matters. Concatenating rank dimensions ($\alpha$-mode) maximizes expressiveness but can cause weight drift on very long lists. Mean pooling ($\beta$-mode) sacrifices some fine-grained capacity but provides excellent stability when generalizing to list lengths unseen during training.
  • Structural vs. Semantic Transfer: PRISMR's structural fix (eliminating parse collapse) is task-general and transfers zero-shot to entirely different domains. However, the semantic ranking capability remains domain-specific and requires brief in-domain fine-tuning to align with new feature distributions.

Key Results

  • Eradicating Parse Collapse: On the Baby Products dataset with a list length of $N=50$, the base Qwen3-VL model suffered a catastrophic parse rate of just 2.0%. PRISMR achieved a 99.9% to 100% parse rate across both synthesis modes.
  • Ranking Quality: PRISMR significantly outperformed strong baselines. At $N=50$, PRISMR ($\alpha$-mode) achieved an NDCG@10 of 0.947, compared to 0.597 for RankGPT and 0.789 for Constrained-decoding Base.
  • Inference Efficiency: Because PRISMR avoids repeatedly calculating long-context attention over the entire candidate list during decoding, it scales much better. At $N=50$, PRISMR achieved up to a 1.7× speedup over constrained decoding baselines.
Figure 3: Left: NDCG@10 vs. list length N. Both PRISMR modes are produced by the same checkpoint trained at N=50 (dotted line). α-mode performs best in range (N ≤50), while β-mode is more robust for extrapolation (N > 50), motivating the default switch at N=50. The resulting PRISMR envelope outperforms the trained baselines across all N. Right: per-product latency on a single B200 (batch size 1, mean over 5 products). PRISMR is consistently faster than constrained- decoding Base, with speed-up increasing from 1.1× at N=10 to 1.7× at N=50.

Limitations and Future Work

While PRISMR pushes empirical parse rates close to 100%, its format compliance is not formally guaranteed by hard decoding constraints. Additionally, the authors note that review ranking is inherently subjective, meaning no single ground-truth ordering can perfectly capture all human preferences when candidate qualities are highly similar.

Future work could explore tighter integration with hard decoding constraints or extending this hypernetwork-based internalization paradigm to other long-context multimodal tasks beyond ranking.

Summary updated

Abstract

Generative listwise ranking with Large Multimodal Models (LMMs) aims to capture global list context in a single forward pass, but its effectiveness degrades in long-context multimodal scenarios. We identify a recurring failure mode, parse collapse, where the autoregressive decoder produces fluent yet incomplete rankings by silently omitting candidates and terminating early. This failure stems from limited context utilization rather than simple formatting mistakes, making prompt engineering and constrained decoding insufficient. We propose PRISMR (Parameterized Representation Internalization for Semantic Multimodal Ranking), a framework that replaces transient in-context list processing with parametric structural conditioning. PRISMR uses a lightweight hypernetwork to encode multimodal candidates in parallel and generate item-specific LoRA weights, which are synthesized into an instance-specific adapter for a LMM. This paradigm enables more robust internalization of list structure while preserving the base model. We further introduce a large-scale multimodal review-ranking benchmark for evaluation. Experiments demonstrate that PRISMR substantially reduces parse collapse, improves listwise ranking performance, and transfers effectively across domains and instruction-tuned backbones.
BibTeX
@misc{jiang2026prismr,
  author = {Hao Jiang and Xin Li and Annan Wang and Zhi Yang and Haoxiang Zhang and Yichi Zhang and Weisi Lin},
  title = {{PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization}},
  year = {2026},
  eprint = {2606.12942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.12942},
}

Loading related content...

Figure