TL;DR
Generative listwise ranking with Large Multimodal Models (LMMs) breaks down over long contexts due to "parse collapse," where models silently omit candidates and terminate early. PRISMR solves this by using a lightweight hypernetwork to encode multimodal candidates in parallel into instance-specific LoRA weights, moving candidate information from fragile prompt tokens into structured parameter space. This approach eliminates parse collapse, boosts ranking accuracy, and cuts inference latency compared to traditional constrained decoding.
What Problem Does This Paper Address?
Generative listwise ranking uses Large Multimodal Models (LMMs) to evaluate an entire list of candidates in a single forward pass, capturing global context and item-to-item interactions. However, when candidate lists grow long and contain interleaved text and dense visual features, LMMs suffer from severe attention dilution and the "lost in the middle" phenomenon.
The authors identify a catastrophic failure mode called parse collapse: the autoregressive decoder produces fluent text but fails to adhere to structural constraints, silently omitting candidates or terminating early. This isn't just a simple formatting error that prompt engineering or constrained decoding can easily fix; it is a fundamental context-utilization bottleneck. Furthermore, as shown in the paper's analysis, adding more images to the context directly exacerbates this collapse.
Methods
The authors introduce PRISMR (Parameterized Representation Internalization for Semantic Multimodal Ranking). Instead of forcing the LMM to digest a massive, complex multimodal prompt, PRISMR offloads candidate processing to a lightweight, shared hypernetwork ($H_\phi$).
- Parallel Encoding: The hypernetwork processes each multimodal candidate (text + images) independently and in parallel, predicting low-rank LoRA factors ($A_i, B_i$) for the base LMM.
- Adapter Synthesis: The per-item LoRA weights are combined into a single composite weight increment ($\Delta W$) mounted onto the frozen base LMM. PRISMR introduces two zero-cost test-time synthesis modes:
- $\alpha$-mode (Rank-Dimension Concatenation): Sums the updates ($\sum B_i A_i$), allowing the effective rank to scale with list size. This provides high capacity for in-distribution list lengths ($N \le 50$).
- $\beta$-mode (Mean Pooling): Averages the updates ($\frac{1}{N} \sum B_i A_i$), keeping the update scale bounded. This ensures stability during length extrapolation ($N > 50$).
- Optimization: The model is trained using a joint objective: token-level distillation (Negative Log-Likelihood) to mirror a teacher model's scoring format, combined with a listwise LambdaRank-style NDCG loss to enforce correct relative ordering.
Datasets
The paper introduces a large-scale, domain-specific multimodal review-ranking benchmark derived from Amazon Reviews 2023.
- Structure: Contains 9,962 products and 880,554 user reviews across three categories: Baby Products, Amazon Fashion, and Software. Each example consists of 10 to 100 reviews containing titles, text, and user-uploaded images.
- Ground Truth: Listwise supervision is provided by a frontier teacher model, validated to have a 90.6% pairwise agreement with human annotators.
For zero-shot and cross-domain transfer evaluation, the authors also test on two off-the-shelf multimodal Information Retrieval benchmarks: INQUIRE-Rerank (nature photos) and MMDocIR (PDF document screenshots).
Key Insights and Ideas
- Parametric vs. In-Context Conditioning: Moving candidate data out of the prompt context window and into the model's weight space (via dynamically generated LoRAs) fundamentally bypasses the attention limits that cause parse collapse.
- The Capacity-Robustness Trade-off: How you aggregate dynamically generated weights matters. Concatenating rank dimensions ($\alpha$-mode) maximizes expressiveness but can cause weight drift on very long lists. Mean pooling ($\beta$-mode) sacrifices some fine-grained capacity but provides excellent stability when generalizing to list lengths unseen during training.
- Structural vs. Semantic Transfer: PRISMR's structural fix (eliminating parse collapse) is task-general and transfers zero-shot to entirely different domains. However, the semantic ranking capability remains domain-specific and requires brief in-domain fine-tuning to align with new feature distributions.
Key Results
- Eradicating Parse Collapse: On the Baby Products dataset with a list length of $N=50$, the base Qwen3-VL model suffered a catastrophic parse rate of just 2.0%. PRISMR achieved a 99.9% to 100% parse rate across both synthesis modes.
- Ranking Quality: PRISMR significantly outperformed strong baselines. At $N=50$, PRISMR ($\alpha$-mode) achieved an NDCG@10 of 0.947, compared to 0.597 for RankGPT and 0.789 for Constrained-decoding Base.
- Inference Efficiency: Because PRISMR avoids repeatedly calculating long-context attention over the entire candidate list during decoding, it scales much better. At $N=50$, PRISMR achieved up to a 1.7× speedup over constrained decoding baselines.
Limitations and Future Work
While PRISMR pushes empirical parse rates close to 100%, its format compliance is not formally guaranteed by hard decoding constraints. Additionally, the authors note that review ranking is inherently subjective, meaning no single ground-truth ordering can perfectly capture all human preferences when candidate qualities are highly similar.
Future work could explore tighter integration with hard decoding constraints or extending this hypernetwork-based internalization paradigm to other long-context multimodal tasks beyond ranking.
Summary updated