When a language model summarizes hundreds of product reviews, it usually sees only a small selected subset of them. Systems that answer from retrieved text, called retrieval-augmented generation, fetch a few passages and pass them to the model. Review corpora are heavily skewed: most e-commerce reviews are positive, and most airline tweets are complaints. The selection step therefore decides which opinions survive into the summary. The methods that work best for factual question answering keep the most popular content and prune low-salience passages as noise, and in a skewed corpus they systematically drop the small, specific, real complaints a buyer most needs to see, such as "the latch broke after two weeks." This note proposes a selection step that tells a rare-but-real complaint apart from genuine noise. The test it uses is corroboration: a rare complaint is repeated independently by a few reviewers, while noise is idiosyncratic or off-topic.
The core idea fits in two sentences. A review passage earns a guaranteed place in the context window when at least a few other reviews say the same specific thing, and "the same specific thing" is judged jointly by exact rare-word overlap and by meaning-level similarity. Neither signal alone is enough. The approach is hybrid retrieval, meaning sparse keyword matching combined with dense neural embeddings, but it is used to define what counts as corroborated evidence rather than to rank documents against a query.
Core Claim
Existing systems split along a clear fault line. Scalable retrieval-based opinion summarizers aggregate by frequency. HERCULES maps review sentences to discrete paths in a learned tree and relies on roughly 8–10 reviews per path. HIRO uses a tf-idf-style prevalence score over a similar hierarchical index. Both select by consensus, so they discard claims held by 2–5% of reviewers. Polarity Calibration (PoCa) documents the downstream symptom, which it calls polarity bias amplification: summaries become more one-sided than their inputs. PoCa fixes this with reinforcement learning on a small fixed input set, with no retrieval stage. It motivates our work, but it acts after the minority evidence is already gone in a retrieval setting.
Methods built to resist distracting context make the problem sharper. Superposition Prompting processes documents in parallel paths and prunes low-salience ones. CompAct iteratively compresses retrieved text and stops early once it judges the evidence sufficient. Both were validated only on factual question answering, where irrelevant documents simply lack the answer. In opinion synthesis, a five-percent complaint cluster has low collective salience and looks exactly like what these methods are designed to discard.
Our claim is that corroboration under a hybrid lexical-and-semantic match separates minority signal from noise, and that it does so better than any single-signal or polarity-only rule. We further claim this matters most for the specific, aspect-level content of the minority view, not merely its overall polarity. A selector that only balances positive against negative can match polarity statistics while still filling the minority slots with generic negativity such as "terrible, don't buy."
Research Program
The selector works per entity, over all of that entity's reviews, in four steps.
First, we find lexical anchors. These are words and phrases that are unusually frequent in this entity's reviews relative to its product category, such as part names, error codes, or flight numbers. We measure this with a smoothed log-odds ratio of in-entity versus in-category frequency. These are the tokens that dense encoders tend to blur, per the entity-blindness and length-capacity findings of Luan et al..
Second, we form candidate claim groups. Two review sentences are linked if they share an anchor and their dense embeddings exceed a similarity threshold. Groups are connected components of this graph. The shared anchor stops dense similarity from merging distinct complaints into one generic negative cluster. The dense threshold stops a shared word from linking unrelated statements: "the latch is sturdy" and "the latch snapped" share an anchor but should be split, which we handle by also requiring agreement from a sentence-level polarity estimate. We also build a dense-only grouping in parallel, for paraphrased claims with no anchor; these must reach a higher support threshold.
Third, we apply the corroboration filter. A group is kept as signal if it is supported by at least $m$ distinct reviews, with $m$ a small constant we sweep from 2 to 4, far below HERCULES's consensus requirement. Single-review groups and groups whose anchors come mostly from off-category text are treated as noise.
Fourth, we allocate the context budget. Given a budget of $B$ passages, each surviving group first receives one slot, filled by its most representative sentence (the one closest to the group's dense centroid). The remaining slots go to groups in proportion to their support. The summary therefore keeps the majority's weight but cannot lose a corroborated minority claim to that weight.
The generator is a fixed instruction-tuned language model that receives the selected passages with their group sizes, such as "(mentioned by 6 of 410 reviewers)," and is asked to summarize with proportions stated. Reporting support counts directly addresses the forced-connection and overconfidence failures noted by Cho et al.. The model is told how much evidence each claim has, rather than being left to infer importance from position.
The method is deliberately polarity-agnostic. It never asks which class is the minority, so it should help negative complaints in positive-skewed Amazon data and rare praise in negative-skewed airline data alike.
Baselines
The naive reference point is to put a random sample of reviews into the same passage budget, and also, where it fits, to give the full review set to a long-context model with an instruction to be balanced. This measures what selection buys at all.
The strongest existing scalable systems are HIRO and HERCULES, run from their released configurations on SPACE and AmaSum. They are the closest competitors: they solve the same many-reviews problem with retrieval, and their consensus scoring is exactly the mechanism we argue erases minorities. We will also run HIRO with its prevalence threshold lowered, so the comparison is not just "a lower threshold."
PoCa is the strongest existing method on polarity faithfulness. On AmaSum it reports polarity-distance error (root-mean-square error between input and summary polarity) falling from 0.2154 to 0.1824, with ROUGE-1 at 28.44 versus 29.23 for its base model (PoCa). Those figures come from its own flan-T5 setup on small input sets, so we will report them as cited. We will also run PoCa on our selected inputs, to test whether calibration and selection are complementary.
Two saliency-pruning readers are included to show the erasure effect directly: Superposition-style path pruning and CompAct-style early-stopping compression. Both get the same retrieved pool.
The key competitor is stratified quota selection. We split reviews by star rating, an oracle-quality polarity label, and alternatively by an off-the-shelf sentiment classifier. We then fill the budget by class proportion with a guaranteed minimum per class, choosing passages within each class by dense centrality. Three ablations isolate our ingredients: dense-only grouping, sparse-only grouping, and anchor boosting without the corroboration filter, which tests whether corroboration is what keeps noise out.
Every system uses the same generator model, the same passage budget $B$ (swept from 10 to 40), the same review pools, and the same tuning budget: a small grid for each method's thresholds on a held-out development split of entities. "Beating" a baseline means higher minority-claim recall at equal or lower unsupported-claim rate, with no more than about one ROUGE point lost against references.
Why not just…?
Why not just stratify by sentiment class and sample proportionally? This is the real threat, and the experiments are designed around it. Polarity stratification will likely match our method on polarity-distance error, since that is what it optimizes. Its weakness, by our hypothesis, is inside the minority class. Generic negative reviews are the most central ones, so they win the minority slots, while specific complaints are rarer even among negatives. The classifier version also inherits the imbalance problem itself: Tiwari et al. report negative-class F1 of only 0.62–0.65 versus 0.88–0.90 for neutral. The decisive comparison is minority-claim recall at fixed budget. If star-stratified sampling matches us there, the hybrid corroboration mechanism is unnecessary and we will say so.
Why not just give everything to a long-context model and ask for balance? PoCa reports that prompting GPT-3.5 and GPT-4 does not prevent polarity bias amplification. The distraction literature, including lost-in-the-middle effects and UDA, predicts that buried specifics are overlooked. We still run this baseline on SPACE, whose 100-review entities fit in context, because frontier models may have improved. A tie there would narrow our claim to corpora larger than the context window, such as AmaSum's roughly 560 reviews per entity.
Why not just lower HERCULES's or HIRO's frequency threshold? Lowering the threshold admits rare claims, but it also admits rare noise, because frequency alone cannot tell them apart. Their latent tree paths are dense-quantized and may still merge distinct specific complaints. The lowered-threshold HIRO run, together with the no-corroboration ablation, measures whether the unsupported-claim rate rises as minority recall rises.
Why not just use RL calibration like PoCa? PoCa corrects the overall polarity of what the generator writes. It cannot restore a claim that never reached the generator. We test the combination, PoCa applied to our selected inputs, and expect gains to add if selection and calibration address different stages.
Proposed Experiments
Standard reference summaries do not label minority claims, so we build that evaluation layer. For AmaSum, SPACE, and Amazon 5-core entities, a language model extracts atomic claims from every review, and these are clustered. A claim cluster counts as minority if it is supported by 1–5% of an entity's reviews and its polarity is opposite to the majority. Humans verify a subset of 200 entities. Minority-claim recall is the fraction of verified minority claims entailed by the summary, judged with SummaC-style entailment checks plus human spot checks. The unsupported-claim rate is the fraction of summary claims entailed by fewer than $m$ input reviews or by none.
A controlled stress test complements this. We subsample entities to imbalance ratios of 80/20, 90/10, and 97/3. We also inject two kinds of noise: off-topic reviews borrowed from other products in the category, and single-review idiosyncratic complaints. This lets us check that noise does not get promoted as the minority shrinks.
For direction-agnosticism, we use the US Airlines 2016 tweets, grouped per airline and per week, where praise is the minority. For political perspective balance, we use NeuS. We also report ROUGE against references and polarity-distance error, so results are comparable with PoCa's AmaSum and NeuS numbers.
The decisive checks are three. First, at 97/3 imbalance, our selector must beat star-stratified quota selection on minority-claim recall without a higher unsupported-claim rate. Second, removing either the sparse anchor or the dense similarity requirement must measurably hurt: the dense-only version should merge specific complaints, and the sparse-only version should miss paraphrases. If neither ablation hurts, hybridity is decoration. Third, the saliency-pruning readers should show minority recall falling as pruning gets more aggressive, giving the first direct measurement of where factual question-answering noise filtering fails on subjective synthesis.
References
- Attributable and Scalable Opinion Summarization
- Hierarchical Indexing for Retrieval-Augmented Opinion Summarization
- Polarity Calibration for Opinion Summarization
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Twitter-based Opinion Mining for Flight Service utilizing Machine Learning
- U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis