Standard generative models are trained to maximize the likelihood of the next token. However, the sequence with the highest joint probability is not always the most accurate, factual, or logically sound. This discrepancy—often stemming from exposure bias or the limitations of maximum likelihood estimation—has led to a powerful paradigm: generating a diverse set of candidate outputs and using a secondary "judge" to select the best one.
The Limits of Likelihood
When a model generates text, it typically uses decoding strategies like beam search or top-p sampling to find high-probability sequences. But as Bryan Wilie and colleagues observe in PICK, the "best" response in terms of conversational relevance or factuality is often present in the candidate pool but buried below the top-ranked sequence.
The core intuition is that recognition is easier than generation. A model might struggle to generate a perfectly factual sentence on its first try, but a specialized scoring mechanism can often identify which of ten generated sentences is the most accurate. This approach allows researchers to optimize for specific quality dimensions—like logical validity in argumentative conclusions Anette Frank or grammatical correctness Hwee Tou Ng—that raw likelihood scores ignore.
The Reranking Pipeline
The reranking workflow generally follows a three-step process:
- Generation: The primary model produces $k$ candidates using sampling or beam search.
- Scoring: A secondary "Quality Estimation" (QE) model or metric evaluates each candidate.
- Selection: The candidate with the highest score is chosen as the final output.
In Neural Quality Estimation of Grammatical Error Correction, Shamil Chollampatt and Hwee Tou Ng demonstrated this by adapting "predictor-estimator" architectures to rank GEC corrections. Similarly, Gueter Josmy Faure and team used a lightweight module called ACE to rerank answers for complex video reasoning in MovieCORE, showing that even a small, efficient model can significantly boost the performance of a much larger primary model.
Reference-Free Evaluation
A major challenge in reranking is how to score a candidate when the "ground truth" is unknown. Traditional metrics like BLEU require a human-written reference, which isn't available at inference time.
To solve this, researchers use reference-free metrics. For example, Alireza Mohammadshahi introduced RQUGE, which evaluates a generated question by checking if a separate model can correctly answer it. If the answer matches the source text, the question is high quality. In translation, António Farinhas and Andre Martins show that Minimum Bayes Risk (MBR) decoding can be used to select the translation that is most "central" or consistent across many samples, effectively using the consensus of the model itself as a quality signal.
Calibration and Test-Time Scaling
If we know we are going to rerank at inference time, can we train the model to be better at it? This is known as calibration-tuning. Griffin Adams and colleagues explored this in What are the Desired Characteristics of Calibration Sets?, finding that teaching a model to rank its own potential errors helps mitigate exposure bias.
This concept is part of a broader trend toward test-time scaling: the idea that we can trade more computation at inference time (by generating and scoring more candidates) for higher quality. Yice Zhang uses a "pseudo-label scorer" to filter synthetic data and rerank predictions in sentiment analysis, proving that this verification layer is essential for high-precision tasks.
The Judge’s Dilemma
While reranking is powerful, it is only as good as the judge. In the JETTS benchmark, Yilun Zhou and Shafiq Joty found that while large language models are competent at reranking full responses, they often struggle with more granular tasks like step-by-step reasoning.
Furthermore, the work of Andrea Pugnana on AUC-based Selective Classification suggests that sometimes the best "reranking" decision is to reject the output entirely if no candidate meets a quality threshold. This "reject option" ensures that the system only provides answers when its internal ranking quality is high, maximizing the reliability of the final output.