In the evaluation of machine-generated text, we often face a choice between two goals: ensuring everything the model says is correct (precision) and ensuring the model says everything it was supposed to say (recall).
For decades, the field has leaned heavily on precision-based metrics like BLEU. These metrics act like a strict editor, penalizing the model for every word that doesn't appear in a reference translation. However, a growing body of research suggests that this focus is misplaced. When humans judge the quality of a translation or a summary, they are far more forgiving of extra words than they are of missing information.
The Precision Trap
The fundamental weakness of precision-heavy metrics is that they reward "playing it safe." As Aaron Li-Feng Han, Derek F. Wong, and colleagues point out in Language-independent Model for Machine Translation Evaluation with Reinforced Factors, metrics that focus on precision allow models to achieve high scores by selectively outputting only high-confidence words while omitting critical information.
Consider a translation that is grammatically perfect but leaves out the word "not." A precision-based metric might give this a high score because every word produced was "correct," even though the meaning is now the exact opposite of the source. This imbalance, as noted by Boxing Chen and Roland Kühn in AMBER: A Modified BLEU, Enhanced Ranking Metric, fails to capture the "completeness" that humans value.
Recall as a Proxy for Adequacy
If precision measures fluency, recall measures adequacy—the degree to which the essential meaning of the source is preserved. Bogdan Babych and Tony Hartley demonstrated in Extending the BLEU MT Evaluation Method with Frequency Weightings that prioritizing the recall of "salient" content words (using statistical weights like tf.idf) creates a metric that aligns much more closely with human judgments.
This intuition holds across different tasks:
- Summarization: Piji Li and colleagues found that recall-oriented metrics correlate more strongly with human quality judgments because comprehensive coverage of key information is more critical than extreme conciseness (A Training-free and Reference-free Summarization Evaluation Metric).
- Grammatical Error Correction: In the GREEN metric, Shota Koyama and others argue that prioritizing recall prevents the system from being over-rewarded for "doing nothing" (outputting an unedited sentence).
- High-Stakes Retrieval: In legal or privacy contexts, Moniba Keymanesh and Micha Elsner argue that it is better to provide a slightly wordy summary than to risk omitting a vital legal detail (Privacy Policy Question Answering Assistant).
The Mechanism of Weighted Matching
To make recall work effectively, researchers have moved away from "exact match" constraints. Hwee Tou Ng and Yee Seng Chan introduced MAXSIM, which models evaluation as a maximum weight bipartite matching problem. By allowing "fuzzy" matches—where a model gets credit for using a synonym or a different grammatical form of a reference word—the metric can more accurately measure if the information was recalled, even if the exact vocabulary differs.
Similarly, the TESLA metric by Chang Liu and Daniel Dahlmeier uses linear programming to find the best possible alignment between a translation and a reference, prioritizing recall to better reflect human perceptions of fluency and accuracy.
When Precision Still Matters
The shift toward recall isn't universal. Yvette Graham noted in Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE that precision-based variants can sometimes align better with human assessments of summary quality, particularly when avoiding redundancy is a priority. Furthermore, Ori Shapira and Mohit Bansal caution that pure recall metrics can overlook "noise" or irrelevant content that degrades the user experience (Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation).
However, the consensus across these diverse papers is that for a metric to be "human-like," it must first ensure the message was delivered. As Boxing Chen suggests in Multi-level Evaluation for Machine Translation, recall is the primary differentiator of quality for lower-tier outputs; before we worry about the perfect arrangement of words, we must first ensure the correct words are present.