SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation

Shrikant Kendre, Austin Xu, Honglu Zhou, Michael Ryoo, Shafiq Joty, Juan Carlos Niebles

Introduction

Evaluating the accuracy of Question-Answering (QA) systems remains a significant challenge in natural language processing. Traditional metrics such as Exact Match (EM) and ROUGE are often too rigid, penalizing correct answers that use synonyms or different phrasing. Conversely, modern embedding-based metrics like BERTScore frequently suffer from "distributional misalignment," where the length discrepancy between a concise ground-truth answer and a verbose model response leads to inaccurate scoring.

To circumvent these issues, researchers have increasingly adopted "LLM-as-judge" approaches (e.g., using GPT-4o). However, these methods are computationally expensive, slow, and susceptible to hallucinations and "concreteness bias," where the judge is swayed by a model's confident tone rather than factual accuracy. We introduce SMILE (Semantic Metric Integrating Lexical Exactness), a lightweight, high-performance alternative designed to address these limitations.

Methodology: The SMILE Framework

SMILE is a composite framework that evaluates QA outputs through a two-step process designed to ensure both semantic relevance and factual precision:

  1. Synthetic Answer Generation (Preprocessing): To resolve the length mismatch problem, SMILE utilizes a lightweight language model (such as Llama-3.2-3B) to transform short ground-truth labels into full sentences. This aligns the style of the reference with the typical output of an AI assistant, creating a more effective baseline for comparison.
  2. Composite Scoring: The final evaluation is derived from two distinct sub-scores:
  3. Semantic Subscore ($s_s$): This measures the overall conceptual similarity by comparing the embeddings (using models like ember-v1) of the model’s response and the synthetic reference.
  4. Keyword Subscore ($s_l$): This component specifically rewards the presence of the exact ground-truth answer or highly similar keywords, ensuring the model provides the specific factual information required.

The final SMILE score is a weighted average of these components, providing a tunable balance between general meaning and lexical exactness.

Key Insights: Alignment and Lexical Precision

A primary insight of this work is that massive models are not required for effective evaluation. A small 3B-parameter model can successfully rephrase ground truths to align distributions, significantly improving the accuracy of subsequent mathematical comparisons. Furthermore, while semantic embeddings are powerful, lexical exactness remains critical for factoid QA. SMILE’s keyword subscore prevents the metric from being "tricked" by fluent but factually incorrect responses.

Qualitative comparison of SMILE against other metrics across diverse QA benchmarks.

As shown in the qualitative analysis above, SMILE correctly identifies valid answers in cases where GPT-4o may hallucinate due to verbosity or where embedding-only metrics (like sBERT) overrate relevant but incorrect answers.

Experimental Results and Efficiency

SMILE was evaluated across nine established benchmarks spanning Natural Language QA (e.g., HotpotQA), Visual QA (e.g., DocVQA), and Video QA (e.g., TGIF). To validate the metric, a "golden evaluation set" of 225 samples was annotated by four domain experts.

Performance and Correlation

SMILE achieved a Pearson correlation of 0.790 with human judgments, surpassing both GPT-4o (0.784) and GPT-3.5 (0.453). Unlike LLM-based judges, SMILE is prompt-independent; its mathematical structure ensures stable scoring regardless of minor phrasing changes in the evaluation request.

Efficiency and Cost

SMILE offers a significant advantage in terms of resource consumption:

  • Latency: SMILE is approximately 9x faster than querying an LLM via API.
  • Cost: While benchmarking a single dataset with GPT-4o can cost upwards of $12, SMILE has virtually zero marginal cost after initial preprocessing and can run efficiently on a standard CPU.
SMILE occupies the 'sweet spot' of high correlation and low latency.

Conclusion and Future Work

SMILE demonstrates that a composite, lightweight approach can outperform expensive LLM-based judges in QA evaluation. By combining synthetic alignment with a dual-focus on semantics and keywords, it provides a robust, stable, and cost-effective metric for researchers.

Future work will focus on extending SMILE to long-form creative writing and multi-hop reasoning tasks, as well as integrating context-awareness to ensure the metric can identify errors that are dependent on the provided source text or images.

Summary updated

BibTeX
@article{kendre2026smile,
  author = {Shrikant Kendre and Austin Xu and Honglu Zhou and Michael Ryoo and Shafiq Joty and Juan Carlos Niebles},
  title = {{SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2026},
}
Figure