Traditional training for sequence generation relies on a rigid comparison between a model's output and a ground-truth reference. Whether using Maximum Likelihood Estimation (MLE) or Reinforcement Learning (RL) with metrics like BLEU or ROUGE, these systems suffer from a "brittleness" problem: they reward exact word overlaps and penalize valid paraphrases. If a model says "the film was excellent" instead of "the movie was great," a lexical reward function treats it as a failure.
To overcome this, researchers have shifted toward Semantic Similarity Rewards. Instead of matching tokens, these methods reward models based on how closely the meaning of their output aligns with a target, typically measured in a continuous embedding space.
The Failure of Lexical Overlap
Lexical metrics like BLEU and ROUGE are discrete and sparse. They cannot recognize synonyms or capture the underlying intent of a sentence. As noted by Hang Li and Xin Jiang in their work on paraphrase generation, these metrics reward surface-level repetition rather than semantic equivalence. This leads to "exposure bias," where a model becomes overly dependent on its own previous (potentially flawed) tokens during generation because it was never taught to prioritize global meaning over local word choice.
Moving to the Embedding Space
The core intuition of semantic rewards is to project both the generated text and the reference into a high-dimensional vector space using a pre-trained encoder. The reward is then calculated as the cosine similarity between these two vectors.
Zhou Zhao and Zhu Zhang demonstrated this in long-form video question answering, where they used embedding similarity to encourage models to capture the semantic essence of complex events. By rewarding the model for being "close enough" in meaning, the RL agent explores a wider variety of linguistically valid responses. This approach has been echoed in specialized fields like genomics; Zhizheng Wang and Qiao Jin used semantic encoders to reward Gene-R1 for conceptual accuracy, ensuring the model captures scientific truth even when its terminology varies from the gold standard.
Learned Evaluators as Reward Functions
Beyond simple cosine similarity, researchers are using "learned metrics"—neural models specifically trained to mimic human judgment. Kang Min Yoo and Raphael Shu showed that optimizing Neural Machine Translation (NMT) models using BLEURT (a learned metric) as a reward signal leads to translations that humans find significantly more adequate than those optimized for BLEU.
These learned evaluators provide a "denser" reward signal. In tasks like keyphrase generation, Yichao Luo and Qi Zhang utilized a transformer-based reward function to recognize "near-misses." This allows the RL process to distinguish between a completely irrelevant word and a synonym, providing a smoother optimization landscape that helps the model converge faster.
Foundation Models as Semantic Judges
A recent trend leverages the "common sense" of large-scale foundation models to provide zero-shot rewards. In robotics, Yuchen Cui and Scott Niekum used CLIP similarity scores as proxy rewards, allowing robots to learn tasks from images or sketches without hand-coded reward functions.
Similarly, the LMA3 framework by Cédric Colas and Pierre-Yves Oudeyer uses an LLM as a "semantic judge." The agent describes its own actions in text, and the LLM compares that description to a goal. This transforms the RL problem from matching raw sensor data to matching high-level concepts, enabling agents to learn a massive repertoire of human-relevant skills through "cultural transmission."
Stability and Consistency
While semantic rewards are powerful, they can be noisy if the underlying embedding model is biased. To counter this, researchers like Yuze Liu have proposed hybrid rewards that combine fuzzy textual similarity with embedding-based scores to stabilize training.
Furthermore, semantic rewards are being used to enforce consistency. Faisal Hamman introduced group similarity rewards for Retrieval-Augmented Generation (RAG), where the model is rewarded for producing semantically similar answers across different phrasings of the same question. This ensures that the model's "knowledge" remains stable, regardless of how a user asks a query.
By shifting the optimization target from what words are said to what those words mean, semantic similarity rewards allow RL agents to move past rote memorization toward a more robust, human-like understanding of language and tasks.