Mixed Objective Optimization for Language Generation

122 concepts96 papers1 proposal

When training language models to perform goal-oriented tasks—such as translating a sentence, writing executable code, or maintaining an empathetic dialogue—we encounter a fundamental tension between two training paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

SFT trains models token-by-token using Maximum Likelihood Estimation (MLE). While this ensures the model learns the grammatical and syntactic structures of human language, it suffers from exposure bias: because the model is only exposed to ground-truth prefixes during training (teacher forcing), its errors compound rapidly when generating text autonomously at inference time. Furthermore, MLE treats all incorrect tokens equally, failing to optimize for sequence-level goals like translation quality (BLEU) or code execution success.

RL solves this by optimizing sequence-level rewards directly, allowing the model to explore alternative paths and learn from the functional outcomes of its generations. However, unconstrained RL introduces a severe failure mode: language degeneration (or reward hacking). Left to maximize a sparse reward function—such as passing a unit test or satisfying a sentiment classifier—a language model will quickly discover degenerate, ungrammatical shortcuts that maximize the reward metric but result in complete gibberish.

To bridge this gap, researchers utilize mixed objective optimization (or hybrid supervised-reinforcement learning). By combining token-level supervised losses with sequence-level reinforcement learning rewards, this approach uses the supervised objective as a linguistic anchor, forcing the model to optimize for task success within the strict boundaries of natural human expression.


The Supervised Loss as a Linguistic Anchor

The most direct way to implement a mixed objective is through a joint loss function that linearly interpolates between the token-level cross-entropy loss and the sequence-level policy gradient reward:

$$\mathcal{L}_{\text{hybrid}} = \alpha \mathcal{L}_{\text{RL}} + (1 - \alpha) \mathcal{L}_{\text{SFT}}$$

In this formulation, the SFT term acts as a dense regularizer. Even if a highly repetitive or ungrammatical sequence manages to trick a reward model, the SFT loss will heavily penalize the model because such transitions are highly improbable under the human data distribution.

This joint optimization prevents the model from drifting into incoherent states. For instance, in neural machine translation, Lijun Wu et al. demonstrated that integrating a token-level MLE loss directly into a sequence-level RL objective preserves linguistic fluency while allowing the model to maximize global translation quality metrics like BLEU. Similarly, Go Yasui et al. showed that a small cross-entropy component is essential for maintaining training stability when using semantic textual similarity models as rewards.

This balancing act is equally vital in open-domain and empathetic dialogue systems. Pure RL optimization can cause chatbots to output safe, repetitive phrases or lose grammatical coherence entirely. To combat this, Jamin Shin et al. integrated an MLE loss term to ensure that a chatbot optimizing for long-term user sentiment look-ahead remains grammatically correct and contextually relevant. The authors of Reinforcing an Image Caption Generator Using Off-Line Human Feedback observed a similar stabilization effect, noting that the hybrid loss prevents the model from losing its foundational linguistic capabilities during offline policy updates.


Interleaved Training Curricula

Rather than combining losses into a single equation at every step, an alternative approach alternates between supervised and reinforcement learning updates throughout the training cycle. This strategy periodically pulls the model back toward the human language manifold after periods of strategic exploration.

   ┌─────────────────────────────────────────────────────────┐
   │                                                         │
   ▼                                                         │
[ SFT Update ] ──► Anchors model to human syntax             │ (Iterative Loop)
   │                                                         │
   ▼                                                         │
[ RL Update  ] ──► Explores strategies for task success ─────┘

In task-oriented visual dialogue, where an agent must ask questions to identify an object in an image, pure RL agents often invent idiosyncratic, private codes to coordinate efficiently, rendering their output incomprehensible to humans. Mingyang Zhou et al. proved that alternating between RL epochs for task success and supervised epochs for language modeling serves as a powerful regularization technique, keeping the communication protocol grounded in natural language. This concept is mirrored in multi-agent environments; Abhinav Gupta et al. utilized hybrid supervised-reinforcement grounding to prevent population-based agents from developing private languages, ensuring they remain capable of coordinating with humans.

This curriculum-based approach is also highly effective for structured text generation. In code translation, Prithwish Jana et al. designed a framework that alternates between policy optimization (using compiler and symbolic execution feedback) and standard supervised fine-tuning. This prevents the model from losing its general linguistic structure while it learns to satisfy rigid execution constraints.

To make this process more granular, Yijun Tian et al. introduced a token-budgeting approach during intermediate training. Instead of applying RL globally, their method applies reinforcement learning exclusively to specific reasoning-heavy tokens while maintaining standard next-token prediction for the rest of the sequence, allowing the model to develop specialized logical capabilities without degrading its broad knowledge base.


Mitigating Structural and Stylistic Drift

Mixed objective optimization is uniquely suited for tasks where the evaluation metric is non-differentiable or where the model must satisfy strict structural constraints without sacrificing naturalness.

Code and Formal Language Generation

When generating regular expressions or code blocks, a model faces the problem of program aliasing—multiple distinct syntactic strings can represent identical logical operations. Standard SFT penalizes a model for generating a valid solution if it does not match the exact character sequence of the training label. By incorporating RL rewards based on functional execution alongside an MLE loss, frameworks can navigate this ambiguity.

Zexuan Zhong et al. demonstrated that this combination allows a model to receive credit for functionally correct regular expressions that deviate syntactically from the ground truth. To ensure that these explored patterns do not devolve into unreadable or hallucinated syntax, Chenhui Mao et al. used a hybrid objective to enforce adherence to standard coding conventions while optimizing for unit test execution success.

Controlled Text Generation and Summarization

In abstractive summarization, models trained purely on token-matching objectives tend to blindly copy source text rather than synthesizing it. Romain Paulus addressed this by combining MLE with policy gradients to optimize for non-differentiable metrics like n-gram novelty, forcing the model to generate original phrasing while maintaining readability. When factual accuracy is the primary constraint, Clément Rebuffel et al. showed that combining a data-grounding RL reward with an MLE loss prevents the model from producing disjointed or robotic text while strictly enforcing truthfulness.

Similarly, when steering the style, length, or lexical attributes of a model, mixed objectives ensure that the constraint satisfaction does not ruin text quality:

  • Length Control: Renlong Jie et al. integrated an SFT loss term into an RL loop optimizing for strict output lengths, proving that models can be forced to follow rigid structural rules without losing the linguistic performance established during pre-training.
  • Stylistic Steering: Hrituraj Singh et al. utilized a cross-entropy loss term alongside a stylistic reward function to prevent a model from drifting into incoherent word lists when attempting to match an author's specific lexical profile.
  • Politeness and Attribute Control: Tong Niu et al. combined an RL loss driven by a politeness classifier with a standard maximum-likelihood objective, allowing dialogue agents to dial up manners while remaining contextually coherent.

By anchoring the fluid, goal-seeking exploration of reinforcement learning with the structural discipline of supervised learning, mixed objective optimization allows language models to achieve functional utility without losing their most critical asset: the ability to generate natural, human-readable language.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_supervised_reinforcement_learning_b5cd37f5,
  title = {hybrid supervised-reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-supervised-reinforcement-learning/rdr_b5cd37f526e7416e8bcc060b76add001},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_supervised_reinforcement_learning_b5cd37f5_v1,
  title = {hybrid supervised-reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-supervised-reinforcement-learning/rdr_b5cd37f526e7416e8bcc060b76add001/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake