Degeneration and Collusion in Selective Rationalization Models

16 concepts10 papers1 proposal

Selective rationalization is a popular paradigm for making deep learning models interpretable. In a standard "select-then-predict" pipeline, the model is split into two cooperative components: a selector (or generator) that extracts a short, continuous snippet of text from the input (the rationale), and a predictor that makes the final classification using only that snippet.

The intuition behind this architecture is elegant: by forcing the predictor through a strict information bottleneck, the selected text must serve as the justification for the model's decision. If the predictor can accurately guess the sentiment of a movie review based only on three highlighted words, those three words must be highly diagnostic.

However, training these two components cooperatively introduces a severe, systemic vulnerability. Instead of learning to extract human-understandable evidence, the selector and predictor often learn to collude, inventing a secret code to optimize task accuracy without providing any real explanation.

The Mechanics of Model Collusion

To understand why collusion happens, we must look at the optimization dynamics of cooperative training. The selector and predictor are trained jointly to minimize prediction loss. The easiest way for the network to minimize this loss is not by parsing complex human semantics, but by finding statistical shortcuts.

Imagine a student (the selector) and a teacher (the predictor) taking a test together. If they are graded solely on whether the final answer is correct, the student doesn't need to highlight the most informative paragraphs in the textbook. Instead, the student can simply highlight the first word of the page if the answer is "A", and the second word if the answer is "B". The teacher learns to decode this positional signal instantly.

In neural networks, this exact behavior is called rationalization degeneration. As Yiming Zheng et al. demonstrate in The Irrationality of Neural Rationale Models, the selector component frequently bypasses genuine evidence extraction. Instead, it computes the final prediction internally and encodes that label directly into the rationale via non-semantic artifacts.

[Input Text] ---> [ Selector ] ---(Secret Code: e.g., Punctuation/Position)---> [ Predictor ] ---> [ High Accuracy ]
                         \                                                            /
                          `-------------------( Bypasses Semantics )-----------------`

These secret codes can take several forms:

The root cause of this degeneration is a mismatch in learning speeds. In Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint, Wei Liu et al. reveal that the predictor component naturally overfits to early, uninformative noise before the generator has time to discover real semantic patterns. This creates a destructive feedback loop: once the predictor starts relying on the selector's accidental noise, the selector is rewarded for producing more noise, and the model never learns to ground its choices in human logic. This phenomenon is further corroborated by the authors in MGR: Multi-generator Based Rationalization.

Strategies for Enforcing Honesty

To eliminate collusion, researchers have developed methods to break the secret communication channel, forcing the selector and predictor to rely strictly on human-intelligible semantics.

1. Aligning Representations and Learning Rates

If collusion is driven by the predictor overfitting to early noise, one solution is to mathematically restrain the predictor's ability to overfit. In Decoupled Rationalization with Asymmetric Learning Rates, the authors show that simply slowing down the predictor's learning rate relative to the generator enforces a smoothness constraint (Lipschitz continuity). This prevents the predictor from reacting wildly to subtle punctuation codes, forcing it to wait for the generator to find stable, semantic phrases.

Another approach modifies the architecture so the components cannot maintain separate, hidden communication channels. In FR: Folded Rationalization with a Unified Encoder, Wei Liu et al. unify the text encoder across both the explanation generator and the final predictor. By folding the architecture into a single encoder, the model cannot easily split its internal representations to pass hidden codes. Similarly, Marc Felix Brinner et al. propose the Rationalizing Transformer Predictor (RTP), which unifies classification and rationale selection into an end-to-end differentiable single-model pass to eliminate training instabilities and prevent the selector from bypassing the reasoning process.

2. Semantic and Distributional Constraints

Rather than just changing how the models learn, we can explicitly penalize them if the extracted rationale diverges from the meaning of the original text.

[Full Input Text]  ---> [ Encoder ] ---> Full Features ----.
                                                              \  (Must Match)
[Selected Rationale] -> [ Encoder ] ---> Rationale Features --'

In Distribution Matching for Rationalization (DMR), Yongfeng Huang et al. introduce a framework that forces the extracted rationales to share the same internal feature distributions and final predictions as the original full text. If the selector tries to pass a single period as a code, the feature distributions will not match the full text, and the model will be penalized.

Similarly, in Enhancing the Rationale-Input Alignment for Self-explaining Rationalization, Wei Liu et al. introduce DAR (Discriminatively Aligned Rationalization). This framework uses a frozen, pretrained expert model to measure the semantic alignment between the input and the rationale, preventing the "rationale shift" that occurs when components begin prioritizing non-semantic shortcuts.

3. Adversarial Interventions

A third approach introduces an adversary whose sole job is to catch the model cheating. In Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasets, Wei Liu et al. introduce the A2I (Attack to Inspection and Instruction) framework. A2I deploys an adversarial attacker that actively searches for non-semantic shortcuts (like commas or token positions) that the model might be using to predict labels. Once these cheating patterns are exposed, the framework instructs the model to ignore them.

This builds on earlier game-theoretic foundations like Class-wise Adversarial Rationalization (CAR), introduced by Shiyu Chang et al. in A Game Theoretic Approach to Class-wise Selective Rationalization, which frames rationale extraction as a competitive game between factual and counterfactual evidence generators, forcing the model to justify why a piece of text supports one specific outcome over another.

Key Foundational Work

Distribution Matching for Rationalization (2021) — contributed 1 concept elements to this direction.
A Game Theoretic Approach to Class-wise Selective Rationalization (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_rationalization_degeneration_e9d23858,
  title = {rationalization degeneration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rationalization-degeneration/rdr_e9d23858ab164bfcbb373eb353d2b723},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_rationalization_degeneration_e9d23858_v1,
  title = {rationalization degeneration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rationalization-degeneration/rdr_e9d23858ab164bfcbb373eb353d2b723/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake