Regeneration Learning: A Learning Paradigm for Data Generation

Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, Yoshua Bengio

Introduction: The Information Mismatch

In the landscape of machine learning, tasks are often categorized by the relationship between input ($X$) and output ($Y$). In Data Understanding (e.g., image classification), the input $X$ (a high-resolution image) contains significantly more information than the output $Y$ (a single label like "dog"). We use Representation Learning to filter out the noise and extract the essential "dog-ness" into a compact representation $X'$.

Conversely, Data Generation (e.g., text-to-image) presents the opposite challenge: the input $X$ (the word "dog") contains far less information than the target $Y$ (a detailed photo). This creates an "ill-posed" problem where one input can map to infinite valid outputs, often leading to blurry results or overfitting when models attempt to map $X$ directly to $Y$.

Information relationships between source (X) and target (Y) across different learning tasks.

The Regeneration Learning Paradigm

To address this mismatch, the authors propose Regeneration Learning (RL), a formal counterpart to representation learning. RL decomposes the complex generation task into three distinct stages:

  1. Abstraction ($Y \to Y'$): The target $Y$ is converted into a compact, semantic abstraction $Y'$. This can be explicit (e.g., mel-spectrograms in audio) or implicit (e.g., latent codes in a VQ-VAE).
  2. Mapping ($X \to Y'$): The model learns to generate the simplified abstraction $Y'$ from the source $X$. Because $Y'$ is more semantic and lower-dimensional than $Y$, this mapping is significantly easier to learn.
  3. Regeneration ($Y' \to Y$): A "renderer" or decoder reconstructs the full data $Y$ from the abstraction $Y'$. Crucially, this stage can be trained using self-supervised learning on massive amounts of target data alone, without requiring paired $(X, Y)$ samples.
The full conceptual pipeline of Regeneration Learning, mirroring Representation Learning through intermediate abstractions.

This framework highlights a symmetry in AI: while representation learning abstracts the source ($X \to X'$), regeneration learning abstracts the target ($Y \to Y'$). The core "learning" happens in the mapping between these two abstract spaces ($X' \to Y'$).

A Unified Theory for Generative AI

Regeneration Learning provides a formal vocabulary for why many modern state-of-the-art models succeed. For instance, Latent Diffusion Models (like Stable Diffusion) do not operate on raw pixels; they generate a latent representation ($Y'$) which is then "regenerated" into an image via a VAE decoder. Similarly, in Text-to-Speech (TTS), systems often map text to a spectrogram ($Y'$) before a vocoder renders the final waveform ($Y$).

By targeting $Y'$, models can ignore "uncorrelated information"—details in the target that have no relationship to the source. In video synthesis, for example, a person's specific facial texture or background is often irrelevant to the speech they are delivering. Using an intermediate 3D face model as $Y'$ allows the system to focus solely on speech-driven movement.

Future Challenges

While RL offers a powerful framework, several research questions remain. Identifying the "optimal" abstraction $Y'$ is a balancing act: if it is too simple, vital details are lost; if it is too complex, the mapping from $X$ remains difficult. Additionally, researchers must address the "training-inference mismatch," where the renderer ($Y' \to Y$) is trained on perfect ground-truth abstractions but must handle noisy, predicted abstractions during real-world use.

Summary updated

Abstract

Machine learning methods for conditional data generation usually build a mapping from source conditional data X to target data Y. The target Y (e.g., text, speech, music, image, video) is usually high-dimensional and complex, and contains information that does not exist in source data, which hinders effective and efficient learning on the source-target mapping. In this paper, we present a learning paradigm called regeneration learning for data generation, which first generates Y' (an abstraction/representation of Y) from X and then generates Y from Y'. During training, Y' is obtained from Y through either handcrafted rules or self-supervised learning and is used to learn X-->Y' and Y'-->Y. Regeneration learning extends the concept of representation learning to data generation tasks, and can be regarded as a counterpart of traditional representation learning, since 1) regeneration learning handles the abstraction (Y') of the target data Y for data generation while traditional representation learning handles the abstraction (X') of source data X for data understanding; 2) both the processes of Y'-->Y in regeneration learning and X-->X' in representation learning can be learned in a self-supervised way (e.g., pre-training); 3) both the mappings from X to Y' in regeneration learning and from X' to Y in representation learning are simpler than the direct mapping from X to Y. We show that regeneration learning can be a widely-used paradigm for data generation (e.g., text generation, speech recognition, speech synthesis, music composition, image generation, and video generation) and can provide valuable insights into developing data generation methods.
BibTeX
@inproceedings{tan2024regeneration,
  author = {Xu Tan and Tao Qin and Jiang Bian and Tie-Yan Liu and Yoshua Bengio},
  title = {{Regeneration Learning: A Learning Paradigm for Data Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30271},
}
Figure