Leveraging Natural Supervision for Language Representation Learning and Generation

Mingda ChenarXiv 2022 · DOI · arXiv

Abstract

Traditional approaches to Natural Language Processing (NLP) often rely on massive, unstructured text corpora, treating language as a flat sequence of tokens. This thesis argues that such "brute force" methods overlook the rich, implicit structures humans embed in data. By leveraging "natural supervision"—including Wikipedia metadata, paraphrase pairs, and document hierarchies—this research introduces novel training objectives and benchmarks. The results demonstrate significant improvements in discourse coherence, controllability in text generation, and the ability of models to handle complex, long-form narratives.

Introduction

The recent evolution of NLP has been driven by scaling models and datasets. However, standard pretraining tasks often fail to capture the sophisticated knowledge embedded in the structure of our data. A Wikipedia article, for instance, is not merely a string of words; it contains a table of contents, hyperlinks, and a rigorous category hierarchy. Similarly, human-generated paraphrases provide a signal that meaning can remain constant even when syntax changes. This research addresses the inefficiencies of standard pretraining, the difficulty of controlling generative output, and the limitations of existing evaluation benchmarks.

Addressing the Limitations of Standard Pretraining

Discourse Coherence and Sentence Ordering

A primary critique of early models like BERT is the "Next Sentence Prediction" (NSP) task. The author identifies that NSP is often too simple; models can solve it by detecting broad topic shifts rather than understanding linguistic coherence. To rectify this, the research introduces Sentence Ordering Prediction (SOP) for the ALBERT (A Lite BERT) architecture. In SOP, the model must distinguish between two consecutive sentences in their original order versus their swapped order. This "difficulty spike" forces the model to learn finer-grained discourse relationships, leading to state-of-the-art results on the GLUE benchmark and SQuAD.

Disentangling Syntax and Semantics

Achieving controllability in text generation requires separating what is being said from how it is said. The author proposes the vMF-Gaussian Variational Autoencoder (VGVAE), a dual-latent variable model. By utilizing a von Mises-Fisher (vMF) distribution to capture semantics and a Gaussian distribution for syntax, and training on paraphrase pairs, the model learns to isolate these factors. This mathematical disentanglement enables "sentential style transfer," allowing researchers to map the meaning of one sentence onto the grammatical structure of another.

Harnessing Natural Supervision

The thesis explores how existing data structures can serve as proxy labels for complex reasoning tasks:

  • Wikipedia as a Knowledge Map: The author utilizes Wikipedia’s category graphs to create WikiNLI, an entailment dataset. For example, the hierarchy implies that a "Jazz" category entails the parent category "Music." Pretraining on this naturally occurring logic improved BERT’s performance on the Multi-Genre Natural Language Inference (MNLI) task by 2.3% in low-resource settings.
  • Intermediate Self-Supervised Finetuning: To bridge the gap between pretraining and human instruction following, the research introduces a "warm-up" stage. Models are trained on self-supervised tasks, such as predicting the final phrase of a sentence, before being finetuned. This approach improved 1.3B parameter models on SuperGLUE few-shot tasks by an average of 4–5 points.

New Benchmarks for Long-Form Generation

To move beyond short, news-centric snippets, the author introduced several datasets designed to test narrative consistency and data-to-text synthesis:

  1. WikiTableT: A massive dataset pairing Wikipedia sections with tabular data for long-form generation.
  2. SummScreen: A summarization dataset using TV show transcripts (e.g., The Big Bang Theory) and fan-written recaps. This reveals that narrative summarization requires deeper social and logical inference than news summarization, as plot points are often implied through dialogue.
  3. TVStoryGen: A dataset focused on generating detailed episode recaps from brief summaries and character descriptions, testing long-term consistency.

Key Insights and Results

The research demonstrates that structured supervision is a viable substitute for manual labeling. Key findings include:

  • NSP Inefficiency: Standard NSP allows models to be "lazy," whereas SOP forces a deeper understanding of logical flow.
  • Logic in the Wild: Wikipedia’s category hierarchy provides a massive, free tree of logical entailments that can teach models common sense.
  • Narrative Complexity: Summarizing stories is fundamentally different from news; models must navigate character consistency and implied subtext, which remains a significant challenge.

Conclusion and Future Directions

While the methods proposed—such as SOP and VGVAE—significantly advance the state of representation learning, hurdles remain. Models still struggle with "character believability" in long-form stories and occasionally "hallucinate" information in data-to-text tasks. Future work aims to create unified architectures capable of ingesting multiple streams of natural supervision—discourse, entities, and paraphrases—simultaneously to create more robust and controllable AI systems.

Summary updated

Abstract

Recent breakthroughs in Natural Language Processing (NLP) have been driven by language models trained on a massive amount of plain text. While powerful, deriving supervision from textual resources is still an open question. For example, language model pretraining often neglects the rich, freely-available structures in textual data. In this thesis, we describe three lines of work that seek to improve the training and evaluation of neural models using naturally-occurring supervision. We first investigate self-supervised training losses to help enhance the performance of pretrained language models for various NLP tasks. Specifically, we alter the sentence prediction loss to make it better suited to other pretraining losses and more challenging to solve. We design an intermediate finetuning step that uses self-supervised training to promote models' ability in cross-task generalization. Then we describe methods to leverage the structures in Wikipedia and paraphrases. In particular, we propose training losses to exploit hyperlinks, article structures, and article category graphs for entity-, discourse-, entailment-related knowledge. We propose a framework that uses paraphrase pairs to disentangle semantics and syntax in sentence representations. We extend the framework for a novel generation task that controls the syntax of output text with a sentential exemplar. Lastly, we discuss our work on tailoring textual resources for establishing challenging evaluation tasks. We introduce three datasets by defining novel tasks using various fan-contributed websites, including a long-form data-to-text generation dataset, a screenplay summarization dataset, and a long-form story generation dataset. These datasets have unique characteristics offering challenges to future work in their respective task settings.
BibTeX
@misc{chen2022leveraging,
  author = {Mingda Chen},
  title = {{Leveraging Natural Supervision for Language Representation Learning and Generation}},
  year = {2022},
  eprint = {2207.10617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.10617},
}
Figure