Abstract
Traditional approaches to Natural Language Processing (NLP) often rely on massive, unstructured text corpora, treating language as a flat sequence of tokens. This thesis argues that such "brute force" methods overlook the rich, implicit structures humans embed in data. By leveraging "natural supervision"—including Wikipedia metadata, paraphrase pairs, and document hierarchies—this research introduces novel training objectives and benchmarks. The results demonstrate significant improvements in discourse coherence, controllability in text generation, and the ability of models to handle complex, long-form narratives.
Introduction
The recent evolution of NLP has been driven by scaling models and datasets. However, standard pretraining tasks often fail to capture the sophisticated knowledge embedded in the structure of our data. A Wikipedia article, for instance, is not merely a string of words; it contains a table of contents, hyperlinks, and a rigorous category hierarchy. Similarly, human-generated paraphrases provide a signal that meaning can remain constant even when syntax changes. This research addresses the inefficiencies of standard pretraining, the difficulty of controlling generative output, and the limitations of existing evaluation benchmarks.
Addressing the Limitations of Standard Pretraining
Discourse Coherence and Sentence Ordering
A primary critique of early models like BERT is the "Next Sentence Prediction" (NSP) task. The author identifies that NSP is often too simple; models can solve it by detecting broad topic shifts rather than understanding linguistic coherence. To rectify this, the research introduces Sentence Ordering Prediction (SOP) for the ALBERT (A Lite BERT) architecture. In SOP, the model must distinguish between two consecutive sentences in their original order versus their swapped order. This "difficulty spike" forces the model to learn finer-grained discourse relationships, leading to state-of-the-art results on the GLUE benchmark and SQuAD.
Disentangling Syntax and Semantics
Achieving controllability in text generation requires separating what is being said from how it is said. The author proposes the vMF-Gaussian Variational Autoencoder (VGVAE), a dual-latent variable model. By utilizing a von Mises-Fisher (vMF) distribution to capture semantics and a Gaussian distribution for syntax, and training on paraphrase pairs, the model learns to isolate these factors. This mathematical disentanglement enables "sentential style transfer," allowing researchers to map the meaning of one sentence onto the grammatical structure of another.
Harnessing Natural Supervision
The thesis explores how existing data structures can serve as proxy labels for complex reasoning tasks:
- Wikipedia as a Knowledge Map: The author utilizes Wikipedia’s category graphs to create WikiNLI, an entailment dataset. For example, the hierarchy implies that a "Jazz" category entails the parent category "Music." Pretraining on this naturally occurring logic improved BERT’s performance on the Multi-Genre Natural Language Inference (MNLI) task by 2.3% in low-resource settings.
- Intermediate Self-Supervised Finetuning: To bridge the gap between pretraining and human instruction following, the research introduces a "warm-up" stage. Models are trained on self-supervised tasks, such as predicting the final phrase of a sentence, before being finetuned. This approach improved 1.3B parameter models on SuperGLUE few-shot tasks by an average of 4–5 points.
New Benchmarks for Long-Form Generation
To move beyond short, news-centric snippets, the author introduced several datasets designed to test narrative consistency and data-to-text synthesis:
- WikiTableT: A massive dataset pairing Wikipedia sections with tabular data for long-form generation.
- SummScreen: A summarization dataset using TV show transcripts (e.g., The Big Bang Theory) and fan-written recaps. This reveals that narrative summarization requires deeper social and logical inference than news summarization, as plot points are often implied through dialogue.
- TVStoryGen: A dataset focused on generating detailed episode recaps from brief summaries and character descriptions, testing long-term consistency.
Key Insights and Results
The research demonstrates that structured supervision is a viable substitute for manual labeling. Key findings include:
- NSP Inefficiency: Standard NSP allows models to be "lazy," whereas SOP forces a deeper understanding of logical flow.
- Logic in the Wild: Wikipedia’s category hierarchy provides a massive, free tree of logical entailments that can teach models common sense.
- Narrative Complexity: Summarizing stories is fundamentally different from news; models must navigate character consistency and implied subtext, which remains a significant challenge.
Conclusion and Future Directions
While the methods proposed—such as SOP and VGVAE—significantly advance the state of representation learning, hurdles remain. Models still struggle with "character believability" in long-form stories and occasionally "hallucinate" information in data-to-text tasks. Future work aims to create unified architectures capable of ingesting multiple streams of natural supervision—discourse, entities, and paraphrases—simultaneously to create more robust and controllable AI systems.
Summary updated