Abstract: Adrian Benton, Hanyang Li, Igor Malioutov. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction
While modern NLP models identify parts of speech (POS) in standard text with high accuracy, they often struggle with news headlines. Headlines are written in "headlinese"—a distinct register characterized by omitted articles, dropped auxiliary verbs, and "garden path" phrasing. These linguistic shortcuts break the syntactic rules that standard models rely on. For example, headlines frequently omit copulas or use infinitives to describe the future (e.g., "President to visit"), leading to significant errors in downstream tasks like Open Information Extraction (Open IE).
Methods
The researchers propose two primary innovations to address the lack of annotated headline data:
- Cross-Register Projection: To create training data without expensive manual labeling, the authors utilized the Google Sentence Compression corpus. They identified "extractive" headlines—those that are subsequences of the news story's lead sentence. By tagging the grammatically complete lead sentence with a standard tagger, they projected these tags onto the corresponding words in the headline. This process created a "silver" dataset of approximately 44,000 headlines (GSCproj).
- Multi-Domain BiGRU-CRF: The tagging model treats standard text and headlines as two different domains. While sharing a core encoder (such as BERT), the model uses domain-specific decoder layers (CRF heads). This allows it to learn the specific syntactic quirks of headlines without losing its proficiency in standard English.
Key Results
The multi-domain BERT model achieved a 23% relative error reduction per token compared to a baseline trained only on standard text. Crucially, the improved POS tags significantly boosted the performance of downstream systems. When integrated into a syntax-based Open Information Extraction system (PredPatt), precision nearly doubled, rising from 29.8% to 56.9%.
The model specifically reduced "malformed predicates," where standard taggers misidentify the main verb in a headline as a noun or an auxiliary. The study also found that the model trained on projected tags generalized well to unconstrained headlines (like those from the NYT) that were not simple sub-sentences of the body text.
Conclusion
This research highlights that "news" is not a monolithic domain; the register shift between news articles and headlines is syntactically profound. By leveraging cross-register projection and multi-domain modeling, the authors successfully bridged the gap to "headlinese," providing a scalable method for improving NLP performance on compressed text formats.
Reference: Benton, A., Li, H., & Malioutov, I. (2021). Cross-Register Projection for Headline Part of Speech Tagging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Summary updated