TL;DR
PropBank, a foundational resource for semantic role labeling (SRL), has evolved significantly over two decades. This paper details its expansion to include non-verbal predicates, a wider range of genres and languages, and improved data structures and quality control, making it a more robust and versatile tool for NLP research.
What Problem Does This Paper Address?
Semantic Role Labeling (SRL) aims to identify the predicate-argument structure of sentences, answering questions like "who did what to whom, when, where, and why?" PropBank, a key resource for SRL, was initially built on the Penn Treebank and focused primarily on verbs in news articles. However, as NLP models and applications became more sophisticated, the need arose for PropBank to:
- Expand beyond verbs: Natural language often uses adjectives, nouns, and multi-word expressions as predicates.
- Increase domain and genre diversity: Models trained solely on news text struggle to generalize to other domains like conversational speech, web text, or specialized fields.
- Support more languages: To build truly global NLP systems, SRL resources need to be available in multiple languages.
- Improve consistency and reliability: For robust model training and evaluation, annotated datasets must be high-quality and consistently produced.
- Facilitate broader adoption and contribution: Making the resource more accessible and easier to contribute to is crucial for its continued growth.
This paper addresses these challenges by detailing the evolution of PropBank to meet these growing demands.
Methods
The paper outlines several key methodological advancements and expansions for PropBank:
- Expansion of Predicate Types: PropBank has moved beyond just verbs to include rolesets for adjectives, nouns (often through light verb constructions or nominalizations), prepositions, and even affixes. This was achieved by unifying predicate types and allowing multiple predicates (aliases) to share a single roleset, increasing lexical coverage.
- XML Schema Evolution: The frame files, which define predicate rolesets, have transitioned from a simpler DTD to a more robust XML schema. This allows for better validation, disambiguation, and a more structured representation of data, including improved handling of multi-word expressions (MWES) and examples.
- Enhanced Data Quality and Consistency: Significant effort has been invested in ensuring consistency and reliability. This includes automated processes, manual correction, and updated guidelines. The serialization of annotations has also been decoupled from strict syntactic parse trees, allowing for more flexible annotation.
- Improved Annotation Structure: New features like
<lexlinks>and<rolelinks>facilitate mappings to other lexical resources (VerbNet, FrameNet, WordNet), creating a richer, interconnected semantic network.<Usage>tags help manage different versions and project-specific subsets of the data. - Syntax-Agnostic Examples: Example sentences within the frame files are now presented in a syntax-agnostic format, resolving them to their actual text spans. This makes them more adaptable to various parsing paradigms and easier for machine processing.
Datasets
PropBank has expanded significantly, now encompassing a wide array of corpora and languages:
- Original: Penn Treebank (Wall Street Journal)
- Expanded English Corpora:
- Brown Corpus: For out-of-domain testing.
- OntoNotes: A large, multi-genre corpus including broadcast news, conversations, web text, telephone conversations, and religious texts.
- English Web Treebank (EWT): Web text from blogs, newsgroups, emails, and reviews.
- BOLT: Informal text from SMS, chat, and discussion forums.
- LORELEI: Newswire, phrasebook, and elicitation corpus.
- Flickr-8k: Image captions.
- ClearEarth: Earth sciences domain data (sea ice blogs/news, academic articles, educational wikis, earthquake reports).
- THYME: Clinical notes (cancer-related narratives).
- Multilingual Corpora: PropBank has been extended to languages including Chinese, Arabic, Hindi, Urdu, Korean, Finnish, Turkish, Persian, Russian, and Brazilian Portuguese.
- New Benchmarks: The paper highlights that the combination of OntoNotes, EWT, and BOLT corpora, with their diverse genres and non-verbal predicates, can serve as a more challenging benchmark for SRL evaluation than previous datasets.
Key Insights and Ideas
- PropBank's Maturation: PropBank has evolved from a verb-centric resource for news text into a comprehensive, multi-genre, multilingual framework for semantic role labeling.
- The Power of Unified Rolesets: Allowing multiple predicate types (verbs, nouns, adjectives) to share a single roleset significantly increases lexical coverage and reduces redundancy, making the resource more efficient.
- Interconnectedness of Lexical Resources: Linking PropBank rolesets to other resources like VerbNet and FrameNet creates a richer, more interconnected semantic network, benefiting both annotation and downstream NLP tasks.
- The Importance of Data Quality: The emphasis on consistency, reliability, and improved annotation structures (like syntax-agnostic examples) is crucial for building robust SRL models and ensuring reproducible research.
- XML as a Robust Choice: The paper defends the continued use of XML for its schema definition and validation capabilities, arguing it remains a versatile choice for complex declarative data specifications.
Key Results
The paper doesn't present novel model performance results but focuses on the evolution and expansion of PropBank itself. Key "results" are the availability of:
- Expanded Corpora: PropBank now covers a much wider range of genres (news, web text, conversational, clinical, etc.) and languages.
- Broader Predicate Coverage: Non-verbal predicates (adjectives, nouns, prepositions, affixes) are now included, significantly increasing the scope of what can be annotated.
- Improved Data Quality: The latest release (PropBank 3.4.0) offers enhanced consistency and reliability due to updated guidelines and quality control measures.
- Accessible Resources: A prominent web presence and a GitHub organization facilitate access to frame files and annotations, promoting open and reproducible research.
- New Benchmarking Opportunities: The expanded datasets provide a more challenging and realistic evaluation environment for SRL systems, testing their generalization capabilities across diverse linguistic phenomena. Table 3 shows baseline performance on four main corpora, with scores ranging from 77.3% to 86.7% F1, demonstrating the task's difficulty and the impact of diverse genres.
Limitations and Future Work
- Source Text Distribution: While annotations are available, the underlying source text for some corpora cannot be redistributed due to copyright restrictions, requiring users to obtain it separately from sources like the LDC.
- Manual Annotation Imperfection: The authors acknowledge that manually annotated data is never perfect and that errors are always found.
- Future Work:
- Continued evolution of the PropBank lexicon to cater to the growing, global, distributed, and diverse community.
- Further development of tools and infrastructure for easier contribution and issue resolution via the GitHub organization.
- Ongoing efforts to improve the quality and consistency of annotations.
- The paper suggests that testing against the combined OntoNotes, EWT, and BOLT corpora offers a more challenging SRL evaluation.
Summary updated