Abstract: Semantic Role Labelling (SRL) is the process of automatically finding the semantic roles of terms in a sentence. It is an essential task towards creating a machine-meaningful representation of textual information. One public linguistic resource commonly used for this task is the FrameNet Project. FrameNet is a human and machine-readable lexical database containing a considerable number of annotated sentences, those annotations link sentence fragments to semantic frames. However, while the annotations across all the documents covered in the dataset link to most of the frames, a large group of frames lack annotations in the documents pointing to them. In this paper, we present a data augmentation method for FrameNet documents that increases by over 13% the total number of annotations. Our approach relies on lexical, syntactic, and semantic aspects of the sentences to provide additional annotations. We evaluate the proposed augmentation method by comparing the performance of a state-of-the-art semantic-role-labelling system, trained using a dataset with and without augmentation.
The Challenge of Semantic Role Labeling
To achieve true "Machine Reading," computers must map unstructured text into structured semantic roles. This task, known as Semantic Role Labeling (SRL), relies heavily on FrameNet, a lexical database where "frames" represent concepts (like Arrest or Giving) and "frame elements" (FEs) represent the roles within them (like Authority, Suspect, or Donor).
However, FrameNet suffers from a significant data sparsity issue. While it contains thousands of frames, many have very few—or zero—annotated examples in the provided documents. This "long tail" problem makes it difficult to train machine learning models to recognize less common or domain-specific roles. This research aims to fill these gaps by leveraging the internal structure of FrameNet itself, avoiding the high cost of manual human annotation.
Proposed Data Augmentation Methodology
The core strategy involves transferring annotations from well-documented frames to related, under-documented ones. To ensure the validity of these transfers, the authors define three levels of Frame Element Equivalence:
- Lexical: Roles are considered equivalent if they share the same name (e.g., "Creator" in one frame is likely the same as "Creator" in a related sub-frame).
- Syntactic: Using the spaCy NLP library, the system identifies the path from a word to its target in a dependency tree. If two roles in different frames share the same syntactic path, they are matched.
- Semantic: The system uses Boxer, a semantic analyzer, to generate a "neo-Davidsonian" logical representation. Roles sharing the same path in this logical structure are considered equivalent.
The augmentation process exploits both hierarchical relations (e.g., "Inherits from" or "Subframe of") and non-hierarchical relations (e.g., "Uses" or "See also"). If all annotations in a sentence for Frame X can be mapped to equivalent roles in a related Frame Y, a new training example is generated.
Experimental Setup and Results
The study utilized FrameNet version 1.5, specifically a subset of 78 documents containing 5,946 annotated sentences and 48,133 frame element annotations. The performance was evaluated using the Open-Sesame system, a state-of-the-art frame semantic parser.
Key Findings
- Data Volume: The method successfully increased the total number of annotations in the FrameNet documents by over 13%.
- Performance Gains: Training the Open-Sesame model on the augmented dataset led to consistent improvements. The baseline F1 score of 0.6004 rose to 0.6109 when using lexical matching on non-hierarchical relations.
- Simplicity vs. Complexity: Surprisingly, simple lexical matching proved most effective. While syntactic and semantic parsing are more sophisticated, they are prone to "cascading errors" where initial parsing mistakes lead to noisy augmented data.
- Relation Types: Transferring data across hierarchical lines (inheritance) was safer, but non-hierarchical relations (like "Uses") provided a larger volume of data, helping the model generalize better despite the added noise.
Conclusion and Future Work
This research demonstrates that the internal "inter-frame" connections in FrameNet contain significant untapped potential for generating synthetic training data. While a ~1% absolute increase in F1 might seem modest, it represents a significant gain in the context of SRL, where high-quality data is notoriously difficult to acquire.
Future work will focus on integrating FrameNet with other linguistic databases like WordNet or PropBank to create a more robust "Predicate Matrix" and refining the underlying syntactic and semantic parsers to reduce noise during the augmentation phase.
Summary updated