Computer Science & Information Technology (CS & IT)

Breno Camargo Carvalho, Aline Paes, Bernardo Gonçalves, Qihao Lin, Jinyu Cai, Genggeng Liu, David L. Hu, Jianfa Gao, Yu Sun, Fangyan Zhang, Chun-Peng Chang, Wen-Jen Ho, Yung-Chieh Hung, Kuei-Chun Chiang, Bill Zhao, Valentyn Yanchuk, Andrii Tkachuk, D.s. Antoniuk, Tetiana А. Vakaliuk, Anna Humeniuk, Yifei Yu, Prazwal Chhabra, Rizwan Ali, Vikram Pudi, O Etzioni, M Banko, M Cafarella, O Abend, A Rappoport, C Baker, C Fillmore, J Lowe, D Das, D Chen, A Martins, N Schneider, N Noah, Smith, J Bos, B Cronin, J Pennington, R Socher, C Manning, L Shi, R Mihalcea, A.-m Giuglea, A Moschitti, E Laparra, G Rigau, S Tonelli, C Giuliano, K Tymoshenko, R Green, B Dorr, P Resnik, P Kingsbury, M Palmer, K Kipper, A Korhonen, N Ryant, C Fellbaum, M Lopez De Lacalle, I Aldabe, E Pavlick, T Wolfe, P Rastogi, C Callison-Burch, M Dredze, B Van Durme, J Ganitkevitch, H Sergieh, I Gurevych, D Vrandecic, M Krotzsch, S Reddy, O Tackstr Om, M Collins, T Kwiatkowski, M Steedman, M Lapata, I Beltagy, C Chau, G Boleda, D Garrette, K Erk, R Mooney, M RichardsonOpenalex Unknown 2020 · DOI · OpenAlex

Abstract: Semantic Role Labelling (SRL) is the process of automatically finding the semantic roles of terms in a sentence. It is an essential task towards creating a machine-meaningful representation of textual information. One public linguistic resource commonly used for this task is the FrameNet Project. FrameNet is a human and machine-readable lexical database containing a considerable number of annotated sentences, those annotations link sentence fragments to semantic frames. However, while the annotations across all the documents covered in the dataset link to most of the frames, a large group of frames lack annotations in the documents pointing to them. In this paper, we present a data augmentation method for FrameNet documents that increases by over 13% the total number of annotations. Our approach relies on lexical, syntactic, and semantic aspects of the sentences to provide additional annotations. We evaluate the proposed augmentation method by comparing the performance of a state-of-the-art semantic-role-labelling system, trained using a dataset with and without augmentation.

The Challenge of Semantic Role Labeling

To achieve true "Machine Reading," computers must map unstructured text into structured semantic roles. This task, known as Semantic Role Labeling (SRL), relies heavily on FrameNet, a lexical database where "frames" represent concepts (like Arrest or Giving) and "frame elements" (FEs) represent the roles within them (like Authority, Suspect, or Donor).

However, FrameNet suffers from a significant data sparsity issue. While it contains thousands of frames, many have very few—or zero—annotated examples in the provided documents. This "long tail" problem makes it difficult to train machine learning models to recognize less common or domain-specific roles. This research aims to fill these gaps by leveraging the internal structure of FrameNet itself, avoiding the high cost of manual human annotation.

Proposed Data Augmentation Methodology

The core strategy involves transferring annotations from well-documented frames to related, under-documented ones. To ensure the validity of these transfers, the authors define three levels of Frame Element Equivalence:

  1. Lexical: Roles are considered equivalent if they share the same name (e.g., "Creator" in one frame is likely the same as "Creator" in a related sub-frame).
  2. Syntactic: Using the spaCy NLP library, the system identifies the path from a word to its target in a dependency tree. If two roles in different frames share the same syntactic path, they are matched.
  3. Semantic: The system uses Boxer, a semantic analyzer, to generate a "neo-Davidsonian" logical representation. Roles sharing the same path in this logical structure are considered equivalent.
Figure 3: A dependency tree generated by spaCy, used to determine syntactic equivalence between frame elements.

The augmentation process exploits both hierarchical relations (e.g., "Inherits from" or "Subframe of") and non-hierarchical relations (e.g., "Uses" or "See also"). If all annotations in a sentence for Frame X can be mapped to equivalent roles in a related Frame Y, a new training example is generated.

Figure 5: Illustration of hierarchical frame relations (Sub-frame) and the mapping of frame elements like 'Creator' between related frames.

Experimental Setup and Results

The study utilized FrameNet version 1.5, specifically a subset of 78 documents containing 5,946 annotated sentences and 48,133 frame element annotations. The performance was evaluated using the Open-Sesame system, a state-of-the-art frame semantic parser.

Key Findings

  • Data Volume: The method successfully increased the total number of annotations in the FrameNet documents by over 13%.
  • Performance Gains: Training the Open-Sesame model on the augmented dataset led to consistent improvements. The baseline F1 score of 0.6004 rose to 0.6109 when using lexical matching on non-hierarchical relations.
  • Simplicity vs. Complexity: Surprisingly, simple lexical matching proved most effective. While syntactic and semantic parsing are more sophisticated, they are prone to "cascading errors" where initial parsing mistakes lead to noisy augmented data.
  • Relation Types: Transferring data across hierarchical lines (inheritance) was safer, but non-hierarchical relations (like "Uses") provided a larger volume of data, helping the model generalize better despite the added noise.

Conclusion and Future Work

This research demonstrates that the internal "inter-frame" connections in FrameNet contain significant untapped potential for generating synthetic training data. While a ~1% absolute increase in F1 might seem modest, it represents a significant gain in the context of SRL, where high-quality data is notoriously difficult to acquire.

Future work will focus on integrating FrameNet with other linguistic databases like WordNet or PropBank to create a more robust "Predicate Matrix" and refining the underlying syntactic and semantic parsers to reduce noise during the augmentation phase.

Summary updated

Abstract

Semantic Role Labelling (SRL) is the process of automatically finding the semantic roles of terms in a sentence. It is an essential task towards creating a machine-meaningful representation of textual information. One public linguistic resource commonly used for this task is the FrameNet Project. FrameNet is a human and machine-readable lexical database containing a considerable number of annotated sentences, those annotations link sentence fragments to semantic frames. However, while the annotations across all the documents covered in the dataset link to most of the frames, a large group of frames lack annotations in the documents pointing to them. In this paper, we present a data augmentation method for FrameNet documents that increases by over 13% the total number of annotations. Our approach relies on lexical, syntactic, and semantic aspects of the sentences to provide additional annotations. We evaluate the proposed augmentation method by comparing the performance of a state-of-the-art semantic-role-labelling system, trained using a dataset with and without augmentation.
BibTeX
@inproceedings{carvalho2020computer,
  author = {Breno Camargo Carvalho and Aline Paes and Bernardo Gonçalves and Qihao Lin and Jinyu Cai and Genggeng Liu and David L. Hu and Jianfa Gao and Yu Sun and Fangyan Zhang and Chun-Peng Chang and Wen-Jen Ho and Yung-Chieh Hung and Kuei-Chun Chiang and Bill Zhao and Valentyn Yanchuk and Andrii Tkachuk and D.S. Antoniuk and Tetiana А. Vakaliuk and Anna Humeniuk and Yifei Yu and Prazwal Chhabra and Rizwan Ali and Vikram Pudi and O Etzioni and M Banko and M Cafarella and O Abend and A Rappoport and C Baker and C Fillmore and J Lowe and D Das and D Chen and A Martins and N Schneider and N Noah and Smith and J Bos and D Das and N Schneider and D Chen and N Smith and C Baker and C Fillmore and B Cronin and J Pennington and R Socher and C Manning and L Shi and R Mihalcea and A.-M Giuglea and A Moschitti and E Laparra and G Rigau and S Tonelli and C Giuliano and K Tymoshenko and R Green and B Dorr and P Resnik and P Kingsbury and M Palmer and K Kipper and A Korhonen and N Ryant and M Palmer and C Baker and C Fellbaum and M Lopez De Lacalle and E Laparra and I Aldabe and G Rigau and E Pavlick and T Wolfe and P Rastogi and C Callison-Burch and M Dredze and B Van Durme and J Ganitkevitch and B Durme and C Callison-Burch and H Sergieh and I Gurevych and D Vrandecic and M Krotzsch and S Reddy and O Tackstr Om and M Collins and T Kwiatkowski and D Das and M Steedman and M Lapata and I Beltagy and C Chau and G Boleda and D Garrette and K Erk and R Mooney and M Richardson},
  title = {{Computer Science \& Information Technology (CS \& IT)}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.5121/csit.2020.10120},
}
Figure