Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions

Hakyung Sung, Kristopher KyleEMNLP 2024 · DOI

Introduction: The Challenge of Argument Structure

In linguistics, Argument Structure Constructions (ASCs) serve as the fundamental "blueprints" of a sentence. They represent form-meaning pairings where the overall meaning of a sentence depends on the interaction between a verb and its arguments.

Traditional "bottom-up" natural language processing (NLP) tools, such as dependency parsers, often struggle to distinguish these nuances because they focus primarily on word-level relationships. For instance, consider the sentences "She ran to the mountains" and "She ran in the mountains." Syntactically, they appear nearly identical, yet they convey fundamentally different relationships: the former describes a motion toward a goal, while the latter describes a location where an action takes place.

Figure 1: Comparison of dependency structures for motion (goal) vs. location constructions.

This paper addresses the challenge of building automated systems that can reliably identify these "top-down" constructions. Such systems are particularly vital for Second Language Acquisition (SLA) research, where tracking how learners progress from simple structures to complex, native-like constructions is a core objective.

Experimental Design

The researchers compared three distinct methodologies using two types of pre-trained language models (PLMs): RoBERTa (an encoder-only model) and GPT-4 (a generative decoder model). The goal was to determine whether modern models could resolve the syntactic ambiguities that stymie traditional parsers.

Figure 2: Overview of the three experimental setups comparing human-annotated data, zero-shot prompting, and synthetic data generation.
  1. Supervised Learning (Experiment 1): RoBERTa was fine-tuned for a multi-class Named Entity Recognition (NER) task, where the "entities" were specific ASC types. The team compared models trained on "silver" data (semi-automatically labeled) versus "gold" data (manually verified by human experts).
  2. Prompt-Guided Annotation (Experiment 2): The study tested GPT-4’s ability to label ASCs directly using zero-shot, 3-shot, and 10-shot prompting techniques.
  3. Synthetic Data Generation (Experiment 3): GPT-4 was used to generate a balanced set of synthetic training sentences for each ASC type. This data was then used to train RoBERTa to see if "AI-authored" data could effectively augment or replace human-annotated datasets.

Key Insights and Results

The Superiority of Gold-Standard Data

Despite the current enthusiasm for generative AI, the model trained on human-annotated "gold" data consistently outperformed all other configurations. The RoBERTa model trained on a mix of native (L1) and learner (L2) gold data achieved weighted F1-scores between 0.912 and 0.928. In contrast, GPT-4’s best performance (10-shot) reached a weighted F1 of 0.631. While impressive for a model with no specific task training, it lagged significantly behind the supervised baseline.

The "Complexity Gap" in Synthetic Data

A critical finding was that GPT-4-generated sentences were significantly shorter and less complex than human-produced ones. Synthetic sentences often lacked the multiple clauses and subtle nuances found in real-world web text or learner data. While synthetic augmentation provided a slight boost for rare constructions (like Transitive Resultatives), it could not match the performance of high-quality human labels.

Contextual Knowledge

The success of the RoBERTa models suggests that the ability of PLMs to store sentence-level contextual knowledge allows them to resolve the ambiguities inherent in ASCs. Furthermore, the models performed best on non-native (L2) speaker data when that specific data type was included in the training set, highlighting the necessity of representative datasets in educational NLP.

Conclusion

While GPT-4 is a powerful tool for data assistance and zero-shot labeling, supervised fine-tuning of encoder models on high-quality, human-annotated data remains the gold standard for precise linguistic analysis. Future work will focus on integrating these identification models into automated writing evaluation systems to provide direct feedback to language learners.

Summary updated

BibTeX
@inproceedings{sung2024leveraging,
  author = {Hakyung Sung and Kristopher Kyle},
  title = {{Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.415},
}

Loading related content...

Figure