TL;DR
This paper tackles the challenge of creating cost-effective training data for complex tasks like conversational Semantic Frame Analysis (SFA) by exploring the optimal mix of human-annotated and LLM-generated data. They propose a novel method for synthesizing this data and find that for lower budgets, a higher proportion of LLM-generated data leads to better cost-efficiency, with fully synthesized data (PP) proving to be a viable and cheaper alternative.
What Problem Does This Paper Address?
Training supervised learning models (SLMs) requires substantial amounts of labeled data, which is often expensive and time-consuming to create through human annotation. While Large Language Models (LLMs) show promise in generating synthetic data at a lower cost, the quality of LLM-generated data can be a bottleneck, especially for complex tasks like Semantic Frame Analysis (SFA). SFA involves identifying "triggers" (main actions) and their associated "arguments" (details of the event) within dialogues. This paper addresses the critical question of how to balance the trade-off between the high quality but high cost of human-annotated data and the lower quality but significantly cheaper LLM-generated data, particularly within constrained budgets, to achieve optimal performance for SFA.
Methods
The authors propose a novel method for synthesizing training data for conversational SFA. This method generates two types of LLM-generated data:
- Human-Pseudo (HP) data: This involves using an LLM to assign SFA labels to human-annotated dialogues.
- Pseudo-Pseudo (PP) data: This is fully synthesized, with both the dialogues and their SFA labels generated by an LLM.
They then create mixtures of human-annotated data (Human-Human, HH) with either HP or PP data. The core of their approach involves training a Semantic Frame Analysis model (using the JaMIE architecture) on these data mixtures across various budget levels. By evaluating the performance at different ratios of human to LLM-generated data, they aim to identify the optimal combination for cost-efficiency. They also employ a multi-step prompting scheme for LLMs to perform pseudo-labeling, which involves entity tagging, trigger detection, and relation detection.
Datasets
- EIDC dataset (cooking section): This dataset, containing Japanese interview dialogues with SFA annotations, was used for both training and evaluation. The authors specifically refer to the cooking section of this dataset.
Key Insights and Ideas
- Budget-dependent optimal data ratio: The ideal proportion of LLM-generated data to human data is not fixed but depends heavily on the available budget. As the budget decreases, leaning more on LLM-generated data becomes more cost-effective.
- LLM-generated data can harm performance: When budgets are sufficiently large, relying too heavily on LLM-generated data can actually degrade performance compared to using only human-annotated data.
- Fully synthesized data (PP) is competitive: The study found that PP data, which is entirely LLM-generated (dialogues and labels), is highly competitive with HP data (human dialogues, LLM labels). This suggests that replacing human-annotated dialogues with LLM-generated ones is a viable and cost-effective strategy.
- PP data is more cost-effective: PP data is significantly cheaper to produce than HP data, making it a superior option from a cost-sensitive perspective.
- LLMs as data augmenters: LLM-generated data (both HP and PP) can effectively augment human data, especially in low-resource settings, boosting performance when human data is limited.
Key Results
- Cost-efficiency: For trigger detection, optimal cost-efficiency was achieved by combining HH and HP data when the budget was below $6,400. For argument detection, this threshold was $3,200. In these lower-budget scenarios, a higher proportion of HP data led to better performance.
- PP vs. HP: At a budget of $1,600, PP data achieved a slightly higher performance in trigger detection (0.596) compared to HP data (0.571), demonstrating its cost-effectiveness.
- Data Augmentation: When trained on all LLM-generated data (HP or PP) and then augmented with human data, performance improved. This effect was more pronounced with lower budgets for human data. The performance gap between using PP and HP data for augmentation was less than 0.02 F1 score, despite PP being significantly cheaper.
Limitations and Future Work
- Task and Domain Specificity: The study focused on SFA in the cooking domain of Japanese interview dialogues. Determining the exact ratio of human to LLM-generated data remains challenging and depends on the specific task, dataset characteristics, and budget constraints.
- Generalizability: The authors hope that future work will extend their findings to other domains and tasks to validate the generalizability of their results.
Summary updated