FPT: Feature Prompt Tuning for Few-shot Readability Assessment

Ziyang Wang, Sanwoo Lee, Hsiu-Yuan Huang, Yunfang WuNAACL 2024 · DOI

TL;DR

This paper introduces Feature Prompt Tuning (FPT), a novel method for few-shot readability assessment that injects rich linguistic features into prompt-based tuning. FPT significantly outperforms existing methods by embedding linguistic features into soft prompts and using a novel calibration loss to preserve their inherent similarity relationships, leading to more robust and accurate readability predictions.

What Problem Does This Paper Address?

Readability assessment (RA) is crucial for tasks like selecting appropriate reading materials and supporting learners. While traditional RA relied on handcrafted linguistic features, modern approaches often use neural networks. However, these methods can be data-hungry and their performance can be unstable, especially in few-shot settings where labeled data is scarce. Existing prompt-based tuning methods, while effective for many NLP tasks in few-shot scenarios, often overlook the crucial role of linguistic knowledge in RA. Previous attempts to incorporate linguistic features have shown inconsistent results in few-shot settings. This paper aims to bridge this gap by developing a prompt-based tuning framework that effectively leverages linguistic features for few-shot RA.

Methods

The core innovation of this paper is Feature Prompt Tuning (FPT). Here's how it works:

  • Feature Extraction: Linguistic features are extracted from the input text using established toolkits (e.g., lingfeat for English, zhfeat for Chinese). These features capture various aspects like word length, sentence structure, and vocabulary complexity.
  • Feature Embedding: These extracted linguistic features are then transformed into trainable soft prompts. This is achieved using a MultiHeadMLP, which maps the linguistic features into distinct vector spaces, allowing the model to learn different aspects of their contribution to readability.
  • Similarity Calibration Loss: A key challenge in RA is that readability levels have an inherent ranking order. FPT introduces a novel calibration loss that ensures the embedded linguistic features preserve their original similarity relationships across different readability classes. This is crucial because if two texts have similar linguistic characteristics, their embedded features should also be similar. This loss function is based on the ListMLE ranking loss.
  • Alternating Training: FPT employs an alternating training procedure. It iteratively optimizes the model parameters using the standard classification loss and then updates the feature embeddings by minimizing the calibration loss. This ensures the model learns to classify accurately while maintaining the integrity of the linguistic feature space.

Datasets

The authors evaluated their method on three datasets:

  • ChineseLR: A Chinese dataset with five difficulty levels, collected from textbooks.
  • WeeBit: An English dataset, often used as a benchmark for RA, derived from the Weekly Reader corpus.
  • Cambridge: An English dataset comprising reading passages from Cambridge English Exams.

The experiments were conducted under a few-shot setting, where a small number of instances (k=1, 2, 4, 8, 16) per class were sampled from the training and development sets.

Key Insights and Ideas

  • Linguistic Features are Crucial for Few-Shot RA: The study strongly suggests that linguistic features are not just helpful but essential for robust readability assessment, especially when data is limited. Simply relying on general-purpose language models without incorporating this domain-specific knowledge leads to performance degradation.
  • Preserving Similarity is Key: The novel similarity calibration loss is a critical component. It ensures that the model doesn't just learn to classify but also understands the inherent relationships between different readability levels, mirroring how linguistic features naturally group texts.
  • Prompt Tuning Can Be Enhanced: FPT demonstrates that prompt tuning, a powerful few-shot learning technique, can be significantly improved by intelligently integrating domain-specific knowledge like linguistic features.
  • Chinese RA Benefits More from Linguistic Features: The experiments showed a more pronounced improvement of FPT on the Chinese dataset compared to English datasets, suggesting that Chinese RA tasks might be more reliant on explicit linguistic cues.

Key Results

  • Significant Performance Gains: FPT consistently outperformed all baseline prompt-based methods (Hard Prompt, Soft Prompt, Hybrid Prompt, P-tuning) across all datasets and few-shot settings. For example, in the 2-shot setting, FPT achieved a relative performance gain of 43.9% over the traditional soft prompt method on the Chinese dataset and 5.50% on the English Weebit dataset.
  • Outperforming Fusion Methods: FPT also surpassed previous feature fusion methods, demonstrating its effectiveness in integrating linguistic features with PLMs. In the 2-shot setting, FPT outperformed the best previous fusion methods by up to 11.66 points on the Cambridge dataset.
  • Competitive with LLMs: Notably, FPT, with its 110M parameters, significantly outperformed the large language model GPT-3.5-turbo-16k on English datasets in few-shot settings, highlighting its efficiency and effectiveness. However, the LLM struggled with longer texts, particularly on the Chinese dataset, underscoring the need for specialized approaches.
  • Ablation Study Confirms Contributions: Removing the similarity calibration module led to a noticeable performance drop, validating its importance. Further removal of the feature prompt component also resulted in a steep performance decline, confirming the value of incorporating linguistic features.

Limitations and Future Work

  • Handling Long Texts: The authors acknowledge that their model, like other MLM-based approaches, has limitations in processing very long texts, which are common in Chinese readability evaluation datasets. This is an area for future research.
  • Interpretability: MLM-based models are often "black boxes," making it challenging to understand the decision-making process. Improving interpretability is a future direction.
  • Linguistic Feature Quality: The performance of FPT is contingent on the quality of extracted linguistic features. Ensuring high-quality feature extraction, especially for languages with complex morphology or limited resources, remains a challenge.
  • Dataset Diversity: The study used only one Chinese dataset. Future work could explore more diverse Chinese RA datasets.

Summary updated

BibTeX
@inproceedings{wang2024fpt,
  author = {Ziyang Wang and Sanwoo Lee and Hsiu-Yuan Huang and Yunfang Wu},
  title = {{FPT: Feature Prompt Tuning for Few-shot Readability Assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.16},
}
Figure