InterpretARA: Enhancing Hybrid Automatic Readability Assessment with Linguistic Feature Interpreter and Contrastive Learning

Jinshan Zeng, Xianchao Tong, Xianglong Yu, Wenyan Xiao, Qing HuangAAAI 2024 · DOI

Introduction

Automatic Readability Assessment (ARA) is a critical task in Natural Language Processing, aimed at determining the difficulty level of a text for a target audience. Applications range from selecting appropriate reading materials for language learners to ensuring medical and legal documents are accessible to the general public. Traditionally, ARA has relied on two distinct paradigms: traditional linguistic features (e.g., sentence length, syllable counts) and modern deep learning representations (e.g., BERT embeddings).

While hybrid models attempt to combine these approaches, they often struggle to integrate raw numerical features with high-dimensional semantic vectors. Deep learning models, while powerful at capturing context, often treat scalar linguistic values as mere numbers rather than structural indicators of complexity. Furthermore, standard classification approaches often ignore the ordinal relationship between readability levels. To address these challenges, we present InterpretARA, a novel hybrid model that leverages a linguistic interpreter and contrastive learning to achieve state-of-the-art performance.

Methodology

InterpretARA is designed to bridge the gap between mathematical feature calculation and semantic understanding. The architecture consists of four primary components: multi-level text representation, a linguistic feature interpreter, orthogonal projection fusion, and a contrastive learning objective.

The InterpretARA architecture overview.

Linguistic Feature Interpreter (LFI)

The core innovation of InterpretARA is the Linguistic Feature Interpreter (LFI). Instead of feeding raw numerical values directly into the classifier, the LFI transforms these features into descriptive natural language sentences using predefined templates. For instance, a feature indicating an average of 15 syllables per sentence is converted into the string: "The average count of syllables per sentence is 15."

Examples of linguistic feature templates used by the LFI.

By converting scalars into text, the model can utilize a pre-trained BERT encoder to "read" and interpret the linguistic rules, leveraging its vast pre-trained knowledge of language structure to weight these features more effectively.

Multi-Level Deep Representations

Readability is influenced by both global context and local structural complexity. InterpretARA extracts features at two scales:

  1. Document-level Representation: Captures the overall theme and global coherence using a standard BERT encoder.
  2. Segment-level Representation: Processes the text in smaller chunks to identify fine-grained complexity, such as particularly dense paragraphs or complex sentence clusters that might be smoothed over in a document-wide average.

Orthogonal Projection Fusion

A common pitfall in hybrid models is feature redundancy; deep representations often implicitly capture some of the information present in traditional linguistic features. To ensure that each feature set provides unique value, InterpretARA employs an Orthogonal Projection scheme. This mathematical filter projects the linguistic features into a space orthogonal to the deep representations, effectively removing overlapping information and allowing the final linear layer to focus on the most discriminative, non-redundant signals.

Contrastive Learning

To better capture the "gradient" of readability, the model is trained using a contrastive learning objective. This encourages the model to minimize the distance between representations of texts at the same readability level while maximizing the distance between texts of different levels. This approach helps the model learn the relative difficulty scales inherent in datasets like Newsela or Cambridge.

Experimental Evaluation

InterpretARA was evaluated across five diverse corpora, covering both English and Chinese:

  • English: Weebit, Cambridge, Newsela, and CLEAR.
  • Chinese: CMT (based on mainland Chinese textbooks).

The results demonstrate significant improvements over existing state-of-the-art (SOTA) models such as BigBird and BERT-FP-LBL. Notably, on the Cambridge dataset, InterpretARA achieved a 16.67% improvement in accuracy over a baseline BERT model. On the Chinese CMT corpus, it reached 44.87% accuracy, outperforming the baseline by over 5 percentage points.

Ablation studies confirmed the importance of the LFI. Removing the interpreter led to a significant drop in F1 scores (e.g., a loss of 8.44 points on Cambridge), proving that "explaining" linguistic features to the model is far more effective than providing them as raw numbers.

Conclusion

InterpretARA demonstrates that deep learning models are more effective at ARA when linguistic rules are presented in a format they are designed to handle: natural language. By combining this semantic interpretation with multi-level representations and orthogonal fusion, the model sets a new benchmark for readability assessment. Future work will explore more sophisticated attention-based fusion methods and the application of this framework to low-resource languages.

Summary updated

BibTeX
@inproceedings{zeng2024interpretara,
  author = {Jinshan Zeng and Xianchao Tong and Xianglong Yu and Wenyan Xiao and Qing Huang},
  title = {{InterpretARA: Enhancing Hybrid Automatic Readability Assessment with Linguistic Feature Interpreter and Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29921},
}
Figure