Abstract
Research has explored the use of automatic text simplification (ATS), which consists of techniques to make text simpler to read, to provide reading assistance to Deaf and Hard-of-hearing (DHH) adults with various literacy levels. Prior work in this area has identified interest in and benefits from ATS-based reading assistance tools. However, no prior work on ATS has gathered judgements from DHH adults as to what constitutes complex text. Thus, following approaches in prior NLP work, this paper contributes new word-complexity judgements from 11 DHH adults on a dataset of 15,000 English words that had been previously annotated by L2 speakers, which we also augmented to include automatic annotations of linguistic characteristics of the words. Additionally, we conduct a supplementary analysis of the interaction effect between the linguistic characteristics of the words and the groups of annotators. This analysis highlights the importance of collecting judgements from DHH adults for training ATS systems, as it revealed statistically significant interaction effects for nearly all of the linguistic characteristics of the words.
1. Introduction
Automatic Text Simplification (ATS) is a critical field of artificial intelligence focused on enhancing text readability while preserving core meaning. For the Deaf and Hard-of-Hearing (DHH) community, ATS represents a potential game-changer for accessibility. Many DHH individuals face unique literacy challenges stemming from varying levels of early language access, which can lead to different patterns of vocabulary acquisition compared to hearing peers.
Despite the clear benefits, most existing ATS systems are trained on datasets derived from general audiences or non-native (L2) English speakers. This "one size fits all" approach assumes that what is complex for an L2 learner is also complex for a DHH adult. However, linguistic research suggests that DHH readers may perceive word difficulty differently. Without data reflecting the specific lived experiences and linguistic profiles of DHH readers, simplification algorithms risk being ineffective or even patronizing. This paper addresses this gap by introducing the first large-scale lexicon of word-complexity judgements specifically tailored to the DHH community.
2. Methodology
The researchers recruited 11 DHH adults with diverse linguistic backgrounds, ranging from those who use American Sign Language (ASL) exclusively to bilingual ASL/English users. These participants were tasked with rating 15,000 English words—the most frequent terms from Google’s 1T Ngram Corpus—on a 6-point Likert scale, where 1 represented "very simple" and 6 represented "very complex."
To provide a deeper understanding of the factors influencing these judgements, the team augmented the dataset with several automatically computed linguistic features:
- Morphological Features: Word length (number of characters) and syllable count.
- Semantic Features: Number of word senses (meanings), synonyms, hypernyms (broader categories), and hyponyms (more specific terms).
- Lexical Features: Word frequency metrics derived from three major corpora: SubIMDB, Subtlex, and Simple Wikipedia.
Finally, the researchers employed a two-way Analysis of Variance (ANOVA) to investigate the "interaction effect" between the annotator group (DHH vs. L2 speakers) and these linguistic features. This statistical approach determines whether the impact of a feature (like word length) on perceived complexity differs significantly between the two groups.
3. The DHH-Lexical-Dataset
The primary contribution of this work is the DHH-lexical-dataset, which is publicly available on GitHub. This comprehensive resource includes:
- Complexity ratings for 15,000 words from 11 DHH adults.
- Comparative data from a previous study involving 11 L2 English speakers.
- Detailed linguistic feature annotations for every word in the set.
4. Results and Analysis
The study's most significant finding is that complexity is inherently subjective and audience-dependent. The analysis revealed that the relationship between a word's characteristics and its perceived complexity changes based on the reader's background.
As detailed in the table above, the researchers found statistically significant interaction effects (p < 0.001) for nearly all linguistic characteristics, including word length, syllable count, number of senses, hypernyms, and all three measures of word frequency. Only synonyms and hyponyms did not show a significant interaction effect. These results provide empirical evidence that features like "word length" may increase perceived complexity much more sharply for one group than the other.
Furthermore, the data showed that DHH annotators generally rated words as simpler (mean rating of 2.2) compared to L2 speakers (mean rating of 2.7). This suggests that DHH adults may have a different "threshold" for complexity, potentially due to different educational experiences or linguistic strategies.
5. Reliability and Validation
The study achieved an inter-annotator agreement of 0.53 (increasing to 0.55 after outlier removal). These figures are consistent with previous high-quality lexical datasets, indicating that the DHH judgements are a reliable signal for training machine learning models. This validation confirms that the dataset is a robust foundation for developing specialized ATS tools.
6. Discussion and Future Work
The findings underscore the importance of audience-specific NLP. Researchers cannot simply "borrow" datasets from one marginalized or specific user group to solve problems for another.
However, the authors acknowledge several limitations:
- Contextual Factors: Words were rated in isolation. In natural reading, context often aids in understanding complex terms. Future studies should incorporate in-context judgements.
- Demographic Representation: Participants were recruited from a university campus. While diverse in literacy, they may not represent the full spectrum of the DHH population, particularly those with less access to higher education.
- Causal Mechanisms: While the study proves that DHH and L2 speakers differ, it does not explain the underlying cognitive or linguistic reasons why certain features affect DHH readers differently. This remains a vital area for future psycholinguistic inquiry.
By providing this dataset, the authors hope to catalyze the development of more inclusive and effective reading assistance technologies for the DHH community.
Summary updated