TL;DR
This paper introduces a novel "semantic authoring" approach for Augmentative and Alternative Communication (AAC) systems, specifically for generating natural language from Blissymbols. Instead of translating a sequence of symbols, their system dynamically guides the user's symbol selection, leading to more intuitive and efficient message generation for individuals with severe language impairments.
What Problem Does This Paper Address?
Individuals with severe language impairments often struggle to communicate effectively using natural language. Augmentative and Alternative Communication (AAC) systems aim to bridge this gap by providing tools that facilitate communication. A common approach involves using iconic symbols on a communication board, where users select symbols sequentially to form messages. However, translating these telegraphic symbol sequences into fluent natural language can be challenging, often requiring complex parsing and relying heavily on context and pragmatics, which are difficult to infer. Existing AAC-NLG systems typically parse a pre-selected sequence of symbols and then generate a grammatical sentence. This paper addresses the limitations of this post-hoc translation by proposing a system that guides the user's symbol selection process itself, making the generation of meaningful messages more intuitive and less reliant on complex post-hoc interpretation.
Methods
The core of the proposed system is "semantic authoring," which intervenes during the process of composing the input sequence of Blissymbols. This approach is built upon the WYSIWYM (What You See Is What You Mean) paradigm, where the system maintains a model of the discourse context and provides interactive feedback.
Key aspects of their method include:
- Dynamic Display Updates: The communication board (display) is updated on the fly as the user selects symbols. This means that at each step, only symbols that are semantically and syntactically plausible given the previous selections are presented.
- Constraint-Driven Selection: The system uses a set of constraints and rules derived from an ontology to guide symbol selection. For example, if a verb requiring an instrumental theme has been selected, only symbols that can function as instruments will be displayed.
- Semantic Templates: Pre-defined "pragmatic contexts" are offered as semantic templates to simplify message construction for common conversational situations.
- Integrated Lexicon: A comprehensive Bliss lexicon for both Hebrew and English was developed. This lexicon maps symbols to their graphic representations, semantic information, and corresponding words in both languages. It can be searched by keyword or by semantic/graphic components.
- Layered Knowledge Bases: The underlying process relies on layered lexical knowledge bases (LKB) and an ontology that encodes concepts and relations, aiding in content determination and lexical choice.
This approach avoids the need for parsing a telegraphic sequence by constructing a semantic structure explicitly as the user inputs symbols.
Datasets
The paper does not explicitly mention training datasets. For evaluation, they collected a set of 19 sentences written in Bliss and their English correspondents.
Key Insights and Ideas
- Shifting from Translation to Authoring: The most significant insight is the move from a "translate-then-generate" paradigm to a "guide-and-generate" approach. By intervening in the symbol selection process, the system proactively shapes the semantic structure, reducing the burden on the user and the complexity of the generation engine.
- Reducing Cognitive Load: Dynamically filtering symbol choices based on semantic and syntactic constraints significantly narrows down the possibilities at each step, thereby reducing the cognitive load on the user and potentially increasing communication speed.
- Leveraging Bliss Symbol Properties: The system capitalizes on the inherent semantic structure of Blissymbols, where symbols are composed of atomic icons and their graphic representation provides clues about word connectivity.
- Multilingual Generation: The semantic authoring approach allows for fluent output in multiple languages (English and Hebrew) without requiring noisy translation from the Bliss sequence.
Key Results
The paper presents an evaluation focused on message generation speed and semantic coverage.
- Keystroke Savings: While a simple word count ratio showed no improvement (133 choices vs. 122 words), the authors argue that savings are realized by narrowing choice possibilities. When considering morphology markers as additional words (totaling 138), the ratio becomes more favorable.
- User Study: A user study with 10 non-disabled subjects was conducted to evaluate the usability and learning curve of the system. This study provided a "reliable approximation" of these aspects.
- Qualitative Findings: The system aims to provide a more intuitive and efficient way to generate messages for AAC users by guiding their symbol selection. The authors hypothesize that ontologically motivated grouping of symbols could further speed up selection.
Limitations and Future Work
- Prototype Stage: The system is described as a prototype and has not yet been adjusted to interact with alternative pointing devices.
- Limited Evaluation: The evaluation was conducted on non-disabled subjects, and a full extrinsic (task-based) evaluation with actual Bliss users has not yet been performed.
- Future Work:
- Implementing full access for alternative pointing devices.
- Expanding the lexicon for Hebrew generation.
- Adding voice output capabilities.
- Conducting task-based extrinsic evaluations with AAC users.
- Further exploring the impact of ontologically motivated symbol grouping on selection speed.
Summary updated