TL;DR
This paper introduces a novel method for evaluating the "communication efficiency" of spoken dialog translation systems, moving beyond simple accuracy metrics. By comparing system performance to human interpreters in a controlled experiment, they reveal how users adapt their communication strategies to improve task completion, offering a more holistic view of system usability.
What Problem Does This Paper Address?
Evaluating spoken dialog translation systems (like those translating spoken Japanese to English) is challenging. Traditional metrics, such as BLEU scores, are often too technical for non-experts and don't fully capture the user experience. While usability metrics like effectiveness, efficiency, and satisfaction exist, they can be confounded by factors like task difficulty and user personality. This paper specifically tackles the challenge of measuring communication efficiency in task-oriented spoken dialog translation, aiming to provide a more practical and user-centric evaluation method. They want to understand not just if the translation is accurate, but how effectively information is conveyed between users interacting through a translation system.
Methods
The core of this paper is a proposed method for measuring communication efficiency, which expands upon existing work by considering two-way communication. They introduce three metrics:
- Provided Density (PD): Measures the average number of "essential items" (information crucial for task completion) conveyed per informative utterance. This metric specifically looks at the user's linguistic behavior.
- Transmission Density (TD): Measures the number of "essential items" successfully transmitted per informative utterance. This metric considers the system's role in relaying information.
- Transmission Efficiency (TE): The ratio of transmitted items to uttered items, representing the overall performance of the communication mediator (S2ST system or human interpreter).
To validate their method, they conducted two dialog experiments:
- Speech-to-Speech Translation (S2ST) System Experiment: Participants used an actual S2ST system for task completion.
- Wizard of Oz (WOZ) Experiment: Participants interacted with hidden human interpreters who typed translations. This served as a "gold standard" for human-level communication efficiency.
They also developed a "relative performance score" by comparing the S2ST system's performance directly against the WOZ results.
Datasets
- Training Data for Speech Recognition: The paper mentions training data for Automatic Speech Recognition (ASR) in Japanese, English, and Chinese, including the number of speakers, total recording time, number of sentences, word tokens, and vocabulary size. Specific details are provided in Table 1.
- Corpora for Machine Translation: Large corpora from the travel domain were used to construct machine translation modules.
- Dialog Data: Collected from the S2ST and WOZ experiments, consisting of task-based dialogs in Japanese-English (JtoE, EtoJ) and Japanese-Chinese (JtoC, CtoJ) language pairs.
Key Insights and Ideas
- Users Adapt Their Language: When using the S2ST system, users tended to produce shorter utterances without sacrificing the number of essential items conveyed. This suggests users intuitively try to be more concise when interacting with a machine.
- Strategic Communication: Users actively adapt their dialog strategies to improve communication efficiency, especially when the system's performance is lower. For example, shifting to a yes/no question style can be more effective.
- Human-Mediated Communication is the Benchmark: The WOZ experiment, using hidden human interpreters, provides a crucial baseline for evaluating how efficiently information can be conveyed in these task-oriented dialogs.
- Efficiency is More Than Just Accuracy: While ASR and MT accuracy are important (as shown in Table 2), they don't tell the whole story. The TE metric, which combines utterance content and transmission success, offers a more nuanced view of system effectiveness.
Key Results
- S2ST vs. WOZ Performance: The S2ST system achieved a relative performance score of 60% for Japanese-English and 46% for Japanese-Chinese compared to the WOZ (human interpreter) baseline.
- Transmission Efficiency: The Transmission Efficiency (TE) for the S2ST system was 67% for JtoE and 62% for JtoC, while the WOZ achieved 91% and 97% respectively. This highlights the gap between machine and human communication efficiency.
- User Utterance Length: Average utterance length in the S2ST experiment (6.0-6.9 words) was shorter than in the WOZ experiment (9.2-10.5 words), supporting the idea of users adapting their linguistic behavior.
- ASR/MT Performance: The S2ST system showed good ASR word accuracy (over 95%) and reasonable translation accuracy (around 80% for JtoE/JtoC), but a significant percentage of utterances were canceled by users due to poor recognition (up to 20-45%).
Limitations and Future Work
- Domain Specificity: The experiments were conducted within a specific domain (travel, shopping, hotel reservations). The findings might not generalize to all task-oriented dialog scenarios.
- Proper Nouns: The paper acknowledges the difficulty of including all necessary proper nouns in the lexicon, which can lead to recognition and translation errors.
- Future Work: The authors plan to conduct similar research on the usability of natural spoken language interfaces, suggesting a broader scope for future investigations into human-computer spoken communication.
Summary updated