Introduction
In voice recognition agents, such as SK Telecom’s Adot, semantic ambiguity presents a significant challenge to user experience. Ambiguity often arises when a single term possesses multiple, sometimes contradictory, meanings depending on the context. For instance, the Korean word "앞" (ap) can denote "forward" (the front) or "previous" (the antecedent). In a "one-turn" interaction typical of voice assistants, where context is minimal and queries are short (averaging 2.05 words), resolving such ambiguity is difficult.
Rather than attempting to resolve every ambiguity, researchers from SK Telecom and KAIST propose a method to detect it. Detecting ambiguity allows developers to identify "risky" service scenarios and debug training data where human annotators may have mislabeled conflicting intents.
Methodology: Task-Specific Embeddings
The core of this approach involves transitioning from general-purpose embeddings (e.g., Word2Vec, KoGPT2) to Task-Specific Embeddings. These are internal representations extracted from a model specifically trained for the target domain (e.g., the Adot service).
The researchers evaluated three types of task-specific vectors:
- Whole Layer Vector (WLV): A concatenation of all internal layers.
- After Representation Layer Vector (ARLV): Focuses on processing layers by excluding the initial embedding layer.
- Conclusion Layer Vector (CLV): Utilizes only the layer immediately preceding the final output.
By measuring the similarity between new inputs and training data using these vectors, the system flags sentences that are semantically close to multiple categories, indicating potential ambiguity.
Experimental Setup and Datasets
The study utilized several datasets from the Adot service:
- Adot Training Data: 2.2 million Korean sentences across 30 domains.
- Correlation Test Dataset: 913 sentence pairs for comparing model similarity scores against human judgment.
- Ambiguity Test Dataset: 29,000 scenarios used to identify service design flaws.
- Training Data Ambiguity Test Set: 2,300 misclassified sentences used for error analysis.
Key Results and Insights
The experiments revealed that task-specific embeddings, while "biased" toward the service domain, are superior at detecting the specific confusions inherent to that service.
- Performance: The Adot-CLV method achieved a Pearson correlation of 0.628 with user-perceived class relevance, significantly outperforming general models like KoGPT2 (0.394).
- Ambiguity Identification: Out of 29,136 scenarios, the method flagged 243 highly ambiguous cases. Manual inspection confirmed that approximately 47% were genuine semantic ambiguities requiring design changes.
- Data Quality: The method proved effective for data cleaning. By identifying "confusing" data points in misclassified training sets, researchers achieved a correction ratio of over 70% after relabeling and retraining.
The study also noted a divergence in perception: while service experts focus on rigid boundaries, ordinary users focus on semantic meaning. The CLV aligned most closely with user perception.
Limitations and Future Directions
The current research focused on short Korean imperative sentences. Future work will explore the scalability of this method to longer texts, other languages, and larger Transformer-based architectures. Additionally, the researchers aim to automate the handling of detected ambiguity, perhaps through system-generated clarification questions to the user.
Summary updated