Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings

Jong Myoung Kim, Young-Jun Lee, Sangkeun Jung, Ho-Jin Choi

Introduction

In voice recognition agents, such as SK Telecom’s Adot, semantic ambiguity presents a significant challenge to user experience. Ambiguity often arises when a single term possesses multiple, sometimes contradictory, meanings depending on the context. For instance, the Korean word "앞" (ap) can denote "forward" (the front) or "previous" (the antecedent). In a "one-turn" interaction typical of voice assistants, where context is minimal and queries are short (averaging 2.05 words), resolving such ambiguity is difficult.

Semantic ambiguity of the Korean word "ap" in a voice command context.

Rather than attempting to resolve every ambiguity, researchers from SK Telecom and KAIST propose a method to detect it. Detecting ambiguity allows developers to identify "risky" service scenarios and debug training data where human annotators may have mislabeled conflicting intents.

Methodology: Task-Specific Embeddings

The core of this approach involves transitioning from general-purpose embeddings (e.g., Word2Vec, KoGPT2) to Task-Specific Embeddings. These are internal representations extracted from a model specifically trained for the target domain (e.g., the Adot service).

The researchers evaluated three types of task-specific vectors:

  1. Whole Layer Vector (WLV): A concatenation of all internal layers.
  2. After Representation Layer Vector (ARLV): Focuses on processing layers by excluding the initial embedding layer.
  3. Conclusion Layer Vector (CLV): Utilizes only the layer immediately preceding the final output.

By measuring the similarity between new inputs and training data using these vectors, the system flags sentences that are semantically close to multiple categories, indicating potential ambiguity.

t-SNE visualization comparing general-purpose embeddings (sent2vec, KoGPT2) with task-specific embeddings (WLV, ARLV, CLV).

Experimental Setup and Datasets

The study utilized several datasets from the Adot service:

  • Adot Training Data: 2.2 million Korean sentences across 30 domains.
  • Correlation Test Dataset: 913 sentence pairs for comparing model similarity scores against human judgment.
  • Ambiguity Test Dataset: 29,000 scenarios used to identify service design flaws.
  • Training Data Ambiguity Test Set: 2,300 misclassified sentences used for error analysis.

Key Results and Insights

The experiments revealed that task-specific embeddings, while "biased" toward the service domain, are superior at detecting the specific confusions inherent to that service.

  • Performance: The Adot-CLV method achieved a Pearson correlation of 0.628 with user-perceived class relevance, significantly outperforming general models like KoGPT2 (0.394).
  • Ambiguity Identification: Out of 29,136 scenarios, the method flagged 243 highly ambiguous cases. Manual inspection confirmed that approximately 47% were genuine semantic ambiguities requiring design changes.
  • Data Quality: The method proved effective for data cleaning. By identifying "confusing" data points in misclassified training sets, researchers achieved a correction ratio of over 70% after relabeling and retraining.

The study also noted a divergence in perception: while service experts focus on rigid boundaries, ordinary users focus on semantic meaning. The CLV aligned most closely with user perception.

Limitations and Future Directions

The current research focused on short Korean imperative sentences. Future work will explore the scalability of this method to longer texts, other languages, and larger Transformer-based architectures. Additionally, the researchers aim to automate the handling of detected ambiguity, perhaps through system-generated clarification questions to the user.

Summary updated

Abstract

Ambiguity is a major obstacle to providing services based on sentence classification. However, because of the structural limitations of the service, there may not be sufficient contextual information to resolve the ambiguity. In this situation, we focus on ambiguity detection so that service design considering ambiguity is possible. We utilize similarity in a semantic space to detect ambiguity in service scenarios and training data. In addition, we apply task-specific embedding to improve performance. Our results demonstrate that ambiguities and resulting labeling errors in training data or scenarios can be detected. Additionally, we confirm that it can be used to debug services
BibTeX
@inproceedings{kim2023semantic,
  author = {Jong Myoung Kim and Young-Jun Lee and Sangkeun Jung and Ho-Jin Choi},
  title = {{Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.41},
}
Figure