TL;DR
This paper introduces two novel metrics, "concept-comparison" and "concept-projection," to quantify how "domain-independent" semantic classes (concepts) are across different dialogue system tasks. By measuring the similarity of lexical contexts, these methods help identify concepts that can be reused, significantly reducing the effort needed to build new dialogue systems.
What Problem Does This Paper Address?
Building robust speech understanding modules for new dialogue system domains is a time-consuming and labor-intensive process. Traditionally, semantic classes (concepts like "destination city" or "movie genre") must be manually defined for each new task. This paper tackles the challenge of domain portability: how can we leverage knowledge from existing, well-studied domains to speed up the development of new ones? Specifically, they aim to automatically identify which semantic concepts are general enough to be reused across different tasks, and which are highly specific to a particular domain.
Methods
The authors propose two primary methodologies based on comparing semantic classes across domains using Kullback-Leibler (KL) divergence, a measure of how one probability distribution differs from a second, reference probability distribution.
- Concept-Comparison Method: This method compares pairs of semantic classes, one from each of two different domains. It estimates the similarity of the lexical (syntactic) contexts in which phrases belonging to these classes appear. For example, it might compare how "city names" are used in a travel domain versus how "movie genres" are used in a movie domain. The core idea is that if two concepts are similar, their associated words and phrases will appear in similar linguistic environments. They use context-dependent bigram language models to calculate KL distances between these environments.
- Concept-Projection Method: This method focuses on a single semantic class from one domain and assesses its "domain independence" by projecting it into the lexical environment of another domain. It measures how similarly the words within that concept are used across the two domains. A low KL distance here suggests the concept is domain-independent, as its usage patterns are consistent.
Both methods rely on calculating KL distances between probability distributions of word contexts. The "distance" between concepts is then derived from these KL divergences, with smaller distances indicating greater domain independence.
Datasets
The authors evaluated their metrics using manually generated semantic classes from three distinct domains:
- Carmen: A children's game corpus collected from a Wizard of Oz study.
- Movie: An information retrieval service for movie titles, show times, and theater/city names.
- Travel: A corpus similar to ATIS, used for flight, car, and hotel reservations.
The corpora were relatively small, with fewer than 2500 sentences each. No new benchmarks were introduced; instead, existing manually defined classes were used for evaluation.
Key Insights and Ideas
- Lexical Context is Key: The fundamental hypothesis is that semantically similar phrases will appear in similar syntactic (lexical) contexts, even across different domains. This is the bedrock of their metrics.
- Domain Independence is Quantifiable: The paper demonstrates that the degree of domain independence for semantic classes can be measured using statistical methods like KL divergence.
- Identifying Reusable Concepts: The metrics provide a rank-ordering of concepts, allowing developers to identify which ones are likely to be reusable and require less manual effort when building a new dialogue system.
- Beyond Literal Definitions: The metrics can reveal similarities in how concepts are used by people in natural speech, even if their formal definitions differ. For instance, "YES" and "GREET" might show similarity due to conversational patterns, not just their semantic meaning.
Key Results
- Both the concept-comparison and concept-projection methods were found to be good indicators of domain independence for manually generated concepts.
- For the Travel <=> Carmen comparison, the projected distances were very similar to the compared distances for concepts like <WANT>, <YES>, and <CITY>, suggesting these are indeed domain-independent.
- Concepts like <CARDINAL> and <MONTH> (specific to Travel) showed high KL distances when projected to Carmen, indicating domain specificity.
- The <WANT> concept was identified as highly domain-independent, as people tend to express requests similarly across tasks.
- The <CITY> class, while containing different sets of cities, showed small KL distances due to similar lexical contexts of use.
- The paper highlights how concepts like <YES> and <GREET> can appear similar in usage patterns due to conversational pragmatics, even if semantically distinct.
Limitations and Future Work
- Manual Class Generation: The evaluation relied on manually generated semantic classes. Future work could explore automatic or semi-automatic methods for concept acquisition and then apply these metrics.
- Small Corpora: The study used relatively small corpora. Larger datasets might yield more robust results.
- Informal Evaluation: The evaluation was informal, relying on experimenter intuition. More rigorous quantitative validation would be beneficial.
- Future Directions: The authors suggest extending this work to evaluate automatic derivation of semantic classes, explore soft classification (assigning words to multiple concepts), and build task hierarchies. The ultimate goal is to create a powerful tool for designers to import knowledge from similar, previously studied tasks.
Summary updated