TL;DR
This paper introduces the Information Exchange (IE) system, a novel approach to building organization-embedded question-answering (QA) systems. IE systems act as intermediaries, allowing clients to ask natural language questions and retrieving answers from an organization's existing expertise, thus making organizational knowledge more accessible and reusable.
What Problem Does This Paper Address?
Organizations increasingly want to make their expertise accessible online via the World Wide Web. However, traditional methods like query languages, complex natural language understanding systems, or simple intelligent browsers are often inadequate. Clients may lack the technical skills for query languages, knowledge engineering is expensive and doesn't scale, and intelligent browsers are too restrictive. The authors aim to solve this by creating a deployable QA technology that organizations can easily embed to answer questions from diverse client groups using natural language.
Methods
The core of the IE system is an "organization-embedded question-answering intermediary." This system operates in two stages:
- Organization Modelling: The system models the organization's structure, often a hierarchy of units and their areas of expertise, as a hierarchy of topics.
- Expertise Acquisition: Experts create "information accounts" under relevant topics. The system then uses a case-based reasoning (CBR) approach, where existing question-answer pairs (Q&A's) are treated as "cases."
Key techniques employed include:
- Hierarchical Indexing: The topic hierarchy is used to create a hierarchy of vector spaces, making search more efficient.
- Predictive Indexing: This technique uses a semantic network (WordNet) to expand query terms with related terms, going beyond simple synonyms or inclusion, and is fully automated.
- Combined Term Weighting: A novel metric that combines both semantic (part of speech, polysemy, depth) and statistical (TF-IDF, condensation clustering) properties of terms to determine their importance.
- Abstraction-Based Retrieval: The system aims to answer questions at the appropriate level of abstraction for the client.
- Negative Evidence Acquisition: Experts can mark questions as non-relevant, which helps the system learn from its failures and avoid similar mistakes in the future.
When a client asks a question, the system converts it into a vector and searches the relevant vector space. If a match is found, the answer is retrieved. If no suitable answer exists, the question is passed to an expert, and their answer is then stored for future use.
Datasets
The paper describes an evaluation using 105 questions about Common Lisp and AI, collected from 20 computer science undergraduates. These questions were selected because they had at least one relevant answer in the system's Q&A collection. The system was populated with Q&A's from a CIE expert.
Key Insights and Ideas
- Bridging the Expertise Gap: The IE system effectively acts as a bridge, translating natural language queries into a format that can access an organization's stored knowledge, making expertise accessible to a broader audience.
- Learning from Failure: The "negative evidence acquisition" is a crucial insight, allowing the system to improve by explicitly learning from what doesn't work, rather than just relying on positive feedback.
- Hybrid Term Weighting: Combining semantic and statistical information for term weighting is a powerful idea, acknowledging that both the meaning and the frequency of words are important for accurate retrieval.
- Organization as a Social Agent: The system is designed to be embedded within an organization, acting as a social agent that services both clients seeking information and experts looking to reduce their workload.
Key Results
The evaluation focused on the "Average Number of Interactions" (ANI) required for a client to find the first relevant answer.
- The system achieved an ANI of 1.0 for one subject, indicating a very efficient search. Other subjects had ANIs ranging from 1.2 to 3.2.
- Experiments with different weighting schemes (varying the importance of
w_tfidfandw_ifcc) showed that for smaller collections,w_tfidfwas more effective, while for larger collections,w_ifcc(which incorporates condensation clustering) performed better.
Limitations and Future Work
- Limitations: The paper acknowledges that the system's performance is influenced by the term ambiguity in client questions. High ambiguity requires more interactions for clarification. The evaluation was conducted on a specific domain (Common Lisp and AI) and with a limited number of Q&A's.
- Future Work: The authors suggest further experimentation with the relative importance of different term weighting metrics (
a2anda3) on larger collections. They also implicitly point to the ongoing challenge of building robust and scalable QA systems for diverse organizational needs.
Summary updated