Automatic Learning Assistant in Telugu

Meghana Bommadi, Shreya Terupally, Radhika MamidiACL 2021

TL;DR

This paper introduces an automated learning assistant designed to generate questions and evaluate answers in Telugu, a low-resource language. By combining rule-based and AI approaches, the system aims to make educational content more accessible and engaging for Telugu speakers, addressing a significant gap in current NLP-driven educational tools.

What Problem Does This Paper Address?

While virtual assistants and AI-powered learning tools are prevalent in high-resource languages like English, there's a notable lack of similar resources for many Indian languages, including Telugu. This gap limits access to personalized learning experiences and automated assessment for a significant portion of the global population. The authors highlight that building a fully AI-based solution for low-resource languages is often unfeasible due to the scarcity of large datasets. Existing rule-based systems might also miss crucial nuances. Therefore, the paper tackles the challenge of creating an effective learning assistant for Telugu by developing a hybrid approach that leverages both linguistic rules and machine learning techniques.

Methods

The core of the system is built around three key steps:

  1. Summarization: To distill the essential information from a text, the system employs two unsupervised summarization techniques:
  2. Word Frequency-based Summarization (WFBS): This method identifies sentences containing the most frequent words, assuming these words carry the most crucial information.
  3. TextRank: A graph-based ranking algorithm that models sentences as nodes and their similarity as edges. It iteratively ranks sentences based on their connectivity and similarity to the overall text.
  4. Question Generation: This is the most novel aspect. The system generates questions based on identified answer phrases within a sentence. It uses a combination of:
  5. Part-of-Speech (POS) Tagging and Universal Dependency (UD) Parsing: These NLP techniques are used to identify grammatical roles and relationships within sentences.
  6. Language-Specific Rules: A comprehensive set of rules, categorized by question type (quantifiers, adjectives, adverbs, possession, time-place, direct/reported speech, verbs, and subjects), are applied to transform identified answer phrases into appropriate Telugu questions. These rules leverage POS and UD tags along with linguistic context.
  7. Answer Evaluation: The system evaluates user-provided answers using two methods:
  8. Telugu Answer Evaluation: For Telugu inputs, it uses string matching against the stored correct answer.
  9. Multilingual Answer Evaluation: For answers in other languages, it employs multilingual sentence embeddings (using Sentence Transformers) to compare the semantic similarity between the user's answer and the expected answer.

Datasets

  • Primary Dataset: A Telugu stories dataset sourced from "kathalu.wordpress.com". This dataset was chosen for its variety in themes, vocabulary, and sentence lengths, making it suitable for generating diverse questions. It comprises 21 stories with an average of 56 sentences and 281 words per story, categorized as "Moral Stories for Children."
  • Testing Dataset: Stories by Prof. N. Lakshmi Aiyar, consisting of 5 stories with an average of 190 sentences and 1060 words per story, categorized as "Realistic Fiction."
  • POS Tagger Training Data: A Telugu corpus of approximately 3.5 million tokens was used for pre-training the POS tagger.
  • UD Tagging Data: The UD_Telugu-MTG dataset was used for training the Bi-LSTM UD tagging model.

Key Insights and Ideas

  • Hybrid Approach for Low-Resource Languages: The paper demonstrates that a mixed rule-based and AI approach can be highly effective for low-resource languages where purely data-driven methods are challenging. This hybrid model balances linguistic precision with the flexibility of machine learning.
  • Leveraging Linguistic Structure for Question Generation: The detailed rule-based system for question formation, informed by POS and UD tags, is a significant contribution. It shows how grammatical structures can be systematically transformed into interrogative forms.
  • Adaptability of Summarization: Using both Word Frequency and TextRank for summarization provides a flexible way to extract key information, which is crucial for identifying potential answer phrases for question generation.
  • Multilingual Answer Evaluation: The integration of multilingual sentence embeddings for answer evaluation is a smart way to handle cross-lingual learning scenarios, making the assistant more versatile.

Key Results

The system achieved 96.28% accuracy in question and answer generation. The authors report that the generated questions are grammatically correct and resemble academic questions found in textbooks. Manual error analysis indicated that most generated questions were legible and contextually significant, with only a small percentage being completely erroneous or illegible. The system successfully generated questions for simple sentences, requiring minimal further linguistic analysis.

Limitations and Future Work

  • Anaphora Resolution: The paper acknowledges that anaphora resolution (identifying what pronouns or noun phrases refer to) is a limitation, leading to some inaccuracies in answer evaluation.
  • UD Tagging Accuracy: While the UD tagging model achieved 85% accuracy, the authors note that this is acceptable given the scarcity of labeled data and the infrequent labeling of "subj" tags.
  • Domain Specificity: The authors suggest that results might vary if the system is tested on domains other than stories, as the rules were constructed based on observation-based data within a particular domain.
  • Future Work: The authors plan to extend their work by:
  • Implementing anaphora resolution.
  • Expanding the system to cover more question types.
  • Improving the UD tagging model.
  • Extending the system to other domains.
  • Testing the system for questions on minor details and Named Entities (NE) / Common Nouns (CN).

Summary updated

BibTeX
@inproceedings{bommadi2021automatic,
  author = {Meghana Bommadi and Shreya Terupally and Radhika Mamidi},
  title = {{Automatic Learning Assistant in Telugu}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure