TL;DR
This paper introduces Sanskrit Voyager, a unified web platform that leverages advanced NLP techniques to make the vast corpus of Sanskrit literature accessible for interactive reading and linguistic analysis. It addresses the fragmentation of existing tools by integrating dictionary lookup, text parsing, and corpus search into a single, user-friendly interface, significantly lowering the barrier to entry for both learners and researchers.
What Problem Does This Paper Address?
Sanskrit, an ancient language central to many religious and philosophical traditions, has a rich and extensive literary corpus. However, accessing and analyzing these texts computationally is hindered by several challenges:
- Fragmentation of Tools: Existing digital libraries and platforms often provide plain text with minimal annotation, or focus on specific aspects like cross-text search, but lack unified, interactive environments for reading and analysis.
- Linguistic Complexity: Sanskrit's complex morphology, including inflections, sandhi (euphonic word merging), and compounding, obscures word boundaries and complicates dictionary lookup and search.
- Expertise Required: Current tools often demand significant linguistic expertise or manual normalization, making them inaccessible to newcomers. Users frequently need to combine multiple platforms for basic tasks.
- Limited Accessibility: Proprietary transliterations, restrictive licenses, and inconsistent formatting further limit the accessibility of digital Sanskrit resources.
Sanskrit Voyager aims to bridge this gap by offering a comprehensive, integrated, and user-friendly platform that handles these complexities automatically.
Methods
Sanskrit Voyager employs a modular client-server architecture with a Python backend and a React frontend. Key NLP techniques and architectural choices include:
- Cascading Analysis Pipeline: A multi-stage system for text processing that combines fast, rule-based methods for simple terms with statistical models (including ByT5-Sanskrit) for more complex sandhi and compound analysis. This approach aims for efficiency and robustness.
- Automatic Normalization: The system automatically normalizes input, handling various transliterations, inflections, sandhi, and compound forms, eliminating the need for manual preprocessing.
- Multi-Dictionary Integration: It aggregates results from six major Sanskrit lexica, offering a comprehensive view of word meanings.
- Interactive Text Processing: Users can upload texts or books, which are then automatically segmented, parsed, and annotated on-click. Subwords and citations are interactive, linking directly to dictionary entries or corpus searches.
- Advanced Corpus Search: The platform utilizes PostgreSQL with advanced full-text indexing and fuzzy matching to enable efficient searching across large corpora, even with inflected or sandhi-merged forms.
- Resource Efficiency: The backend is designed to run efficiently on modest hardware, demonstrating a focus on accessibility and scalability.
Datasets
The platform integrates over 900 digitized Sanskrit texts from major corpora like GRETIL and SARIT. It also leverages the BuddhaNexus (now DharmaNexus) dataset for its corpus search functionality, with plans to integrate additional datasets. The paper mentions the Sandhikosh benchmark for evaluating sandhi splitting, but notes its limitations in covering compounds. A custom evaluation was designed for compound analysis.
Key Insights and Ideas
- Unified Platform is Key: The most significant insight is that integrating disparate functionalities (dictionary, parsing, reading, search) into a single, cohesive platform dramatically improves usability and accessibility for Sanskrit studies.
- Handling Linguistic Complexity Automatically: By automating the normalization of inflections, sandhi, and compounds, Sanskrit Voyager democratizes access to texts, removing a major hurdle for non-experts.
- Interactive Exploration: Making every analyzed component (subwords, stems, roots) clickable and interactive transforms text analysis from a static process into an engaging exploration.
- Cascading Approach for Efficiency: Combining rule-based and statistical methods in a cascading pipeline offers a practical solution for balancing accuracy and processing speed across a diverse range of linguistic phenomena.
Key Results
- Parsing Accuracy: The system achieved over 92.4% accuracy on complex compounds and 95.9% accuracy on the entire Yoga Sūtra text.
- Corpus Search Recall: Sanskrit Voyager demonstrated substantially higher search recall than BuddhaNexus on challenging queries, finding significantly more matches for terms like "ālayavijñāna" (472 vs. 73).
- User Engagement: Early user evaluation indicates strong engagement, with users averaging over 9 minutes per session and assigning high scores for usefulness and ease of use.
- Efficiency: Dictionary lookups complete in under 100 ms, and even complex compounds are processed within 3 seconds. The backend runs efficiently on a single droplet with 2 vCPUs and 4GB RAM.
Limitations and Future Work
- Dictionary Gaps: Parsing errors are often attributed to dictionary gaps, particularly for rare or Buddhist technical terms.
- Limited Corpus Coverage (Currently): While the platform integrates many texts, the corpus search currently focuses on the BuddhaNexus dataset, with plans to expand.
- Machine Translation: While machine translation is offered for a subset of books, it's based on state-of-the-art LLMs, and further development of evaluation metrics is needed to determine the best models.
- Future Work: Planned enhancements include integrating more Sanskrit digital libraries, developing advanced features like embedding-based retrieval, expanding dictionary coverage (including traditional lexical resources like koshas and Dhātupātha), and refining machine translation models. The developers are also committed to incorporating community feedback for continuous improvement.
Summary updated