TL;DR
This paper introduces Sakshm AI, an intelligent tutoring system designed to enhance coding education for engineering students in India. By leveraging Socratic questioning through its AI chatbot, Disha, Sakshm AI aims to foster critical thinking and deeper understanding, moving beyond traditional AI tools that often provide direct answers. The study analyzes user data and feedback to demonstrate Sakshm AI's impact on problem-solving behaviors and engagement.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has opened new avenues for educational tools, particularly in programming. While LLMs can generate code, explain concepts, and offer personalized feedback, existing AI tools for programming education face several challenges. These include a lack of Socratic guidance, limited contextual memory, insufficient adaptive feedback, and the burden of prompt engineering. Sakshm AI aims to address these limitations by providing a more structured, context-aware, and pedagogically sound approach to AI-assisted coding education, focusing on fostering independent problem-solving skills rather than rote memorization or direct solution retrieval.
Methods
Sakshm AI employs a multi-faceted approach to AI-assisted coding education:
- Socratic Tutoring (Disha Chatbot): The core of Sakshm AI is its Disha chatbot, which uses a Socratic questioning framework. Instead of providing direct answers, Disha guides students through problems with context-aware hints and structured feedback, maintaining conversational memory and adapting to the user's progress.
- LLM-Based Code Analysis: Sakshm AI utilizes LLMs to analyze student code submissions for correctness, efficiency, and code quality. This analysis informs the feedback provided to students.
- Comprehensive Question Bank: The platform features a large bank of over 450 Data Structures and Algorithms (DSA) problems, categorized by difficulty and tagged with relevant companies, allowing for targeted practice.
- Integrated Development Environment (IDE): A user-friendly IDE supports multiple programming languages and includes features like test case validation and custom test case creation.
- Automated Performance Reports: Upon submission, students receive detailed reports on correctness, efficiency, and code quality, along with recommendations for improvement.
- Mixed-Methods Evaluation: The study combines quantitative analysis of platform logs and engagement trends with qualitative insights from structured surveys and in-depth interviews with users.
Datasets
The study utilized data from 1,170 registered participants on the Sakshm AI platform. This data included platform logs, engagement trends, and problem-solving behavior. Additionally, a structured survey was administered to 45 active users, and 25 in-depth interviews were conducted. No new benchmarks were introduced; the study focused on analyzing user interactions with the existing Sakshm AI platform.
Key Insights and Ideas
- Socratic Guidance is Key: Users found Disha's Socratic approach more beneficial for fostering critical thinking and problem-solving than platforms that offer direct answers (like ChatGPT). This approach encourages deeper understanding and reduces reliance on simply copying solutions.
- Context Matters: Sakshm AI's ability to maintain conversational context was highly valued, differentiating it from tools where users have to re-explain their problem or code repeatedly.
- Balancing AI and Human Touch: While AI offers 24/7 accessibility and speed, users still recognize the irreplaceable value of human tutors for personalized insights and deeper understanding, suggesting a hybrid approach is optimal.
- Engagement Peaks in Afternoon/Evening: User activity, particularly chat usage, shows a clear pattern, peaking in the afternoon and early evening, aligning with typical study schedules.
- Socratic Help for Medium Difficulty: Users tend to rely more on the Socratic chatbot for medium-difficulty problems, suggesting it's most effective when students need guidance but not a complete solution. Harder problems might require more direct intervention or different support strategies.
Key Results
- Quantitative Analysis: The study analyzed 1,170 users, revealing that users who engaged with chat support generally showed better problem-solving efficiency and more structured engagement. Peak chat activity occurred between 1 PM and 5 PM.
- Qualitative Analysis:
- 15 out of 25 interviewees appreciated Sakshm AI's Socratic approach for fostering independent thinking.
- Users found the platform's features like company-specific questions, complexity analysis, and integrated hints unique and valuable for interview preparation.
- While generally positive, users suggested improvements such as more varied problem types, enhanced feedback detail, and better UI features like autosave and dark mode.
- Users reported that Sakshm AI's feedback system was helpful for understanding code quality and efficiency, promoting professional coding habits.
Limitations and Future Work
- Limitations:
- Sakshm AI is currently an independent learning platform, not integrated into formal curricula.
- It supports only four programming languages and is still in its developmental stage.
- The Disha chatbot is only active during DSA problem-solving sessions.
- The study's seven-month duration limits long-term conclusions.
- The dataset is predominantly from top-tier engineering colleges, potentially not representing diverse experiences.
- Future Work:
- Expand to interdisciplinary applications (data science, finance, etc.).
- Integrate peer learning and collaboration tools.
- Conduct longitudinal studies to assess long-term impact.
- Incorporate gamification and rewards for sustained motivation.
- Enhance accessibility features (multilingual support, assistive technologies).
- Develop a scaffolded teaching model alongside Socratic guidance for complex problems.
- Introduce advanced career-oriented tools (resume builder, mock interviews).
- Broaden language support and problem types (real-world scenarios, advanced topics).
- Upgrade the feedback system with more actionable insights and comparative analytics.
Summary updated