Abstract
This study examines the adoption of rubrics in grading criteria at Japanese universities. Between 2016 and 2020, the percentage of universities explicitly stating grading criteria in syllabi increased to 98.1%. However, the use of rubrics for all subjects at the undergraduate level only rose from 2.9% to 6.1%, indicating challenges in rubric adoption. A two-step approach was proposed to address faculty members' difficulties in creating rubrics: 1) introducing a reference rubric (R2) based on common learning outcomes in diploma policies, and 2) having faculty members create rubrics for their individual learning outcomes. Successful implementation of this approach was reported at a university where rubrics were introduced for all subjects. Furthermore, the study evaluated the comparison between R2 and syllabi rubrics using doc2vec, pre-trained by Wikipedia text data, and cosine similarity to assess the degree of optimization for educational goals of each departments.
Introduction: The Implementation Gap in Japanese Higher Education
In recent years, the Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT) has emphasized transparency in student assessment. While nearly 98% of Japanese universities now explicitly state grading criteria in their syllabi, a significant "implementation gap" remains regarding rubrics. Rubrics—detailed scoring tools that define specific expectations for different levels of achievement—are only used across all undergraduate subjects in approximately 6% of institutions.
The primary barriers to adoption include the high administrative burden on faculty, difficulties in maintaining inter-rater reliability, and a general lack of pedagogical training among higher education instructors. This paper proposes a solution through the lens of "Eduinformatics."
Methodology: The Eduinformatics Approach
The researchers define "Eduinformatics" as the intersection of education and informatics, where informatics provides the problem-solving methods for educational challenges.
The Two-Step Rubric Development Framework
To lower the barrier for faculty, the study implemented a two-step approach:
- Reference Rubric (R2): The institution provides a standardized baseline rubric aligned with university-wide diploma policies.
- Individual Customization: Faculty members adapt this R2 to reflect the specific learning outcomes of their individual courses.
Evaluation via Machine Learning
To evaluate the effectiveness of this customization, the authors employed Doc2Vec, an unsupervised machine learning algorithm. Unlike simple keyword matching, Doc2Vec captures semantic meaning by representing text as fixed-length vectors. By pre-training the model on Japanese Wikipedia, the researchers could calculate the cosine similarity between the standardized R2 and the final faculty-produced rubrics. This quantitative measure reveals how much faculty "optimized" the template for their specific departmental needs.
Results and Discussion
National Trends in Rubric Adoption
Data from MEXT questionnaires between 2011 and 2020 highlight the slow growth of comprehensive rubric usage across different university sectors (National, Public, and Private).
As shown in the national data, while the use of rubrics for some classes has increased significantly, the "all classes" metric remains low. In contrast, by applying the two-step approach, Kobe Tokiwa University achieved 100% rubric inclusion in all class syllabi by 2020.
Semantic Analysis of Customization
The Doc2Vec analysis revealed that departments with more experience in rubric-based assessment, such as Nursing, showed lower cosine similarity to the reference rubric. This suggests a higher degree of "pedagogical maturity," where faculty actively modified the template to better suit clinical and professional goals rather than simply copy-pasting the provided scaffold.
Conclusion
The two-step approach effectively bridges the gap between policy requirements and faculty implementation. By providing a "scaffold" (the R2) and using Eduinformatics tools like Doc2Vec for evaluation, universities can move beyond mere compliance toward meaningful, optimized assessment criteria. Future work will focus on automating data extraction from unstructured syllabus formats and linking these rubric metrics to actual student performance data.
Summary updated