In technical education and hiring, uncompilable code submissions are frequently discarded, despite often containing sound algorithmic logic. This paper presents a novel framework for grading uncompilable code by decoupling syntax from semantics. By employing "Make Compilable" (MC) and "Rule Relaxation" (RR) techniques, the system extracts features from imperfect code to build a grading model. Results demonstrate that the system achieves a 0.71 correlation with human experts, effectively matching human consistency and enabling the identification of high-potential candidates who would otherwise be filtered out by strict compilers.
Introduction
Automated grading systems typically rely on functional test cases, requiring code to be syntactically perfect to run. However, uncompilable code is prevalent, representing approximately 34% of MOOC submissions and 46% of job application pools. Rejecting these submissions outright leads to "evaluator blindness," where candidates with strong algorithmic thinking are overlooked due to minor syntax errors. This research aims to assess the underlying "logic" of a program independently of its "syntax."
Methods
The proposed system utilizes two primary strategies to process broken code:
- Make Compilable (MC): A rule-based system combined with an n-gram token predictor attempts to fix common syntax errors like missing semicolons or unbalanced parentheses.
- Rule Relaxation (RR): This method generates "Imperfect" Abstract Syntax Trees (AST) and Symbol Tables (ST) by skipping unparseable segments and ignoring strict type-checking or declaration errors.
Features such as keywords, expressions, and control contexts are extracted from these structures. A LASSO regression model is then trained to predict grades. A key innovation is "Distribution Correction" (DC), which allows a model trained on compilable code (CC) to be adapted for uncompilable code (UC) with minimal manual labeling, reducing the need for manual annotation by approximately 33%.
Results and Discussion
The system was evaluated on 2,606 Java programs across five tasks. The hybrid RRMC approach proved robust, parsing 95% of submissions.
As shown in the performance data, the system's correlation with human graders (0.71) nearly matches the inter-human correlation (0.72). Furthermore, the "Distribution Correction" technique (CC Model with dist. Fixed) significantly improves the Mean Absolute Error (MAE) compared to a raw CC model, validating the feasibility of model reuse.
In a real-world hiring scenario with a Chinese tech company, the system identified that 15.5% of uncompilable submissions actually possessed near-correct logic. By utilizing this tool, the company was able to hire 19% more candidates who would have been automatically rejected by a traditional compiler-based filter.
Conclusion
By focusing on algorithmic intent rather than syntactic perfection, this system provides a more equitable and comprehensive assessment of programming skill. The automated system achieves human-level consistency, offering a scalable solution for large-scale education and recruitment. Future work will focus on expanding language support and providing more granular, line-by-line feedback.
Summary updated