Abstract
This paper introduces the Automated Cognitive Modeler (ACM), a machine learning system designed to automatically generate production system models of human cognitive processes. Focusing on the domain of multi-column subtraction, ACM treats cognition as a search through a problem space and learns to "reverse engineer" the specific procedural errors, or "bugs," in a student's logic. Unlike previous diagnostic systems, ACM does not require a pre-defined library of errors, instead constructing models directly from performance data.
Introduction
In the early 1980s, cognitive simulation—the creation of computer programs to mimic human mental processes—was a manual and labor-intensive task. Researchers typically observed behavioral data, such as a student's incorrect answers on math problems, and manually intuited the rules that might explain such errors. While AI tools like DEBUGGY existed for diagnosing errors, they relied heavily on "bug libraries"—fixed lists of known common mistakes. The authors of this paper sought a more general, automated method to discover a student's internal "program" directly from their performance data, whether correct or erroneous.
Methods: The ACM Algorithm
The ACM approach is built upon two foundational pillars of cognitive science: the Production System Hypothesis, which models behavior as condition-action rules, and the Problem Space Hypothesis, which views cognition as a search through possible states using specific operators.
The ACM algorithm operates in three distinct stages:
- Exhaustive Search: For a given problem (e.g., $93 - 25$) and a student's specific answer (e.g., $72$), ACM searches the problem space to identify a "solution path" that leads to that exact result.
- Instance Generation: Once a path is identified, the moves on that path are labeled as positive instances for the rules that proposed them. Moves leading away from the path are labeled as negative instances.
- Discrimination Learning: ACM employs a learning algorithm (similar to ID3) to identify the specific conditions that distinguish positive instances from negative ones. For instance, it might learn that a student applies a "find-difference" operator only when the top digit is larger than the bottom digit.
One of the core insights of ACM is treating algorithmic behavior as "frozen" search. A rigid procedure is viewed as a search process where rules have become so specific that only one move is suggested at any given time. By breaking the task down and learning application conditions for individual operators (like "decrement" or "add-ten") independently, ACM makes the complex task of modeling human cognition computationally manageable.
Evaluation and Results
The system was evaluated using the domain of multi-column subtraction, utilizing empirical analyses of over 100 systematic errors compiled by VanLehn and colleagues. ACM was tested on 20 representative problems for each of 11 common "bugs."
ACM successfully modeled all 11 common subtraction bugs. Once the system learned the refined rules for a specific "buggy" student, it could regenerate the student's exact incorrect solution paths for new problems without further searching. Notably, for the "smaller from larger" bug—where a student subtracts the smaller digit from the larger regardless of position—ACM produced a parsimonious two-rule model that perfectly predicted the error.
Limitations and Future Work
Despite its success, ACM faced several challenges. In 1984, searching the problem space was computationally expensive; modeling 20 problems could take up to two CPU hours on a Vax 750, and deeper search paths led to combinatorial explosions. Additionally, when multiple paths led to the same incorrect answer, ACM defaulted to the shortest path, which may not always be the most psychologically plausible explanation.
The system also required a human-defined problem space (states and operators) upfront. While it could learn when to use an operator, it could not yet invent new ones. Future work suggested extending ACM to handle multiple problem spaces and incorporating reaction time data to better constrain model selection.
Summary updated