Abstract
Parallel test assembly has long been an important yet challenging topic in educational assessment. Cognitive diagnosis models (CDMs) are a new class of assessment models and have drawn increasing attention for being able to measure examinees' ability in detail. However, few studies have been devoted to the parallel test assembly problem in CDMs (CDM-PTA). To fill the gap, this paper models CDM-PTA as a subset-based bi-objective combinatorial optimization problem. Given an item bank, it aims to find a required number of tests that achieve optimal but balanced diagnostic performance, while satisfying important practical requests in the aspects of test length, item type distribution, and overlapping proportion. A set-based multiobjective particle swarm optimizer based on decomposition (S-MOPSO/D) is proposed to solve the problem. To coordinate with the property of CDM-PTA, S-MOPSO/D utilizes an assignment-based representation scheme and a constructive learning strategy. Through this, promising solutions can be built efficiently based on useful assignment patterns learned from personal and collective search experience on neighboring scalar problems. A heuristic constraint handling strategy is also developed to further enhance the search efficiency. Experimental results in comparison with three representative approaches validate that the proposed algorithm is effective and efficient.
Introduction
In modern educational assessment, Cognitive Diagnosis Models (CDMs) are considered the gold standard for providing fine-grained feedback on specific skills mastered by students. However, high-stakes scenarios—such as preventing cheating or conducting pre- and post-tests—require the assembly of parallel tests. These are multiple versions of a test that use different questions but yield nearly identical diagnostic results.
Manually assembling these tests is an "NP-hard" combinatorial challenge. Developers must maximize the diagnostic quality of every test while ensuring they are all equally difficult, all while adhering to strict constraints like test length, item type distribution (e.g., multiple choice vs. essay), and limited item overlap. Traditional methods designed for simpler models do not translate well to the complex, attribute-based nature of CDMs. This paper introduces a specialized algorithm, S-MOPSO/D, to automate this process.
Methodology: The S-MOPSO/D Approach
The authors propose a Set-based Multiobjective Particle Swarm Optimizer based on Decomposition (S-MOPSO/D). This approach treats test assembly as a bi-objective optimization problem, simultaneously targeting Average Diagnostic Quality (accuracy) and Quality Variance (equivalence across tests).
Key components of the S-MOPSO/D framework include:
- Decomposition (/D): The complex bi-objective problem is broken into a series of simpler "scalar" subproblems, allowing the particles (candidate solutions) to search the trade-off space more effectively.
- Assignment-Based Representation: Unlike standard PSO which uses continuous numbers, this version uses discrete sets of "item-to-test" assignments.
- Constructive Learning: Particles build new solutions by "learning" successful assignment patterns from their own history and their neighbors' best results.
- Heuristic Repair Strategy: If a generated test violates a constraint, a smart repair function uses "utility ratios" to swap items, fixing the test while simultaneously improving its quality.
Experimental Evaluation
The researchers evaluated S-MOPSO/D using a synthetic item bank designed to mimic realistic conditions, consisting of 300 items measuring 5 distinct attributes. The bank included multiple choice (MC), blank-filling (BF), and short essay (SE) items. The reduced reparameterized unified model (r-RUM) served as the underlying psychometric framework.
S-MOPSO/D was compared against a random baseline (RND), a problem-specific heuristic (p-CDI), and a state-of-the-art evolutionary algorithm (NSGA-II).
Key Results and Insights
- Pareto Dominance: S-MOPSO/D consistently found a better "Pareto front," identifying solutions that were both higher quality and more balanced than competing methods.
- Lower Error Rates: In simulations with 20,000 examinees, tests assembled by S-MOPSO/D resulted in significantly lower Pattern-Level ($E_1$) and Attribute-Level ($E_2$) Diagnostic Errors.
- Simultaneous Assembly Advantage: Most existing methods build tests sequentially. This paper demonstrates that building them all at once (simultaneously) prevents the "leftover item" problem, where the final test assembled is significantly weaker than the first.
- Strategic Overlap: While educators often prefer zero overlap, the authors argue that allowing a small, controlled percentage of shared items significantly boosts the quality of parallel tests, especially when the item bank is small.
Conclusion
S-MOPSO/D proves to be a robust and effective tool for the automated assembly of parallel cognitive diagnosis tests. By treating test assembly as a pattern-learning problem rather than a simple selection task, it achieves superior balance and accuracy. Future work will focus on improving computational efficiency through parallel implementation and incorporating more complex constraints, such as item exposure control to ensure no single question is overused across test sessions.
Summary updated