Abstract
Assemble the panels of computerized adaptive multistage testing by the bottom-up and the top-down approach, and simulate the administration of the assembled panels. The full documentation and tutorials are at <a href="https://github.com/xluo11/Rmst" target="_top">https://github.com/xluo11/Rmst</a>. Reference: Luo and Kim (2018) <a href="https://doi.org/10.1111%2Fjedm.12174" target="_top">doi:10.1111/jedm.12174</a>.
Introduction
In educational and psychological measurement, Computerized Adaptive Testing (CAT) represents the gold standard for efficiency. However, its implementation is often hindered by complex item-selection algorithms and the difficulty of performing human quality control on unique test forms. Multistage Testing (MST) provides a robust alternative by grouping items into modules. This structure allows for human review of test forms while maintaining the ability to adapt difficulty to an examinee’s performance between stages.
The Rmst package addresses the significant combinatorial challenge of assembling MST panels. It ensures that every potential route through the test adheres to specific statistical targets, such as Test Information Functions (TIF), and non-statistical constraints like content balance.
Methods and Implementation
The Rmst package utilizes Mixed Integer Programming (MIP) to automate the assembly process. By converting test specifications into linear constraints and objectives, it leverages external solvers like lpsolve and glpk to identify optimal item selections from a pool.
Assembly Strategies
The framework supports two primary assembly strategies:
- Top-down Approach: Objectives and constraints are applied to entire routes (sequences of modules). This ensures the overall test experience is optimized for the examinee.
- Bottom-up Approach: Objectives and constraints are applied to individual modules, offering granular control over specific item sets.
The package supports standard Item Response Theory (IRT) models, including the 3-Parameter Logistic (3PL) model for dichotomous items, and the Generalized Partial Credit Model (GPCM) and Graded Response Model (GRM) for polytomous items.
Key Features and Simulation
A standout feature of Rmst is its flexible route management. Users can explicitly add or remove specific routes using the mst_route function, which is critical for preventing jarring jumps in difficulty (e.g., from the easiest Stage 1 module to the hardest Stage 2 module).
The mst_constraint function is equally versatile, handling both categorical attributes (e.g., "Content Area") and continuous attributes (e.g., "Word Count"). This makes the package highly adaptable to diverse test blueprints. Furthermore, the mst_sim engine allows researchers to run "virtual examinees" through assembled panels using Expected A Posteriori (EAP) estimation to evaluate performance and routing logic.
Results and Applications
The documentation demonstrates the package's utility through complex MST designs, such as:
- 1-2-2 Design: A three-stage test branching into two difficulty levels. The example demonstrates maximizing information at specific $\theta$ levels for "easy" versus "hard" routes.
- 1-2-3 Design: A structure where specific routes are pruned to control test flow.
Simulation outputs provide detailed logs comparing true versus estimated ability, administered items, and routing decisions, offering researchers deep insights into the reliability and validity of their test designs.
Conclusion
While Rmst provides a powerful toolset for MST assembly, its efficiency is currently tied to the performance of external MIP solvers. Future developments might include heuristic assembly methods, such as the Big-Shadow algorithm, to handle extremely large item pools or highly complex constraint sets where exact MIP solutions may be computationally intensive.
Summary updated