Abstract
This article examines the psychometric quality of the tests applied in the External Independent Evaluation (EIE) system introduced in Ukraine. By analyzing data from the 2009 and 2010 testing cycles, the study identifies significant flaws in test design, including high guessing probabilities and low task-to-total correlations, which challenge the objectivity and reliability of standardized university admissions.
The Challenge of Objective Testing
Since 2008, Ukraine has transitioned to the External Independent Evaluation (EIE) as the primary mechanism for university admissions, aiming to foster objectivity and mitigate corruption. However, testing is not a "magic bullet." A test is a model of knowledge, and like any model, it is prone to inaccuracies. The core challenge lies in the gap between the theoretical advantages of standardized testing—fairness and efficiency—and the practical reality where poorly designed questions allow students to pass through guesswork or fail to accurately measure intended learning outcomes.
Psychometric Methodology
To evaluate test quality, the authors utilize several standard psychometric indices:
- Cronbach’s Alpha ($\alpha$): A measure of internal consistency. While a value of 0.8 is typically the threshold for a "good" test, high values can be misleading if the underlying score distribution is stochastic.
- Complexity Index ($p_i$): The ratio of the actual score to the maximum possible score for a specific task.
- Correlation Coefficient ($R_i$): Measures how well an individual task correlates with the total test score. A low $R_i$ suggests a task does not measure the same underlying knowledge as the rest of the exam.
- Discrimination Index ($I_i$): Evaluates a task's ability to differentiate between high-performing and low-performing students.
Data Analysis and Distribution Challenges
The study utilizes official nation-wide statistics from the EIE-2009 and EIE-2010 campaigns, covering thousands of students across disciplines ranging from Mathematics to Foreign Languages. One of the primary findings is that test score distributions often deviate significantly from the expected "normal" bell curve.
As seen in Figure 1, while Geography follows a relatively normal distribution, Mathematics-1 is heavily skewed, and English displays a distinct bimodal peak. These deviations can invalidate standard statistical processing methods if not properly accounted for by test designers.
The "Guessing" Trap and Task Quality
A major flaw identified in the multiple-choice design is the high probability of passing through sheer guesswork. In some subjects, the distribution of answers for specific tasks was nearly uniform, suggesting students were choosing options at random rather than applying knowledge.
As detailed in Figure 3, the probability of guessing the minimum passing score was alarmingly high in 2009, reaching 62% for Physics and 59% for Mathematics. While these figures improved in 2010, they remained substantial. Furthermore, the "correlation crisis" is evident in subjects like History of Ukraine and Geography, where over 75% of tasks had a correlation coefficient $R_i \le 0.3$, implying that the majority of individual questions did not effectively contribute to a coherent measurement of student ability.
Conclusion and Future Directions
While most EIE tests showed high overall reliability scores (Cronbach $\alpha$ between 0.80 and 0.96), these metrics can mask a high fraction of "bad" tasks. The inability to pre-test secret exam questions on representative groups remains a significant hurdle for validity.
To improve the system, the authors propose moving toward more sophisticated task types, such as:
- Numerical answers that allow for partial credit based on proximity to the correct value.
- Dual-answer tasks with strict time limits to minimize guessing.
- Weighted scoring based on the empirical complexity of each task.
Ultimately, the EIE must move beyond "testing for the sake of testing" and embrace rigorous, data-driven psychometric refinement to ensure true educational objectivity.
Summary updated