TL;DR
This paper investigates how the number and type of examples used in security training affect problem-solving abilities. While the study found no statistically significant differences in performance based on varying example types or quantities, it highlights the challenges in designing effective security training and suggests future research directions to improve statistical power and generalizability.
What Problem Does This Paper Address?
Information system developers and administrators often struggle to apply security best practices due to a lack of context-specific knowledge. Evaluating the effectiveness of security tools and training methods is crucial, but the way examples are presented in training stimuli can significantly influence learning outcomes. This paper tackles the challenge of understanding how the number and type of examples used in security training impact a practitioner's ability to solve security-related problems, drawing on theories of analogical transfer from psychology.
Methods
The researchers designed an experiment based on the theory of analogical transfer, which posits that people solve new problems by applying knowledge gained from past examples. They conducted a human-subjects study using an online survey with two groups: a "3-Example group" and a "2-Example group." Participants were recruited from universities and had to have completed at least one computer security course or possess multiple years of industry experience.
The experiment consisted of two phases:
- Training Phase: Participants were shown "worked-out examples" of secure coding problems. The independent variables manipulated here were the number of examples (2 vs. 3) and the types of examples, specifically varying "schemas" (the underlying problem-solving principle) and "surfaces" (the superficial features of the problem).
- Test Phase: Participants were given three test problems to solve, assessing their ability to apply the learned concepts.
Performance was measured by counting the number of "intended" (correct) answers. Statistical analysis primarily used Fisher's Exact test due to the categorical nature of the data.
Datasets
The study did not introduce new benchmark datasets. Instead, it created its own training and test problems within the domain of secure coding. These problems were categorized by schema (error handling, input validation, integer security) and surface (source code in C, Java, PHP, or natural language text).
Key Insights and Ideas
- Analogical Transfer Theory as a Framework: The study grounds its hypotheses in analogical transfer theory, suggesting that understanding how people map past experiences (base analogues) to new problems (target analogues) is key to effective training.
- Surface vs. Schema Importance: The research explores whether focusing on varied "surfaces" (different ways a problem looks) or varied "schemas" (different underlying principles) is more effective for learning.
- Challenges in Measuring Security Training Effectiveness: The paper highlights the difficulty in isolating the impact of training examples from other factors like individual differences, motivation, and cognitive load.
- Potential for "Out-of-the-Box" Thinking: The observation that participants sometimes provided "extended-in-context" answers (technically correct but not the intended solution) suggests a form of creative problem-solving that warrants further investigation.
- Textual Representations May Increase Ambiguity: The study hints that using natural language text for problem descriptions might lead to more unintended answers, possibly due to increased ambiguity or the opportunity for inferring external information.
Key Results
Despite the theoretical underpinnings suggesting that more examples or varied examples might improve performance, the study found no statistically significant differences in participant performance between the 3-Example and 2-Example groups for any of the test problems.
- Hypothesis H1 (More Surfaces per Schema): Participants trained on more surfaces for a single schema did not perform significantly better than those trained on fewer surfaces.
- Hypothesis H2 (More Schemas): Participants trained on more schemas did not perform significantly better than those trained on fewer schemas.
- Surface Representation: Performance on a test problem using natural language text was lower than on a problem using source code, though this difference was not statistically significant.
- Participant Dropout: There was no significant difference in dropout rates between the groups, but participants tended to drop out more during the training phase.
- Overall Performance: No significant difference was found in the overall number of test problems participants solved correctly between the two groups.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Low Statistical Power: The primary limitation identified is the low statistical power of the experiment. The sample size was insufficient to detect statistically significant differences, even if they existed. Future work should aim for much larger sample sizes (hundreds or thousands of participants) to achieve adequate power.
- Generalizability: The study's participants were primarily students. To improve generalizability, future research should recruit security professionals.
- Environmental Factors: The use of an online survey means environmental factors (e.g., distractions, screen size) could not be controlled, potentially increasing variability. Future studies could explore laboratory settings or account for these factors.
- Factorial Design: To better isolate the effects of schema and surface, a factorial design with multiple levels for each independent variable is recommended.
- Investigating "Extended-in-Context" Answers: The authors suggest further research into why participants provide these "out-of-the-box" answers, as it might indicate a valuable form of creative problem-solving.
- Fatigue and Order Effects: The order of test problems was not randomized, which could have introduced fatigue effects. Future studies should randomize test problem order.
- Controlling for Experience: While participant experience was collected, its impact on performance was not a primary focus of the statistical analysis. Future work could explore this more deeply.
Summary updated