TL;DR
Evaluating engineering creativity is notoriously subjective, but new eye-tracking research reveals that visual sketches act as a critical "anchor" for human judgment. By comparing how novice raters process text vs. sketches, researchers found that visual elements significantly boost agreement between raters and lead to more focused, convergent attention patterns.
What Problem Does This Paper Address?
In engineering, the "gold standard" for measuring creativity is the Consensual Assessment Technique (CAT). It’s based on a simple premise: a design is creative if a panel of independent experts says it is. However, CAT has two major blind spots. First, it only captures the final score (the "what"), ignoring the underlying cognitive process (the "how"). Second, most studies look at ideas in a single format—just a sketch or just a description.
In the real world, engineering ideas are messy and multimodal, ranging from scribbled notes to CAD models. This paper investigates how the modality of an idea (text-only, sketch-only, or a combination) affects both the reliability of human ratings and the physical way our eyes scan the information.
Methods
The researchers conducted an empirical study with 13 novice raters (graduate engineering students). They used a set of 10 conceptual design ideas for a milk frother, presented in three formats:
- Text-only: A short written description.
- Sketch-only: A hand-drawn diagram.
- Text + Sketch: The original combination of both.
To capture the "implicit" side of evaluation, the team used a Tobii Pro Fusion eye tracker. They focused on two primary metrics:
- Intraclass Correlation Coefficient (ICC2): A statistical measure of how much the raters agreed with each other (interrater reliability).
- Normalized Mean Pairwise Distance (NMPD): A metric developed to measure "visual convergence"—essentially, how close the raters' longest eye fixations were to one another on the screen.
Datasets
The study utilized a publicly available dataset of conceptual design ideas for milk frothers originally created by Starkey et al. (2016). This dataset is a staple in design research because it contains a wide variety of innovative (and sometimes wacky) solutions to a simple engineering problem.
Key Insights and Ideas
- The Visual Anchor: When presented with both text and a sketch, raters spent 94% of their longest fixations on the sketch. Even when text was available, the visual representation dominated the cognitive effort.
- Text is a Secondary Source: The researchers found that text often serves as a "secondary" source of information. In some cases, raters didn't even look at the text description when evaluating usefulness or novelty if a sketch was present.
- Agreement Requires a Picture: Reliability for text-only descriptions was consistently poor. Without a visual cue, raters' interpretations of the "creativity" of an idea diverged wildly.
- Attention $\neq$ Agreement: Interestingly, one group of raters (Group C) showed high visual convergence (they all looked at the same spot) but still had poor rating agreement. This suggests that while a sketch helps people focus on the same feature, they may still interpret the value of that feature differently.
Key Results
- Interrater Reliability (IRR): Sketches significantly improved agreement. For "Creativity" ratings, the average ICC2 for Text-only was a dismal 0.07, while Sketch-only reached 0.46.
- Visual Convergence: Raters were much more "in sync" when looking at multimodal designs. The NMPD (where lower is better) for Text + Sketch was 28.95, compared to 82.93 for Text-only.
- Modality Matters for Novelty: The highest agreement for novelty was found in the Sketch-only and Text + Sketch modalities, suggesting that visual "geometric and topological information" is vital for recognizing if an idea is truly new.
Limitations and Future Work
The study was limited by a relatively small sample size of 13 novice raters and focused on a 2D screen environment. The authors note that real engineering involves 3D prototypes where senses like touch and hearing come into play.
Future work aims to:
- Compare Novices vs. Experts: Do experts rely on text more than novices, or is the "visual bias" universal?
- AI Integration: Use these eye-gaze patterns to train AI models to better "understand" and evaluate engineering designs the way humans do.
- Complex Modalities: Expand the research to include CAD models and physical prototypes to see if the "sketch-only" advantage holds up as designs become more high-fidelity.
Summary updated