TL;DR
This paper introduces a novel approach to analyze survey data by combining powerful NLP models (like BERT) with explainability techniques. This allows for a deeper understanding of how students make career choices, by jointly analyzing both free-text answers and structured survey responses, and revealing the underlying factors influencing these decisions.
What Problem Does This Paper Address?
Traditional methods for analyzing survey data often rely on shallow machine learning techniques or manual human judgment, which can be time-consuming, biased, and fail to capture the full richness of the data. Surveys frequently contain a mix of closed-ended (e.g., multiple-choice) and open-ended (free-text) questions. While closed-ended questions are good for statistical analysis, open-ended answers provide valuable qualitative insights. The challenge is to effectively integrate and analyze both types of data to extract meaningful knowledge and understand complex relationships, such as the factors influencing students' career aspirations. Existing NLP models, while powerful, are often "black boxes," making it difficult to understand why they make certain predictions.
Methods
The authors propose a multi-modal architecture that leverages a DistilBERT transformer for processing open-ended text answers. This model is used to extract contextual information and generate embeddings. For the closed-ended, numerical survey responses, Fully Connected (FC) layers are employed. These two streams of information are then concatenated and fed into further FC layers to make predictions.
To address the "black box" nature of these deep learning models, the paper utilizes post-hoc explainability techniques:
- SHAP (SHapley Additive exPlanations): This method is used to understand the importance of individual input features (both text and numerical) for specific predictions (instance-level) and for the model's overall behavior (global-level). They analyze SHAP values at different layers of the model to gain a holistic view.
- ConceptSHAP: This technique helps to identify higher-level, human-interpretable concepts that the model learns from the data. It extracts clusters of features that share similar activation patterns and summarizes them with concept vectors, allowing researchers to understand what abstract ideas the model is using for its predictions.
Datasets
The study uses the EMS 1.0 dataset, a longitudinal survey of engineering students' career goals. This dataset comprises data from 7,197 students across 27 US universities, collected in 2015. It includes a mix of closed-ended questions (categorized into 8 topics based on the Social Cognitive Career Theory framework) and two open-ended questions (Q22 about future plans and "Inspire" about the survey's influence).
Key Insights and Ideas
- Synergy of Text and Numerical Data: Combining both open-ended text responses and structured numerical survey data leads to improved prediction performance compared to using either data type alone.
- BERT's Power for Open-Ended Questions: DistilBERT effectively extracts contextual information from free-text answers, revealing factors that traditional methods might miss.
- Explainability is Crucial: Post-hoc explainability methods like SHAP and ConceptSHAP are essential for understanding why the model makes its predictions, bridging the gap between model performance and human interpretability.
- Identifying Key Career Influencers: The model, with the help of explainability, can pinpoint specific factors (both from text and numerical responses) that significantly influence students' career choices, such as entrepreneurial behavior.
- Concept-Level Understanding: ConceptSHAP reveals that the model learns abstract concepts related to career planning, such as "clarity of plans," "career characteristics," and "plan timeline," which align with human intuition and previous qualitative research.
Key Results
- The combined model achieved a macro F1 score of 73.64% (classification) and 76.23% (regression) when using Q22 text input and the full set of SCCT topics.
- Numerical variables alone were generally more predictive, but the text data provided complementary insights.
- The "Inspire" text variable, surprisingly, had a negative impact on performance, likely due to its brevity.
- SHAP analysis showed that numerical features were generally more important than text embeddings for the model's predictions, with specific entrepreneurial-related features being highly influential.
- ConceptSHAP identified four key concepts, including "clarity of plans," "career characteristics," and "plan timeline," with the latter being a novel finding not previously identified in similar research.
Limitations and Future Work
- "Inspire" Variable Performance: The negative impact of the "Inspire" question on model performance suggests that very short free-text answers might not contribute positively to prediction tasks.
- Dataset Specificity: While the EMS dataset is rich, future work could explore applying this methodology to other survey domains to assess its generalizability.
- Further Exploration of Concepts: The authors suggest further investigation into the extracted concepts and their relationships to human judgment.
- Code Release: The authors have released their code to facilitate further research in this area.
Summary updated