TL;DR
This paper introduces Aux-LSTM, a novel neural network architecture for user attribute classification that leverages the relationships between related attributes to improve performance. By treating one attribute classification as a main task and another as an auxiliary task, Aux-LSTM learns a shared representation that boosts accuracy, particularly when attributes are correlated.
What Problem Does This Paper Address?
User attribute classification, such as predicting gender, age, or profession from social media content, has been a focus of research. However, existing methods typically tackle each attribute in isolation, ignoring the fact that these attributes are often interconnected. For instance, a user's age might strongly suggest their profession (e.g., a 19-year-old is likely a student). This paper addresses this gap by proposing a method for "joint learning" where the classification of one user attribute can actively help improve the classification of another related attribute.
Methods
The core of the proposed solution is the Aux-LSTM model. This architecture is built upon Long Short-Term Memory (LSTM) networks, which are well-suited for processing sequential data like text. The Aux-LSTM model works by:
- Shared Auxiliary LSTM Layer: A key innovation is the use of an auxiliary LSTM layer that is shared between the main task (the attribute to be predicted) and an auxiliary task (a related attribute). This shared layer learns a representation that captures commonalities between the tasks.
- Auxiliary Representation Learning: The auxiliary LSTM layer processes the input text for both tasks. For the auxiliary task, its output is used to generate an "auxiliary representation."
- Joint Representation: For the main task, the model combines the representation from its own dedicated LSTM layer with the learned auxiliary representation. This combined representation is then fed into a fully-connected layer and a dropout layer.
- Joint Learning Objective: The model is trained using a combined cost function that is a weighted sum of the losses from both the main and auxiliary tasks. This encourages the model to optimize for both simultaneously.
The paper also compares Aux-LSTM against two baseline joint learning approaches: Meta-Learning (a two-stage SVM approach) and Multi-task LSTM (a DNN-based multi-task learning framework with shared word representations).
Datasets
The authors collected their dataset from Tencent Micro-blog, a popular Chinese social networking service. They crawled user information (name, profession, age, gender) and posted messages. To ensure data quality, they removed users who posted fewer than 50 messages. The final dataset comprised 25,000 users.
- Gender Classification: Used users with publicly available gender information.
- Profession Classification: Used users with available profession information, focusing on the top seven most frequent categories.
- Age Classification: Used users whose ages ranged from 18 to 28, resulting in 11 age categories.
The data was split into an 80% training set and a 20% test set, with an additional 10% of the training data set aside for validation. Word embeddings were pre-trained using word2vec on the entire 25,000-user dataset.
Key Insights and Ideas
- Attribute Correlation is Crucial: The most significant finding is that joint learning is only beneficial when the user attributes are genuinely related. The paper demonstrates that attempting to jointly learn gender and age classification yielded no improvement, and sometimes even a slight performance drop. This suggests that these two attributes might not have a strong inherent correlation in the data.
- Profession is a Good Auxiliary Task: Conversely, the profession attribute showed a positive correlation with both age and gender. Jointly learning profession with either age or gender consistently improved classification performance for the main task. This highlights the power of leveraging related information.
- Aux-LSTM Outperforms Baselines: The proposed Aux-LSTM model consistently outperformed the Meta-Learning and Multi-task LSTM baselines when joint learning was beneficial. This suggests that the specific way Aux-LSTM learns and integrates the auxiliary representation is more effective.
- Deep Learning for User Attributes: The paper reinforces the trend that deep learning approaches (CNN, PCNN, and especially LSTM) are superior to traditional methods like SVM and ME for user attribute classification.
Key Results
- Individual Classification: LSTM achieved the best performance among individual classification methods for gender, age, and profession, outperforming SVM, ME, CNN, and PCNN.
- Joint Learning (Gender as Main Task):
- Joint learning with age was not helpful. Aux-LSTM showed a minimal performance loss (0.869 to 0.867 Macro-F1).
- Joint learning with profession significantly improved performance. Aux-LSTM (Gender+Prof) achieved a Macro-F1 of 0.903, a notable improvement over individual LSTM (0.869).
- Joint Learning (Age as Main Task):
- Joint learning with gender was not helpful. Aux-LSTM showed a slight performance loss (0.280 to 0.279 Macro-F1).
- Joint learning with profession was beneficial. Aux-LSTM (Age+Prof) achieved a Macro-F1 of 0.314, a 3.4% promotion compared to individual LSTM.
- Joint Learning (Profession as Main Task):
- Joint learning with age was beneficial. Aux-LSTM (Prof+Age) achieved a Macro-F1 of 0.373, a 3.4% promotion over individual LSTM.
- Joint learning with gender was also beneficial. Aux-LSTM (Prof+Gender) achieved a Macro-F1 of 0.380, a 4.1% promotion over individual LSTM.
- Significance: Statistical significance tests (p-value < 0.05) confirmed that the improvements achieved by Aux-LSTM over the baselines were significant when joint learning was beneficial.
Limitations and Future Work
- Attribute Relatedness: The paper explicitly notes that the effectiveness of joint learning is highly dependent on the relatedness of the attributes. The study found that gender and age are not related, limiting the benefits of joint learning between them.
- Two Attributes: The current Aux-LSTM model is designed for joint learning of two attributes.
- Future Work: The authors propose extending Aux-LSTM to handle joint learning across three or more attributes. They also aim to incorporate more user attributes, such as location, into their joint learning framework.
Summary updated