Introduction
In social media analysis, categorizing users—whether for political stance detection or identifying automated bots—typically relies on two primary signals: the content of their posts ("what they say") and their interaction network ("what they do"). However, these two modalities often provide contradictory information. For example, a user might tweet about progressive topics like "clean energy" while exclusively following and retweeting conservative politicians.
Traditional Graph Neural Networks (GNNs) often "blindly" fuse these modalities, which can lead to biased or incorrect predictions if one source is misleading. Furthermore, most models act as "black boxes," failing to explain why they prioritized one data source over another for a specific individual.
The CAMUE Framework
To address these challenges, researchers developed CAMUE (Contribution-Aware Multimodal User Embedding). This framework is designed for "explainable fusion" by utilizing a learnable attention gate that dynamically adjusts the influence of text and graph data for every user.
The architecture consists of three main components:
- Text Encoder: Uses models like BERT or GloVe to transform tweets into vector representations.
- Graph Encoder: Employs a Relational Graph Convolutional Network (R-GCN) to capture the nuances of different interaction types, such as follows, mentions, and retweets.
- Attention Gate Learner: This core innovation takes outputs from both encoders and learns two weights: $\alpha$ (for the graph) and $\beta$ (for the text).
Unlike standard concatenation methods, CAMUE uses these weights to prioritize the more "trustworthy" modality. If a user’s text is nonsensical or neutral, the model learns to down-weight $\beta$ and rely more heavily on the graph structure ($\alpha$).
Key Insights and Results
The model was evaluated on two primary Twitter-based datasets: TIMME (political ideology prediction) and TwiBot-20-Sub (bot detection).
"Doing" Usually Beats "Saying"
For the majority of users—especially bots and politicians—the graph structure proved to be a more reliable indicator than text. The authors found that $\alpha > \beta$ for roughly 70-77% of users across different categories. For instance, in the case of Elon Musk (using pre-2020 data), the graph was a much stronger indicator of his political leanings than his varied tweet content.
Performance and Robustness
CAMUE consistently outperformed single-modality baselines and simple fusion methods:
- Political Prediction (TIMME): Achieved an accuracy of 0.961 and an F1-score of 0.960.
- Bot Detection: Reached an accuracy of 0.782, significantly higher than using text (0.731) or graph (0.735) alone.
- Robustness to Noise: When paired with a weak text encoder (GloVe), CAMUE maintained high accuracy (0.954) by learning to ignore the poor text data, whereas standard fusion models dropped to 0.840.
Limitations and Future Work
While CAMUE provides personalized explanations via its "contribution map," it currently treats user behavior as static. Future work could incorporate temporal dynamics to track how a user’s reliability shifts over time. Additionally, integrating other modalities like images and videos could further enhance the model's predictive power and explainability.
Summary updated