Overview
While deep learning has proven immensely powerful in data-rich environments, many real-world applications suffer from limited training data. To overcome this, practitioners often turn to multi-task learning (MTL) or pre-training, leveraging auxiliary tasks to boost performance on a primary task. However, this approach is fraught with challenges, most notably the risk of "negative transfer," where auxiliary updates actively interfere with the primary task.
This paper introduces ATTITTUD (Auxiliary Task Training with Influence from Target Task Update Direction), a novel, model-agnostic framework that decomposes auxiliary task gradients into three distinct components based on their impact on the primary task: those that help, those that harm, and those that are neutral. By selectively re-weighting these components, ATTITTUD provides a scalable and effective way to harness auxiliary data without compromising the primary objective.
The Challenge of Negative Transfer
In data-starved scenarios, borrowing statistical regularities from auxiliary tasks is a common strategy. Yet, treating all tasks symmetrically or relying on coarse metrics—like the cosine similarity between average gradients—often fails to prevent negative transfer. Auxiliary tasks do not always align perfectly with the primary task; their gradients can sometimes push the model parameters in detrimental directions. There is a critical need for a fine-grained, primary-task-centric approach that filters and manipulates auxiliary updates to ensure they are strictly beneficial or, at worst, harmless.
Methodology: Decomposing the Gradient
The core innovation of ATTITTUD lies in its geometric decomposition of the auxiliary gradient ($\mathbf{g}_{aux}$) relative to the primary task's gradient subspace. The framework splits the auxiliary update into three orthogonal directions:
- The Good ($\mathbf{g}^+_{aux}$): Components that align with the primary task's gradient subspace, actively helping to minimize the primary loss.
- The Bad ($\mathbf{g}^-_{aux}$): Components that directly oppose the primary task's gradient, causing interference and negative transfer.
- The Neutral ($\mathbf{g}^\perp_{aux}$): Components orthogonal to the primary task's gradient subspace, having no immediate first-order impact on the primary loss.
The ATTITTUD Algorithm
To make this decomposition computationally feasible for large-scale neural networks, ATTITTUD employs several advanced techniques:
- Subspace Projection: Rather than relying on a single average primary gradient, the method constructs a subspace spanned by per-example primary gradients (the Jacobian). This captures a richer neighborhood of helpful directions.
- Randomized SVD: Computing the full Jacobian is prohibitively expensive. ATTITTUD utilizes randomized Singular Value Decomposition (SVD) and efficient Jacobian-vector products to approximate the top-$k$ singular vectors of the primary task's Jacobian.
- Flexible Weighting: The framework introduces hyperparameters ($\eta$) to independently scale the decomposed components. For instance, practitioners can invert the "Bad" gradients ($\eta_- = -1$) to make them constructive, or retain the "Neutral" components to encourage general feature learning without immediate harm.
Experimental Setup
The authors rigorously evaluated ATTITTUD across diverse domains to demonstrate its versatility:
- Text Classification: Sentiment analysis using Amazon Helpfulness and IMDb Movie Review datasets, focusing on out-of-distribution auxiliary data.
- Image Classification: Tasks included MultiCifar100 (treating super-classes as primary) and a severely data-starved "Cat-vs-Dog" subset of CIFAR-10 (only 50 examples per class).
- Medical Imaging: Transfer learning from ImageNet (auxiliary) to the ChexPert dataset (primary, 5k examples).
Key Findings and Results
ATTITTUD consistently outperformed strong baselines, including vanilla pre-training, standard MTL, and PCGrad:
- Selfish Optimization: Unlike standard MTL, which seeks a Pareto optimal balance across all tasks, ATTITTUD is intentionally asymmetric, prioritizing the primary task above all else.
- The Value of Neutral Updates: Surprisingly, retaining "neutral" components ($\mathbf{g}^\perp_{aux}$) often proved beneficial. While they do not immediately reduce the primary loss, they help the model learn generalized features that improve overall robustness.
- Performance Gains:
- In text classification (Amazon/IMDb), ATTITTUD achieved an F1 score of 68.5, surpassing TAPT's 67.8.
- On the low-resource Cat-vs-Dog task, it reached 67.1% accuracy, significantly beating PCGrad (64.2%) and vanilla pre-training (64.5%).
- In medical imaging (ChexPert), the method achieved an 83.3% AUC, outperforming standard ImageNet pre-training (81.4%).
Limitations and Future Directions
While highly effective, ATTITTUD introduces new hyperparameters, specifically the scaling factors ($\eta$) and the subspace rank ($k$).
As illustrated above, the optimal size of the subspace ($k$) varies depending on the data regime. In low-resource settings, a smaller $k$ is preferable to prevent overfitting, whereas high-resource settings can leverage a larger $k$ for more nuanced gradient manipulation. Future work will likely focus on automating the selection of these hyperparameters and further exploring the theoretical underpinnings of using singular vectors for variance reduction in gradient descent.
Summary updated