Small Towers Make Big Differences

Yuyan Wang, Zhe Zhao, Bo Dai, Christopher Fifty, Dong Lin, Lichan Hong, Ed H. ChiarXiv 2020 · arXiv

Abstract

Multi-task learning (MTL) aims to leverage shared representations across multiple objectives. However, it faces a fundamental tension between optimization conflict and generalization decay. While increasing model size helps resolve optimization conflicts, it often leads to overfitting and poor generalization. We present a novel approach using "Under-Parameterized Self-Auxiliaries"—tiny, low-capacity towers that share the same labels as the primary task towers. By requiring the shared backbone to satisfy both high-capacity and low-capacity predictors, the model is regularized toward more robust, universal features. Our results across synthetic and real-world datasets demonstrate that this method effectively shifts the Pareto frontier, particularly for large-scale models, with no added cost during inference.

Introduction

Multi-task learning is a cornerstone of modern machine learning, enabling single models to handle diverse objectives simultaneously. In an ideal scenario, tasks provide mutual regularization, leading to better generalization than single-task counterparts. In practice, however, MTL often suffers from "negative transfer" where tasks compete for capacity in the shared backbone.

To mitigate this, practitioners often resort to over-parameterization. While larger models provide the "room" necessary for conflicting tasks to coexist, they frequently hit a point of diminishing returns. As models grow, they may stop learning shared representations and instead begin memorizing tasks independently, leading to a regression in the Pareto frontier—the set of optimal trade-offs between task performances.

The Under-Parameterized Self-Auxiliary Method

The proposed solution is deceptively simple: for each task, we add a second, significantly smaller "auxiliary" tower. These towers are "self-auxiliaries" because they predict the exact same labels as the main task towers, requiring no additional data.

Architecture of the proposed multi-task learning setup with self-auxiliary towers.

The key is that these auxiliary towers are intentionally under-parameterized—for example, using a single fully-connected layer or a narrow bottleneck. During training, the shared backbone must produce features that are useful for both the high-capacity main tower and the low-capacity auxiliary tower. This acts as a powerful implicit regularizer, forcing the backbone to prioritize robust, generalizable features over task-specific memorization. Crucially, these auxiliary towers are discarded at inference time, ensuring zero additional latency.

Key Insights: Shifting the Pareto Frontier

A central finding of this work is the behavior of the Pareto frontier as model capacity increases. In standard MTL (the "Baseline"), increasing the number of hidden layers initially improves the trade-off between tasks. However, beyond a certain point, the frontier begins to regress toward higher loss values.

Pareto frontiers for different model depths, illustrating how over-parameterization can lead to performance regression and how self-auxiliaries help.

The "Small Towers" approach pushes this frontier back out. By regularizing the learning dynamics, it allows larger models to maintain the optimization benefits of high capacity while preserving the generalization benefits of smaller models. This method is model-agnostic and can be combined with existing gradient-based MTL optimization techniques like PCGrad.

Experimental Results

The researchers evaluated the method on several benchmarks:

  • MultiMNIST & MultiFashionMNIST: On large-scale image classification setups, the method achieved superior accuracy (e.g., 92.80% / 91.03% on MultiMNIST) compared to state-of-the-art baselines.
  • MovieLens 1M: In recommendation tasks, the method significantly reduced error rates for both binary actions ("watch") and numerical ratings.
  • Scaling: The performance gains were most pronounced in larger models, confirming that the method effectively addresses the downsides of over-parameterization.

Conclusion

Adding tiny, under-parameterized auxiliary towers provides a simple yet effective way to balance the multi-task trade-off. By forcing shared layers to support both high and low-capacity predictors, models learn more robust representations that generalize better. This approach offers a practical path for scaling multi-task models without sacrificing performance or increasing inference overhead.

Summary updated

Abstract

Multi-task learning aims at solving multiple machine learning tasks at the same time. A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal. In this paper, we provide some insights on understanding the trade-off between Pareto efficiency and generalization as a result of parameterization in multi-task deep learning models. As a multi-objective optimization problem, enough parameterization is needed for handling task conflicts in a constrained solution space; however, from a multi-task generalization perspective, over-parameterization undermines the benefit of learning a shared representation which helps harder tasks or tasks with limited training examples. A delicate balance between multi-task generalization and multi-objective optimization is therefore needed for finding a better trade-off between efficiency and generalization. To this end, we propose a method of under-parameterized self-auxiliaries for multi-task models to achieve the best of both worlds. It is task-agnostic and works with other multi-task learning algorithms. Empirical results show that small towers of under-parameterized self-auxiliaries can make big differences in improving Pareto efficiency in various multi-task applications.
BibTeX
@misc{wang2020small,
  author = {Yuyan Wang and Zhe Zhao and Bo Dai and Christopher Fifty and Dong Lin and Lichan Hong and Ed H. Chi},
  title = {{Small Towers Make Big Differences}},
  year = {2020},
  eprint = {2008.05808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.05808},
}
Figure