Introduction
Multitask learning (MTL) is a cornerstone of efficient computer vision, typically implemented by sharing early feature-extraction layers across multiple tasks. However, standard MTL architectures often suffer from "siloed" learning once the network branches into task-specific heads. This paper introduces Cross-residual Learning (CRL), an extension to the popular ResNet architecture that allows related tasks to share high-level semantic information deep within the network. The resulting X-ResNet is not only 40% smaller than separate specialized networks but also significantly more accurate.
The Challenge of Task Independence
In many visual recognition scenarios, tasks are semantically intertwined. For instance, identifying a "noun" (e.g., dog) and an "adjective" (e.g., happy) are often treated as independent problems. Yet, the visual features of a "happy dog" are a synergistic combination of both concepts.
Traditional MTL models use a shared "trunk" but "fan out" into independent branches. Once branched, these paths stop communicating, preventing the model from leveraging high-level correlations. The authors propose a way to "cross-pollinate" information between these branches while maintaining task-specific specialization.
Methods: The Cross-residual Network (X-ResNet)
The core innovation is the Cross-residual block. In a standard ResNet, a layer learns a residual mapping $y = F(x) + x$. In an X-ResNet, the shortcut connection for a task $t$ incorporates a weighted combination of features from all $N$ related tasks at that depth.
Mathematically, the output for task $t$ is defined as: $$y^{(t)} = F(x, \{W_i^{(t)}\}) + \sum_{j=1}^{N} W_s^{(j)} x$$ Here, $W_s^{(j)}$ are learned weights that act as a soft gating mechanism, determining how much information from task $j$ should flow into task $t$. This structure provides a form of in-network regularization, forcing the model to find representations that are mutually beneficial across the entire problem domain.
Experimental Setup: Visual Sentiment Ontology
The researchers evaluated their approach using the Visual Sentiment Ontology (VSO), focusing on detecting Adjective-Noun Pairs (ANPs) such as misty woods or shiny cars. The dataset consists of over 384,000 social images from Flickr, organized into a hierarchy of 167 nouns, 117 adjectives, and 553 specific ANP combinations. This hierarchy is ideal for testing CRL because the ANP task is a direct semantic composition of the noun and adjective tasks.
The authors compared three primary architectural configurations:
- ResNet-50 (x3): Three completely independent ResNet-50 models.
- $X_0$-ResNet-50: A multitask model with a shared trunk that branches into three independent heads (no cross-residuals).
- X-ResNet-50: The proposed model with cross-residual connections between the task branches.
Key Results and Efficiency
The experiments yielded several critical insights:
- Superior Accuracy: The X-ResNet with learned cross-weights consistently outperformed both the $X_0$ baseline and independent models. On the most challenging task—ANP detection—the cross-residual connections provided a 10.4% relative improvement in Top-1 accuracy.
- Parameter Efficiency: By using a single branched network instead of three separate models, the authors reduced the total parameter count by 40% (from ~72M to ~43M) while improving performance.
- The Power of Learned Weights: Simply using identity connections for cross-talk was found to be less effective than learned scaling layers ($W_s$). This suggests that the network needs to dynamically calibrate the degree of information sharing between specific tasks.
Conclusion
Deep Cross Residual Learning demonstrates that high-level feature sharing is just as important as low-level sharing in multitask environments. By allowing task-specific branches to "talk" to one another through learned residual shortcuts, X-ResNet achieves a superior balance of accuracy and efficiency, proving that joint learning is more powerful than siloed learning for semantically related visual tasks.
Summary updated