Abstract
Vision-Language Pre-Trained (VLP) models, such as CLIP, have demonstrated remarkable effectiveness in learning generic visual representations. Several approaches aim to efficiently adapt VLP models to downstream tasks with limited supervision, aiming to leverage the acquired knowledge from VLP models. However, these methods suffer from either introducing biased representations or requiring high computational complexity, which hinders their effectiveness in fine-tuning the CLIP model. Moreover, when a model is trained on data specific to a particular domain, its ability to generalize to uncharted domains diminishes. In this work, we propose Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period. Specifically, we estimate Gaussian distributions to model visual features of the few-shot support images to capture the knowledge from the support set. The cosine similarity between query image and the feature distribution of support images is used as the prediction of visual adapter. Subsequently, the visual adapter's prediction merges with the original CLIP prediction via a residual connection, resulting in the final prediction. Our extensive experimental results on visual reasoning for human object interaction demonstrate that our proposed TT-DNA outperforms existing state-of-the-art methods by large margins.
Introduction
Adapting large-scale Vision-Language Pre-Trained (VLP) models like CLIP to specialized tasks often requires significant fine-tuning or large datasets. This is particularly challenging for complex visual reasoning tasks, such as distinguishing between a person "eating" an orange versus "holding" or "washing" it. Existing adaptation methods, or "adapters," frequently encounter two primary issues: individual instance bias and high computational complexity. Methods like Tip-Adapter, for instance, compare query images to every individual image in a support set, making them sensitive to outliers. Furthermore, many current techniques introduce numerous parameters, increasing the difficulty of deployment.
To address these challenges, we introduce the Test-Time Distribution Learning Adapter (TT-DNA). TT-DNA moves away from instance-based comparison, instead modeling the "essence" of a category as a Gaussian distribution. This approach aims to capture generic class representations that generalize better to uncharted domains without the need for extensive training.
Methodology
The proposed framework includes two variants: a training-free version (TT-DNA) and a fine-tuned version (TT-DNA-F).
Distribution Modeling and Synthesis
Rather than treating support images as discrete points, TT-DNA estimates a Gaussian distribution (defined by mean and covariance) for the visual features of positive and negative examples. To mitigate the "small sample size" problem common in few-shot learning, the researchers employ DALL-E to generate synthetic images from text prompts. These synthetic images augment the support set, allowing for the estimation of a more robust, "rectified" distribution.
Inference and Fusion
During inference, the visual adapter calculates the cosine similarity between the query image's features and the means of the estimated distributions. This prediction is then fused with CLIP’s standard text-based prediction using a residual connection.
TT-DNA-F: The Fine-tuned Variant
For scenarios where additional performance is required, TT-DNA-F introduces a Query Model, where distribution means act as learnable weights, and a Textual Adapter. The textual adapter is a learnable matrix applied after the encoder, which preserves CLIP’s foundational knowledge while optimizing for task-specific nuances.
Experimental Setup: Bongard-HOI
The primary evaluation benchmark is Bongard-HOI, a dataset specifically designed for visual reasoning in Human-Object Interactions (HOI).
As illustrated in the figure above, a Bongard problem presents six positive examples and six negative examples of a specific interaction. The model must determine if a query image fits the positive concept. This task is significantly more difficult than standard classification as it requires a deep understanding of the relationship between humans and objects.
Results and Discussion
Experimental results demonstrate that modeling classes as distributions provides a more comprehensive and generic representation than instance-based methods.
- Performance: TT-DNA-F achieved a state-of-the-art average accuracy of 70.21% on Bongard-HOI, surpassing the previous best, BDC-Adapter (68.25%).
- Generalization: The model showed consistent performance across unseen actions and objects, indicating that distribution learning effectively captures transferable knowledge.
- Efficiency: Even the training-free TT-DNA outperformed several CLIP-based methods that require training, such as TPT.
Conclusion
TT-DNA offers a robust and efficient path for adapting VLP models to complex reasoning tasks. By leveraging Gaussian distributions and synthetic data augmentation, it overcomes the biases inherent in few-shot instance matching. Future work may explore more complex distribution models beyond Gaussians to capture even more intricate feature spaces.
Summary updated