Introduction
Vision-Language (V+L) tasks, such as image captioning and visual question answering, are currently dominated by massive, multi-layered Transformer architectures. While powerful, models like Oscar face significant challenges: they are computationally expensive, behave as "black boxes," and often underutilize the very data intended to bridge the gap between modalities. This research focuses on optimizing the Oscar architecture to be leaner, more accurate, and more interpretable through knowledge distillation and innovative feature re-engineering.
The Oscar Architecture and the Role of Object Tags
The core innovation of the Oscar model lies in its use of object tags as "anchors" to align visual features with textual tokens. As shown in Figure 1, the input data consists of three components: word tokens, object tags (detected via Faster R-CNN), and region features.
Despite this design, the authors observed that models often ignore high-confidence tags during caption generation. To address this, the researchers explored three primary optimization strategies: knowledge distillation, feature re-engineering, and a copy mechanism.
Methodology
1. Knowledge Distillation for Efficiency
To reduce the computational burden, the 12-layer Oscar model was compressed into a 6-layer "student" version. This process utilized a multi-faceted loss function—combining Cross-Entropy, Mean Squared Error (MSE) on logits, and Cosine Embedding on hidden states—to ensure the student captured the teacher's nuances. A key finding was the need to carefully balance these losses, as MSE can easily dominate and hinder the generation of coherent text.
2. Feature Re-engineering and Tag Confidence
The researchers hypothesized that incorporating the confidence scores of object detectors would improve performance. They experimented with reordering tags by confidence and embedding confidence values directly into the input. While reordering helped training convergence, direct embedding proved more complex to optimize within the existing embedding space.
3. Copy Mechanism and Interpretability
A "copy mechanism" was implemented to allow the model to directly select an object tag for inclusion in a caption. This bypasses the need to predict the word from a fixed vocabulary, which is particularly useful for handling synonyms or ensuring salient objects are not overlooked. Furthermore, by plotting attention weights, the authors could visualize which tokens the model prioritized, helping diagnose cases where the model ignored explicit tags in favor of generic context.
Experimental Results and Ablation Studies
The study primarily utilized the MS-COCO dataset, following the standard Karpathy split for benchmarking. The results demonstrated that the copy mechanism improved the CIDEr score to 124.1, surpassing the baseline Oscar's 123.7.
As illustrated in Figure 2, ablation studies revealed critical insights into the model's dependencies. Removing image features (finetune_noImgFeat) resulted in a drastic performance drop compared to removing object tags (finetune_noObjectTag). This confirms that while tags are valuable anchors, raw visual region features remain the primary driver of performance. Meanwhile, the distilled 6-layer model reached a CIDEr score of 80.4, representing a significant step toward making these models runnable on more modest hardware.
Conclusion and Future Directions
This work demonstrates that V+L models can be made more efficient and effective through targeted architectural refinements. While the distilled model does not yet match the full teacher, it provides a path toward accessibility. Future research will explore dynamic pointer networks for more sophisticated tag selection and the integration of spatial grounding to better align visual coordinates with text descriptions.
Summary updated