TL;DR
This paper introduces "Lifelong Scene Graph Generation" (LSGG), a new task that addresses the challenge of continuously learning to generate scene graphs without forgetting previously acquired knowledge. They propose a novel in-context prompt learning approach that leverages pretrained language models and a knowledge-aware prompt retrieval mechanism to achieve this.
What Problem Does This Paper Address?
Scene graph generation (SGG) aims to represent visual scenes as structured graphs of objects and their relationships. Traditional SGG models are trained in a "closed-set" manner, meaning they are trained on a fixed set of object and predicate categories. This approach faces significant limitations in real-world applications where new categories (especially predicates) can emerge over time.
The core problem is catastrophic forgetting: when a model is trained on new data, it tends to forget what it learned from previous data. This is particularly challenging in SGG because:
- Long-tailed predicate distribution: Some predicates are very rare, making few-shot learning difficult.
- Streaming data: New data arrives sequentially, and models need to adapt without access to all past data.
Existing methods struggle with this, either requiring complete retraining on all data or suffering from forgetting. This paper proposes a new task, Lifelong Scene Graph Generation (LSGG), to tackle this problem head-on.
Methods
The authors propose a framework for LSGG with three main components:
- Visual Features to Symbolic Representations: They use a cross-modality model (like GLIP) to extract visual features (context, relationship, subject, object). These features are then transformed into textual tokens using a transformer-based encoder (VL2T). This converts visual information into a format understandable by language models.
- Knowledge-Aware Prompt Learning: Instead of a single prompt, they learn multiple "knowledge-aware prompts." Each prompt is associated with a "knowledge word" and a set of "exemplars" (past training samples). This allows for more specialized prompts that capture different types of knowledge.
- In-Context Exemplar Selection: When a new task arrives, they use a knowledge-based retrieval strategy to find the most relevant prompts and their associated exemplars. These are then combined with the current input to form an "in-context prompt" for a pretrained language model (like GPT-2). This leverages the power of in-context learning to guide the model's predictions without requiring new parameter training.
The overall loss function combines three terms: a prompt learning loss, a knowledge word learning loss, and a predicate classification loss.
Datasets
- Visual Genome (VG): The primary dataset used for training and evaluation. It's a standard benchmark for SGG.
- Open Images (v6): Used for additional evaluation to demonstrate the model's performance on a different dataset.
The authors also define a new task setting, Lifelong Scene Graph Generation (LSGG), where the predicate categories are presented in a streaming fashion across multiple tasks. They partition the VG dataset's predicates into five sequential tasks to emulate this lifelong learning scenario.
Key Insights and Ideas
- Lifelong Scene Graph Generation (LSGG) as a distinct task: The paper formalizes LSGG, highlighting the critical need for models to learn incrementally without forgetting.
- Leveraging Pretrained Language Models (PLMs) for SGG: The core idea is to use PLMs (like GPT-2) as powerful knowledge bases for SGG, transforming visual features into textual representations that PLMs can process.
- Knowledge-Aware Prompts: Moving beyond a single prompt, learning multiple specialized prompts that capture different types of knowledge is crucial for effective lifelong learning in SGG.
- In-Context Learning for Rehearsal: Using a few selected past examples as "in-context" demonstrations within the prompt serves as an effective rehearsal mechanism to combat forgetting.
- Visual-to-Textual Transformation is Key: Converting visual features into symbolic textual representations is essential for bridging the gap between vision and language models in this context.
Key Results
- Superior Performance in LSGG: The proposed ICSGG model significantly outperforms state-of-the-art SGG models on the new LSGG task across various metrics (mR@K, R@K, M@K) on both VG and Open Images datasets. For example, on the PREDCLS task, ICSGG exceeded the second-best model VS³ by 2.8 on average mR@50/100.
- Reduced Forgetting: The model demonstrates significantly lower Forgetting Measure (FM) values compared to conventional SGG models and even recent methods like VS³, indicating its effectiveness in retaining knowledge.
- Robustness to Data Starvation: The in-context prompt learning strategy shows strong few-shot learning capabilities, performing well even with limited examples from new tasks.
- Ablation Studies Confirm Component Effectiveness: Experiments show that each component of the proposed framework (knowledge-aware prompts, in-context exemplar selection, buffer size) contributes positively to the overall performance. Removing the in-context learning strategy led to a significant performance drop.
- Qualitative Improvements: Visualizations show that the model can predict novel predicates and generate more informative scene graphs, even when the textual representations are not perfectly grammatical sentences.
Limitations and Future Work
- Object Detection Network is Frozen: The current framework uses a pretrained object detection network (GLIP) whose parameters are frozen. The authors suggest that jointly training the object detection network in a one-step framework could be a future direction.
- Separate Training of Components: While the VL2T transformer and prompt tokens are learned, the language model (GPT-2) is used off-the-shelf. Future work could explore fine-tuning the language model, although their experiments suggest this might not always be beneficial due to potential catastrophic forgetting in the PLM itself.
- Focus on Predicates: The current work focuses on lifelong learning for novel predicates, assuming object labels are predetermined. Extending this to novel object categories would be a natural next step.
Summary updated