Goal-Driven Sequential Data Abstraction

Umar Riaz Muhammad, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yi-Zhe Song

Abstract

Traditional data abstraction methods often produce static, "one-size-fits-all" summaries that fail to account for varying user objectives. We introduce Goal-Driven Sequential Abstraction (GDSA), a domain-agnostic framework that leverages reinforcement learning to summarize videos, sketches, and text based on specific, quantifiable goals. By framing abstraction as a sequential decision-making process, GDSA identifies the most informative "Atomic Units" to satisfy a target objective—such as sentiment analysis or category recognition—without the need for expensive human-annotated training labels. Our results demonstrate that GDSA consistently outperforms state-of-the-art baselines across multiple data modalities.

Introduction

Data abstraction aims to distill large datasets into their most essential components. However, the definition of "essential" is inherently subjective and context-dependent. For instance, a security professional and a wildlife biologist would likely prioritize different segments of the same surveillance footage. Current summarization models face two primary challenges: label scarcity, as human-generated "gold standard" summaries are costly to produce, and rigid goals, where models struggle to adapt to changing user requirements.

Figure 1: Overview of the GDSA framework showing how different goals (e.g., category vs. attribute) lead to different abstractions of the same sequential input.

GDSA addresses these hurdles by treating abstraction as a goal-oriented task where an agent learns to select data points that maximize a specific reward, effectively providing a flexible and weakly supervised solution for multi-modal summarization.

Methodology: The GDSA Framework

The GDSA model decomposes input data into Atomic Units (AUs)—such as video segments, pen strokes, or sentences. Unlike traditional models that process data linearly, GDSA views the entire collection of AUs as a pool, allowing the agent to select the most informative units regardless of their original temporal position.

Reinforcement Learning and Goal Rewards

The core of GDSA is a reinforcement learning agent trained to maximize a "Goal Reward." Instead of mimicking human behavior, the agent is rewarded based on how well a pre-trained classifier can perform a task (e.g., object recognition) using only the selected AUs.

Figure 2: The GDSA architecture, featuring an RL agent that processes candidate AUs and state embeddings to select the most informative units for a given goal.

Key Technical Innovations

  • Holistic Processing: By considering all available units simultaneously, the model can identify the "essence" of the data more efficiently.
  • Curriculum Learning Reward: To ensure robust learning, the reward function is competitive. The agent must outperform both random selection and the original human-ordered sequence to receive a positive reward. An annealing parameter gradually increases the difficulty during training.
  • State Embeddings: The model utilizes Recurrent Neural Networks (RNNs) to maintain a representation of the units already chosen, the remaining candidates, and the specific goal category.

Experimental Evaluation

The model was evaluated across three distinct domains to test its versatility:

  1. Sketches (QuickDraw): 70,000 sketches across 9 categories.
  2. Video (TVSum): 50 videos covering diverse categories like "parades" or "cooking."
  3. Text (Amazon Reviews): 1,400 reviews per category across 9 product types.

Key Results

GDSA consistently outperformed state-of-the-art baselines, particularly under tight data "budgets":

  • Sketch Recognition: At a 25% stroke budget, GDSA achieved 71.92% accuracy, surpassing the previous best of 65.05%.
  • Video Summarization: With a 25% budget on the TVSum dataset, GDSA reached 74.0% accuracy, significantly higher than the 64.0% achieved by DQSN.
  • Text Sentiment: For Amazon reviews, GDSA achieved 86.08% sentiment recognition accuracy at a 25% sentence budget, nearly matching the performance of the full text (86.66%).

Discussion and Insights

A significant finding of this research is that goals fundamentally change the summary. When the objective for a video changes from "Parade" to "Humans," the model adaptively shifts its selection from marching bands to spectators. Furthermore, the model's order independence allows it to prioritize distinctive features—like the eyes in a face sketch—even if they were added late in the original sequence.

Because the rewards are derived from pre-trained classifiers, GDSA operates under weak supervision, eliminating the need for full human-made summaries during training.

Limitations and Future Work

While effective, the current model relies on a fixed selection budget. Future iterations could incorporate a confidence-based stopping criterion, allowing the model to determine the optimal summary length dynamically. Additionally, while GDSA selects the most informative units, the resulting sequence may lack narrative "flow" for human consumption; current implementations mitigate this by re-ordering selected units chronologically for the final output. Future research will explore multi-goal optimization to satisfy complex, overlapping objectives.

Summary updated

BibTeX
@inproceedings{muhammad2019goal,
  author = {Umar Riaz Muhammad and Yongxin Yang and Timothy M. Hospedales and Tao Xiang and Yi-Zhe Song},
  title = {{Goal-Driven Sequential Data Abstraction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00016},
}
Figure