Adjusting Image Attributes of Localized Regions with Low-level Dialogue

Tzu-Hsiang Lin, Alexander I. Rudnicky, Trung Bui, Doo Soon Kim, Jean Oh

Abstract

Natural Language Image Editing (NLIE) aims to use natural language instructions to edit images. Since novices are inexperienced with image editing techniques, their instructions are often ambiguous and contain high-level abstractions that tend to correspond to complex editing steps to accomplish. Motivated by this inexperience aspect, we aim to smooth the learning curve by teaching the novices to edit images using low-level commanding terminologies. Towards this end, we develop a task-oriented dialogue system to investigate low-level instructions for NLIE. Our system grounds language on the level of edit operations, and suggests options for a user to choose from. Though compelled to express in low-level terms, a user evaluation shows that 25% of users found our system easy-to-use, resonating with our motivation. An analysis shows that users generally adapt to utilizing the proposed low-level language interface. In this study, we identify object segmentation as the key factor to user satisfaction. Our work demonstrates the advantages of the low-level, direct language-action mapping approach that can be applied to other problem domains beyond image editing such as audio editing or industrial design.

Introduction

Natural Language Image Editing (NLIE) is notoriously difficult because of the "novice gap." Most users lack the technical vocabulary of professional editors, leading to Image Edit Requests (IERs) that are ambiguous ("make it pop"), abstract, or imprecise.

Current AI approaches usually fall into two camps: attempting to map high-level intent to low-level operations (a difficult one-to-many mapping problem), or using end-to-end image generation (GANs), which is often data-hungry and struggles with localized edits. This paper proposes a middle ground: using dialogue to teach novices "in-domain" terminology, smoothing the learning curve by grounding the interaction in direct, low-level language-action mapping.

The ImaDial System Architecture

The researchers developed ImaDial, a modular system consisting of four main components that work in a feedback loop with the user.

The modular architecture of the ImaDial system, illustrating the flow from user utterance to image edit execution.
  • State Tracker: Uses an LSTM-based Natural Language Understanding (NLU) unit to extract "slots" from user text. It identifies the Action (e.g., increase), Referent (e.g., the large cow), Attribute (e.g., saturation), and Value (e.g., 50).
  • Dialogue Manager: Employs a rule-based policy to guide the user. It follows a specific flow: Requesting a region $\rightarrow$ Querying the vision engine $\rightarrow$ Confirming the mask with the user $\rightarrow$ Requesting attributes/values $\rightarrow$ Executing the edit.
  • Vision Engine: Utilizes MattNet, a modular neural network that resolves referring expressions (like "the person on the left") into a localized image mask.
  • Image Edit Engine: A localized editing tool built with OpenCV that applies the requested transformations (brightness, contrast, hue, saturation, or lightness) specifically to the masked region.

Experimental Results and User Evaluation

The study utilized the ILLC-IER Dataset, a new collection of 2,537 precise imperative requests, alongside RefCOCO for training the Vision Engine. Evaluation involved a user study resulting in 83 dialogues (1,359 utterances).

A key contribution is the Vision Accuracy metric: $$\text{Vision Accuracy} = \frac{\#\text{Execute}}{\#\text{Query}}$$ This measures how often a user accepted a suggested mask and proceeded to an edit, which correlated more strongly with user ratings than raw performance metrics like Intersection over Union (IoU).

User agreement ratings across different system components, highlighting areas of satisfaction and friction.

User feedback indicated that while the NLU for referring expressions and the Vision Engine were generally well-received, the NLU for attributes and values faced more disagreement. Crucially, the analysis identified object segmentation as the primary bottleneck for user satisfaction. When the system fails to correctly "see" the target object, the dialogue flow is disrupted, regardless of NLU performance.

Conclusion and Future Work

The system received a "Fair to Good" rating overall, with 25.3% of users specifically praising its ease of use. Interestingly, the number of turns per edit didn't decrease significantly over time, suggesting that users successfully adapted their vocabulary to the system rather than just learning to be faster.

Future work will focus on:

  • Slot Carry-over: Implementing "memory" for active regions so users don't have to re-identify objects for sequential edits.
  • Vision Bounds: Expanding the pre-trained categories of the Vision Engine.
  • Multimodality: Incorporating gestures (pointing) and UI elements (sliders) alongside text for a more fluid experience.

Summary updated

Abstract

Natural Language Image Editing (NLIE) aims to use natural language instructions to edit images. Since novices are inexperienced with image editing techniques, their instructions are often ambiguous and contain high-level abstractions that tend to correspond to complex editing steps to accomplish. Motivated by this inexperience aspect, we aim to smooth the learning curve by teaching the novices to edit images using low-level commanding terminologies. Towards this end, we develop a task-oriented dialogue system to investigate low-level instructions for NLIE. Our system grounds language on the level of edit operations, and suggests options for a user to choose from. Though compelled to express in low-level terms, a user evaluation shows that 25% of users found our system easy-to-use, resonating with our motivation. An analysis shows that users generally adapt to utilizing the proposed low-level language interface. In this study, we identify that object segmentation as the key factor to the user satisfaction. Our work demonstrates the advantages of the low-level, direct language-action mapping approach that can be applied to other problem domains beyond image editing such as audio editing or industrial design.
BibTeX
@inproceedings{lin2020adjusting,
  author = {Tzu-Hsiang Lin and Alexander I. Rudnicky and Trung Bui and Doo Soon Kim and Jean Oh},
  title = {{Adjusting Image Attributes of Localized Regions with Low-level Dialogue}},
  booktitle = {LREC},
  year = {2020},
  url = {https://openreview.net/forum?id=hTlbej20HT},
}
Figure