TL;DR
This paper introduces InternGPT (iGPT), a novel interactive framework that combines the power of large language models (LLMs) like ChatGPT with non-verbal user input (pointing gestures) to perform complex vision-centric tasks. By integrating pointing with language, iGPT significantly enhances the efficiency and accuracy of human-AI interaction for image and video manipulation, editing, and generation.
What Problem Does This Paper Address?
Current AI systems for vision-centric tasks, while increasingly capable, often struggle with complex visual scenarios that require fine-grained control and precise manipulation. Existing interactive systems primarily rely on purely linguistic instructions, which can be inefficient and ambiguous when dealing with multiple objects or intricate details in images and videos. Describing specific regions or actions verbally can be cumbersome and lead to misunderstandings. The paper aims to bridge this gap by enabling more intuitive and precise interaction through the combination of natural language and pointing gestures, mimicking how humans naturally interact with visual information.
Methods
InternGPT (iGPT) is built upon three core components:
- Perception Unit: This unit interprets user pointing gestures (clicks, drags, strokes) on images and videos. It leverages existing open-source models like SAM (Segment Anything Model) for object segmentation and OCR for text extraction, enabling precise object selection, movement, and content creation.
- LLM Controller: This component acts as the brain of the system, processing user language commands. It decomposes complex tasks into smaller subtasks, plans their execution order, and selects appropriate tools from the toolkit. A key innovation here is an auxiliary control mechanism designed to improve the LLM's ability to invoke APIs, especially when parsing arguments, by refining abstract instructions into specific commands.
- Open-World Toolkit: This is a versatile collection of off-the-shelf models and applications, including vision foundation models (e.g., InternImage, InternVideo), vision-language models (e.g., Husky, BLIP-2), and general tools like calculators and search engines. This allows iGPT to handle a wide range of vision-centric tasks.
The framework operates through continuous interaction, maintaining conversation history to improve task execution. It supports three levels of interaction: basic (e.g., "caption this photo"), language-guided (e.g., "remove the black dog near the table"), and pointing-language enhanced (for tasks requiring high precision).
A significant contribution is the development of Husky, a large vision-language model fine-tuned for high-quality multi-modal dialogue. Husky is a 7B parameter model that reportedly achieves impressive results, impressing ChatGPT-3.5-turbo with 93.89% GPT-4 quality in certain evaluations.
Datasets
The paper does not explicitly list specific datasets used for training the core iGPT framework. However, it mentions leveraging existing open-source projects like SAM and OCR for the perception unit. The Husky model is trained using data generated by GPT-4 and instruction-following data. The evaluation of Husky uses the COCO validation images for quantitative assessment.
Key Insights and Ideas
- Pointing is a powerful complement to language: The paper strongly advocates that non-verbal cues like pointing are not just shortcuts but essential for efficient and accurate interaction in complex visual tasks, especially when dealing with multiple objects.
- LLMs need more than just language: The auxiliary control mechanism highlights that even advanced LLMs can benefit from structured support to reliably interact with external tools and APIs, particularly for argument parsing.
- Unified interactive framework is key: iGPT's success lies in its seamless integration of perception, language understanding, and a diverse toolkit, creating a holistic system for vision-centric tasks.
- Efficiency gains are substantial: User studies indicate that iGPT significantly reduces the number of attempts and prompt lengths required for complex tasks compared to purely language-based systems.
Key Results
- Improved Efficiency: User studies showed that iGPT required fewer attempts (e.g., 1.0 vs. 2.7 for "remove something" in one-object scenarios) and shorter prompts to achieve satisfactory results compared to Visual ChatGPT.
- Higher User Preference: Participants in the user study preferred iGPT due to its efficiency and user-friendliness.
- Husky's Performance: The Husky model achieved 93.89% GPT-4 quality when evaluated by ChatGPT-3.5-turbo on detailed quantitative results, demonstrating strong reasoning capabilities.
- Versatile Applications: iGPT successfully demonstrated capabilities in interactive image editing, visual question answering, image generation, and video highlight interpretation.
Limitations and Future Work
- Model Performance Dependency: iGPT's effectiveness is heavily reliant on the quality and accuracy of the underlying open-sourced models.
- Scalability Challenges: Handling increasingly complex interactions with a large number of instances might pose challenges for response times and accuracy.
- Adaptability to Novel Scenarios: The system might struggle in uncommon scenarios not covered by the training data of its employed models.
- User Interface Learning Curve: While designed for user-friendliness, some users might still face difficulties with the combined pointing and language instructions.
- Compatibility: Achieving seamless integration across diverse devices and platforms could be a challenge.
The authors express a commitment to ongoing updates and improvements for exceptional performance, aiming to establish iGPT as a foundational benchmark for visual interactive systems. They also welcome community contributions.
Summary updated