POINTS1.5: Building a Vision-Language Model towards Real World Applications

Yuan Liu, Le Tian, Xiao Zhou, Xinyu Gao, Kavio Yu, Yang Yu, Jie ZhouarXiv 2024 · DOI · arXiv

Introduction

POINTS1.5 is a state-of-the-art 7B-parameter vision-language model (VLM) developed by Tencent. Designed to bridge the performance gap between open-source models and closed-source giants like GPT-4o, POINTS1.5 currently leads the OpenCompass leaderboard for models under 10 billion parameters. By integrating a NaViT-style native dynamic resolution vision encoder and implementing a rigorous bilingual data-cleaning pipeline, the model achieves exceptional performance in complex tasks such as optical character recognition (OCR), mathematical reasoning, and document analysis.

Overcoming the Bottlenecks of Open-Source VLMs

The development of POINTS1.5 was driven by the identification of three primary bottlenecks in current open-source VLM research:

  1. Spatial Disruption: Traditional VLMs often use fixed-resolution vision encoders (e.g., standard CLIP). To process high-resolution images, these models split images into "tiles," which can break the spatial relationships between visual elements. This is particularly detrimental for text-heavy documents or intricate diagrams.
  2. Bilingual Scarcity: While English-language VLM datasets are abundant, high-quality, large-scale Chinese datasets remain scarce, limiting the global utility of many open-source models.
  3. Data Noise: Many instruction-tuning datasets contain grammatical errors or "image-independent" questions—queries that can be answered by a language model without visual context. This leads to model hallucinations and poor grounding.

Architectural and Methodological Innovations

POINTS1.5 utilizes a standard LLaVA-style architecture (Vision Encoder + Projector + LLM) but introduces several critical enhancements to address the aforementioned bottlenecks.

Native Dynamic Resolution

Instead of a standard CLIP encoder, POINTS1.5 adopts a NaViT-style architecture. This allows the model to process images of any resolution natively without the need for tiling or down-sampling. By preserving the original spatial structure, the model maintains a coherent view of the entire visual input, which is essential for high-fidelity OCR and document understanding.

The "Tokenizer" Analogy

The authors treat the vision encoder as a visual "tokenizer." Their research indicates that the vision encoder should remain fixed; unfreezing it during pre-training was found to actually degrade overall performance. This suggests that a high-quality, stable visual representation is more beneficial than task-specific fine-tuning of the encoder itself.

Figure 1: Performance of POINTS1.5-7B compared to other vision-language models on the OpenCompass leaderboard.

Data Quality and Filtering Strategies

Data quality was prioritized over raw quantity. The training process involved a massive, cleaned corpus of 2 million image-caption pairs for pre-training and a diverse instruction-tuning set.

  • Bilingual Expansion: The Chinese data ratio was significantly increased, utilizing the CapFusion method to merge original captions with VLM-generated descriptions for 1 million Chinese pairs.
  • Grammar and Logic Filtering: Using Qwen2.5-72B, the team detected and discarded samples with grammatical errors. They found that removing bad samples entirely was more effective than attempting to correct them, resulting in the removal of approximately 15% of the tuning data.
  • Image-Independence Check: To combat hallucinations, the authors filtered out questions that could be answered by an LLM without looking at the image. While this led to a slight dip in general text reasoning, it significantly improved the model's visual grounding.
  • Model Soup: To maximize performance, the final model weights were derived using "Model Soup," averaging multiple top-performing checkpoints trained on different data subsets.

Results and Benchmark Performance

POINTS1.5-7B achieves state-of-the-art results for its size class across several key benchmarks:

  • OpenCompass Leaderboard: It ranks #1 among all models under 10B parameters.
  • Mathematical Reasoning: The model scored 66.4 on MathVista, outperforming the much larger InternVL2-26B (58.3) and rivaling the 72B-parameter Molmo.
  • OCR Capabilities: It achieved a score of 83.2 on OCRBench, demonstrating superior ability in extracting text from complex real-world images like receipts and technical diagrams.
Figure 2: Examples of POINTS1.5's capabilities in complex math problem solving and high-accuracy Chinese OCR.

Conclusion and Future Directions

POINTS1.5 demonstrates that architectural choices like dynamic resolution and rigorous data filtering can allow a 7B-parameter model to compete with much larger architectures. While the model currently faces some category imbalances in its tuning data (e.g., a heavy lean toward OCR), it sets a new standard for efficient, high-performance VLMs. Future work will focus on scaling this architecture to larger language models and refining data balancing strategies to reduce hallucinations further without sacrificing general reasoning capabilities.

Summary updated

BibTeX
@misc{liu2024points,
  author = {Yuan Liu and Le Tian and Xiao Zhou and Xinyu Gao and Kavio Yu and Yang Yu and Jie Zhou},
  title = {{POINTS1.5: Building a Vision-Language Model towards Real World Applications}},
  year = {2024},
  eprint = {2412.08443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08443},
}
Figure