DeepSeek-VL: Towards Real-World Vision-Language Understanding
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→我们提出了 DeepSeek-VL，这是一个面向真实世界视觉和语言理解应用的开源视觉语言模型。我们的方法围绕三个关键维度展开：首先，我们努力确保数据的多样性、可扩展性，并广泛覆盖真实世界场景，包括网页截图、PDF、OCR、图表和基于知识的内容，旨在全面代表实际应用环境。其次，我们从真实用户场景中创建了一个用例分类法，并据此构建了指令微调数据集。使用该数据集进行微调显著提升了模型在实际应用中的用户体验。考虑到效率和大多数真实世界场景的需求，DeepSeek-VL 采用了一种混合视觉编码器，能够高效处理高分辨率图像（1024 x 1024），同时保持相对较低的计算开销。这一设计选择确保了模型在各种视觉任务中捕捉关键语义和细节信息的能力。我们认为，一个优秀的视觉语言模型首先应具备强大的语言能力。为了在预训练过程中保持大语言模型的能力，我们探索了一种有效的视觉语言预训练策略，即从一开始就整合大语言模型训练，并仔细管理视觉和语言模态之间观察到的竞争动态。DeepSeek-VL 系列（包括 1.3B 和 7B 模型）作为视觉语言聊天机器人在真实世界应用中展现了卓越的用户体验，在相同模型规模下，在广泛的视觉语言基准测试中取得了最先进或具有竞争力的性能，同时在以语言为中心的基准测试中保持了稳健的性能。我们已公开提供 1.3B 和 7B 模型，以促进基于该基础模型的创新。
We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.