SegGPT: 在上下文中分割一切

SegGPT: Segmenting Everything In Context

王鑫龙 Xinlong Wang · BAAI · 2023-04-06 · arXiv:2304.03284 ↗ · 被引 275

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们提出了 SegGPT，一个用于在上下文中分割一切的通用模型。我们将各种分割任务统一到一个通用的上下文学习框架中，通过将不同类型的分割数据转换为相同格式的图像来适应它们。SegGPT 的训练被表述为一个上下文着色问题，每个数据样本具有随机颜色映射。目标是根据上下文完成不同的任务，而不是依赖特定的颜色。训练后，SegGPT 可以通过上下文推理在图像或视频中执行任意分割任务，例如对象实例、材质、部分、轮廓和文本。SegGPT 在广泛的任务上进行了评估，包括少样本语义分割、视频对象分割、语义分割和全景分割。我们的结果在定性和定量上都显示出在分割域内和域外目标方面的强大能力。

We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images. The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample. The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos via in-context inference, such as object instance, stuff, part, contour, and text. SegGPT is evaluated on a broad range of tasks, including few-shot semantic segmentation, video object segmentation, semantic segmentation, and panoptic segmentation. Our results show strong capabilities in segmenting in-domain and out-of-domain targets, either qualitatively or quantitatively.

核心贡献 · Key contributions

• 首个通过上下文学习实现多种分割任务的通用模型。
  First generalist model for diverse segmentation tasks via in-context learning.
• 将分割任务统一为具有随机颜色映射的上下文着色框架。
  Unifies segmentation tasks into an in-context coloring framework with random color mapping.
• 无需微调即可执行图像或视频中的任意分割任务。
  Performs arbitrary segmentation tasks in images or videos without fine-tuning.
• 提出上下文集成策略（空间和特征）用于多示例提示。
  Proposes context ensemble strategies (spatial and feature) for multi-example prompting.
• 支持无需更新模型参数的上下文调优，用于专门任务。
  Enables in-context tuning for specialized tasks without updating model parameters.
• 在域内和域外分割基准上表现强劲。
  Strong performance on in-domain and out-of-domain segmentation benchmarks.

局限 · Limitations

• 随机着色方案使域内任务更难，导致性能下降。
  Random coloring scheme makes in-domain tasks harder, causing performance drop.
• 并非旨在所有基准上超越专用模型。
  Not designed to outperform specialist models on all benchmarks.
• 性能受限于模型大小和数据可用性以进行规模扩张。
  Performance limited by model size and data availability for scaling.
• 上下文集成策略在低分辨率图像上可能导致信息丢失。
  Context ensemble strategies may cause information loss with low-resolution images.
• 域内任务的上下文调优仍落后于专用方法。
  In-context tuning for in-domain tasks still lags behind specialist methods.

论文章节 · Sections（共 18）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2.1 视觉分割 2.1 Visual Segmentation
• 2.2 视觉通才 2.2 Vision Generalist
• 2.3 上下文视觉学习 2.3 In-Context Visual Learning
• 3 方法 3 Approach
• 3.1 上下文着色 3.1 In-Context Coloring
• 3.2 上下文集成 3.2 Context Ensemble
• 3.3 上下文调优 3.3 In-Context Tuning
• 4 实验 4 Experiment
• 4.1 训练数据 4.1 Training Data
• 4.2 单样本训练细节 4.2 One-Shot Training Details
• 4.3 定性结果 4.3 Qualitative Results
• 4.4 与专用方法比较 4.4 Comparison with Specialist Methods
• 4.5 消融研究 4.5 Ablation Study
• 4.6 上下文调优 4.6 In-Context Tuning
• 5 讨论与结论 5 Discussion and Conclusion
• 致谢 Acknowledgement

阅读逐段中英对照全文 →