我们提出了 Segment Anything Model 2（SAM 2），这是一个用于解决图像和视频中可提示视觉分割的基础模型。我们构建了一个数据引擎，通过用户交互改进模型和数据，收集了迄今为止最大的视频分割数据集。我们的模型是一个简单的 Transformer 架构，带有流式内存，用于实时视频处理。基于我们的数据训练的 SAM 2 在广泛的任务中表现出强大的性能。在视频分割中，我们观察到更好的准确性，使用的交互次数比先前方法少 3 倍。在图像分割中，我们的模型比 Segment Anything Model（SAM）更准确，速度快 6 倍。我们相信，我们的数据、模型和见解将成为视频分割及相关感知任务的重要里程碑。我们正在发布我们的主要模型、数据集以及模型训练和演示的代码。
We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing our main model, dataset, as well as code for model training and our demo.
核心贡献 · Key contributions
将可提示分割从图像扩展到视频，采用统一模型。 Extends promptable segmentation from images to video with a unified model.
提出带记忆的流式 Transformer 架构，支持实时视频处理。 Introduces a streaming transformer architecture with memory for real-time video processing.
构建人机协同数据引擎，收集最大视频分割数据集 SA-V。 Builds a data engine with human-in-the-loop to collect the largest video segmentation dataset (SA-V).
视频分割中交互次数减少 3 倍，精度优于先前方法。 Achieves better accuracy with 3x fewer interactions in video segmentation vs prior work.
图像分割中比 SAM 精度更高，推理速度快 6 倍。 Outperforms SAM in image segmentation with 6x faster inference.
在 17 个视频和 37 个图像基准上展示强零样本泛化能力。 Demonstrates strong zero-shot generalization across 17 video and 37 image benchmarks.
局限 · Limitations
在镜头切换或拥挤场景中可能无法分割物体。 May fail to segment objects across shot changes or in crowded scenes.
难以跟踪极细或快速移动的物体。 Struggles with tracking very thin or fast-moving objects.
单独处理每个物体，缺乏物体间通信。 Processes each object separately without inter-object communication.
数据引擎依赖人工标注者进行验证和修正。 Data engine relies on human annotators for verification and correction.
对长期遮挡和长视频的评估有限。 Limited evaluation on long-term occlusion and extended videos.