In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. In addition, Painter significantly outperforms recent generalist models on several challenging tasks.
核心贡献 · Key contributions
提出 Painter 通用模型，将视觉任务输出重定义为图像，并使用图像对作为任务提示。 Proposes Painter, a generalist model that redefines vision task outputs as images and uses image pairs as task prompts.
将七种不同视觉任务（深度、分割、关键点、恢复）统一到单一掩码图像建模框架中。 Unifies seven diverse vision tasks (depth, segmentation, keypoint, restoration) into a single masked image modeling framework.
在多个基准上达到与专用模型竞争的性能，在 NYUv2 深度估计上达到最先进水平。 Achieves competitive performance against task-specific models on multiple benchmarks, with state-of-the-art on NYUv2 depth estimation.
展示了使用视觉提示对域内和域外任务进行上下文学习的能力。 Demonstrates in-context learning capability for both in-domain and out-of-domain tasks using visual prompts.
表明跨任务联合训练在大多数任务上优于单独训练。 Shows that joint training across tasks can improve performance compared to separate training on most tasks.
引入简单的提示选择和学习方法，进一步提升了任务性能。 Introduces a simple prompt selection and learning method that further improves task performance.
局限 · Limitations
由于输入分辨率较低，全景分割性能落后于专用模型。 Performance on panoptic segmentation lags behind specialized models due to lower input resolution.
基于图像的接口不自然适用于建模离散语言信号。 The image-based interface is not naturally suited for modeling discrete language signals.
联合训练在关键点检测上性能略有下降，表明存在任务冲突。 Joint training slightly hurts performance on keypoint detection, indicating task conflicts.
该方法需要为每个任务精心设计输出图像表示。 The approach requires careful design of output image representations for each task.
域外泛化仅定性展示，定量评估有限。 Out-of-domain generalization is only qualitatively shown; quantitative evaluation is limited.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 方法3 Approach
3.1 将输出空间重定义为“图像”3.1 Redefining Output Spaces as “Images”