卷积神经网络（CNN）由于其构建模块中的固定几何结构，本质上难以建模几何变换。在这项工作中，我们引入了两个新模块来增强 CNN 的变换建模能力，即可变形卷积和可变形 RoI 池化。两者都基于在模块中增加空间采样位置的偏移量，并从目标任务中学习这些偏移量，无需额外监督。这些新模块可以轻松替换现有 CNN 中的普通对应模块，并通过标准反向传播进行端到端训练，从而形成可变形卷积网络。大量实验验证了我们的方法在目标检测和语义分割等复杂视觉任务上的有效性。代码将开源。
Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation. The code would be released.
核心贡献 · Key contributions
提出可变形卷积和可变形 RoI 池化，增强 CNN 对几何变换的建模能力。 Introduces deformable convolution and deformable RoI pooling to enhance CNNs' geometric transformation modeling.
模块从目标任务学习空间偏移，无需额外监督，实现自适应采样。 Modules learn spatial offsets from target tasks without extra supervision, enabling adaptive sampling.
可变形模块轻量、即插即用，可通过标准反向传播端到端训练。 Deformable modules are lightweight, plug-and-play, and trainable end-to-end via standard backpropagation.
首次证明在深度 CNN 中学习密集空间变换对目标检测和语义分割有效。 First demonstration that learning dense spatial transformations in deep CNNs is effective for object detection and semantic segmentation.
大量实验表明，在 PASCAL VOC 和 COCO 上相比普通 CNN 精度显著提升。 Extensive experiments show significant accuracy improvements over plain CNNs on PASCAL VOC and COCO.
代码已公开，便于复现和进一步研究。 Code is publicly released to facilitate reproducibility and further research.
局限 · Limitations
偏移量按图像位置学习，略微增加模型复杂度和训练时间。 Offsets are learned per image location, increasing model complexity and training time slightly.
可变形卷积可能难以泛化到几何变换差异很大的任务。 Deformable convolution may not generalize well to tasks with very different geometric transformations.
方法假设偏移为分数并使用双线性插值，可能引入近似误差。 The method assumes offsets are fractional and uses bilinear interpolation, which may introduce approximation errors.
性能提升依赖于任务，且随可变形层数增加可能饱和。 Performance gains are task-dependent and may saturate with many deformable layers.
该方法未在 3D 数据或视频任务上测试，适用范围有限。 The approach has not been tested on 3D data or video tasks, limiting its scope.