Partial convolution for padding, inpainting, and image synthesis
Author(s)
Date
2022-09-26Source Title
IEEE Transactions on Pattern Analysis and Machine Intelligence
Print ISSN
0162-8828
Electronic ISSN
1939-3539
Publisher
IEEE
Pages
1 - 15
Language
English
Type
ArticleItem Usage Stats
6
views
views
40
downloads
downloads
Abstract
Partial convolution weights convolutions with binary masks and renormalizes on valid pixels. It was originally proposed for image inpainting task because a corrupted image processed by a standard convolutional often leads to artifacts. Therefore, binary masks are constructed that define the valid and corrupted pixels, so that partial convolution results are only calculated based on valid pixels. It has been also used for conditional image synthesis task, so that when a scene is generated, convolution results of an instance depend only on the feature values that belong to the same instance. One of the unexplored applications for partial convolution is padding which is a critical component of modern convolutional networks. Common padding schemes make strong assumptions about how the padded data should be extrapolated. We show that these padding schemes impair model accuracy, whereas partial convolution based padding provides consistent improvements across a range of tasks. In this paper, we review partial convolution applications under one framework. We conduct a comprehensive study of the partial convolution based padding on a variety of computer vision tasks, including image classification, 3D-convolution-based action recognition, and semantic segmentation. Our results suggest that partial convolution-based padding shows promising improvements over strong baselines.
Keywords
Image inpaintingImage synthesis
Object classification
Padding
Partial convolution
Semantic segmentation