G A N mask R-CNN : instance semantic segmentation benefits from generative adversarial networks

Abstract

In designing instance segmentation ConvNets that reconstruct masks, segmentation is often taken as its literal definition -assigning label to every pixel- for defining the loss functions. That is, using losses that compute the difference between pixels in the predicted (reconstructed) mask and the ground truth mask -a template matching mechanism. However, any such instance segmentation ConvNet is a generator, so we can lay the problem of predicting masks as a GANs game framework: We can think the ground truth mask is drawn from the true distribution, and a ConvNet like Mask R-CNN is an implicit model that infers the true distribution. In GANs terms, Mask R-CNN is the generator who reconstructs a mask as the fake one. We then send the fake mask and the real (ground truth) one to a discriminator (critic). By playing a min-max game, we want Mask R-CNN to fool the critic, and the critic to distinguish between real and fake masks. In this way, we take the advantage of a region proposal network (implemented in Mask R-CNN) to design a generator, and the benefit of a critic network to design a better loss function as opposed to a template matching one. We discuss how we utilize the GANs training stability regiments in practice to make this concept works. We show this GANs framework performs better than the original Mask R-CNN. Furthermore, we show the results give crisper boundaries - a traditional challenge of ConvNets where there is a trade-off between having higher level of semantics and finer boundaries.