Quantitative analysis of cell nuclei in microscopic images is an essential yet challenging source of biological and pathological information. The major challenge is accurate detection and segmentation of densely packed nuclei in images acquired under a variety of conditions. Mask R-CNN-based methods have achieved state-of-the-art nucleus segmentation. However, the current pipeline requires fully annotated training images, which are time consuming to create and sometimes noisy. Importantly, nuclei often appear similar within the same image. This similarity could be utilized to segment nuclei with only partially labeled training examples. We propose a simple yet effective region-proposal module for the current Mask R-CNN pipeline to perform few-exemplar learning. To capture the similarities between unlabeled regions and labeled nuclei, we apply decomposed self-attention to learned features. On the self-attention map, we observe strong activation at the centers and edges of all nuclei, including unlabeled nuclei. On this basis, our region-proposal module propagates partial annotations to the whole image and proposes effective bounding boxes for the bounding box-regression and binary mask-generation modules. Our method effectively learns from unlabeled regions thereby improving detection performance. We test our method with various nuclear images. When trained with only 1/4 of the nuclei annotated, our approach retains a detection accuracy comparable to that from training with fully annotated data. Moreover, our method can serve as a bootstrapping step to create full annotations of datasets, iteratively generating and correcting annotations until a predetermined coverage and accuracy are reached. The source code is available at https://github.com/feng-lab/nuclei.

Introduction

The cell nucleus is a fundamental biological structure containing important information, such as the cell type, density, and viability. Automatically identifying and segmenting nuclei from microscopic images is the first step in many types of quantitative analyses with applications ranging from basic cell biology [1], [2] and systems neuroscience [3], [4] to cancer diagnosis [5]. This task is a challenging instance-segmentation task because it requires the correct detection of all instances of nuclei within an image, along with precise labeling of the pixels belonging to each detected nucleus. Nucleus segmentation methods need to be instance-aware to correctly separate touching nuclei. Meanwhile, such methods need to address difficulties such as a high cell density, low contrast, intensity inhomogeneity, shape variation, weak boundaries, strong gradients inside the nucleus, and ambiguous overlapping. Classical approaches, including thresholding [6]–[۱۰], marker-controlled watersheding [11]–[۱۳], edge detection [14], shape matching [15]–[۱۷] and region merging/ growing [18]–[۲۲], often assume a certain signal pattern of nuclei or cells, such as bright centers, strong boundaries or low signal gradients inside the nuclei. These methods work well for certain datasets but tend to fail in difficult cases in which the assumptions do not hold. Denoising and transformation [9], [13] could be used to improve these methods by creating transformed images that fit the assumptions more closely. Nevertheless, the applicability of such methods is limited by these assumptions.