Optimization of the Preprocessing Method for Edge Detection on Overlapping Cells at PAP Smear Images

Nita Merlina - Universitas Nusa Mandiri Jakarta
Edi Noersasongko - Universitas Dian Nuswantoro Semarang
Pulung Andono - Universitas Dian Nuswantoro Semarang
M Soeleman - Universitas Dian Nuswantoro Semarang
Dwiza Riana - Universitas Nusa Mandiri Jakarta

Citation Format:

DOI: http://dx.doi.org/10.30630/joiv.7.2.1329


The complexity of the cell structure and high overlap cause poor image contrast. Complex imaging factors can make automatic visual interpretation more difficult. Segmentation separates a digital image into different parts with homogeneous attributes so that different areas have different features. The challenges faced in performing nucleus segmentation on Pap Smear (PS) images are poor contrast, the presence of neutrophils, and uneven staining of overlapping cells. This research was conducted to improve image quality in identifying the nucleus accurately. The method used is the Polynomial Contrast Enhancement (PCE) model as an approach to preprocessing. This method functions to change the contrast of the Pap smear image against the overlapping cells so that it becomes a significant contrast in detecting the edge of the nucleus object. The detection process uses the Robert and Prewitt edge detection method to test the identification of the nucleus object on 797 PS Repository images of the University of Nusa Mandiri (RepomedUNM). The accuracy result obtained is 86.8%. Comparing Robert's edge detection and Prewitt's edge detection shows that the PCE approach as a filter method can overcome color contrast problems and detect more accurately. The difficulty in detecting the nucleus from the PS image against the overlapping cells can be solved. This method can distinguish overlapping cells from their core during testing, thus becoming a reference in identifying cells with improved accuracy and testing larger data sets.


Nucleus; overlap cells; PAP smear; edge detection; polynomial contrast enhancement (PCE)

Full Text:



Y. Hamama-Raz, S. Shinan-Altman, and I. Levkovich, "The intrapersonal and interpersonal processes of fear of recurrence among cervical cancer survivors: a qualitative study," Support. Care Cancer, vol. 30, no. 3, pp. 2671–2678, 2022.

E. Black, F. Hyslop, and R. Richmond, "Barriers and facilitators to uptake of cervical cancer screening among women in Uganda: A systematic review," BMC Womens. Health, vol. 19, no. 1, pp. 1–12, 2019.

R. Di Fiore et al., "Cancer Stem Cells and Their Possible Implications in Cervical Cancer: A Short Review," Int. J. Mol. Sci., vol. 23, no. 9, 2022.

W. William, A. Ware, A. H. Basaza-Ejiri, and J. Obungoloch, "A pap-smear analysis tool (PAT) for detection of cervical cancer from pap-smear images," BioMedical Engineering Online, vol. 18, no. 1. 2019.

O. Y. and T. Tuncer, "Exemplar pyramid deep feature extraction based cervical cancer image classification model using pap-smear images," Biomed. Signal Process. Control, vol. 73.

R. Guido, "Secondary Prevention of Cervical Cancer Part 2," Clin. Obstet. & Gynecol., vol. 57, no. 2, pp. 292–301, 2014.

W. A. Jensen, "The Nucleus," Plant Cell, pp. 85–96, 1973.

J. Jordan et al., "European guidelines for quality assurance in cervical cancer screening: Recommendations for clinical management of abnormal cervical cytology, part 1," Cytopathology, vol. 19, no. 6, pp. 342–354, 2008.

F. Kuang, J. Ren, Q. Zhong, F. Liyuan, Y. Huan, and Z. Chen, "The value of apparent diffusion coefficient in the assessment of cervical cancer," Eur. Radiol., vol. 23, no. 4, pp. 1050–1058, 2013.

H. L. Glasgow et al., "Laminin targeting of a peripheral nerve-highlighting peptide enables degenerated nerve visualization," Proc. Natl. Acad. Sci. U. S. A., vol. 113, no. 45, pp. 12774–12779, 2016.

J. S. Torres-Roman et al., "Cervical cancer mortality among young women in Latin America and the Caribbean: trend analysis from 1997 to 2030," BMC Public Health, vol. 22, no. 1, pp. 1–10, 2022.

A.A. Alshehri, T. Daws, S. Ezekiel, "Medical Image Segmentation Using Multifractal Analysis," Int. J. Adv. Sci. Eng. Inf. Technol., vol. 10, no. 2, 2020.

W. William, A. Ware, A. H. Basaza-Ejiri, and J. Obungoloch, "A review of image analysis and machine learning techniques for automated cervical cancer screening from pap-smear images," Comput. Methods Programs Biomed., vol. 164, pp. 15–22, 2018.

L. Zhang, L. Zou, C. Wu, J. Jia, and J. Chen, "Method of famous tea sprout identification and segmentation based on improved watershed algorithm," Comput. Electron. Agric., vol. 184, no. March, p. 106108, 2021.

A. R. Revanda, "Classification of Acute Lymphoblastic Leukemia on White Blood Cell Microscopy Images Based on Instance Segmentation Using Mask R-CNN," vol. 15, no. 5, pp. 625–637, 2022.

T. Wang, H. Lan, L. Niu, Z. Fan, and G. Yang, "Overlapping Cell Segmentation of Cervical Cytology Images Based on Nuclear Radial Boundary Enhancement," Mob. Inf. Syst., vol. 2022, 2022.

K. P. Win, Y. Kitjaidure, K. Hamamoto, and T. M. Aung, "Computer-assisted screening for cervical cancer using digital image processing of pap smear images," Appl. Sci., vol. 10, no. 5, 2020.

F. L. G. Sun, S. Li, Y. Cao, "Cervical cancer diagnosis based on random forest," Int. J. Performability Eng, 2017.

E. Hussain, L. B. Mahanta, C. R. Das, M. Choudhury, and M. Chowdhury, "A shape context fully convolutional neural network for segmentation and classification of cervical nuclei in Pap smear images," Artif. Intell. Med., vol. 107, p. 101897, 2020.

MengZhao, Y. X. HaoWang, H. NingDai, XuguoSun, JinZhang, and M. Pedersen, "SEENS: Nuclei segmentation in Pap smear images with selective edge enhancement," Elsevier, vol. 114, pp. 185–194, 2021.

K. M. Jeevan, A. A. B. Gowda, and P. V. Kumar, "An image enhancement method based on gabor filtering in wavelet domain and adaptive histogram equalization," Indones. J. Electr. Eng. Comput. Sci., vol. 21, no. 1, pp. 146–153, 2021.

S. Zhang, Z. Ma, G. Zhang, T. Lei, R. Zhang, and Y. Cui, "Semantic image segmentation with deep convolutional neural networks and quick shift," Symmetry (Basel)., vol. 12, no. 3, pp. 1–11, 2020.

Y. H. TAO WANG , JINJIE HUANG , DEQUAN ZHENG, "Nucleus Segmentation of Cervical Cytology Images Based on Depth Information," IEE Access, vol. 8, pp. 75846–75859, 2020.

J. M. Curtis G. Northcutt, Anish Athalye, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks," arXiv, 2021.

K.-S. C. Youyi Song, Lei Zhu, Baiying Lei, Senior , Bin Sheng, Qi Dou, Jing Qin, "Constrained Multi-shape Evolution for Overlapping Cytoplasm Segmentation," Cornell Univ., 2020.

T. Wan, S. Xu, C. Sang, Y. Jin, and Z. Qin, "Accurate segmentation of overlapping cells in cervical cytology with deep convolutional neural networks," Neurocomputing, vol. 365, pp. 157–170, 2019.

H. Zhang, H. Zhu, and X. Ling, "Polar coordinate sampling-based segmentation of overlapping cervical cells using attention U-Net and random walk," Neurocomputing, vol. 383, pp. 212–223, 2020.

M. Versaci and F. C. Morabito, "Image Edge Detection: A New Approach Based on Fuzzy Entropy and Fuzzy Divergence," Int. J. Fuzzy Syst., vol. 23, no. 4, pp. 918–936, 2021.

L. T. Shen1, Dianhuai, Xueying Jiang, "A novel Gauss-Laplace operator based on multi-scale convolution for dance motion image enhancement," EAI Endorsed Trans., vol. 9, no. 36, pp. 2–12, 2022.

W. H. and P.-C. Shao, "Variational contrast-saturation enhancement model for effective single image dehazing," Signal Processing, vol. 192, 2022.

Y. A. S. Devi and M. C. Kamsali, "Retinal Image Contrast Enhancement through Pixel Collaboration in Spatial Domain," Int. J. Intell. Eng. Syst., vol. 15, no. 3, pp. 500–513, 2022.

S. Aulia and D. Rahmat, "Brain Tumor Identification Based on VGG-16 Architecture and CLAHE Method," Int. J. Informatics Vis., vol. 6, no. 1, pp. 96–102, 2022.

M. Sharma, S. Kumar Singh, P. Agrawal, and V. Madaan, "Classification of Clinical Dataset of Cervical Cancer using KNN," Indian J. Sci. Technol., vol. 9, no. 28, 2016.

D. Riana et al., "RepoMedUNM: A New Dataset for Feature Extraction and Training of Deep Learning Network for Classification of Pap Smear Images," in International Conference on Neural Information Processing, 2021, pp. 317–325.

X. Wu and C. N. Yang, "A combination of color-black-and-white visual cryptography and polynomial based secret image sharing," J. Vis. Commun. Image Represent., vol. 61, pp. 74–84, 2019.

A. A. Abdullah, A. F. D. Giong, and N. A. H. Zahri, "Cervical cancer detection method using an improved cellular neural network (CNN) algorithm," Indones. J. Electr. Eng. Comput. Sci., vol. 14, no. 1, pp. 210–218, 2019.

S. Saxena, Y. Singh, B. Agarwal, and R. C. Poonia, "Comparative analysis between different edge detection techniques on mammogram images using PSNR and MSE," J. Inf. Optim. Sci., vol. 43, no. 2, pp. 347–356, 2022.

K. A. Korznikov, D. E. Kislov, J. Altman, J. Doležal, A. S. Vozmishcheva, and P. V. Krestov, "Using u-net-like deep convolutional neural networks for precise tree recognition in very high resolution rgb (Red, green, blue) satellite images," Forests, vol. 12, no. 1, pp. 1–17, 2021.


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

JOIV : International Journal on Informatics Visualization
ISSN 2549-9610  (print) | 2549-9904 (online)
Organized by Department of Information Technology - Politeknik Negeri Padang, and Institute of Visual Informatics - UKM and Soft Computing and Data Mining Centre - UTHM
W : http://joiv.org
E : joiv@pnp.ac.id, hidra@pnp.ac.id, rahmat@pnp.ac.id

View JOIV Stats

Creative Commons License is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.