Object detection method based on instance segmentation of satellite image obtained in the conditions of cloud cover
Article Sidebar
Main Article Content
Abstract
Modern achievements in the space industry, combined with the continuous development of remote sensing technologies, form the basis for solving problems in various areas. Medium- and high-resolution satellite imagery often plays a key role in decision-making during crises in hard-to-reach areas. In the process of processing remote sensing data, a significant and still unresolved problem is the reconstruction of clouded images. This article analyses various approaches to cloud removal and data quality improvement. The traditional approaches considered have certain limitations associated with the loss of useful information. Particular attention is paid to deep learning methods, which are gaining popularity in solving cloud removal problems because they produce good results. The article discusses different DNN architectures (convolutional neural networks (CNN), conditional generative adversarial networks (cGAN)) and their modifications, identifies their advantages and disadvantages. A significant advantage of neural networks is their ability to adapt to various conditions and image types. The analysis of the disadvantages of fusing purely optical data led to the conclusion that the best solution to the problem of cloud removal from satellite images is to combine optical and radar data. As a result, the architecture of a model for removing clouds from optical satellite imagery using generative adversarial networks, combined with radar imagery, was developed. The theoretical hypotheses were confirmed by testing the model on the SEN12MS-CR dataset.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Accepted 2025-12-02
Published 2026-02-16
References
Cai, Z. and Vasconcelos, N., 2018. Cascade R-CNN: Delving Into High Quality Object Detection. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. Computer Vision Foundation / IEEE Computer Society, pp.6154–6162. Available from: https://doi.org/10.1109/CVPR.2018.00644. DOI: https://doi.org/10.1109/CVPR.2018.00644
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., Loy, C.C. and Lin, D., 2019. Hybrid Task Cascade for Instance Segmentation. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, pp.4974–4983. Available from: https://doi.org/10.1109/CVPR.2019.00511. DOI: https://doi.org/10.1109/CVPR.2019.00511
Cheng, M., Liu, Y., Lin, W., Zhang, Z., Rosin, P.L. and Torr, P.H.S., 2019. BING: Binarized normed gradients for objectness estimation at 300fps. Computational Visual Media, 5(1), pp.3–20. Available from: https://doi.org/10.1007/S41095-018-0120-1. DOI: https://doi.org/10.1007/s41095-018-0120-1
Cheng, Q., Zhang, H.S.L., Yuan, Q. and Zhen, C., 2014. Cloud removal for remotely sensed images by similar pixel replacement guided with a spatio-temporal MRF model. ISPRS Journal of Photogrammetry and Remote Sensing, 92, pp.54–68. Available from: https://doi.org/10.1016/j.isprsjprs.2014.02.015. DOI: https://doi.org/10.1016/j.isprsjprs.2014.02.015
Dai, J., He, K., Li, Y., Ren, S. and Sun, J., 2016. Instance-Sensitive Fully Convolutional Networks. In: B. Leibe, J. Matas, N. Sebe and M. Welling, eds. Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, Lecture Notes in Computer Science, vol. 9910. Springer, pp.534–549. Available from: https://doi.org/10.1007/978-3-319-46466-4_32. DOI: https://doi.org/10.1007/978-3-319-46466-4_32
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H. and Wei, Y., 2017. Deformable Convolutional Networks. IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, pp.764–773. Available from: https://doi.org/10.1109/ICCV.2017.89. DOI: https://doi.org/10.1109/ICCV.2017.89
Dollár, P., Tu, Z., Perona, P. and Belongie, S.J., 2009. Integral Channel Features. In: A. Cavallaro, S. Prince and D.C. Alexander, eds. British Machine Vision Conference, BMVC 2009, London, UK, September 7-10, 2009. Proceedings. British Machine Vision Association, pp.1–11. Available from: https://doi.org/10.5244/C.23.91. DOI: https://doi.org/10.5244/C.23.91
Ebel, P., Xu, Y., Schmitt, M. and Zhu, X.X., 2022. SEN12MS-CR-TS: A Remote-Sensing Data Set for Multimodal Multitemporal Cloud Removal. IEEE Transactions Geoscience Remote Sensing, 60, pp.1–14. Available from: https://doi.org/10.1109/TGRS.2022.3146246. DOI: https://doi.org/10.1109/TGRS.2022.3146246
Eckardt, R., Berger, C., Tiel, C. and Schmullius, C., 2013. Removal of Optically Thick Clouds from Multi-Spectral Satellite Images Using Multi-Frequency SAR Data. Remote Sensing, 5(6), pp.2973–3006. Available from: https://doi.org/10.3390/rs5062973. DOI: https://doi.org/10.3390/rs5062973
Gidaris, S. and Komodakis, N., 2016. Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization. In: R.C. Wilson, E.R. Hancock and W.A.P. Smith, eds. Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press. Available from: https://bmva-archive.org.uk/bmvc/2016/papers/paper090/index.html.
Girshick, R.B., 2015. Fast R-CNN. 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, pp.1440–1448. Available from: https://doi.org/10.1109/ICCV.2015.169. DOI: https://doi.org/10.1109/ICCV.2015.169
He, K., Gkioxari, G., Dollár, P. and Girshick, R.B., 2017. Mask R-CNN. CoRR, abs/1703.06870. Available from: http://arxiv.org/abs/1703.06870.
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, pp.770–778. Available from: https://doi.org/10.1109/CVPR.2016.90. DOI: https://doi.org/10.1109/CVPR.2016.90
Isola, P., Zhu, J., Zhou, T. and Efros, A.A., 2017. Image-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp.5967–5976. Available from: https://doi.org/10.1109/CVPR.2017.632. DOI: https://doi.org/10.1109/CVPR.2017.632
Kovbasiuk, S.V., Kanevskyy, L.B., Romanchuk, M.P., Chernyshuk, S.V. and Naumchak, L.M., 2023. Object detection method based on aerial image instance segmentation received by unmanned aerial vehicles in the conditions rough for visualization. In: T.A. Vakaliuk and S.O. Semerikov, eds. Proceedings of the 3rd Edge Computing Workshop, Zhytomyr, Ukraine, April 7, 2023, CEUR Workshop Proceedings, vol. 3374. CEUR-WS.org, pp.41–55. Available from: https://ceur-ws.org/Vol-3374/paper03.pdf.
Li, X., Wang, L., Cheng, Q., Wu, P., Gan, W. and Fang, L., 2019. Cloud removal in remote sensing images using nonnegative matrix factorization and error correction. ISPRS Journal of Photogrammetry and Remote Sensing, 148, pp.103–113. Available from: https://doi.org/10.1016/j.isprsjprs.2018.12.013. DOI: https://doi.org/10.1016/j.isprsjprs.2018.12.013
Li, Y., Qi, H., Dai, J., Ji, X. and Wei, Y., 2017. Fully Convolutional Instance-Aware Semantic Segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp.4438–4446. Available from: https://doi.org/10.1109/CVPR.2017.472. DOI: https://doi.org/10.1109/CVPR.2017.472
Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B. and Belongie, S.J., 2017. Feature Pyramid Networks for Object Detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp.936–944. Available from: https://doi.org/10.1109/CVPR.2017.106. DOI: https://doi.org/10.1109/CVPR.2017.106
Lin, T., Goyal, P., Girshick, R.B., He, K. and Dollár, P., 2017. Focal Loss for Dense Object Detection. IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, pp.2999–3007. Available from: https://doi.org/10.1109/ICCV.2017.324. DOI: https://doi.org/10.1109/ICCV.2017.324
Liu, S., Qi, L., Qin, H., Shi, J. and Jia, J., 2018. Path Aggregation Network for Instance Segmentation. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. Computer Vision Foundation / IEEE Computer Society, pp.8759–8768. Available from: https://doi.org/10.1109/CVPR.2018.00913. DOI: https://doi.org/10.1109/CVPR.2018.00913
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C. and Berg, A.C., 2016. SSD: Single Shot MultiBox Detector. In: B. Leibe, J. Matas, N. Sebe and M. Welling, eds. Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, Lecture Notes in Computer Science, vol. 9905. Springer, pp.21–37. Available from: https://doi.org/10.1007/978-3-319-46448-0_2. DOI: https://doi.org/10.1007/978-3-319-46448-0_2
Long, J., Shelhamer, E. and Darrell, T., 2015. Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. IEEE Computer Society, pp.3431–3440. Available from: https://doi.org/10.1109/CVPR.2015.7298965. DOI: https://doi.org/10.1109/CVPR.2015.7298965
Lv, H., Wang, Y. and Shen, Y., 2016. An empirical and radiative transfer model based algorithm to remove thin clouds in visible bands. Remote Sensing of Environment, 179, pp.183–195. Available from: https://doi.org/10.1016/j.rse.2016.03.034. DOI: https://doi.org/10.1016/j.rse.2016.03.034
Meng, F., Yang, X., Zhou, C. and Li, Z., 2017. A Sparse Dictionary Learning-Based Adaptive Patch Inpainting Method for Thick Clouds Removal from High-Spatial Resolution Remote Sensing Imagery. Sensors, 17(9), p.2130. Available from: https://doi.org/10.3390/s17092130. DOI: https://doi.org/10.3390/s17092130
Meng, Q., Borders, B., Cieszewski, C. and Madden, M., 2009. Closest Spectral Fit for Removing Clouds and Cloud Shadows. Photogrammetric Engineering & Remote Sensing, 75, pp.569–576. Available from: https://doi.org/10.14358/PERS.75.5.569. DOI: https://doi.org/10.14358/PERS.75.5.569
Pilkevych, I.A., Romanchuk, M.P., Naumchak, O.M., Fedorchuk, D.L. and Naumchak, L.M., 2025. Improved model for detecting randomly oriented objects on remote sensing images. In: T.A. Vakaliuk and S.O. Semerikov, eds. Proceedings of the 5th Edge Computing Workshop (doors 2025), Zhytomyr, Ukraine, April 4, 2025, CEUR Workshop Proceedings, vol. 3943. CEUR-WS.org, pp.118–126. Available from: https://ceur-ws.org/Vol-3943/paper26.pdf.
Redmon, J., Divvala, S.K., Girshick, R.B. and Farhadi, A., 2016. You Only Look Once: Unified, Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, pp.779–788. Available from: https://doi.org/10.1109/CVPR.2016.91. DOI: https://doi.org/10.1109/CVPR.2016.91
Rees, W., 2012. Physical Principles of Remote Sensing. 3rd ed. Cambridge University Press. Available from: https://doi.org/10.1017/CBO9781139017411. DOI: https://doi.org/10.1017/CBO9781139017411
Ren, S., He, K., Girshick, R.B. and Sun, J., 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In: C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama and R. Garnett, eds. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. pp.91–99. Available from: https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html.
Romanchuk, M., Zavada, A., Naumchak, O., Naumchak, L. and Kosheva, I., 2024. Removing cloudiness on optical space images by a generative adversarial network model using SAR images. Eastern-European Journal of Enterprise Technologies, 5(2 (131)), pp.6–12. Available from: https://doi.org/10.15587/1729-4061.2024.313690. DOI: https://doi.org/10.15587/1729-4061.2024.313690
Saha, S., Bovolo, F. and Bruzzone, L., 2018. Destroyed-buildings detection from VHR SAR images using deep features. In: L. Bruzzone and F. Bovolo, eds. Image and Signal Processing for Remote Sensing XXIV, vol. 10789. International Society for Optics and Photonics, SPIE, p.107890Z. Available from: https://doi.org/10.1117/12.2325149. DOI: https://doi.org/10.1117/12.2325149
Schowengerdt, R., 2007. Remote Sensing, Models and Methods for Image Processing. 3rd ed. Elsevier. Available from: https://doi.org/10.1016/B978-0-12-369407-2.X5000-1. DOI: https://doi.org/10.1016/B978-0-12-369407-2.X5000-1
Shen, H., Li, X., Cheng, Q., Zeng, C., Yang, G., Li, H. and Zhang, L., 2015. Missing Information Reconstruction of Remote Sensing Data: A Technical Review. IEEE Geoscience and Remote Sensing Magazine, 3(3), pp.61–85. Available from: https://doi.org/10.1109/MGRS.2015.2441912. DOI: https://doi.org/10.1109/MGRS.2015.2441912
Siravenha, A., Sousa, D., Bispo, A. and Pelaes, E., 2011. Evaluating Inpainting Methods to the Satellite Images Clouds and Shadows Removing. Signal Processing, Image Processing and Pattern Recognition - International Conference, SIP 2011, Held as Part of the Future Generation Information Technology Conference FGIT 2011, in Conjunction with GDC 2011, Jeju Island, Korea, December 8-10, 2011. Proceedings, Communications in Computer and Information Science, vol. 260. Springer, pp.56–65. Available from: https://doi.org/10.1007/978-3-642-27183-0_7. DOI: https://doi.org/10.1007/978-3-642-27183-0_7
Sun, M., Kim, B.S., Kohli, P. and Savarese, S., 2014. Relating Things and Stuff via ObjectProperty Interactions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7), pp.1370–1383. Available from: https://doi.org/10.1109/TPAMI.2013.193. DOI: https://doi.org/10.1109/TPAMI.2013.193
Tan, M., Pang, R. and Le, Q.V., 2020. EfficientDet: Scalable and Efficient Object Detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation / IEEE, pp.10778–10787. Available from: https://doi.org/10.1109/CVPR42600.2020.01079. DOI: https://doi.org/10.1109/CVPR42600.2020.01079
Tighe, J., Niethammer, M. and Lazebnik, S., 2014. Scene Parsing with Object Instances and Occlusion Ordering. 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014. IEEE Computer Society, pp.3748–3755. Available from: https://doi.org/10.1109/CVPR.2014.479. DOI: https://doi.org/10.1109/CVPR.2014.479
Tu, Z., Chen, X., Yuille, A. and Zhu, S., 2005. Image parsing: Unifying segmentation, detection, and recognition. International Journal of Computer Vision, 63(2), pp.113–140. Available from: https://doi.org/10.1007/S11263-005-6642-X. DOI: https://doi.org/10.1007/s11263-005-6642-x
Wang, J., Chen, K., Yang, S., Loy, C.C. and Lin, D., 2019. Region Proposal by Guided Anchoring. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, pp.2965–2974. Available from: https://doi.org/10.1109/CVPR.2019.00308. DOI: https://doi.org/10.1109/CVPR.2019.00308
Wang, L., Xu, X., Yu, Y., Yang, R., Gui, R., Xu, Z. and Pu, F., 2019. SAR-to-Optical Image Translation Using Supervised Cycle-Consistent Adversarial Networks. IEEE Access, 7, pp.129136–129149. Available from: https://doi.org/10.1109/ACCESS.2019.2939649. DOI: https://doi.org/10.1109/ACCESS.2019.2939649
Windy.com. Available from: https://www.windy.com/?50.452,30.529,5.
Xie, S., Girshick, R.B., Dollár, P., Tu, Z. and He, K., 2017. Aggregated Residual Transformations for Deep Neural Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp.5987–5995. Available from: https://doi.org/10.1109/CVPR.2017.634. DOI: https://doi.org/10.1109/CVPR.2017.634
Xu, M., Jia, X., Pickering, M. and Jia, S., 2019. Thin cloud removal from optical remote sensing images using the noise-adjusted principal components transform. ISPRS Journal of Photogrammetry and Remote Sensing, 149, pp.215–225. Available from: https://doi.org/10.1016/j.isprsjprs.2019.01.025. DOI: https://doi.org/10.1016/j.isprsjprs.2019.01.025
Yang, B., Yan, J., Lei, Z. and Li, S.Z., 2016. CRAFT Objects from Images. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, pp.6043–6051. Available from: https://doi.org/10.1109/CVPR.2016.650. DOI: https://doi.org/10.1109/CVPR.2016.650
Yao, J., Fidler, S. and Urtasun, R., 2012. Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation. 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012. IEEE Computer Society, pp.702–709. Available from: https://doi.org/10.1109/CVPR.2012.6247739. DOI: https://doi.org/10.1109/CVPR.2012.6247739
Yu, C., Chen, L., Su, L., Fan, M. and Li, S., 2011. Kriging interpolation method and its application in retrieval of MODIS aerosol optical depth. 2011 19th International Conference on Geoinformatics. pp.1–6. Available from: https://doi.org/10.1109/GeoInformatics.2011.5981052. DOI: https://doi.org/10.1109/GeoInformatics.2011.5981052
Zhang, Q., Yuan, Q., Zeng, C., Li, X. and Wei, Y., 2018. Missing Data Reconstruction in Remote Sensing Image With a Unified Spatial–Temporal–Spectral Deep Convolutional Neural Network. IEEE Transactions on Geoscience and Remote Sensing, 56(8), pp.4274–4288. Available from: https://doi.org/10.1109/TGRS.2018.2810208. DOI: https://doi.org/10.1109/TGRS.2018.2810208
Zhang, X., Zhang, T., Wang, G., Zhu, P., Tang, X., Jia, X. and Jiao, L., 2023. Remote Sensing Object Detection Meets Deep Learning: A metareview of challenges and advances. IEEE Geoscience and Remote Sensing Magazine, 11(4), pp.8–44. Available from: https://doi.org/10.1109/MGRS.2023.3312347. DOI: https://doi.org/10.1109/MGRS.2023.3312347
Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J., 2017. Pyramid Scene Parsing Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, pp.6230–6239. Available from: https://doi.org/10.1109/CVPR.2017.660. DOI: https://doi.org/10.1109/CVPR.2017.660
Zhu, J., Park, T., Isola, P. and Efros, A.A., 2017. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, pp.2242–2251. Available from: https://doi.org/10.1109/ICCV.2017.244. DOI: https://doi.org/10.1109/ICCV.2017.244
Zhu, X.X., Montazeri, S., Ali, M., Hua, Y., Wang, Y., Mou, L., Shi, Y., Xu, F. and Bamler, R., 2021. Deep Learning Meets SAR: Concepts, models, pitfalls, and perspectives. IEEE Geoscience and Remote Sensing Magazine, 9(4), pp.143–172. Available from: https://doi.org/10.1109/MGRS.2020.3046356. DOI: https://doi.org/10.1109/MGRS.2020.3046356