Novel green-fruit detection algorithm based on D2D framework

Jinmeng Wei, Yanhui Ding, Jie Liu, Muhammad Zakir Ullah, Xiang Yin, Weikuan Jia

Abstract


In the complex orchard environment, the efficient and accurate detection of object fruit is the basic requirement to realize the orchard yield measurement and automatic harvesting. Sometimes it is hard to differentiate between the object fruits and the background because of the similar color, and it is challenging due to the ambient light and camera angle by which the photos have been taken. These problems make it hard to detect green fruits in orchard environments. In this study, a two-stage dense to detection framework (D2D) was proposed to detect green fruits in orchard environments. The proposed model was based on multi-scale feature extraction of target fruit by using feature pyramid networks MobileNetV2 +FPN structure and generated region proposal of target fruit by using Region Proposal Network (RPN) structure. In the regression branch, the offset of each local feature was calculated, and the positive and negative samples of the region proposals were predicted by a binary mask prediction to reduce the interference of the background to the prediction box. In the classification branch, features were extracted from each sub-region of the region proposal, and features with distinguishing information were obtained through adaptive weighted pooling to achieve accurate classification. The new proposed model adopted an anchor-free frame design, which improves the generalization ability, makes the model more robust, and reduces the storage requirements. The experimental results of persimmon and green apple datasets show that the new model has the best detection performance, which can provide theoretical reference for other green object detection.
Keywords: green-fruit detection, D2D framework, automatic harvesting, MobileNetV2+FPN, binary mask prediction, anchor-free
DOI: 10.25165/j.ijabe.20221501.6943

Citation: Wei J M, Ding Y H, Liu J, Ullah M Z, Yin X, Jia W K. Novel green-fruit detection algorithm based on D2D framework. Int J Agric & Biol Eng, 2022; 15(1): 251–259.

Keywords


green-fruit detection, D2D framework, automatic harvesting, MobileNetV2+FPN, binary mask prediction, anchor-free

Full Text:

PDF

References


Jia W, Zhang Y, Lian J, Zheng Y J, Zhao D A, Li C J. Apple harvesting robot under information technology: A review. International Journal of Advanced Robotic Systems, 2020; 17(3): 1729881420925310. doi: 10.1177/179881420925310.

Xiong Y, Ge Y, Grimstad L, From P J. An autonomous

strawberry-harvesting robot: Design, development, integration, and field evaluation. Journal of Field Robotics, 2020; 37(2): 202–224.

Tang Y C, Chen M Y, Wang C L, Luo L F, Li J H, Lian G P, et al. Recognition and localization methods for vision–based fruit picking robots: A review. Frontiers in Plant Science, 2020; 11: 510. doi: 10.3389/ fpls.2020.00510.

Fu L S, Gao F F, Wu J Z, Li R, Karkee M, Zhang Q. Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review. Computers and Electronics in Agriculture, 2020; 177: 105687. doi: 10.1016/j.compag.2020.105687.

Ilea D E, Whelan P F. Image segmentation based on the integration of colour–texture descriptors-A review. Pattern Recognition, 2011; 44(10-11): 2479–2501.

Sharma P, Suji J. A review on image segmentation with its clustering techniques. International Journal of Signal Processing, Image Processing and Pattern Recognition, 2016; 9(5): 209–218.

Jia W K, Zheng Y J, Zhao D A, Yin Xiang, Liu X Y, Du R C. Preprocessing method of night vision image application in apple harvesting robot. Int J Agric & Biol Eng, 2018; 11(2): 158–163.

Arefi A, Motlagh A M, Mollazade K, Teimourlou R F. Recognition and localization of ripen tomato based on machine vision. Australian Journal of Crop Science, 2011; 5(10): 1144–1149.

Linker R, Cohen O, Naor A. Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 2012; 81: 45–57.

Liao W, Zheng L H, Li M Z, Sun H, Yang W. Green apple recognition in natural illuminations based on random forest algorithm. Transactions of the Chinese Society for Agricultural Machinery, 2017; 48(S1): 86–91. (in Chinese)

Tian Y Y, Duan H C, Luo R, Zhang Y, Jia W K, Lian J, et al. Fast recognition and location of target fruit based on depth information. IEEE Access, 2019; 7: 170553–170563.

Li B R, Long Y, Song H B. Detection of green apples in natural scenes based on saliency theory and Gaussian curve fitting. Int J Agric & Biol Eng, 2018; 11(1): 192–198.

Koirala A, Walsh K B, Wang Z L, McCarthy C. Deep learning-method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 2019; 162: 219–234.

Boogaard F P, Rongen K H, Kootstra G W. Robust node detection and tracking in fruit-vegetable crops using deep learning and multi-view imaging. Biosystems Engineering, 2020; 192: 117–132.

Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.

Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.

Jia W K, Tian Y Y, Luo R, Zhang Z H, Lian J, Zheng Y J. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Computers and Electronics in Agriculture, 2020; 172: 105380. doi: 10.1016/j.compag.2020.105380.

Wang D, He D. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network. Transactions of the CSAE, 2019; 35(3): 156–163. (in Chinese)

Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L C. MobilenetV2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Lake City, USA: IEEE, 2018; pp.4510–4520. doi: 10.1109/CVPR.2018.00474.

Lin T Y, Dollár P, Girshick R, He B. Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA: IEEE, 2017; pp.935–944. doi: 10.1109/CVPR.207.106.

Girshick R. Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile: IEEE, 2015; pp.1440–1448.

Cao J, Cholakkal H, Anwer R M, Khan F S, Pang Y, Shao L. D2Det: Towards high quality object detection and instance segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.11485–11494. doi: 10.1109/CVPR42600.2020.01150.

Howard A, Sandler M, Chu G, Chen L C, Chen B, Tan M X, et al. Searching for MobilenetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE, 2019; pp.1314–1324. arXiv: 1905.02244v1.

Zhang Y Q, Chu J, Leng L, Miao J. Mask-refined R-CNN: A network for

refining object details in instance segmentation. Sensors, 2020; 20(4): 1010. doi: 10.3390/s20041010.

Zhang K, Sun M, Han T X, Yuan X F, Guo L R, Liu T. Residual networks of residual networks: Multilevel residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 2017; 28(6): 1303–1314.

Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016; 39(6): 1137–1149.

Tian Z, Shen C H, Chen H, He T. FCOS: Fully Convolutional One-stage Object Detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019; pp. 9627–9636. doi: 10.1109/

ICCV.2019.00972.

Dai J F, Qi H Z, Xiong Y W, Li Y, Zhang G D, Hu H, et al. Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017; pp.764–773. doi: 10.1109/ICCV.2017.89.

He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. 2017; pp.2961–2969.

Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y. SSD: Single shot multibox detector. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Springer, Cham, 2016; 9905: 21–37. doi: 10.1007/978-3-319-46448-0_2.




Copyright (c) 2022 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office