Enhancing Soybean Fertilization Optimization with Prioritized Experience Replay and Noisy Networks in Deep Q-Networks
DOI:
https://doi.org/10.26555/jiteki.v11i2.30690Keywords:
Deep Q-Network, Fertilizatio, Noisy Network, Prioritized Experience Replay, Precision farming, Soil nutrient managementAbstract
This study focuses on the optimization of reinforcement learning in the Deep Q Network algorithm. This is achieved using the prioritized experience replay algorithm and Noisy Network optimization. The main goal is to optimize fertilization so that it can adapt to its environment and avoid over-fertilization. This study uses the prioritized experience replay algorithm and Noisy Network optimization to create an agent in RL that is able to explore and exploit optimally so that it can improve the precision of fertilization in soybeans. This methodology includes several steps, including data preparation, creating an environment that matches real-world conditions, and validating changes in soil nutrient conditions. The RL model was trained with PER and NN, with performance evaluated using cumulative reward, convergence speed, action distribution, and Mean Squared Error (MSE). The main results of the study show that DQN-PER NN achieves the highest cumulative reward, approaching 600,000 in 1000 episodes, outperforming standard DQN, A2C, and PPO. It also converges faster at episode 230, indicating superior adaptability. In addition, the results of this study indicate that the model that has been created is able to recommend a dose of SP36 fertilizer of 150 kg/ha, urea fertilizer of 100 kg/ha, and KCL fertilizer of 125 kg/ha. Compared with the A2C and PPO methods, the dose of urea fertilizer is reduced by 14%, KCL fertilizer is reduced by 33%, while for SP36 the difference is 23%. In Conclusion this model effectively distributes actions based on environmental conditions, which supports sustainable agriculture. In conclusion, the integration of PER and NN into DQN significantly improves exploration and decision making, and optimizes soybean fertilization. This model not only improves harvest efficiency but also encourages sustainable agricultural practices.
References
[1] M. N. O. Sadiku, T. J. Ashaolu, and S. M. Musa, “International Journal of Scientific Advances Big Data in Agriculture”, [Online]. Available: www.ijscia.com.
[2] M. Kijima and K. Fujita, “Effectiveness of Reward Functions for Deep Reinforcement Learning in Chick-Feeding System,” in 15th International Congress on Advanced Applied Informatics Winter (IIAI-AAI-Winter), pp. 236–241, 2023, https://doi.org/10.1109/IIAI-AAI-Winter61682.2023.00051.
[3] H. Elver. Global Food Systems in Crisis. pp. 332–347, 2024, https://doi.org/10.4324/9781032699028-24.
[4] F. Kahar, I. Fadah, E. Wahyudi, and A. Wardhono, “Does Soybean in Indonesia Still Competitive?: A Policy Analysis Matrix Approach,” Jurnal Ekonomi dan Studi Pembangunan, vol. 15, p. 89, Nov. 2023, https://doi.org/10.17977/um002v15i22023p089.
[5] F. Hellal and M. Abdelhamid, “Nutrient Management practices for enhancing Soybean (Glycine max L.) production,” Acta Biolo Colomb, vol. 18, pp. 3–14, May 2013. https://dialnet.unirioja.es/servlet/articulo?codigo=9770226.
[6] H. Shaaibu and H. M. Rabiu, “Effects Of Npk And Poultry Manure Rates On Growth Performance And Yield Of Soybean (Glycine Max L. Merril),” Fudma Journal Of Sciences, vol. 7, no. 5, pp. 64–70, Oct. 2023, https://doi.org/10.33003/fjs-2023-0705-1228.
[7] O. Cahyono and S. Minardi, “Effect of Fast Dissolved Phosphorus Fertilizer on the Growth, Seed Product, and Phosphorus Uptake Efficiency of Soybean (Glycine max L.),” Agrivita, vol. 44, no. 1, pp. 21–30, 2022, https://doi.org/10.17503/agrivita.v44i1.3002.
[8] U. Mutammimah, S. Minardi, S. Suryono, O. Cahyono, and S. Sudadi, “Efektivitas Pupuk Organik pada Nitrogen, Fosfor, dan Produksi Kedelai di Tanah Masam,” Agrista: Jurnal Ilmiah Mahasiswa Agribisnis UNS, vol. 4, no. 1, pp. 221–230, 2020, https://www.neliti.com/publications/365797/efektivitas-pupuk-organik-pada-nitrogen-fosfor-dan-produksi-kedelai-di-tanah-mas.
[9] G. Giusti, Y. María, and G. Almeida, “Environmental impacts assessment of maize, soybean, and wheat production in the southwest of São Paulo State: alternative scenarios for the substitution of chemical fertilization,” Revista Engenharia na Agricultura - REVENG, vol. 30, pp. 328–346, Oct. 2022, https://doi.org/10.13083/reveng.v30i1.13864.
[10] N. Thenveettil, R. Bheemanahalli, K. Reddy, W. Gao, and K. Reddy, “Temperature and elevated CO2 alter soybean seed yield and quality, exhibiting transgenerational effects on seedling emergence and vigor,” Front Plant Sci, vol. 15, Jul. 2024, https://doi.org/10.3389/fpls.2024.1427086.
[11] E. Lopez, S. Alvarez Prado, J. Rotundo, and J. Gerde, “Unravelling the environmental drivers determining the residual fraction of soybean seed,” Crop Sci, vol. 65, no.1, p. e21406 Nov. 2024, https://doi.org/10.1002/csc2.21406.
[12] N. Vennampally et al., “Envirotype-based delineation of environmental effects and genotype × environment interactions in Indian soybean (Glycine max, L.),” Sci Rep, vol. 14, May 2024, https://doi.org/10.1038/s41598-024-62613-y.
[13] Y. Dong, Z. Fu, Y. Peng, Y. Zheng, H. Yan, and L. Xinxing, “Precision fertilization method of field crops based on the Wavelet-BP neural network in China,” J Clean Prod, vol. 246, p. 118735, Oct. 2019, https://doi.org/10.1016/j.jclepro.2019.118735.
[14] M. Chen et al., “A reinforcement learning approach to irrigation decision-making for rice using weather forecasts,” Agric Water Manag, vol. 250, p. 106838, 2021, https://doi.org/https://doi.org/10.1016/j.agwat.2021.106838.
[15] R. Tao et al., Optimizing Crop Management with Reinforcement Learning and Imitation Learning. 2023. https://doi.org/10.24963/ijcai.2023/691.
[16] M. Madondo et al., “A SWAT-based Reinforcement Learning Framework for Crop Management,” arXiv preprint, 2023. [Online]. Available: https://arxiv.org/abs/2302.04988.
[17] Md. A. Jabed and M. A. Azmi Murad, “Crop yield prediction in agriculture: A comprehensive review of machine learning and deep learning approaches, with insights for future research and sustainability,” Heliyon, vol. 10, no. 24, p. e40836, 2024, https://doi.org/10.1016/j.heliyon.2024.e40836.
[18] J. Liu, L. Lu, X. Zong, and B. Xie, “Nonlinear relationships in soybean commodities Pairs trading-test by deep reinforcement learning,” Financ Res Lett, vol. 58, p. 104477, 2023, https://doi.org/10.1016/j.frl.2023.104477.
[19] M. Vogt and B. Rosman, “Analyzing Reinforcement Learning Algorithms for Nitrogen Fertilizer Management in Simulated Crop Growth,” in Proceedings of 43rd Conference of the South African Institute of Computer Scientists and Information Technologists, vol. 85, pp. 146–157, 2022, https://doi.org/10.29007/1v4x.
[20] M. G. J. Kallenberg, H. Overweg, R. van Bree, and I. N. Athanasiadis, “Nitrogen management with reinforcement learning and crop growth models,” Environmental Data Science, vol. 2, p. e34, 2023, https://www.cambridge.org/core/journals/environmental-data-science/article/nitrogen-management-with-reinforcement-learning-and-crop-growth-models/358749FAFAA4990B1448DAB7F48D641C.
[21] Z. Wang, K. Jha, and S. Xiao, “Continual Reinforcement Learning for Intelligent Agricultural Management under Climate Changes,” Computers, Materials and Continua, vol. 81, no. 1, pp. 1319–1336, 2024, https://doi.org/10.32604/cmc.2024.055809.
[22] A. Din, M. Y. Ismail, B. Shah, M. Babar, F. Ali, and S. U. Baig, “A deep reinforcement learning-based multi-agent area coverage control for smart agriculture,” Computers and Electrical Engineering, vol. 101, p. 108089, 2022, https://doi.org/10.1016/j.compeleceng.2022.108089.
[23] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized Experience Replay,” 2016. [Online]. Available: https://arxiv.org/abs/1511.05952.
[24] D. Panda, O. Turner, S. Das, and M. Abusara, “Prioritized experience replay based deep distributional reinforcement learning for battery operation in microgrids,” J Clean Prod, vol. 434, p. 139947, Nov. 2023, https://doi.org/10.1016/j.jclepro.2023.139947.
[25] A. Mitra, G. J. Pappas, and H. Hassani, “Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning,” arXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2301.00944.
[26] K. Asadi, S. Sabach, Y. Liu, O. Gottesman, and R. Fakoor, “TD Convergence: An Optimization Perspective,” arXiv preprint, 2023. [Online]. Available: https://arxiv.org/abs/2306.17750.
[27] S. Han, W. Zhou, J. Lu, J. Liu, and S. Lü, “NROWAN-DQN: A stable noisy network with noise reduction and online weight adjustment for exploration,” Expert Syst Appl, vol. 203, p. 117343, 2022, https://doi.org/10.1016/j.eswa.2022.117343.
[28] A. Choudhari, D. Bhoyar, and W. P. Badole, ”Design of an IoT based Monitoring System Utilizing Deep Reinforcement Learning and Ensemble Learning for Precision Agriculture,” International Conference on Sustainable Computing and Smart Systems (ICSCSS), pp. 845-852, 2024, https://doi.org/10.1109/ICSCSS60660.2024.10625266.
[29] H. Chen et al., “Reinforcement learning-based assimilation of the WOFOST crop model,” Smart Agricultural Technology, vol. 9, p. 100604, 2024, https://doi.org/10.1016/j.atech.2024.100604.
[30] A. Upadhyaya and Rahul, “Optimizing Plant Health with Q-Learning: A Deep Reinforcement Learning Approach,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), pp. 1–5, 2024, https://doi.org/10.1109/ICCCNT61001.2024.10725377.
[31] D. Elavarasan and P. M. D. Vincent, “Crop Yield Prediction Using Deep Reinforcement Learning Model for Sustainable Agrarian Applications,” IEEE Access, vol. 8, pp. 86886–86901, 2020, https://doi.org/10.1109/ACCESS.2020.2992480.
[32] W. Su, M. Gao, X. Gao, X. Zhu, and D. Fang, “Adaptive decision-making with deep Q-network for heterogeneous unmanned aerial vehicle swarms in dynamic environments,” Computers and Electrical Engineering, vol. 119, p. 109621, 2024, https://doi.org/10.1016/j.compeleceng.2024.109621.
[33] Y. Lu, X. Yu, Z. Hu, and X. Wang, “Convolutional neural network combined with reinforcement learning-based dual-mode grey wolf optimizer to identify crop diseases and pests,” Swarm Evol Comput, vol. 94, p. 101874, 2025, https://doi.org/10.1016/j.swevo.2025.101874.
[34] S. Deivasigamani, A. J. M. Rani, R. Natchadalingam, P. Vijayakarthik, G. B. S. Kumar, and P. C. S. Reddy, “Crop Yield Prediction Using Deep Reinforcement Learning,” in 2023 Second International Conference on Trends in Electrical, Electronics, and Computer Engineering (TEECCON), pp. 137–142, 2023, https://doi.org/10.1109/TEECCON59234.2023.10335861.
[35] W.-T. Sung and I. G. T. Isa, “A Deep Reinforcement Learning Approach in IoT-based Smart Plantation Monitoring System,” in 2023 International Automatic Control Conference (CACS), 2023, pp. 1–6. https://doi.org/10.1109/CACS60074.2023.10326165.
[36] C. Zhang, M. Yao, W. Chen, S. Zhang, D. Chen, and Y. Wu, “Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy,” Security and Communication Networks, vol. 2021, 2021, https://doi.org/10.1155/2021/9956773.
[37] R. S. Alonso, I. Sittón-Candanedo, R. Casado-Vara, J. Prieto, and J. M. Corchado, “Deep Reinforcement Learning for the management of Software-Defined Networks in Smart Farming,” in 2020 International Conference on Omni-layer Intelligent Systems (COINS), pp. 1–6, 2020, https://doi.org/10.1109/COINS49042.2020.9191634.
[38] H. M. Yassine, K. Roufaida, V. P. Shkodyrev, M. Abdelhak, L. Zarour, and R. Khaled, “Intelligent Farm Based on Deep Reinforcement Learning for optimal control,” in 2022 International Symposium on iNnovative Informatics of Biskra (ISNIB), pp. 1–6, 2022, https://doi.org/10.1109/ISNIB57382.2022.10076088.
[39] M. Ramicic and A. Bonarini, “Correlation minimizing replay memory in temporal-difference reinforcement learning,” Neurocomputing, vol. 393, pp. 91–100, 2020, https://doi.org/10.1016/j.neucom.2020.02.004.
[40] L. Wang and X. Wang, “Enhanced Deep Reinforcement Learning Strategy for Energy Management in Plug-in Hybrid Electric Vehicles with Entropy Regularization and Prioritized Experience Replay,” Energy Engineering, vol. 121, no. 12, pp. 3953–3979, 2024, https://doi.org/https://doi.org/10.32604/ee.2024.056705.
[41] H. Hassani, S. Nikan, and A. Shami, “Improved exploration–exploitation trade-off through adaptive prioritized experience replay,” Neurocomputing, vol. 614, p. 128836, 2025, https://doi.org/10.1016/j.neucom.2024.128836.
[42] C. Wang et al., “Prioritized sum-tree experience replay TD3 DRL-based online energy management of a residential microgrid,” Appl Energy, vol. 368, p. 123471, 2024, https://doi.org/10.1016/j.apenergy.2024.123471.
[43] M. Paniri, M. B. Dowlatshahi, and H. Nezamabadi-pour, “Ant-TD: Ant colony optimization plus temporal difference reinforcement learning for multi-label feature selection,” Swarm Evol Comput, vol. 64, p. 100892, 2021, https://doi.org/10.1016/j.swevo.2021.100892.
[44] J. Gao, X. Li, W. Liu, and J. Zhao, “Prioritized Experience Replay Method Based on Experience Reward,” in International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), pp. 214–219, 2021, https://doi.org/10.1109/MLISE54096.2021.00045.
[45] J. Park, J. Choi, S. Nah, and D. Kim, “Distributional and hierarchical reinforcement learning for physical systems with noisy state observations and exogenous perturbations,” Eng Appl Artif Intell, vol. 123, p. 106465, 2023, https://doi.org/10.1016/j.engappai.2023.106465.
[46] P. Yadav, A. Mandal, and K. Dhakal, “Comparative Study Of Evaluation Of Soil Fertility Status In Rice Zone, Morang,” Tropical Agroecosystems, vol. 2, no. 1, pp. 12-25, Apr. 2021, https://doi.org/10.26480/taec.01.2021.12.25.
[47] M. A. Budiyanto and M. H. Lubis, “Physical reviews of solar radiation models for estimating global solar radiation in Indonesia,” Energy Reports, vol. 6, pp. 1206–1211, 2020, https://doi.org/10.1016/j.egyr.2020.11.053.
[48] M. A. Raza et al., “Sugarcane/soybean intercropping enhances crop productivity, nutrient uptake, and net economic return with reduced inputs,” Field Crops Res, vol. 314, p. 109428, 2024, https://doi.org/10.1016/j.fcr.2024.109428.
[49] G. Verma et al., “Productivity, soil health, and carbon management index of soybean-wheat cropping system under double zero-tillage and natural-farming based organic nutrient management in north-Indian plains,” Science of The Total Environment, vol. 917, p. 170418, 2024, https://doi.org/10.1016/j.scitotenv.2024.170418c.
[50] Y. Liu et al., “An integrated tool for cost-effectively applying nutrient management practices for corn, soybeans, and wheat,” Science of The Total Environment, vol. 955, p. 177110, 2024, https://doi.org/10.1016/j.scitotenv.2024.177110.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Alfian Fakhrezi, Gelar Budiman, Doan Perdana
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with JITEKI agree to the following terms:
- Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
This work is licensed under a Creative Commons Attribution 4.0 International License
