Adaptive Cooling System Control in Data Center with Reinforcement Learning
DOI:
https://doi.org/10.26555/jiteki.v11i1.30671Keywords:
Reinforcement Learning, Data Center, Monitoring, Flask, Energy Efficiency, Thermal OptimizationAbstract
Data center cooling system is consuming large amounts of power, which requires effective control to reduce operational costs and deliver optimal server performance. The high power consumption occurs because traditional cooling methods struggle to adapt dynamically to workloads, causing wasteful power consumption. Therefore, this study aimed to explore the use of machine learning methods to improve energy efficiency for data center cooling system. For the experiment, an RL (Reinforcement Learning) model was designed to adjust cooling parameters with dynamic environmental changes. The method focused on optimizing energy efficiency while maintaining stable temperature and humidity control. By applying RL-based control method to PAC system, this study contributed original results that validated the effectiveness of RL-simulated data center environments. Specifically, the stages included developing system model, creating simulations using the PAC control system, and training an RL model with environmental conditions. Data were collected from simulations and analyzed to test the model performance, and the outcomes were presented using a real-time monitoring interface with Flask. The results showed that the RL model achieved an average reward of 4.76 (between -5 and 5), a convergence rate 13.2, a sampling efficiency 10.15, and a stability score 2.6. The model effectively reduced temperature and increased humidity during stressed data center operations. When compared with a fixed cooling system, RL showed superior adaptability to workload variations and reduced unnecessary energy consumption. However, scalability to real data center remained an issue, which required more than simulation validation. In conclusion, the RL-based method optimized efficiency of cooling system, showing the potential to improve energy savings and operational resilience in data center environments.
References
[1] S. Ketabi, H. Chen, H. Dong, and Y. Ganjali, “A Deep Reinforcement Learning Framework for Optimizing Congestion Control in Data Centers,” in NOMS 2023-2023 IEEE/IFIP Network Operations and Management Symposium, pp. 1-7, 2023, https://doi.org/10.48550/arXiv.2301.12558.
[2] M. Yenugula, “Data center power management using neural network,” International Journal of Advanced Academic Studies, vol. 3, pp. 320–325, Jan. 2021, https://doi.org/10.33545/27068919.2021.v3.i1d.1124.
[3] R. Gunawan, T. Andhika, . S., and F. Hibatulloh, “Monitoring System for Soil Moisture, Temperature, pH and Automatic Watering of Tomato Plants Based on Internet of Things,” Telekontran : Jurnal Ilmiah Telekomunikasi, Kendali dan Elektronika Terapan, vol. 7, no. 1, pp. 66–78, Apr. 2019, https://doi.org/10.34010/telekontran.v7i1.1640.
[4] K. Bilal et al., “A Comparative Study Of Data Center Network Architectures,” 26th EUROPEAN conference on modelling and simulation, ECMS, May 2012, https://doi.org/10.7148/2012-0526-0532.
[5] C. Blad, S. Bøgh, and C. S. Kallesøe, “Data-driven Offline Reinforcement Learning for HVAC-systems,” Energy, vol. 261, p. 125290, 2022, https://doi.org/10.1016/j.energy.2022.125290.
[6] M. Biemann, F. Scheller, X. Liu, and L. Huang, “Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control,” Appl Energy, vol. 298, p. 117164, 2021, https://doi.org/10.1016/j.apenergy.2021.117164.
[7] S. Wassermann, T. Cuvelier, P. Mulinka, and P. Casas, “Adaptive and Reinforcement Learning Approaches for Online Network Monitoring and Analysis,” IEEE Transactions on Network and Service Management, vol. 18, no. 2, pp. 1832–1849, 2021, https://doi.org/10.1109/TNSM.2020.3037486.
[8] T. A. Nakabi and P. Toivanen, “Deep reinforcement learning for energy management in a microgrid with flexible demand,” Sustainable Energy, Grids and Networks, vol. 25, p. 100413, 2021, https://doi.org/10.1016/j.segan.2020.100413.
[9] H. Che, Z. Bai, R. Zuo, and H. Li, “A Deep Reinforcement Learning Approach to the Optimization of Data Center Task Scheduling,” Complexity, vol. 2020, pp. 1–12, Aug. 2020, https://doi.org/10.1155/2020/3046769.
[10] H. Li et al., “Modeling the Relationship Between Air Conditioning Load and Temperature Based on Machine Learning,” in 4th International Conference on Intelligent Control, Measurement and Signal Processing (ICMSP), pp. 524–528, 2022, https://doi.org/10.1109/ICMSP55950.2022.9859169.
[11] N. Sulaiman, M. P. Abdullah, H. Abdullah, M. Zainudin, and A. Yusop, “Fault detection for air conditioning system using machine learning,” IAES International Journal of Artificial Intelligence (IJ-AI), vol. 9, p. 109, Mar. 2020, https://doi.org/10.11591/ijai.v9.i1.pp109-116.
[12] J. Hao, D. W. Gao, and J. J. Zhang, “Reinforcement Learning for Building Energy Optimization Through Controlling of Central HVAC System,” IEEE Open Access Journal of Power and Energy, vol. 7, pp. 320–328, 2020, https://doi.org/10.1109/OAJPE.2020.3023916.
[13] L. Yu, D. Xie, C. Huang, T. Jiang, and Y. Zou, “Energy Optimization of HVAC Systems in Commercial Buildings Considering Indoor Air Quality Management,” IEEE Trans Smart Grid, vol. 10, no. 5, pp. 5103-5113, Oct. 2018, https://doi.org/10.1109/TSG.2018.2875727.
[14] A. Chatterjee and D. Khovalyg, “Dynamic indoor thermal environment using Reinforcement Learning-based controls: Opportunities and challenges,” Build Environ, vol. 244, p. 110766, 2023, https://doi.org/10.1016/j.buildenv.2023.110766.
[15] C. Chen et al., “Deep Reinforcement Learning-Based Joint Optimization Control of Indoor Temperature and Relative Humidity in Office Buildings,” Buildings, vol. 13, p. 438, Feb. 2023, https://doi.org/10.3390/buildings13020438.
[16] X. Zhong, Z. Zhang, R. ZHANG, and C. Zhang, “End-to-End Deep Reinforcement Learning Control for HVAC Systems in Office Buildings,” Designs (Basel), vol. 6, p. 52, Jun. 2022, https://doi.org/10.3390/designs6030052.
[17] C. Zhou et al., “Simulator-Based Reinforcement Learning for Data Center Cooling Optimization,” In Deployable RL: From Research to Practice@ Reinforcement Learning Conference, 2024, https://openreview.net/forum?id=3hZL9Vv0Ay.
[18] Y. Peng et al., “Energy Consumption Optimization for Heating, Ventilation and Air Conditioning Systems Based on Deep Reinforcement Learning,” IEEE Access, vol. 11, pp. 88265–88277, 2023, https://doi.org/10.1109/ACCESS.2023.3305683.
[19] T. Bian and Z.-P. Jiang, “Reinforcement Learning and Adaptive Optimal Control for Continuous-Time Nonlinear Systems: A Value Iteration Approach,” IEEE Trans Neural Netw Learn Syst, vol. 33, no. 7, pp. 2781–2790, 2022, https://doi.org/10.1109/TNNLS.2020.3045087.
[20] Y. Wang, M. Xiao, and Z. Wu, “Safe Transfer-Reinforcement-Learning-Based Optimal Control of Nonlinear Systems,” IEEE Trans Cybern, vol. 54, no. 12, pp. 7272–7284, 2024, https://doi.org/10.1109/TCYB.2024.3485697.
[21] O. Al-Ani and S. Das, “Reinforcement Learning: Theory and Applications in HEMS,” Energies, vol. 15, no. 17, p. 6392, 2022, https://doi.org/10.3390/en15176392.
[22] X. Xu, Y. Jia, Y. Xu, Z. Xu, S. Chai, and C. S. Lai, “A Multi-Agent Reinforcement Learning-Based Data-Driven Method for Home Energy Management,” IEEE Trans Smart Grid, vol. 11, no. 4, pp. 3201–3211, 2020, https://doi.org/10.1109/TSG.2020.2971427.
[23] Y. Wang, Y. Sun, B. Cheng, G. Jiang, and H. Zhou, “DQN-Based Chiller Energy Consumption Optimization in IoT-Enabled Data Center,” in IEEE 23rd International Conference on Communication Technology (ICCT), pp. 985–990, 2023, https://doi.org/10.1109/ICCT59356.2023.10419683.
[24] Q. Zhang, C.-B. Chng, K. Chen, P.-S. Lee, and C.-K. Chui, “DRL-S: Toward safe real-world learning of dynamic thermal management in data center,” Expert Syst Appl, vol. 214, p. 119146, 2023, https://doi.org/10.1016/j.eswa.2022.119146.
[25] Y. Ran, H. Hu, Y. Wen, and X. Zhou, “Optimizing Energy Efficiency for Data Center via Parameterized Deep Reinforcement Learning,” IEEE Trans Serv Comput, vol. 16, no. 2, pp. 1310–1323, 2023, https://doi.org/10.1109/TSC.2022.3184835.
[26] G. Obaido et al., “Supervised machine learning in drug discovery and development: Algorithms, applications, challenges, and prospects,” Machine Learning with Applications, vol. 17, p. 100576, 2024, https://doi.org/10.1016/j.mlwa.2024.100576.
[27] Q. Zhang, M. H. B. Mahbod, C.-B. Chng, P.-S. Lee, and C.-K. Chui, “Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method,” IEEE Trans Cybern, vol. 54, no. 2, pp. 865–876, 2024, https://doi.org/10.1109/TCYB.2022.3178084.
[28] Z. Cao, R. Wang, X. Zhou, and Y. Wen, “Toward Model-Assisted Safe Reinforcement Learning for Data Center Cooling Control: A Lyapunov-based Approach,” In Proceedings of the 14th ACM International Conference on Future Energy Systemspp,pp. 333-346, 2023, https://doi.org/10.1145/3575813.3597343.
[29] G. Wei, M. Chi, Z.-W. Liu, M. Ge, C. Li, and X. Liu, “Deep Reinforcement Learning for Real-Time Energy Management in Smart Home,” IEEE Syst J, vol. 17, no. 2, pp. 2489–2499, 2023, https://doi.org/10.1109/JSYST.2023.3247592.
[30] T. Hua, J. Wan, S. Jaffry, Z. Rasheed, L. Li, and Z. Ma, “Comparison of Deep Reinforcement Learning Algorithms in Data Center Cooling Management: A Case Study,” in IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 392–397, 2021, https://doi.org/10.1109/SMC52423.2021.9659100.
[31] Z. Chen, J. Hu, G. Min, C. Luo, and T. El-Ghazawi, “Adaptive and Efficient Resource Allocation in Cloud Datacenters Using Actor-Critic Deep Reinforcement Learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 8, pp. 1911–1923, 2022, https://doi.org/10.1109/TPDS.2021.3132422.
[32] X. Wang et al., “Deep Reinforcement Learning: A Survey,” IEEE Trans Neural Netw Learn Syst, vol. 35, no. 4, pp. 5064–5078, 2024, https://doi.org/10.1109/TNNLS.2022.3207346.
[33] D. Lee, S. Koo, I. Jang, and J. Kim, “Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation,” Energies (Basel), vol. 15, p. 2834, Apr. 2022, https://doi.org/10.3390/en15082834.
[34] L. Yu et al., “Deep Reinforcement Learning for Smart Home Energy Management,” IEEE Internet Things J, p. 1, Dec. 2019, https://doi.org/10.1109/JIOT.2019.2957289.
[35] Y. Li, C. Yu, M. Shahidehpour, T. Yang, Z. Zeng, and T. Chai, “Deep Reinforcement Learning for Smart Grid Operations: Algorithms, Applications, and Prospects,” Proceedings of the IEEE, vol. 111, no. 9, pp. 1055–1096, 2023, https://doi.org/10.1109/JPROC.2023.3303358.
[36] M. Otterlo and M. Wiering, “Reinforcement Learning and Markov Decision Processes,” Reinforcement Learning: State of the Art, pp. 3–42, Jan. 2012, https://doi.org/10.1007/978-3-642-27645-3_1.
[37] S. El Hamdani, S. Loudari, S. Novotny, P. Bouchner, and N. Benamar, “A Markov Decision Process Model for a Reinforcement Learning-based Autonomous Pedestrian Crossing Protocol,” in 3rd IEEE Middle East and North Africa COMMunications Conference (MENACOMM), pp. 147–151, 2021, https://doi.org/10.1109/MENACOMM50742.2021.9678310.
[38] W. Zhan et al., “Deep-Reinforcement-Learning-Based Offloading Scheduling for Vehicular Edge Computing,” IEEE Internet Things J, vol. 7, no. 6, pp. 5449–5465, 2020, https://doi.org/10.1109/JIOT.2020.2978830.
[39] T. T. Nguyen and V. J. Reddi, “Deep Reinforcement Learning for Cyber Security,” IEEE Trans Neural Netw Learn Syst, vol. 34, no. 8, pp. 3779–3795, 2023, https://doi.org/10.1109/TNNLS.2021.3121870.
[40] H. Yang et al., “Deep Reinforcement Learning Based Intelligent Reflecting Surface for Secure Wireless Communications,” in GLOBECOM IEEE Global Communications Conference, pp. 1–6, 2020, https://doi.org/10.1109/GLOBECOM42002.2020.9322615.
[41] A. Singh, R. Akash, and G. R. V, “Flower Classifier Web App Using Ml & Flask Web Framework,” in 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), pp. 974–977, 2022, https://doi.org/10.1109/ICACITE53722.2022.9823577.
[42] C. Wang, Z. Wang, D. Dong, X. Zhang, and Z. Zhao, “A Novel Reinforcement Learning Framework for Adaptive Routing in Network-on-Chips,” in IEEE 23rd Int Conf on High Performance Computing & Communications; 7th Int Conf on Data Science & Systems; 19th Int Conf on Smart City; 7th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 336–344, 2021, https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys53884.2021.00069.
[43] J. G. Sukumar, M. S. R. Reddy, N. Sambangi, S. Abhishek, and A. T, “Enhancing salary projections: a supervised machine learning approach with flask deployment,” in 5th International Conference on Inventive Research in Computing Applications (ICIRCA), pp. 693–700, 2023, https://doi.org/10.1109/ICIRCA57980.2023.10220707.
[44] S. Gros and M. Zanon, “Data-Driven Economic NMPC Using Reinforcement Learning,” IEEE Trans Automat Contr, vol. 65, no. 2, pp. 636–648, 2020, https://doi.org/10.1109/TAC.2019.2913768.
[45] Q. Chou, W. Fan, and J. Zhang, “A Reinforcement Learning Model for Virtual Machines Consolidation in Cloud Data Center,” in 6th International Conference on Automation, Control and Robotics Engineering (CACRE), pp. 16–21, 2021, https://doi.org/10.1109/CACRE52464.2021.9501288.
[46] M. Duggan, K. Flesk, and E. Howley, “A Reinforcement Learning Approach for Dynamic Selection of Virtual Machines in Cloud Data Centres,” In 2016 sixth international conference on innovative computing technology (INTECH), pp. 92-97, 2016, https://doi.org/10.1109/INTECH.2016.7845053.
[47] R. Wang and M. Bonney, “Novel Data Acquisition Utilising a Flask Python Digital Twin Operational Platform,” In Special Topics in Structural Dynamics & Experimental Techniques, Volume 5: Proceedings of the 40th IMAC, A Conference and Exposition on Structural Dynamics, pp. 7–13, 2023, https://doi.org/10.1007/978-3-031-05405-1_2.
[48] G. Schäfer, M. Schirl, J. Rehrl, S. Huber, and S. Hirlaender, “Python-Based Reinforcement Learning on Simulink Models,” In International Conference on Soft Methods in Probability and Statistics, pp. 449–456, 2024, https://doi.org/10.1007/978-3-031-65993-5_55.
[49] H. Zhou, K. Jiang, X. Liu, X. Li, and V. C. M. Leung, “Deep Reinforcement Learning for Energy-Efficient Computation Offloading in Mobile-Edge Computing,” IEEE Internet Things J, vol. 9, no. 2, pp. 1517–1530, 2022, https://doi.org/10.1109/JIOT.2021.3091142.
[50] S. Sierla, H. Ihasalo, and V. Vyatkin, “A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems,” Energies (Basel), vol. 15, p. 3526, May 2022, https://doi.org/10.3390/en15103526.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Ericha Septya Dinata, Sofia Naning Hertiana, Erna Sri Sugesti
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with JITEKI agree to the following terms:
- Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
This work is licensed under a Creative Commons Attribution 4.0 International License
