MODEL AND ALGORITHM OF AN ADAPTIVE INTELLIGENT AGENT FOR URBAN DELIVERY OPTIMIZATION IN A WEB-ORIENTED DYNAMIC ENVIRONMENT
DOI:
https://doi.org/10.28925/2663-4023.2026.35.1332Keywords:
intelligent agent; urban logistics; dynamic routing; web-oriented system; Q-learning; Markov decision process; simulation modelling.Abstract
The article investigates a mathematical model and an adaptive intelligent-agent algorithm for automated route selection in urban delivery under changing traffic conditions. The urban logistics system is represented as a web-oriented environment in which order data, current road congestion and incident information are transmitted through REST APIs to a server-side agent module that can be implemented in Python. The route-selection problem is formalized as a non-stationary Markov decision process. The reward function accounts for travel duration, late delivery, road incidents and route-switching costs. Q-learning is enhanced by a window-based detector of declining mean reward, temporary increases in exploration and learning rates, and partial discounting of accumulated Q-values. A simulation study covered 300 decision cycles, three consecutive phases of changing road conditions and 200 independent runs. A fixed policy, standard Q-learning and the proposed adaptive variant were compared. The adaptive agent achieved an on-time delivery rate of 80.0%, an average travel time of 24.55 minutes and an integral adaptability index of 0.707. The corresponding values were 76.8%, 25.22 minutes and 0.670 for standard Q-learning, and 50.5%, 29.41 minutes and 0.426 for the fixed policy. The mean policy stabilization time after an environmental change decreased from 36.9 to 25.6 cycles. The results show that change detection and controlled partial forgetting improve the agent’s robustness to traffic drift. The proposed model and algorithm can serve as a basis for a web-based decision-support system for urban logistics. The comparison further demonstrates that the proposed mechanism is most effective after repeated environmental changes, when previously learned action values no longer correspond to current traffic conditions.
Keywords: intelligent agent; urban logistics; dynamic routing; web-oriented system; Q-learning; Markov decision process; simulation modelling.
Downloads
References
Mardešić, N., Erdelić, T., Carić, T., & Đurasević, M. (2024). Review of stochastic dynamic vehicle routing in the evolving urban logistics environment. Mathematics, 12(1), Article 28. https://doi.org/10.3390/math12010028
Aguayo, O., Sepúlveda, S., & Mazo, R. (2024). Variability management in self-adaptive systems through deep learning: A dynamic software product line approach. Electronics, 13(5), Article 905. https://doi.org/10.3390/electronics13050905
Alfonso, I., Garcés, K., Castro, H., & Cabot, J. (2021). Self-adaptive architectures in IoT systems: A systematic literature review. Journal of Internet Services and Applications, 12, Article 14. https://doi.org/10.1186/s13174-021-00145-8
Gheibi, O., Weyns, D., & Quin, F. (2021). Applying machine learning in self-adaptive systems: A systematic literature review. ACM Transactions on Autonomous and Adaptive Systems, 15(3), Article 9, 1–37. https://doi.org/10.1145/3469440
Mahdavi-Hezavehi, S. M., Weyns, D., Avgeriou, P., Calinescu, R., Mirandola, R., & Perez-Palacin, D. (2021). Uncertainty in self-adaptive systems: A research community perspective. ACM Transactions on Autonomous and Adaptive Systems, 15(4), Article 10, 1–36. https://doi.org/10.1145/3487921
Hildebrandt, F. D., Thomas, B. W., & Ulmer, M. W. (2023). Opportunities for reinforcement learning in stochastic dynamic vehicle routing. Computers & Operations Research, 150, Article 106071. https://doi.org/10.1016/j.cor.2022.106071
Pan, W., & Liu, S. Q. (2023). Deep reinforcement learning for the dynamic and uncertain vehicle routing problem. Applied Intelligence, 53, 405–422. https://doi.org/10.1007/s10489-022-03456-w
Zhou, C., Ma, J., Douge, L., Chew, E. P., & Lee, L. H. (2023). Reinforcement learning-based approach for dynamic vehicle routing problem with stochastic demand. Computers & Industrial Engineering, 182, Article 109443. https://doi.org/10.1016/j.cie.2023.109443
Konovalenko, A., & Hvattum, L. M. (2024). Optimizing a dynamic vehicle routing problem with deep reinforcement learning: Analyzing state-space components. Logistics, 8(4), Article 96. https://doi.org/10.3390/logistics8040096
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023). Reinforcement learning algorithms: A brief survey. Expert Systems with Applications, 231, Article 120495. https://doi.org/10.1016/j.eswa.2023.120495
Hinder, F., Vaquet, V., & Hammer, B. (2024a). One or two things we know about concept drift–A survey on monitoring in evolving environments. Part A: Detecting concept drift. Frontiers in Artificial Intelligence, 7, Article 1330257. https://doi.org/10.3389/frai.2024.1330257
Hinder, F., Vaquet, V., & Hammer, B. (2024b). One or two things we know about concept drift–A survey on monitoring in evolving environments. Part B: Locating and explaining concept drift. Frontiers in Artificial Intelligence, 7, Article 1330258. https://doi.org/10.3389/frai.2024.1330258
Metzger, A., Quinton, C., Mann, Z. Á., Baresi, L., & Pohl, K. (2024). Realizing self-adaptive systems via online reinforcement learning and feature-model-guided exploration. Computing, 106(4), 1251–1272. https://doi.org/10.1007/s00607-022-01052-x
Al-Shaikh, R. Z., Al-Nayar, M. M. J., & Hasan, A. M. (2025). Reinforcement learning algorithms for adaptive load-balancing for web applications. South African Computer Journal, 37(2), 121–145. https://doi.org/10.18489/sacj.v37i2.20753
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Анна Андрух, Ігор Тищенко
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
