IMPROVING THE METHOD OF ATTACK DETECTION IN THE MQTT SEGMENT OF A SMART HOME IoT NETWORK WITH ADAPTATION TO A NEW NETWORK USING LEGITIMATE TRAFFIC
DOI:
https://doi.org/10.28925/2663-4023.2026.35.1375Keywords:
cybersecurity; Internet of Things; MQTT; intrusion detection; smart home; IoT gateway; cross-network transfer; threshold calibration.Abstract
The MQTT protocol is the main means of message exchange between sensors, actuators and the gateway of a smart home, so attacks on the MQTT broker directly affect the availability and integrity of the IoT system. Known machine-learning methods for detecting such attacks report an accuracy of 99–100% when training and testing are performed on traffic from the same testbed, whereas in practice a detector is deployed in a network whose traffic was not used for training and must operate at a given false alarm rate. The paper improves the method of attack detection in the MQTT segment of an IoT network: it relies on window-level protocol features of broker traffic that do not depend on network identifiers and absolute time, and on normalising the features and the decision threshold by the legitimate traffic of the target network without using its attack labels: the features are normalised by the median and the interquartile range of legitimate calibration windows, and the threshold is set as an order statistic of their scores for the given false alarm rate. The research methodology uses four independent datasets (MQTTset, MQTT-IoT-IDS2020, MQTTEEB-D, MQTT_UAD): reproduction of known approaches under their authors' conditions, evaluation of packet features with a chronological split and in another network, and evaluation of the improved method for 12 "training network – target network" pairs at a false alarm rate of 1%. The improved method achieves a mean attack detection rate of 0.710 (median 0.905) versus 0.532 for threshold-only calibration, 0.549 for a detector trained only on legitimate traffic of the target network and at most 0.124 for packet features; the result holds for window sizes of 5–30 s and calibration on 10–50% of the time. The method is implemented in C as an IoT gateway module that processes traffic in 180–400 ns per packet with a 21 KB model.
Downloads
References
Alasmari, R., & Alhogail, A. A. (2024). Protecting smart-home IoT devices from MQTT attacks: An empirical study of ML-based IDS. IEEE Access, 12, 25993–26004. https://doi.org/10.1109/ACCESS.2024.3367113
Alatram, A., Sikos, L. F., Johnstone, M., Szewczyk, P., & Kang, J. J. (2023). DoS/DDoS-MQTT-IoT: A dataset for evaluating intrusions in IoT networks using the MQTT protocol. Computer Networks, 231, Article 109809. https://doi.org/10.1016/j.comnet.2023.109809
Apruzzese, G., Laskov, P., & Schneider, J. (2023). SoK: Pragmatic assessment of machine learning for network intrusion detection. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) (pp. 592–614). IEEE. https://doi.org/10.1109/EuroSP57164.2023.00042
Aqachtoul, A., Karam, K., Elamrani, A., Najib, M., Rafalia, N., & Bakhouya, M. (2025). MQTTEEB-D: A real-world IoT cybersecurity dataset for AI-powered threat detection in MQTT networks. Data in Brief, 62, Article 111897. https://doi.org/10.1016/j.dib.2025.111897
Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and don’ts of machine learning in computer security. In Proceedings of the 31st USENIX Security Symposium (pp. 3971–3988). USENIX Association.
Aveleira-Mata, J., Alaiz-Moretón, H., Bayón-Gutiérrez, M., García-Ordás, M. T., Prieto-Fernández, N., & García-Rodríguez, I. (2025). MQTT_UAD: MQTT Under Attack Dataset. A public dataset for the detection of attacks in IoT networks using MQTT protocol. Data in Brief, 63, Article 112167. https://doi.org/10.1016/j.dib.2025.112167
Cantone, M., Marrocco, C., & Bria, A. (2024). Machine learning in network intrusion detection: A cross-dataset generalization study. IEEE Access, 12, 144489–144508. https://doi.org/10.1109/ACCESS.2024.3472907
Cerasuolo, F., Bovenzi, G., & Pescapè, A. (2026). Cross-network transferability of AI-based network intrusion detection systems in heterogeneous Internet of Things environments. Computer Networks, 286, Article 112434. https://doi.org/10.1016/j.comnet.2026.112434
Dadkhah, S., Neto, E. C. P., Ferreira, R., Molokwu, R. C., Sadeghi, S., & Ghorbani, A. A. (2024). CICIoMT2024: A benchmark dataset for multi-protocol security assessment in IoMT. Internet of Things, 28, Article 101351. https://doi.org/10.1016/j.iot.2024.101351
Flood, R., Engelen, G., Aspinall, D., & Desmet, L. (2024). Bad design smells in benchmark NIDS datasets. In 2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P) (pp. 658–675). IEEE. https://doi.org/10.1109/EuroSP60621.2024.00042
Hindy, H., Bayne, E., Bures, M., Atkinson, R., Tachtatzis, C., & Bellekens, X. (2021). Machine learning based IoT intrusion detection system: An MQTT case study (MQTT-IoT-IDS2020 dataset). In Selected papers from the 12th International Networking Conference (Lecture Notes in Networks and Systems, Vol. 180, pp. 73–84). Springer. https://doi.org/10.1007/978-3-030-64758-2_6
Imran, Zuhairi, M. F., Ali, S. M., Shahid, Z., Alam, M. M., & Su’ud, M. M. (2024). Realtime feature engineering for anomaly detection in IoT based MQTT networks. IEEE Access, 12, 25700–25718. https://doi.org/10.1109/ACCESS.2024.3363889
Khan, M. A., Khan, M. A., Jan, S. U., Ahmad, J., Jamal, S. S., Shah, A. A., Pitropakis, N., & Buchanan, W. J. (2021). A deep learning-based intrusion detection system for MQTT enabled IoT. Sensors, 21(21), Article 7016. https://doi.org/10.3390/s21217016
KouhiRonaghi, A., & Habibi Lashkari, A. (2026). MQTTFlowLyzer: Interpretable TabNet-based flow-level MQTT intrusion detection for IoT. The Journal of Supercomputing, 82(5), Article 334. https://doi.org/10.1007/s11227-026-08485-7
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining (pp. 413–422). IEEE. https://doi.org/10.1109/ICDM.2008.17
Nguyen, X.-H., Nguyen, X.-D., Huynh, H.-H., & Le, K.-H. (2022). Realguard: A lightweight network intrusion detection system for IoT gateways. Sensors, 22(2), Article 432. https://doi.org/10.3390/s22020432
Omotosho, A., Qendah, Y., & Hammer, C. (2023). IDS-MA: Intrusion detection system for IoT MQTT attacks using centralized and federated learning. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) (pp. 678–688). IEEE. https://doi.org/10.1109/COMPSAC57700.2023.00093
Roy, U. S., & Minu, M. J. (2026). Unmasking shortcut learning in IoT intrusion detection: A forensic, multi-paradigm evaluation of feature dependence and data leakage [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2609.28725
Ullah, S., Ahmad, J., Khan, M. A., Alshehri, M. S., Boulila, W., Koubaa, A., Jan, S. U., & Iqbal Ch, M. M. (2023). TNN-IDS: Transformer neural network-based intrusion detection system for MQTT-enabled IoT networks. Computer Networks, 237, Article 110072. https://doi.org/10.1016/j.comnet.2023.110072
Vaccari, I., Chiola, G., Aiello, M., Mongelli, M., & Cambiaso, E. (2020). MQTTset, a new dataset for machine learning techniques on MQTT. Sensors, 20(22), Article 6578. https://doi.org/10.3390/s20226578
Nishchemenko, D., & Voloshchuk, O. (2025). Adaptive cleaning of heterogeneous sensor data in smart home systems based on noise classification. Cybersecurity: Education, Science, Technique, 4(28), 740–750. https://doi.org/10.28925/2663-4023.2025.28.844
Zhebka, V. V., Aronov, A. O., Nishchemenko, D. O., & Ananchenko, O. Ye. (2026). Improvement of the multi-objective optimization method for controlling a smart home IoT system, considering cyber risk. Cybersecurity: Education, Science, Technique, 2(34), 820–830. https://doi.org/10.28925/2663-4023.2026.34.1389
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Дмитро Ніщеменко , Наталія Сало, Катерина Нестеренко, Сергій Мамченко
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
