ARCHITECTURAL APPROACHES TO APPLYING LANGUAGE MODELS FOR DETECTING PHISHING AND UNWANTED EMAILS
DOI:
https://doi.org/10.28925/2663-4023.2026.35.1430Keywords:
phishing; phishing email detection; large language models; small language models; cybersecurity; machine learning; hybrid detection systems; calibration; adversarial attacks; Ukrainian-language data.Abstract
Phishing remains one of the most common cyberattack vectors: in 2025, CERT-UA handled twice as many phishing incidents as in 2024. This article aims to systematize architectural approaches to using language models for detecting phishing and unwanted emails and to experimentally compare specific models and routing schemes. The approaches are classified according to how the model is used: task adaptation, extended analysis, and hybrid solutions incorporating classical machine learning. An experiment on 17,479 English emails, with near-duplicates kept together in a group-based split, compared TF-IDF logistic regression, fine-tuned XLM-RoBERTa (three runs), prompt-based Qwen2.5-7B-Instruct without an external provider's API, and cascade and parallel hybrids. Transfer was evaluated on Nazario + Enron, two machine-translated Ukrainian datasets, and 94 authentic Ukrainian messages. On the English test set, logistic regression and XLM-R achieved F1 scores of 0.970 and 0.969, whereas the LLM achieved 0.849 (exploratory because of earlier prompt selection) and required approximately 71 times as much processing time as XLM-R (T4 GPU, batch processing). On translated data, XLM-R retained F1 scores of 0.960 and 0.889, whereas the LLM's false positive rate reached 29.3%. On authentic Ukrainian messages, the LLM achieved F1 = 0.820 versus 0.509 for XLM-R (mean of three runs), mainly on SMS and messenger messages. An absolute uncertainty band on calibrated XLM-R scores escalated 1–12% of emails to the LLM, with cascade F1 scores close to those of XLM-R on English and translated data; most XLM-R errors on authentic messages were not escalated. Prompt injection almost completely disabled the LLM's detection capability; a simple system-prompt instruction mitigated this only partially. With the same prompt and threshold, Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 produced excessive false positives, indicating that the prompt requires tuning for each model. The authentic-message dataset is small and its classes differ in genre, so conclusions for Ukrainian messages remain preliminary.
Downloads
References
National Cybersecurity Coordination Center under the National Security and Defense Council of Ukraine. (2026). Ohliad natsionalnoi systemy kiberbezpeky Ukrainy 2025 [Review of the national cybersecurity system of Ukraine 2025]. https://www.rnbo.gov.ua/files/2026/NATIONAL_CYBER_SCC/kiber_2025.pdf
Poltava Regional Military Administration. (2026). Torik CERT-UA opratsiuvala maizhe 6000 kiberintsydentiv: Kilkist vorozhykh atak zrosla na 37% (za informatsiieiu Derzhspetszviazku) [Last year CERT-UA processed almost 6,000 cyber incidents: The number of hostile attacks increased by 37% (according to the State Service of Special Communications and Information Protection)]. https://poda.gov.ua/news/251006
Chataut, R., Gyawali, P. K., & Usman, Y. (2024). Can AI keep you safe? A study of large language models for phishing detection. In 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 548–554). IEEE. https://doi.org/10.1109/CCWC60891.2024.10427626
Patel, H., Rehman, U., & Iqbal, F. (2024). Evaluating the efficacy of large language models in identifying phishing attempts. In 2024 16th International Conference on Human System Interaction (HSI) (pp. 1–7). IEEE. https://doi.org/10.1109/HSI61632.2024.10613528
Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., & Park, P. S. (2024). Devising and detecting phishing emails using large language models. IEEE Access, 12, 42131–42146. https://doi.org/10.1109/ACCESS.2024.3375882
Koide, T., Nakano, H., & Chiba, D. (2024). ChatPhishDetector: Detecting phishing sites using large language models. IEEE Access, 12, 154381–154400. https://doi.org/10.1109/ACCESS.2024.3483905
Koide, T., Fukushi, N., Nakano, H., & Chiba, D. (2025). ChatSpamDetector: Leveraging large language models for effective phishing email detection. In Security and privacy in communication networks: SecureComm 2024 (pp. 297–319). Springer. https://doi.org/10.1007/978-3-031-94455-0_14
Lee, J., Lim, P., Hooi, B., & Divakaran, D. M. (2024). Multimodal large language models for phishing webpage detection and identification. In 2024 APWG Symposium on Electronic Crime Research (eCrime) (pp. 1–13). IEEE. https://doi.org/10.1109/eCrime66200.2024.00007
Li, Y., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Lim, H. W., & Hooi, B. (2024). KnowPhish: Large language models meet multimodal knowledge graphs for enhancing reference-based phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24) (pp. 793–810). USENIX Association. https://www.usenix.org/conference/usenixsecurity24/presentation/li-yuexin
Yadav, R., Bhosale, R., Rathod, R., Dhotre, P., & Patil, Y. S. (2026). Leveraging large language models for enhanced URL phishing detection. In Information systems for intelligent systems. ISBM 2025 (Smart Innovation, Systems and Technologies, Vol. 467, pp. 478–486). Springer. https://doi.org/10.1007/978-3-032-12983-3_46
Li, W., Manickam, S., Chong, Y. W., & Karuppayah, S. (2025). PhishDebate: An LLM-based multi-agent framework for phishing website detection [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2506.15656
Trad, F., & Chehab, A. (2024). Prompt engineering or fine-tuning? A case study on phishing detection with large language models. Machine Learning and Knowledge Extraction, 6(1), 367–384. https://doi.org/10.3390/make6010018
Goldenits, G., König, P., Raubitzek, S., & Ekelhart, A. (2026). Small language models for phishing website detection: Cost, performance, and privacy trade-offs. Journal of Cybersecurity and Privacy, 6(2), Article 48. https://doi.org/10.3390/jcp6020048
Kulkarni, A., Balachandran, V., Divakaran, D. M., & Das, T. (2025). From ML to LLM: Evaluating the robustness of phishing web page detection models against adversarial attacks. Digital Threats: Research and Practice, 6(2), Article 10. https://doi.org/10.1145/3737295
Zhang, J., Wu, P., London, J., & Tenney, D. (2025). Benchmarking and evaluating large language models in phishing detection for small and midsize enterprises: A comprehensive analysis. IEEE Access, 13, 28335–28352. https://doi.org/10.1109/ACCESS.2025.3540075
Fairbanks, J., & Serra, E. (2024). Generating phishing attacks and novel detection algorithms in the era of large language models. In 2024 IEEE International Conference on Big Data (BigData) (pp. 2314–2319). IEEE. https://doi.org/10.1109/BigData62323.2024.10825007
Shtonda, R., Chernysh, Yu., Tereshchenko, T., Tereshchenko, K., Tsykalo, Yu., & Polishchuk, S. (2024). Klasyfikatsiia ta metody vyiavlennia fishynhovykh atak [Classification and methods of detection of phishing attacks]. Kiberbezpeka: osvita, nauka, tekhnika, 4(24), 69–80. https://doi.org/10.28925/2663-4023.2024.24.6980
Feshchenko, Ye. O., & Zabolotnia, T. M. (2025). Metod avtomatyzovanoho vyiavlennia fishynhu v elektronnykh lystakh na osnovi hibrydnoi neiromerezhevoi arkhitektury [Method for automated detection of phishing in emails based on hybrid neural network architecture]. Naukovi pratsi Vinnytskoho natsionalnoho tekhnichnoho universytetu, (2), 145–154. https://doi.org/10.31649/2307-5376-2025-2-145-154
Tereikovskyi, I., & Tereikovska, L. (2026). Vyiavlennia fishinhu v kanalakh elektronnoi komunikatsii [Phishing detection in electronic communication channels]. Kiberbezpeka: osvita, nauka, tekhnika, 4(32), 141–153. https://doi.org/10.28925/2663-4023.2026.32.1092
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (Vol. 36, pp. 10088–10115). Curran Associates. https://doi.org/10.52202/075280-0441
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., ... Qiu, Z. (2024). Qwen2.5 technical report [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2412.15115
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8440–8451). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.747
Nazario, J. (n.d.). The online phishing corpus [Data set]. https://monkey.org/~jose/phishing/
Liu, Z. (n.d.). Phishing-email-dataset [Data set]. Hugging Face. https://huggingface.co/datasets/zefang-liu/phishing-email-dataset
Metsis, V., Androutsopoulos, I., & Paliouras, G. (2006). Spam filtering with naive Bayes—Which naive Bayes? In Proceedings of the Third Conference on Email and Anti-Spam (CEAS 2006).
Champa, A. I. (2023). Phishing email curated datasets (Version 1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.8339691
Tiedemann, J., & Thottingal, S. (2020). OPUS-MT—Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (pp. 479–480). European Association for Machine Translation. https://aclanthology.org/2020.eamt-1.61/
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In D. Precup & Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70, pp. 1321–1330). PMLR. https://proceedings.mlr.press/v70/guo17a.html
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., ... Ma, Z. (2024). The Llama 3 herd of models [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2407.21783
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., & El Sayed, W. (2023). Mistral 7B [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2310.06825
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Василь Лучик, Віталій Носов, Катерина Вітвіцька, Ірина Манжай
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
