EVALUATION OF THE EFFECTIVENESS OF RAG ARCHITECTURES FOR AUTOMATION OF INFORMATION SECURITY POLICY ANALYSIS
DOI:
https://doi.org/10.28925/2663-4023.2026.34.1309Keywords:
Retrieval-Augmented Generation (RAG), LangChain, ChatGPT-5, information security, Agentic RAG, Advanced RAG, Large Language Models (LLMs)Abstract
This paper presents an empirical comparison of the effectiveness of three key Retrieval-Augmented Generation (RAG) architectures – the baseline (Naive RAG), the enhanced (Advanced RAG), and the agent-oriented (Agentic RAG). The study aims to analyze their performance in generating accurate and comprehensive answers to queries in the critical field of information security, based on internal corporate policies and procedures. The experiments were conducted using a specialized Python framework built with the LangChain library, the ChromaDB vector database, and a hybrid configuration of local (Ollama: LLaMA 3) and cloud-based (OpenAI ChatGPT-5) large language model platforms. The results demonstrate the significant potential of Advanced and Agentic RAG architectures to improve the accuracy, completeness, and contextual relevance of generated responses. The proposed approach proves effective for automating and simplifying the processing of large volumes of documentation by information security professionals and auditors.
Downloads
References
Kret, T. B. (2024). Approaches to threat modeling in the creation of a comprehensive information security system for multi-level intelligent control systems. Computer Systems and Networks, 6(1), 81–88. https://doi.org/10.23939/csn2024.01.081
Martseniuk, Ye. V., Partyka, A. I., & Kret, T. B. (2025). Study of artificial intelligence vulnerabilities and development of a comprehensive organizational security model. Modern Information Security, 1(61), 206–218. https://doi.org/10.31673/2409-7292.2025.018929
Oliinyk, B. V., & Chychkarov, Ye. O. (2025). Methods of implementing retrieval-augmented generation in combination with modern large language models. Scientific Notes of the State University of Telecommunications. Series: Information Technologies, (3), 56–65. https://doi.org/10.31673/2412-1034.2025.03.3267
Gao, Y., Xiong, Y., Gao, X., et al. (2023). Retrieval-augmented generation for large language models: A survey. arXiv. https://doi.org/10.48550/arXiv.2312.10997
International Organization for Standardization. (2022). ISO/IEC 27001:2022 Information security, cybersecurity and privacy protection—Information security management systems—Requirements. ISO.
Jiang, Z., Xu, F. F., Gao, L., et al. (2023). Active retrieval augmented generation. arXiv. https://doi.org/10.48550/arXiv.2305.06983
Karpukhin, V., Oguz, B., Min, S., et al. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) (pp. 6769–6781). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.550
LangChain. (2025). LangChain Python documentation. https://python.langchain.com/en/latest/
Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
Li, Z., Wang, J., Jiang, Z., et al. (2024). DMQR-RAG: Diverse multi-query rewriting for RAG. arXiv. https://doi.org/10.48550/arXiv.2411.13154
Singh, A., Ehtesham, A., Kumar, S., et al. (2025). Agentic retrieval-augmented generation: A survey on agentic RAG. arXiv. https://doi.org/10.48550/arXiv.2501.09136
Yao, S., Zhao, J., Yu, D., et al. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023). https://doi.org/10.48550/arXiv.2210.03629
Zhao, P., Zhang, H., Yu, Q., et al. (2024). Retrieval-augmented generation for AI-generated content: A survey. arXiv. https://doi.org/10.48550/arXiv.2402.19473
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Андрій Винар, Олексій Сведенюк, Остап Мединський
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
