DEVELOPMENT OF A HYBRID CLASSIFIER METHOD FOR DETECTING MALICIOUS REQUESTS BASED ON A COMBINATION OF DETERMINISTIC TOOLS AND GENERATIVE ARTIFICIAL INTELLIGENCE

Abstract

The fast development of GenAI has led to its adoption across various sectors, while also expanding the landscape of cyber threats. This led to rise of critical cybersecurity domain: GenAI Security. Efforts to protect the GenAI currently rely on combination of traditional cybersecurity defense methods and AI-native solutions. However, current experience shows that neither approach is sufficient on its own. While traditional methods are effective against known threats, they fail to cover the semantic threats, which are native to LLM and systems that are based on their usage, including native security mechanisms of LLM. The critical threat to these systems is Prompt Injection, which is currently ranked as the first in the OWASTP GenAI LLM Top 10 2026. It encompasses a variety of techniques, ranging from concealing malicious commands through encoding to role-playing attacks and manipulation. Consequently, identifying the nature of an incoming prompt and the intent behind it has become a vital step in securing GenAI-based systems. This article focuses on developing a method for classifying prompts by integrating traditional defense mechanisms, such as validation and filtering, with generative AI, utilizing the local model Gemma4:e2b-it-qat. The proposed method features a three-layered structure, which consists of deterministic security controls that are applied to input and output stages, with internal semantic analysis layer of two AI-agents acting as an Intention Evaluator and LLM-as-a-Judge. The evaluation held demonstrated a high detection rate for malicious or suspicious requests when the obfuscation wasn’t used for prompt concealment. Further analysis of handling of obfuscated queries revealed the limitations of the developed method and suggested potential improvements. Also, recommendations for its implementation were provided. This method is expected to perform most efficiently in the systems with clearly defined tasks that do net require creative content generation.