AUTOMATED EVALUATION OF ETHICAL ASPECTS IN LARGE LANGUAGE MODELS

Abstract

The growing integration of large language models into critical social domains, including healthcare, education, and decision-making systems, introduces new risks associated with the algorithmic reproduction of bias, social stereotypes, and failures in machine ethics. In cybersecurity, incorrect or uncontrolled outputs from generative models create a new attack surface. Generated content may cause confidential data leakage or enable established security policies to be bypassed.

This study proposes a modular architecture and presents the technical implementation of an automated system for comprehensive ethical auditing of large language models. The system comprises four functional components: a catalogue of structured test scenarios in JSON format, a central control module, adapters for interacting with target models, and analytical detectors for evaluating test outcomes. A unified research toolkit was assembled using six established datasets: CrowS-Pairs, BBQ, Seegull, SPICE, ETHICS, and TruthfulQA. The combined experimental sample contained 6,400 scenarios. Specialised evaluators were developed to process model responses. They combine linguistic text normalisation using regular expressions with string similarity measurement based on the SequenceMatcher algorithm.

The experimental study covered seven models with different access modes and hardware requirements. The sample included five local models - TinyLlama 1.1B Chat, Qwen 2.5 0.5B Instruct, Qwen 2.5 1.5B Instruct, Microsoft Phi-2 2.7B, and DeepSeek-Coder 1.3B - and two cloud-based models, GPT-4o-mini and GPT-4.1-mini. The local models were executed on a test system equipped with an RTX 3060 12 GB GPU, while the cloud-based models were accessed through their respective APIs.

The empirical analysis indicated a consistent relationship between model scale and the ability to comply with ethical requirements within the experimental setting. The cloud-based models achieved the highest overall scores: 73% for GPT-4.1-mini and 70% for GPT-4o-mini. Among the local models, Qwen 2.5 1.5B Instruct produced the most balanced result at 69%. TinyLlama 1.1B Chat and DeepSeek-Coder 1.3B showed limited ability to resolve ethical conflicts, with overall scores close to 50%.

The experiments also exposed substantial weaknesses in the local models. Phi-2 failed the ETHICS subtest by generating fragments of service code and template instructions instead of valid answers. ETHICS was the most difficult subtest for all evaluated models, with an average score of 39.9%, partly because its tasks imposed demanding context retention and response-format requirements. Qwen 2.5 0.5B Instruct could distort the automated score by generating several contradictory answer options in a single response. This behaviour demonstrates the limitations of evaluators based on simple syntactic rules.

Inconsistencies were also found in the Seegull dataset labels. For example, the generalisation “All Nevadans are friendly” is labelled as non-stereotypical, although the models classified it as a stereotype. Such cases systematically reduced model scores, resulting in an average success rate of 45.1% for this dataset. The findings define practical directions for improving automated AI testing systems, particularly the need to move beyond syntactic comparison towards semantic analysis of ethical judgements.