METHODS AND TOOLS FOR FORENSIC MEMORY ANALYSIS TO DETECT HIDDEN PROCESSES IN COMPUTER SYSTEMS

Abstract

The article addresses the problem of detecting hidden processes and injected malicious code in the volatile memory of computer systems running Microsoft Windows 11. The relevance of the research is driven by the rapid growth of fileless attacks, which fully unfold in RAM and leave no traces accessible to classical antivirus and EDR monitoring tools. According to industry reports, the volume of cyberattacks increased by 30 % in Q2 2024, with a significant share constituted by fileless and memory-resident campaigns. The aim of the article is to formalise a methodology for detecting hidden processes through analysis of memory structures — Virtual Address Descriptor (VAD) tree and page organisation — using the Volatility 3 framework with subsequent external validation of the obtained results. The paper systematises the theoretical foundations of Windows virtual memory organisation, performs a comparative analysis of six memory acquisition tools and four analysis frameworks, and traces the evolution of code-hiding techniques. The proposed methodology is designed as a sequential six-stage process: preparation, volatile data acquisition, triage, deep analysis (in three parallel branches: process analysis, memory analysis, and network analysis), cross-verification through external cyberintelligence platforms, and reporting with chain-of-custody documentation. The methodology is integrated with NIST SP 800-86, IETF RFC 3227, and MITRE ATT&CK standards. Experimental verification of the methodology has been performed on a controlled attack scenario using the Metasploit Meterpreter framework with subsequent migration of the malicious code into the address space of the legitimate Windows 11 explorer.exe process via the Reflective DLL Injection technique (T1055.001 in MITRE ATT&CK classification). A robust set of compromise indicators has been identified: two private RWX memory regions, a characteristic x86-64 shellcode prologue with the typical PEB-traversal pattern through the GS segment, the asymmetry of the VAD tree with the absence of file-on-disk linkage for the anomalous regions, and a hardcoded address of the command-and-control server. Independent validation through the VirusTotal platform has confirmed the artefact's attribution to the shellcode/marte family with a detection rate of 17 out of 62 antivirus engines. Comparative analysis with the reference (clean) dump has unequivocally proven that the detected anomalies result from the simulated attack rather than background system activity. The detected artefacts are mapped onto six MITRE ATT&CK matrix techniques: T1105, T1204.002, T1055.001, T1620, T1562.001, and T1071.001, ensuring compatibility of conclusions with the international incident response practice. The scientific novelty consists in the formalisation of a generalised methodology for detecting hidden processes that integrates collection, triage, deep analysis, cross-verification, and reporting stages with explicit MITRE ATT&CK mapping, applied to modern Windows 11 systems. The practical significance of the work lies in the methodology's readiness for direct application in cybersecurity training at bachelor and master levels, in commercial and governmental SOC incident response practice, and as a methodological basis for further research in volatile memory forensics. Prospective research directions include extending the methodology to privilege escalation and credential dumping scenarios, adaptation to Linux and macOS platforms, integration with EDR platforms, and the application of deep learning methods for automated classification of detected artefacts.