MULTI-AGENT APPROACH TO DATA PIPELINE STABILITY USING DISTRIBUTED STORAGE COORDINATION

Abstract

The article analyzes approaches to ensuring the stability of data processing pipelines in analytical systems, which is a critical aspect of modern Big Data and machine learning solutions. The relevance of the study is driven by the rapid growth of data volumes and heterogeneity, increasing architectural complexity of analytical platforms, and rising requirements for reliability and reproducibility under dynamic workloads and distributed computing. Instability in pipelines manifests through cascading failures, performance degradation, data inconsistency, and variability of results, which is particularly critical in Big Data environments. The limitations of centralized orchestration approaches are examined, highlighting their reduced adaptability and single points of failure. The paper substantiates the transition to decentralized multi-agent control mechanisms that provide component autonomy, fault localization, and improved resilience. Special attention is given to the role of distributed storage systems, which serve not only as repositories but also as coordination environments for agent interaction. The use of document-oriented, graph, object, and streaming storage systems enables consistent state management, reproducibility of control actions, and asynchronous coordination of processing workflows. The study summarizes approaches to organizing analytical and predictive mechanisms for stability support, including pipeline monitoring, workload and failure risk forecasting, and adaptive adjustment of processing parameters based on accumulated experience. A concept of a continuous evaluation–forecasting–adaptation cycle is proposed, enabling proactive stability management. A case study in e-commerce confirms the effectiveness of the approach, demonstrating reduced recovery time, increased automation of interventions, and improved adaptability of pipelines to workload and data quality changes. The results can be applied in the design and modernization of Big Data and machine learning systems aimed at enhancing stability, reliability, and reproducibility of analytical processes.