An ML-Enabled Framework for Resilient Data Pipelines with Intelligent Anomaly Detection and Automated Recovery Mechanisms

Abstract

The contemporary data-intensive computing systems demand the establishment of strong data pipelines that can be utilized in order to sustain the active continuous running of the system in case of the occurrence of runtime anomalies and system errors. In this paper, introduce an ML-based framework of resilient data pipelines that combines smart anomaly detection with automated recovery controls to allow self-healing data pipeline processes. The pipeline will be run on the Google Cluster Traces dataset, which is systematically preprocessed, including data cleaning, anomaly labeling, and min-max normalization to generate high-quality inputs for training the model. The suggested model utilizes a hybrid CNN-LSTM model, where former part of workload trace patterns is extracted by the CNN, and the latter part captures the temporal relationships between successive pipeline actions by the LSTM, allowing for accurate anomaly classification. As an experimental analysis of the Google Cluster Traces dataset shows, the proposed CNN-LSTM model achieves an accuracy (acc) of 95.45% and precision (prec), recall (rec), and F1-score (F1) of 93.65%, 96.34%, and 94.51%, respectively, which is superior to all the base models addressed in this work. The framework is intended to be deployed in cloud computing systems and has been shown to apply to distributed, data-intensive systems that require high availability and consistent service reliability. It is also extended to IoT and cyber-physical systems as a direction for future work.