An Ensemble-Based Hybrid Data Drift Detection Framework Integrating Principal Component Analysis (PCA) and Variational Autoencoder (VAE)

This paper presents a novel hybrid data drift detection framework, the Ensemble Modeling-based Hybrid Drift Score (EM-HDS), that integrates Principal Component Analysis (PCA), Variational Autoencoders (VAEs), and ensemble modeling to effectively detect both global structural shifts and localized non-linear variations in feature space. PCA identifies global changes by monitoring variance alignment and principal component transformations, while VAEs enhance sensitivity to localized anomalies through probabilistic modeling of reconstruction errors. The EM-HDS framework integrates complementary techniques using a Random Forest ensemble model, effectively capturing complex, non-linear relationships between PCA and VAE metrics. Experimental evaluations on synthetic datasets with simulated drift and real-world COCO image features demonstrate the robustness and adaptability of the proposed method. EM-HDS delivers superior drift detection performance, significantly improving detection accuracy, particularly in scenarios involving simultaneous global and local drifts, surpassing standalone PCA and VAE approaches. Although the framework requires careful tuning of hyperparameters to adapt to specific datasets, its ability to dynamically adjust to diverse drift patterns makes it a practical and effective solution for real-time monitoring and adaptation in dynamic environments. This research establishes a strong foundation for enhancing the reliability of machine learning models in complex, real-time applications for future work.