Predicting Backup Non-Compliance Using KPI-Derived Labels and Machine Learning: A Proof-of-Concept Study

Syafrial Fachri Pane; Muhammad Amran Hakim Siregar; Muhammad Qinthar Sabilla Almaliki; Andi Alamsyah Ode; Muhammad Baihaqi Siregar; Muhammad Arif Rivaldi; Raditya Rizki Raharja; Ahmad Rifky Ayala

doi:10.63158/journalisi.v8i5.1846

Authors

Syafrial Fachri Pane Universitas Logistik dan Bisnis Internasional
Muhammad Amran Hakim Siregar University of Mitra Bangsa
Muhammad Qinthar Sabilla Almaliki Universitas Logistik dan Bisnis Internasional
Andi Alamsyah Ode Universitas Logistik dan Bisnis Internasional
Muhammad Baihaqi Siregar Universitas Logistik dan Bisnis Internasional
Muhammad Arif Rivaldi Universitas Logistik dan Bisnis Internasional
Raditya Rizki Raharja Universitas Logistik dan Bisnis Internasional
Ahmad Rifky Ayala Universitas Logistik dan Bisnis Internasional

DOI:

https://doi.org/10.63158/journalisi.v8i5.1846

Keywords:

Backup Non-Compliance, KPI-Derived Labels, Machine Learning, Label Leakage, Enterprise IT

Abstract

Ensuring backup compliance is essential for data protection and operational resilience in enterprise IT environments. This study proposes a KPI-aware machine learning approach combining weak supervision, rule-based labeling, and supervised learning to estimate KPI-derived backup non-compliance status. The approach was evaluated on 439 device instances with 33 features using an 80:20 stratified split, producing a test set of 88 instances comprising 86 non-compliant and only 2 compliant proxy labels, indicating extreme class imbalance. Proxy labels were generated from predefined KPI rules without manual annotation or independent compliance audits. Logistic Regression, Random Forest, and XGBoost were evaluated using AUC-PR, F1-score, recall, Brier score, and calibration analysis. Random Forest achieved the strongest comparative results, with an AUC-PR of 0.998, F1-score of 0.994, recall of 1.000, and the lowest Brier score of 0.014. However, calibration remains uncertain because only two compliant instances were available. Therefore, these results indicate preliminary feasibility for reproducing or estimating KPI-derived proxy labels and should not be interpreted as independent validation of real-world non-compliance detection. The approach provides a scalable basis for backup compliance monitoring and preliminary risk prioritization in enterprise IT systems.

Downloads

Download data is not yet available.

References

[1] J. Wang, Z. Dou, L. Huang, X. Guo, T. Zhao, and J. Yang, “Research on enterprise data asset management and disaster recovery backup,” in Proc. 2024 3rd Int. Conf. Cyber Secur., Artif. Intell. Digit. Econ. (CSAIDE), Mar. 2024, pp. 532–537. doi: 10.1145/3672919.3673013.

[2] U. Ojiako et al., “Advancing resilience in infrastructure projects through machine learning-driven models,” Prod. Plan. Control, vol. 37, no. 8, pp. 796–820, 2026, doi: 10.1080/09537287.2025.2583300.

[3] Z. Yuan, M. Zhu, H. Qian, T. Lv, and T. Huang, “Multi-Class Failure Prediction for Distributed Systems Based on KPI Data and Feature Selection,” in 2023 6th Int. Conf. Softw. Eng. Comput. Sci. (CSECS), 2023, pp. 1–6. doi: 10.1109/CSECS60003.2023.10428105.

[4] Y. Yang and H. Wang, “Random Forest-Based Machine Failure Prediction : A Performance Comparison,” Appl. Sci., vol. 15, no. 16, Aug. 2025, doi: 10.3390/app15168841.

[5] A. Singh and J. Battra, “Strategies for Data Backup and Recovery in the Cloud,” Int. J. Performability Eng, vol. 19, no. 11, pp. 728–735, Nov. 2023, doi: 10.23940/ijpe.23.11.p3.728735.

[6] S. K. Parimi and R. Yallavula, “Enterprise Risk Intelligence: Machine Learning Models for Predicting Compliance, Fraud, and Operational Failures,” Int. J. Emerg. Trends Comput. Sci. Inf. Technol., vol. 4, no. 2, pp. 173–181, 2023, doi: 10.63282/3050-9246.ijetcsit-v4i2p117.

[7] M. N. Scoparo and S. M. Bruschi, “Anomaly Detection for Infrastructure Key Performance Indicators under Real-Time Constraints,” in Proc. 2025 IEEE/SBC 37th Int. Symp. Comput. Arch. High Perform. Comput. Workshops (SBAC-PADW), 2025, pp. 100–107. doi: 10.1109/SBAC-PADW69789.2025.00022.

[8] S. Thota, “Secure and Scalable AI-Powered Data Governance Models for Salesforce Cloud-Based Enterprises,” IJAIDSML, vol. 6, no. 2, pp. 180–189, 2025, doi: 10.63282/3050-9262.IJAIDSML-V6I2P120.

[9] J. Gram, J. Drees, S. Stambera, M. Jestädt, M. Di Cagno, and T. Bauernhansl, “Real-Time Anomaly Detection in Manufacturing: A Data-Driven Approach to Enhancing Production Efficiency through Online KPI Monitoring,” Procedia CIRP, vol. 134, pp. 145–150, 2025, doi: 10.1016/j.procir.2025.03.047.

[10] M. Cilinio, T. Saraiva, M. Sousa, P. Vieira, and A. Rodrigues, “An Adaptive Framework for Collective Anomaly Detection in Key Performance Indicators From Mobile Networks,” IEEE Access, vol. 13, pp. 105828–105849, 2025, doi: 10.1109/ACCESS.2025.3581120.

[11] H. Yuliana, Hendrawan, Iskandar, and Y. Musashi, “Estimating Base Station Traffic and Throughput Using Machine Learning Based on Hourly Key Performance Indicator (KPI) Network Analysis,” IEEE Access, vol. 12, pp. 116285–116301, 2024, doi: 10.1109/ACCESS.2024.3447098.

[12] Z. Duan, M. Xi, C. Zhou, and M. Su, “KPIs Anomaly Detection: A Comparative Analysis of Algorithms and Models,” in Proc. IEEE 2nd Int. Conf. Big Data Sci. Eng. (ICBDSE), Aug. 2025, pp. 1–6. doi: 10.1109/ICBDSE65491.2025.11220055.

[13] R. S. Kumar, A. R. Singh, P. L. Narayana, V. S. Chandrika, M. Bajaj, and I. Zaitsev, “Hybrid machine learning framework for predictive maintenance and anomaly detection in lithium-ion batteries using enhanced random forest,” Sci. Rep., vol. 15, no. 1, pp. 1–20, 2025, doi: 10.1038/s41598-025-90810-w.

[14] B. M. Kolluri, “Contextual Anomaly Detection for Business KPIs : A Site Reliability Engineering Approach,” Sarcouncil J. Multidiscip., vol. 5, pp. 5–9, Sep. 2025, doi: 10.5281/zenodo.17238497.

[15] J. Diaz-de-Arcaya, A. I. Torre-Bastida, G. Zárate, R. Miñón, and A. Almeida, “A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey,” ACM Comput. Surv., vol. 56, no. 4, Oct. 2023, doi: 10.1145/3625289.

[16] A. Apicella, F. Isgrò, and R. Prevete, “Don’t push the button! Exploring data leakage risks in machine learning and transfer learning,” Artif. Intell. Rev., vol. 58, no. 11, Nov. 2025, doi: 10.1007/s10462-025-11326-3.

[17] D. A. Bhanage, A. V. Pawar, and K. Kotecha, “IT Infrastructure Anomaly Detection and Failure Handling: A Systematic Literature Review Focusing on Datasets, Log Preprocessing, Machine & Deep Learning Approaches and Automated Tool,” IEEE Access, vol. 9, pp. 156392–156421, 2021, doi: 10.1109/ACCESS.2021.3128283.

[18] H. E.-D. Hassanien and A. Elragal, “Deep Learning for Enterprise Systems Implementation Lifecycle Challenges: Research Directions,” Informatics, vol. 8, no. 1, 2021, doi: 10.3390/informatics8010011.

[19] D. Petrik, A. Untermann, and H. Baars, “Functional Requirements for Enterprise Data Catalogs: A Systematic Literature Review,” in Software Business, S. Hyrynsalmi, J. Münch, K. Smolander, and J. Melegati, Eds., Cham: Springer Nature Switzerland, 2024, pp. 3–18. doi: 10.1007/978-3-031-53227-6_1.

[20] G. Kumar, S. Basri, A. A. Imam, S. A. Khowaja, L. F. Capretz, and A. O. Balogun, “Data harmonization for heterogeneous datasets: A systematic literature review,” Sep. 01, 2021, Appl. Sci. doi: 10.3390/app11178275.

[21] Y. Gong, G. Liu, Y. Xue, R. Li, and L. Meng, “A survey on dataset quality in machine learning,” Inf. Softw. Technol., vol. 162, no. June, p. 107268, 2023, doi: 10.1016/j.infsof.2023.107268.

[22] A. K. Kushwaha, P. S. Chandhirathil, D. Rai, and B. A. Kumar, “Modeling and Real-Time classification of driving behavior in mixed traffic: A naturalistic study using rule-Based labeling and supervised learning,” Expert Syst. Appl., vol. 309, p. 131106, May 2026, doi: 10.1016/j.eswa.2026.131106.

[23] A. M. Martínez-Heredia and S. Ventura, “Weak Supervision: A Survey on Predictive Maintenance,” WIREs Data Min. Knowl. Discov., vol. 15, no. 2, p. e70022, 2025, doi: 10.1002/widm.70022.

[24] T. Silva Filho, H. Song, M. Perello-Nieto, R. Santos-Rodriguez, M. Kull, and P. Flach, “Classifier calibration: a survey on how to assess and improve predicted class probabilities,” Mach. Learn., vol. 112, no. 9, pp. 3211–3260, Sep. 2023, doi: 10.1007/s10994-023-06336-7.

[25] L. Yu, R. Zhou, R. Chen, and K. K. Lai, “Missing Data Preprocessing in Credit Classification: One-Hot Encoding or Imputation?,” Emerg. Markets Finance Trade, vol. 58, no. 2, pp. 472–482, Aug. 2020, doi: 10.1080/1540496X.2020.1825935.

[26] X. Zhang, M. Liu, P. Wang, Z. Cao, and T. Wang, “KPIs Anomaly Detection Through Missing Value Interpolation and Adversarial Training,” Concurr. Comput., vol. 37, no. 27–28, p. e70410, 2025, doi: 10.1002/cpe.70410.

[27] L. Yang, “Predictive modeling of tax compliance risks : A comparative study of machine learning approaches,” PLoS One, pp. 1–19, Sep. 2025, doi: 10.1371/journal.pone.0331715.

[28] A. Zaidi and A. S. M. Al Luhayb, “Two Statistical Approaches to Justify the Use of the Logistic Function in Binary Logistic Regression,” Math. Probl. Eng., vol. 2023, no. 1, Jan. 2023, doi: 10.1155/2023/5525675.

[29] I. D. Mienye and Y. Sun, “A Survey of Ensemble Learning: Concepts, Algorithms, Applications, and Prospects,” IEEE Access, vol. 10, pp. 99129–99149, Sep. 2022, doi: 10.1109/ACCESS.2022.3207287.

[30] E. Richardson, R. Trevizani, J. A. Greenbaum, and H. Carter, “Article The receiver operating characteristic curve accurately assesses imbalanced datasets The receiver operating characteristic curve accurately assesses imbalanced datasets,” Patterns, vol. 5, no. 6, p. 100994, 2024, doi: 10.1016/j.patter.2024.100994.

[31] J. L. Leevy, J. M. Johnson, J. Hancock, and T. M. Khoshgoftaar, “Threshold optimization and random undersampling for imbalanced credit card data,” J. Big Data, vol. 10, no. 1, Dec. 2023, doi: 10.1186/s40537-023-00738-z.

[32] L. Zhao et al., “Understanding decision curve analysis in clinical prediction model research,” Postgrad. Med. J., vol. 100, no. 1185, pp. 512–515, Jul. 2024, doi: 10.1093/postmj/qgae027.

[33] T. R. Chhetri, A. Kurteva, R. J. Delong, R. Hilscher, K. Korte, and A. Fensel, “Data Protection by Design Tool for Automated GDPR Compliance Verification Based on Semantically Modeled Informed Consent,” Sensors, vol. 22, no. 7, Apr. 2022, doi: 10.3390/s22072763.

[34] S. E. Davis, M. E. Matheny, S. Balu, and M. P. Sendak, “A framework for understanding label leakage in machine learning for health care,” J. Am. Med. Inform. Assoc., vol. 31, no. 1, pp. 274–280, Jan. 2024, doi: 10.1093/jamia/ocad178.

[35] M. Ravi and T. Kotla, “Intelligent Compliance and Risk Monitoring using Machine Learning in Enterprise Integration Platforms,” IJCTEC, vol. 13, no. 3, Jun. 2026, doi: 10.15680/IJARETY.2026.1303084.

[36] L. Brugere, Y. Kwon, A. E. Frazier, and P. Kedron, “Improved prediction of tree species richness and interpretability of environmental drivers using a machine learning approach,” For. Ecol. Manage., vol. 539, p. 120972, 2023, doi: 10.1016/j.foreco.2023.120972.

[37] A. S. More and D. P. Rana, “Performance enrichment through parameter tuning of random forest classification for imbalanced data applications,” Mater. Today Proc., vol. 56, pp. 3585–3593, 2022, doi: 10.1016/j.matpr.2021.12.020.

[38] A. Bakdi, N. B. Kristensen, and M. Stakkeland, “Multiple Instance Learning With Random Forest for Event Logs Analysis and Predictive Maintenance in Ship Electric Propulsion System,” IEEE Trans. Industr. Inform., vol. 18, no. 11, pp. 7718–7728, 2022, doi: 10.1109/TII.2022.3144177.

[39] S. Ruan et al., “CPWS: Confident Programmatic Weak Supervision for High-Quality Data Labeling,” ACM Trans. Inf. Syst., vol. 43, no. 4, May 2025, doi: 10.1145/3725730.

[40] L. Barreñada, P. Dhiman, D. Timmerman, A. L. Boulesteix, and B. Van Calster, “Understanding overfitting in random forest for probability estimation : a visualization and simulation study,” Diagn. Progn. Res., pp. 1–14, 2024, doi: 10.1186/s41512-024-00177-1.

[41] D. E. Ehrmann, S. Joshi, S. D. Goodfellow, M. L. Mazwi, and D. Eytan, “Making machine learning matter to clinicians: model actionability in medical decision-making,” Dec. 01, 2023, npj Digit. Med. doi: 10.1038/s41746-023-00753-7.