Optimizing Multiclass Android Malware Family Classification Using SMOTE-Tomek Links and XGBoost
DOI:
https://doi.org/10.63158/journalisi.v8i4.1788Keywords:
Android malware classification, SMOTE-Tomek Links, XGBoost, imbalanced learning, multiclass classificationAbstract
The increasing sophistication of Android malware attacks has created significant challenges for accurate malware family classification, particularly under highly imbalanced data distributions where minority malware families are frequently misclassified. This study presents a robust multiclass Android malware family classification framework by combining SMOTE-Tomek Links hybrid resampling with an optimized Extreme Gradient Boosting (XGBoost) classifier. The proposed framework addresses two critical issues in previous studies: ineffective handling of minority classes and potential data leakage during resampling and model validation. Experiments were conducted using the CCCS-CIC-AndMal-2020 After Reboot dataset containing 25,059 malware samples distributed across 14 malware families. The proposed approach applies stratified data partitioning, leakage-free SMOTE-Tomek Links integration within an imbalanced-learn pipeline, and RandomizedSearchCV-based hyperparameter optimization with 5-fold stratified cross-validation. Evaluation on an independent holdout test set demonstrates that the optimized framework achieves 80.09% accuracy, 79.85% weighted F1-score, 74.00% macro F1-score, and 97.48% OvR ROC-AUC, outperforming baseline XGBoost and Random Forest models. The results confirm that hybrid resampling combined with optimized gradient boosting improves classification reliability, especially in addressing severe class imbalance and enhancing recognition capability across diverse Android malware families.
Downloads
References
[1] C. Palma, A. Ferreira, and M. Figueiredo, “Explainable Machine Learning for Malware Detection on Android Applications †,” Information (Switzerland), vol. 15, no. 1, Jan. 2024, doi: 10.3390/info15010025.
[2] F. Taher, O. AlFandi, M. Al-kfairy, H. Al Hamadi, and S. Alrabaee, “DroidDetectMW: A Hybrid Intelligent Model for Android Malware Detection,” Applied Sciences (Switzerland), vol. 13, no. 13, Jul. 2023, doi: 10.3390/app13137720.
[3] P. Faruki, R. Bhan, V. Jain, S. Bhatia, N. El Madhoun, and R. Pamula, “A Survey and Evaluation of Android-Based Malware Evasion Techniques and Detection Frameworks,” Jul. 01, 2023, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/info14070374.
[4] W. F. Elsersy, A. Feizollah, and N. B. Anuar, “The rise of obfuscated Android malware and impacts on detection methods,” PeerJ Comput. Sci., vol. 8, 2022, doi: 10.7717/PEERJ-CS.907.
[5] J. M. Arif, M. F. A. Razak, S. Awang, S. R. T. Mat, N. S. N. Ismail, and A. Firdaus, “A static analysis approach for Android permission-based malware detection systems,” PLoS One, vol. 16, no. 9 September, Sep. 2021, doi: 10.1371/journal.pone.0257968.
[6] D. Soi, A. Sanna, D. Maiorca, and G. Giacinto, “Enhancing android malware detection explainability through function call graph APIs,” Journal of Information Security and Applications, vol. 80, Feb. 2024, doi: 10.1016/j.jisa.2023.103691.
[7] B. T. Hammad, N. Jamil, I. T. Ahmed, Z. M. Zain, and S. Basheer, “Robust Malware Family Classification Using Effective Features and Classifiers,” Applied Sciences (Switzerland), vol. 12, no. 15, Aug. 2022, doi: 10.3390/app12157877.
[8] M. E. Eren, M. Bhattarai, R. J. Joyce, E. Raff, C. Nicholas, and B. S. Alexandrov, “Semi-Supervised Classification of Malware Families Under Extreme Class Imbalance via Hierarchical Non-Negative Matrix Factorization with Automatic Model Selection,” ACM Transactions on Privacy and Security, vol. 26, no. 4, Nov. 2023, doi: 10.1145/3624567.
[9] N. T. Cam, T. M. Huy, and N. T. Tin, “Malware classification using deep neural networks with Deep Q-Learning and eXplainable artificial intelligence,” Eng. Appl. Artif. Intell., vol. 166, Feb. 2026, doi: 10.1016/j.engappai.2025.113622.
[10] P. Dominic Andrew, A. M. Jose, N. Jayapandian, C. N. Angel, and K. Brar, “Advanced Malware Analysis and Detection Using Deep Neural Networks,” in Conference Proceedings - 2025 IEEE 4th International Conference on Data, Decision and Systems, ICDDS 2025, Institute of Electrical and Electronics Engineers Inc., 2025, pp. 7–12. doi: 10.1109/ICDDS67737.2025.11344689.
[11] W. W. Y. Ng, Z. Liu, J. Zhang, and W. Pedrycz, “Maximizing minority accuracy for imbalanced pattern classification problems using cost-sensitive Localized Generalization Error Model,” Appl. Soft Comput., vol. 104, Jun. 2021, doi: 10.1016/j.asoc.2021.107178.
[12] S. Nemoto, S. Kitada, and H. Iyatomi, “Majority or Minority: Data Imbalance Learning Method for Named Entity Recognition,” IEEE Access, vol. 13, pp. 9902–9909, 2025, doi: 10.1109/ACCESS.2024.3522972.
[13] D. Elreedy, A. F. Atiya, and F. Kamalov, “A theoretical distribution analysis of synthetic minority oversampling technique (SMOTE) for imbalanced learning,” Mach. Learn., vol. 113, no. 7, pp. 4903–4923, Jul. 2024, doi: 10.1007/s10994-022-06296-4.
[14] N. Mohanty, B. K. Behera, C. Ferrie, and P. Dash, “A quantum approach to synthetic minority oversampling technique (SMOTE),” Quantum Mach. Intell., vol. 7, no. 1, Jun. 2025, doi: 10.1007/s42484-025-00248-6.
[15] H. Sun, J. Li, and X. Zhu, “A Novel Expandable Borderline Smote Over-Sampling Method for Class Imbalance Problem,” IEEE Trans. Knowl. Data Eng., vol. 37, no. 5, pp. 2183–2199, 2025, doi: 10.1109/TKDE.2025.3544284.
[16] G. Hou, D. L. Tong, S. Y. Liew, and P. Y. Choo, “Comparative Analysis of Resampling Techniques for Class Imbalance in Financial Distress Prediction Using XGBoost,” Mathematics, vol. 13, no. 13, Jul. 2025, doi: 10.3390/math13132186.
[17] M. Ilham, A. Winarno, M. Lutfi, and A. Indrasetianingsih, “Handling Imbalanced Fraudulent Transaction Data Using SMOTE-Tomek and Random Forest: A Classification Approach,” BEST Journal of Applied Electrical & Science Technology, 2025, doi: 10.36456/best.vol7.no1.10335.
[18] D. Yilmaz Eroglu and M. S. Pir, “Hybrid Oversampling and Undersampling Method (HOUM) via Safe-Level SMOTE and Support Vector Machine,” Applied Sciences (Switzerland), vol. 14, no. 22, Nov. 2024, doi: 10.3390/app142210438.
[19] E. H. Yulianti, O. Soesanto, and Y. Sukmawaty, “Penerapan Metode Extreme Gradient Boosting (XGBOOST) pada Klasifikasi Nasabah Kartu Kredit,” JOMTA Journal of Mathematics: Theory and Applications, vol. 4, no. 1, 2022, doi: 10.31605/jomta.v4i1.1792.
[20] M. Wiens, A. Verone-Boyle, N. Henscheid, J. T. Podichetty, and J. Burton, “A Tutorial and Use Case Example of the eXtreme Gradient Boosting (XGBoost) Artificial Intelligence Algorithm for Drug Development Applications,” Clin. Transl. Sci., vol. 18, no. 3, Mar. 2025, doi: 10.1111/cts.70172.
[21] M. Abdelhaq, S. K. Palanisamy, M. Gopinath, V. G. S. Manasa, M. A. Ram, and S. K. MD, “A hybrid XGBoost–SVM ensemble framework for robust cyber-attack detection in the internet of medical things (IoMT),” Sci. Rep., vol. 16, no. 1, Dec. 2026, doi: 10.1038/s41598-026-37832-0.
[22] H. Elwahsh et al., “Hyperparameter optimization of XGBoost and hybrid CnnSVM for cyber threat detection using modified Harris hawks algorithm,” PeerJ Comput. Sci., vol. 11, Sep. 2025, doi: 10.7717/peerj-cs.3169.
[23] M. Maghanaki, S. Keramati, F. F. Chen, and M. Shahin, “Systematic Evaluation of Machine Learning and Deep Learning Models for IoT Malware Detection Across Ransomware, Rootkit, Spyware, Trojan, Botnet, Worm, Virus, and Keylogger,” Sensors, vol. 26, no. 6, Mar. 2026, doi: 10.3390/s26061750.
[24] P. Anand, P. Nandhini, J. J. Christy, and K. Shiyamala, “Cyber threat estimation and prevention using xgboost,” in ViTECoN 2023 - 2nd IEEE International Conference on Vision Towards Emerging Trends in Communication and Networking Technologies, Proceedings, Institute of Electrical and Electronics Engineers Inc., 2023. doi: 10.1109/ViTECoN58111.2023.10157276.
[25] I. Singh Makkar, A. Kumar Sinha, T. Pratap, and A. Midhun Kumar, “Optimizing Android Malware Detection for Resource-Constrained Devices via a Two-Stage ML Pipeline,” in 2025 International Conference on Intelligent Computing and Knowledge Extraction, ICICKE 2025, Institute of Electrical and Electronics Engineers Inc., 2025. doi: 10.1109/ICICKE65317.2025.11136486.
[26] Z. Khalid, M. Zeeshan, and I. Ullah, “FGSM-CW Hardened Static Android-Malware Detection with Incremental PCA on the CCCS-CIC AndMal-2020 Corpus,” in 2025 IEEE 22nd International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and IoT, HONET 2025, Institute of Electrical and Electronics Engineers Inc., 2025, pp. 122–127. doi: 10.1109/HONET67928.2025.11318437.
[27] D. F. Duarte and A. A. Bortoli, “Empirical Evaluation of SMOTE in Android Malware Detection with Machine Learning: Challenges and Performance in CICMalDroid 2020,” Feb. 2026, doi: 10.5281/zenodo.15620300.
[28] N. A. Azhar, M. S. Mohd Pozi, A. M. Din, and A. Jatowt, “An investigation of SMOTE based methods for imbalanced datasets with data complexity analysis,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 7, pp. 6651–6672, Jul. 2023, doi: 10.1109/TKDE.2022.3179381.
[29] N. A. Azhar, M. S. Mohd Pozi, A. M. Din, and A. Jatowt, “An investigation of SMOTE based methods for imbalanced datasets with data complexity analysis,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 7, pp. 6651–6672, Jul. 2023, doi: 10.1109/TKDE.2022.3179381.
[30] A. Rahali, A. H. Lashkari, G. Kaur, L. Taheri, F. Gagnon, and F. Massicotte, “DIDroid: Android malware classification and characterization using deep image learning,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Nov. 2020, pp. 70–82. doi: 10.1145/3442520.3442522.
[31] N. V Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002, doi: 10.1613/jair.953.
[32] I. Tomek, “Two Modifications of CNN,” IEEE Transactions on Systems Man and Communications, pp. 769–772, 1976, doi: 10.1109/TSMC.1976.4309452.
[33] G. Lemaitre, F. Nogueira, and C. K. Aridas, “Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning,” Sep. 2016, [Online]. Available: http://arxiv.org/abs/1609.06570
[34] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, Aug. 2016, pp. 785–794. doi: 10.1145/2939672.2939785.
[35] S. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” Nov. 2017, [Online]. Available: http://arxiv.org/abs/1705.07874
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
