Learning Techniques-Based Malware Detection: A Comprehensive Review

Main Article Content

Sarah Fouad Ali
Musaab Riyadh Abdulrazzaq
Methaq Talib Gaata

Abstract

The rapid proliferation of Internet of Things (IoT) devices has significantly increased the threat landscape, with malwares arising as a critical concern. Advanced learning methods such as machine learning (ML), deep learning (DL), and federated learning (FL) are essential for handling complex IoT data. ML provides tools for pattern identification and detecting anomalies. DL boosts malware detection by automatically extracting features and identifying patterns. FL enables collaborative model training across decentralized devices, ensuring data privacy, which is crucial for diverse IoT systems. This comprehensive review specifically synthesizes ML, DL and FL for malware detection in the IoT environment, highlighting key trends and developments. Additionally, several significant contributions have been provided, including an overview of various types of malwares and their approaches and a comparison with existing studies. Importantly, notable trends and advancements are highlighted, and the current limitations of these learning techniques in malware detection are identified. It concludes by outlining future research directions to develop robust, scalable malware detection mechanisms tailored to safeguard the prosperity of the IoT environment against evolving cyber threats.

Article Details

Section

Articles

How to Cite

Learning Techniques-Based Malware Detection: A Comprehensive Review (S. F. . Ali, M. R. Abdulrazzaq, & M. T. Gaata , Trans.). (2025). Mesopotamian Journal of CyberSecurity, 5(1), 273-300. https://doi.org/10.58496/MJCS/2025/018

References

[1] A. Nuhu, A. F. M. Raffei, M. F. Ab Razak, and A. Ahmad, “Distributed Denial of Service Attack Detection in IoT Networks using Deep Learning and Feature Fusion: A Review,” Mesopotamian J. CyberSecurity, vol. 4, no. 1, pp. 47–70, 2024.

[2] M. N. Jasim and M. T. Gaata, “K-Means clustering-based semi-supervised for DDoS attacks classification,” Bull. Electr. Eng. Informatics, vol. 11, no. 6, pp. 3570–3576, 2022.

[3] B. A. Jaafar, M. T. Gaata, and M. N. Jasim, “Home appliances recommendation system based on weather information using combined modified k-means and elbow algorithms,” Indones. J. Electr. Eng. Comput. Sci., vol. 19, no. 3, pp. 1635–1642, 2020.

[4] M. Riyadh and D. R. Alshibani, “Intrusion detection system based on machine learning techniques,” Indones. J. Electr. Eng. Comput. Sci., vol. 23, no. 2, pp. 953–961, 2021.

[5] M. M. Mijwil, R. Doshi, K. K. Hiran, A.-H. Al-Mistarehi, and M. Gök, “Cybersecurity Challenges in Smart Cities: An Overview and Future Prospects,” Mesopotamian J. cybersecurity, vol. 2022, pp. 1–4, 2022.

[6] M. Mijwil, Y. Filali, M. Aljanabi, M. Bounabi, and H. Al-Shahwani, “The purpose of cybersecurity governance in the digital transformation of public services and protecting the digital environment,” Mesopotamian J. cybersecurity, vol. 2023, pp. 1–6, 2023.

[7] D. Singh and S. Khurana, “Malware Detection in IoT Devices Using Machine Learning: A Review,” in 2024 International Conference on Computational Intelligence and Computing Applications (ICCICA), IEEE, 2024, pp. 203–209.

[8] S. S. Hussain, M. F. Ab Razak, and A. Firdaus, “Deep Learning Based Hybrid Analysis of Malware Detection and Classification: A Recent Review,” J. Cyber Secur. Mobil., pp. 91–134, 2024.

[9] J. BOODAI, A. ALQAHTANI, and K. RIAD, “DEEP LEARNING FOR MALWARE DETECTION: LITERATURE REVIEW,” J. Theor. Appl. Inf. Technol., vol. 102, no. 4, 2024.

[10] M. Ghahramani, R. Taheri, M. Shojafar, R. Javidan, and S. Wan, “Deep Image: A precious image based deep learning method for online malware detection in IoT Environment,” Internet of Things, p. 101300, 2024.

[11] A. Brown, M. Gupta, and M. Abdelsalam, “Automated machine learning for deep learning based malware detection,” Comput. Secur., vol. 137, p. 103582, 2024.

[12] F. Mofidi, S. G. Hounsinou, and G. Bloom, “L-IDS: A Multi-Layered Approach to Ransomware Detection in IoT,” in 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), IEEE, 2024, pp. 387–396.

[13] A. Javed, I. Rashid, S. Tahir, S. Saeed, A. M. Almuhaideb, and K. Alissa, “AdamW+: Machine Learning Framework to Detect Domain Generation Algorithms for Malware,” IEEE Access, 2024.

[14] M. Riyadh, B. J. Ali, and D. R. Alshibani, “IDS-MIU: An Intrusion Detection System Based on Machine Learning Techniques for Mixed type, Incomplete, and Uncertain Data Set.,” Int. J. Intell. Eng. Syst., vol. 14, no. 3, 2021.

[15] N. Z. Gorment, A. Selamat, L. K. Cheng, and O. Krejcar, “Machine learning algorithm for malware detection: taxonomy, current challenges and future directions,” IEEE Access, 2023.

[16] C. P. Chenet, A. Savino, and S. Di Carlo, “A survey on hardware-based malware detection approaches,” IEEE Access, 2024.

[17] M. U. Rana, M. A. Shah, and O. Ellahi, “Malware Persistence and Obfuscation: An Analysis on Concealed Strategies,” in 2021 26th International Conference on Automation and Computing (ICAC), IEEE, 2021, pp. 1–6.

[18] M. Luoma-aho, “Analysis of Modern Malware: obfuscation techniques,” 2023.

[19] K. Brezinski and K. Ferens, “Metamorphic malware and obfuscation: a survey of techniques, variants, and generation kits,” Secur. Commun. Networks, vol. 2023, 2021.

[20] C. Catalano, G. Specchia, and N. G. Totaro, “Enhancing Code Obfuscation Techniques: Exploring the Impact of Artificial Intelligence on Malware Detection,” in International Conference on Product-Focused Software Process Improvement, Springer, 2023, pp. 80–88.

[21] H. J. Asghar et al., “Use of cryptography in malware obfuscation,” J. Comput. Virol. Hacking Tech., vol. 20, no. 1, pp. 135–152, 2024.

[22] J. R. S. Alrzini and D. Pennington, “A review of polymorphic malware detection techniques,” Int. J. Adv. Res. Eng. Technol., vol. 11, no. 12, pp. 1238–1247, 2020.

[23] M. A. Khoje, A. Kaswan, and J. Pawar, “Exploring Polymorphic Malware Analysis Techniques: A Comprehensive Survey”.

[24] R. Hakobyan and T. Jamgharyan, “Polymorphic Malware Analysis Model”.

[25] M. Gopinath and S. C. Sethuraman, “A comprehensive survey on deep learning based malware detection techniques,” Comput. Sci. Rev., vol. 47, p. 100529, 2023.

[26] D. Dholariya, D. Panchal, and P. Singhal, “Anti-Virus Tempering Methodologies,” Int. Res. J. Adv. Eng. Manag., vol. 2, no. 04, pp. 1232–1239, 2024.

[27] V. Priya and A. S. Sofia, “Review on Malware Classification and Malware Detection Using Transfer Learning Approach,” in 2023 5th International Conference on Smart Systems and Inventive Technology (ICSSIT), IEEE, 2023, pp. 1042–1049.

[28] Y. T. Ling, N. F. M. Sani, M. T. Abdullah, and N. A. W. A. Hamid, “Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model,” J. Comput. Virol. Hacking Tech., pp. 1–21, 2022.

[29] A. Kashtalian, S. Lysenko, O. Savenko, A. Nicheporuk, T. Sochor, and V. Avsiyevych, “Multi-computer malware detection systems with metamorphic functionality,” Radioelectron. Comput. Syst., vol. 2024, no. 1, pp. 152–175, 2024.

[30] F. Habib, S. H. Shirazi, K. Aurangzeb, A. Khan, B. Bhushan, and M. Alhussein, “Deep Neural Networks for Enhanced Security: Detecting Metamorphic Malware in IoT Devices,” IEEE Access, 2024.

[31] K. O. Babaagba and J. Wylie, “An Evolutionary based Generative Adversarial Network Inspired Approach to Defeating Metamorphic Malware,” in Proceedings of the Companion Conference on Genetic and Evolutionary Computation, 2023, pp. 1753–1759.

[32] A. Bensaoud, J. Kalita, and M. Bensaoud, “A survey of malware detection using deep learning,” Mach. Learn. With Appl., vol. 16, p. 100546, 2024.

[33] B. Ait Messaad, K. Chetioui, Y. Balboul, and H. Rhachi, “Analyzing and Detecting Malware Using Machine Learning and Deep Learning,” in The International Conference on Artificial Intelligence and Smart Environment, Springer, 2023, pp. 518–525.

[34] M. S. Akhtar and T. Feng, “Evaluation of machine learning algorithms for malware detection,” Sensors, vol. 23, no. 2, p. 946, 2023.

[35] M. J. H. Faruk et al., “Malware detection and prevention using artificial intelligence techniques,” in 2021 IEEE International Conference on Big Data (Big Data), IEEE, 2021, pp. 5369–5377.

[36] A.-A. M. Majid, A. J. Alshaibi, E. Kostyuchenko, and A. Shelupanov, “A review of artificial intelligence based malware detection using deep learning,” Mater. Today Proc., vol. 80, pp. 2678–2683, 2023.

[37] A. Djenna, A. Bouridane, S. Rubab, and I. M. Marou, “Artificial intelligence-based malware detection, analysis, and mitigation,” Symmetry (Basel)., vol. 15, no. 3, p. 677, 2023.

[38] D. Gibert, C. Mateu, and J. Planes, “The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,” J. Netw. Comput. Appl., vol. 153, p. 102526, 2020.

[39] S. A. Roseline, S. Geetha, S. Kadry, and Y. Nam, “Intelligent vision-based malware detection and classification using deep random forest paradigm,” IEEE Access, vol. 8, pp. 206303–206324, 2020.

[40] F. Deldar and M. Abadi, “Deep learning for zero-day malware detection and classification: a survey,” ACM Comput. Surv., vol. 56, no. 2, pp. 1–37, 2023.

[41] A. Abusitta, M. Q. Li, and B. C. M. Fung, “Malware classification and composition analysis: A survey of recent developments,” J. Inf. Secur. Appl., vol. 59, p. 102828, 2021.

[42] V. Khushali, “A Review on Fileless Malware Analysis Techniques,” Int. J. Eng. Res. Technol., vol. 9, no. 05, 2020.

[43] R. Baker del Aguila, C. D. Contreras Pérez, A. G. Silva-Trujillo, J. C. Cuevas-Tello, and J. Nunez-Varela, “Static Malware Analysis Using Low-Parameter Machine Learning Models,” Computers, vol. 13, no. 3, p. 59, 2024.

[44] M. A. Ayub, A. Siraj, B. Filar, and M. Gupta, “RWArmor: a static-informed dynamic analysis approach for early detection of cryptographic windows ransomware,” Int. J. Inf. Secur., vol. 23, no. 1, pp. 533–556, 2024.

[45] S. Talukder, “Tools and techniques for malware detection and analysis,” arXiv Prepr. arXiv2002.06819, 2020.

[46] J. A. Herrera-Silva and M. Hernández-Álvarez, “Dynamic feature dataset for ransomware detection using machine learning algorithms,” Sensors, vol. 23, no. 3, p. 1053, 2023.

[47] G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, V. G. Sujadevi, and P. Poornachandran, “CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection,” Procedia Comput. Sci., vol. 233, pp. 492–503, 2024.

[48] F. A. Aboaoja, A. Zainal, F. A. Ghaleb, B. A. S. Al-rimy, T. A. E. Eisa, and A. A. H. Elnour, “Malware detection issues, challenges, and future directions: A survey,” Appl. Sci., vol. 12, no. 17, p. 8482, 2022.

[49] V. Vasani, A. K. Bairwa, S. Joshi, A. Pljonkin, M. Kaur, and M. Amoon, “Comprehensive analysis of advanced techniques and vital tools for detecting malware intrusion,” Electronics, vol. 12, no. 20, p. 4299, 2023.

[50] U.-H. Tayyab, F. B. Khan, M. H. Durad, A. Khan, and Y. S. Lee, “A survey of the recent trends in deep learning based malware detection,” J. Cybersecurity Priv., vol. 2, no. 4, pp. 800–829, 2022.

[51] D. L. S. Punyasiri, “Signature & Behavior Based Malware Detection,” 2023.

[52] L. S. Fascí, M. Fisichella, G. Lax, and C. Qian, “Disarming visualization-based approaches in malware detection systems,” Comput. Secur., vol. 126, p. 103062, 2023.

[53] K. S. Sangher, A. Singh, and H. M. Pandey, “Signature based ransomware detection based on optimizations approaches using RandomClassifier and CNN algorithms,” Int. J. Syst. Assur. Eng. Manag., pp. 1–17, 2023.

[54] A. Samad, “Hybrid Approaches in Threat Detection: Integrating Traditional Signature-Based Methods with AI and ML Techniques for Enhanced Accuracy,” 2023.

[55] J. Raghunath, S. Kiran, G. S. N. Rao, J. R. A. Kumar, R. Anasuya, and C. S. Kumar, “A MACHINE LEARNING TECHNIQUE TO DETECT BEHAVIOR BASED MALWARE,” Semicond. Optoelectron., vol. 42, no. 1, pp. 1268–1278, 2023.

[56] T. Alsmadi and N. Alqudah, “A Survey on malware detection techniques,” in 2021 International Conference on Information Technology (ICIT), IEEE, 2021, pp. 371–376.

[57] S. K. Sahay, A. Sharma, and H. Rathore, “Evolution of malware and its detection techniques,” in Information and Communication Technology for Sustainable Development: Proceedings of ICT4SD 2018, Springer, 2020, pp. 139–150.

[58] S. Soja Rani and S. R. Reeja, “A survey on different approaches for malware detection using machine learning techniques,” in Sustainable Communication Networks and Application: ICSCN 2019, Springer, 2020, pp. 389–398.

[59] Ö. A. Aslan and R. Samet, “A comprehensive review on malware detection approaches,” IEEE access, vol. 8, pp. 6249–6271, 2020.

[60] Q.-D. Ngo, H.-T. Nguyen, V.-H. Le, and D.-H. Nguyen, “A survey of IoT malware and detection methods based on static features,” ICT Express, vol. 6, no. 4, pp. 280–286, 2020.

[61] H. Alrubayyi, G. Goteng, M. Jaber, and J. Kelly, “Challenges of malware detection in the IoT and a review of artificial immune system approaches,” J. Sens. Actuator Networks, vol. 10, no. 4, p. 61, 2021.

[62] R. Qamar and B. A. Zardari, “An analysis of the Internet of Everything,” Mesopotamian J. CyberSecurity, vol. 2023, pp. 85–92, 2023.

[63] N. A. Bajao and J. Sarucam, “Threats Detection in the Internet of Things Using Convolutional neural networks, long short-term memory, and gated recurrent units,” Mesopotamian J. cybersecurity, vol. 2023, pp. 22–29, 2023.

[64] C. C. Uchenna, N. Jamil, R. Ismail, L. K. Yan, and M. A. Mohamed, “Malware threat analysis techniques and approaches for iot applications: A review,” Bull. Electr. Eng. Informatics, vol. 10, no. 3, pp. 1558–1571, 2021.

[65] S. K. Jagatheesaperumal, M. Rahouti, K. Ahmad, A. Al-Fuqaha, and M. Guizani, “The duo of artificial intelligence and big data for industry 4.0: Applications, techniques, challenges, and future research directions,” IEEE Internet Things J., vol. 9, no. 15, pp. 12861–12885, 2021.

[66] A. A. Almazroi, M. A. Alqarni, M. A. Al-Shareeda, and S. Manickam, “L-CPPA: Lattice-based conditional privacy-preserving authentication scheme for fog computing with 5G-enabled vehicular system,” PLoS One, vol. 18, no. 10, p. e0292690, 2023.

[67] J. E. M. Díaz, “Internet of things and distributed denial of service as risk factors in information security,” in Bioethics in Medicine and Society, IntechOpen, 2020.

[68] J. Jeon, J. H. Park, and Y.-S. Jeong, “Dynamic analysis for IoT malware detection with convolution neural network model,” IEEE Access, vol. 8, pp. 96899–96911, 2020.

[69] E. Amer and I. Zelinka, “A dynamic Windows malware detection and prediction method based on contextual understanding of API call sequence,” Comput. Secur., vol. 92, p. 101760, 2020.

[70] D. Rabadi and S. G. Teo, “Advanced windows methods on malware detection and classification,” in Annual Computer Security Applications Conference, 2020, pp. 54–68.

[71] X. Liu, Y. Lin, H. Li, and J. Zhang, “A novel method for malware detection on ML-based visualization technique,” Comput. Secur., vol. 89, p. 101682, 2020.

[72] J. Hemalatha, S. A. Roseline, S. Geetha, S. Kadry, and R. Damaševičius, “An efficient densenet-based deep learning model for malware detection,” Entropy, vol. 23, no. 3, p. 344, 2021.

[73] X. Huang, L. Ma, W. Yang, and Y. Zhong, “A method for windows malware detection based on deep learning,” J. Signal Process. Syst., vol. 93, pp. 265–273, 2021.

[74] N. A. Azeez, O. E. Odufuwa, S. Misra, J. Oluranti, and R. Damaševičius, “Windows PE malware detection using ensemble learning,” in Informatics, MDPI, 2021, p. 10.

[75] Ö. Aslan, M. Ozkan-Okay, and D. Gupta, “Intelligent behavior-based malware detection system on cloud computing environment,” IEEE Access, vol. 9, pp. 83252–83271, 2021.

[76] V. Rey, P. M. S. Sánchez, A. H. Celdrán, and G. Bovet, “Federated learning for malware detection in IoT devices,” Comput. Networks, vol. 204, p. 108693, 2022.

[77] K. Shaukat, S. Luo, and V. Varadharajan, “A novel deep learning-based approach for malware detection,” Eng. Appl. Artif. Intell., vol. 122, p. 106030, 2023.

[78] M. Nobakht, R. Javidan, and A. Pourebrahimi, “DEMD-IoT: A deep ensemble model for IoT malware detection using CNNs and network traffic,” Evol. Syst., vol. 14, no. 3, pp. 461–477, 2023.

[79] S. H. Khan et al., “A new deep boosted CNN and ensemble learning based IoT malware detection,” Comput. Secur., vol. 133, p. 103385, 2023.

[80] A. El-Ghamry, T. Gaber, K. K. Mohammed, and A. E. Hassanien, “Optimized and efficient image-based IoT malware detection method,” Electronics, vol. 12, no. 3, p. 708, 2023.

[81] Y. Z. Wei, M. Md-Arshad, A. A. Samad, and N. Ithnin, “Comparing Malware Attack Detection using Machine Learning Techniques in IoT Network Traffic,” Int. J. Innov. Comput., vol. 13, no. 1, pp. 21–27, 2023.

[82] E. S. Alomari et al., “Malware detection using deep learning and correlation-based feature selection,” Symmetry (Basel)., vol. 15, no. 1, p. 123, 2023.

[83] P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “API-MalDetect: Automated malware detection framework for windows based on API calls and deep learning techniques,” J. Netw. Comput. Appl., vol. 218, p. 103704, 2023.

[84] S. Alsubai, A. K. Dutta, A. M. Alnajim, R. Ayub, A. M. AlShehri, and N. Ahmad, “Artificial intelligence-driven malware detection framework for internet of things environment,” PeerJ Comput. Sci., vol. 9, p. e1366, 2023.

[85] A. Almaleh, R. Almushabb, and R. Ogran, “Malware API Calls Detection Using Hybrid Logistic Regression and RNN Model,” Appl. Sci., vol. 13, no. 9, p. 5439, 2023.

[86] X. Yang, D. Yang, and Y. Li, “A Hybrid Attention Network for Malware Detection Based on Multi-Feature Aligned and Fusion,” Electronics, vol. 12, no. 3, p. 713, 2023.

[87] M. N. Al-Andoli, K. S. Sim, S. C. Tan, P. Y. Goh, and C. P. Lim, “An ensemble-based parallel deep learning classifier with PSO-BP optimization for malware detection,” IEEE Access, 2023.

[88] H. Alrubayyi, G. Goteng, and M. Jaber, “AIS for Malware Detection in a Realistic IoT System: Challenges and Opportunities,” Network, vol. 3, no. 4, pp. 522–537, 2023.

[89] R. A. Devi and A. R. Arunachalam, “Enhancement of IoT device security using an Improved Elliptic Curve Cryptography algorithm and malware detection utilizing deep LSTM,” High-Confidence Comput., vol. 3, no. 2, p. 100117, 2023.

[90] S. S. Shafin, G. Karmakar, and I. Mareels, “Obfuscated memory malware detection in resource-constrained IoT devices for smart city applications,” Sensors, vol. 23, no. 11, p. 5348, 2023.

[91] A. Khan, G. Choudhary, S. K. Shandilya, D. M. Sharma, and A. K. Sharma, “A hybrid mechanism for advance IoT malware detection,” in International Conference on IoT, Intelligent Computing and Security: Select Proceedings of IICS 2021, Springer, 2023, pp. 247–259.

[92] A. Alamer, “A privacy-preserving federated learning with a secure collaborative for malware detection models using Internet of Things resources,” Internet of Things, vol. 25, p. 101015, 2024.

[93] T. Lei, J. Xue, Y. Wang, T. Baker, and Z. Niu, “An empirical study of problems and evaluation of IoT malware classification label sources,” J. King Saud Univ. Inf. Sci., vol. 36, no. 1, p. 101898, 2024.

[94] K. Shaukat, S. Luo, and V. Varadharajan, “A novel machine learning approach for detecting first-time-appeared malware,” Eng. Appl. Artif. Intell., vol. 131, p. 107801, 2024.

[95] M. Azeem, D. Khan, S. Iftikhar, S. Bawazeer, and M. Alzahrani, “Analyzing and comparing the effectiveness of malware detection: A study of machine learning approaches,” Heliyon, vol. 10, no. 1, 2024.

[96] I. T. Ahmed, B. T. Hammad, and N. Jamil, “A Comparative Performance Analysis of Malware Detection Algorithms Based on Various Texture Features and Classifiers,” IEEE Access, 2024.

[97] W. Al-Khater and S. Al-Madeed, “Using 3D-VGG-16 and 3D-Resnet-18 deep learning models and FABEMD techniques in the detection of malware,” Alexandria Eng. J., vol. 89, pp. 39–52, 2024.

[98] J. Jeon, B. Jeong, S. Baek, and Y.-S. Jeong, “Static Multi Feature-Based Malware Detection Using Multi SPP-net in Smart IoT Environments,” IEEE Trans. Inf. Forensics Secur., 2024.

[99] M. Nobakht, R. Javidan, and A. Pourebrahimi, “SIM-FED: Secure IoT malware detection model with federated learning,” Comput. Electr. Eng., vol. 116, p. 109139, 2024.

Similar Articles

You may also start an advanced similarity search for this article.