Deep learning for malware detection using API call graphs

The increasing use of computers in daily life resulted in a rise in sophisticated computer attacks, with malware continually evolving to bypass traditional security measures. This study focused on leveraging deep learning techniques for malware detection by analysing API call graphs, which represent...

Full description

Bibliographic Details
Main Author: Rajandaran, Usharani
Format: Dissertation
Language:English
Published: Universiti Teknologi Malaysia 2026
Subjects:
Online Access:https://utmik.utm.my/handle/123456789/190846
Abstract Abstract here
Description
Summary:The increasing use of computers in daily life resulted in a rise in sophisticated computer attacks, with malware continually evolving to bypass traditional security measures. This study focused on leveraging deep learning techniques for malware detection by analysing API call graphs, which represent the interactions between programs. Traditional detection methods, relying on fixed patterns, often failed to identify new and adaptive malware. Deep learning, particularly through techniques like Graph Convolutional Neural Networks (GCNNs), offered a promising solution due to its ability to understand complex patterns in API call sequences. This research aimed to develop a deep learning model optimized using Deep Graph Convolutional Neural Networks (DGCNNs) for effective malware detection. By examining the sequence of API calls, the study intended to enhance the accuracy and reliability of malware identification and classification. The objectives included selecting relevant features for the deep learning model, developing a specialized malware detection model using DGCNNs, and evaluating the performance against existing models. Results indicated that the proposed DGCNN model consistently outperformed others in terms of AUC-ROC, F1-score, precision, recall, and accuracy. Specifically, the model achieved an AUC-ROC of 0.9740, an F1-score of 0.9939, precision of 0.9893, recall of 0.9987, and accuracy of 0.9881 on a balanced dataset. The study emphasized the importance of balanced datasets for robust model training and discussed the challenges posed by imbalanced datasets. The findings suggested that DGCNNs were highly effective in behavioural malware detection, providing a robust approach to securing systems against evolving malware threats. Future work will explore more complex architectures and multiclass malware classification to further enhance detection capabilities. followed by the English translation. If the thesis is written in English, the abstract must be written in English and followed by the translation in Bahasa Melayu. The abstract should be brief, written in one paragraph and not exceed one (1) page. An abstract is different from synopsis or summary of a thesis. It should states the field of study, problem definition, methodology adopted, research process, results obtained and conclusion of the research. The abstract can be written using single or one and a half spacing. Example can be seen in Appendix 1 (Bahasa Melayu) and Appendix J (English).