The Illusion of Success: Learning-Based Android Malware Detectors (Replicability Study)

Machine learning (ML) approaches have shown impressive performance in Android malware detection and have become the scalable alternatives to signature and heuristic-based methods. ML-based detectors typically collect large amounts of benign and malware applications from popular data sources, split the collected dataset into training and testing subsets, train a binary classifier, and report test set performance. While these tools report 95%+ recall, several recent works have shed light on flaws in experiment setup or evaluation methods questioning the trustworthiness of their results.

In an effort to further understand the true performance of the tools, we conduct a systematic literature review of tools and design large-scale longitudinal evaluation using widely used datasets and detection approaches. Even after carefully accounting for the known biases in experiment setup and evaluation, we observe that tools achieve significantly diverging classification performance (ranging from 43% to 94% recall) even on different subsets of the same dataset. Surprisingly, we also find that a deliberately flawed classifier that uses unreliable class package names as features performs nearly as good, sometimes better, than the set of tools we evaluated on these datasets, showing the untrustworthiness of tool-reported results.

Our results further establish that there is still much room for improvement in ML-based AMD towards creating semantic-aware detectors and more reliable evaluation methodologies.

This website provides additional information about the paper.