UBCBench
New sources of failures
When manually looking into results reported by the tools, we observed that for all studied tools:
- In 51 DroidBench applications (out of 158), and 9 ICCBench applications (out of 24), failures occur for reasons different from any original target criteria covered by all benchmarks.
- All failures in Google play applications occur for reasons not covered by any benchmarks.
We further identified these failure scenarios that were not covered by existing benchmark applications, and summarized all new criteria attributing to failures in the table below. The table has two columns, which are:
- CriteriaID: All new criteria we identified are indexed in the format U-DB[cat].[index], where [cat] is the category of DroidBench and [index] is an index in that category; TO indicates cases where a tool ran out of time to analyze an app; UN indicates cases where the reason for the failure is unknown.
- CriteriaDescription: This describe the details about newly found criteria.
- Tool: The tool(s) where we identified the new criteria.
| CriteriaID | CriteriaDescription | Tool |
|---|---|---|
| U-DB2.12 | Precisely model Android framework methods. | FlowDroid |
| U-DB4.15 | Detect flows from location-related sources to sinks. | Amandroid |
| U-DB4.16 | Do not treat callback Intent parameters as sources . | Amandroid |
| U-DB4.17 | Resolve setContentView(int) to the correct layout file when the argument is not directly a resource ID. | FlowDroid |
| U-DB4.18 | Collect callbacks registered in fragments. | FlowDroid, Amandroid |
| U-DB7.24 | Handle casting statements. | FlowDroid, Amandroid |
| U-DB10.6 | Recongize sources and sinks invoked using reflection. | FlowDroid |
| U-DB10.7 | Handle reflective calls with method getClass(). | DroidSafe |
| U-DB10.8 | Handle method overloading. | DroidSafe |
| U-DB10.9 | Handle reflective calls using strings defined in resource files. | DroidSafe |
| U-DB10.10 | Handle dynamically-constructed reflective calls. | DroidRA |
| U-DB10.11 | Handle reflective calls to Android framework methods. | DroidRA |
| U-DB10.12 | Handle reflective calls with method getConstructor(). | FlowDroid, DroidRA |
| TO | Fail to detect flow due to running out of time. | - |
| UN | Fail to detect flow due to unknown reasons. | - |
Download link
You can find our newly-developed benchmark suite UBCBench used in the paper here.
Fixed bugs
We identified eight bugs in FlowDroid when performing our analysis. Three of the bugs lead to crashes of FlowDroid. We notified the authors of these bugs, and worked with them to fix them. The below table lists these bugs with the following information:
- CriteriaID: All bugs are indexed in the format BUGx or EX (for those lead to crashes).
- BugDescription: Descrption and patches for a bug.
- AffectedApp: The IDs of benchmark or Google Play apps that are affected by a bug.
| CriteriaID | BugDescription | AffectedApp | ||
|---|---|---|---|---|
| DroidBench | ICCBench | GooglePlay | ||
| BUG1 | Bug of StubDroid's summaries for handling Parcel. Reported by us, fixed by the authors (see commit). | 11 | ||
| Bug of StubDroid's summaries on primitive types. Reported by us, fixed by the authors (see commit). | 13 | |||
| Bug of StubDroid's summaries for handling reflection. Reported by us, fixed by the authors (see commit). | 132, 134, 136, 137, 138 | |||
| BUG2 | Bug in IccTA when handling Broadcast Receivers. Fixed by us (see pull request). | 96 | 11, 12 | |
| Bug in IccTA when handling IntentServices. Fixed by us (see pull request). | 9 | |||
| EX | Exception thrown in IccTA when handling bound Services. Fixed by us (see pull request). | 102 | 20, 21, 22, 23, 24 | 2, 7, 14 |
| Exception thrown in IccTA when handling ICC methods with no parameter. Fixed by us (see pull request). | 1, 16 | |||
| Exception thrown in Soot when handling reflection. Fixed by us (see pull request). | 5, 8, 12, 18, 19 | |||