Benchmarks

Sources and sinks

We provided the list of sources and sinks we used for benchmark apps (SS-Bench) in Section 3.3.2 (Table 3) of our paper.

Selected benchmarks and expected results

This section contains (1) the full list of benchmarks (from DroidBench and ICCBench) that we used in our study, (2) target criterion of each application, and (3) the ground truth we followed, determined by the list of sources/sinks and tool configurations we used. There are seven columns in the table. Here we explain every column and provide suggestions to the readers about how to read the table:

• Category: In DroidBench and ICCBench, authors put benchmarks to different folders based on what aspect of Android/Java the benchmark is designed to test. We extracted the folder name as categories, and numbered them.
• AppID: We numbered all benchmarks in DroidBench we use, from 1 to 158, obeying the order of “Category”. The same to ICCBench benchmarks.
• AppName: The name of the benchmark, which are extracted from DroidBench and ICCBench test suites.
• TargetCriteriaID and TargetCriteriaDesciption: Each of the DroidBench and ICCBench application is designed to test a particular aspect of taint analysis, which we refer to as the Target Criterion (short as Criterion) of the benchmark. Developers generally annotate the criterion in source code. We extract this information from DroidBench and ICCBench applications annotations, which is the column “CriteriaDesciption”. For easy reference, we number each criterion in the format [Test Suite Name][Category ID][subid of application inside this category], for example, if we look at the first row, “DB1.1” means the criterion “Aliases must be computed precisely or a false positive will be found.” is tested by the first benchmark “Merge1” in the first category “Aliasing” in DroidBench test suite.
• Number of designer’s expected flow(s): In DroidBench and ICCBench, developers annotated the number of expected leaks, which are what we extracted to this column.
• Number of our expected flow(s): With our selection of sources, sinks, and configuration parameters discussed in previous subsections, the expected flows for each benchmark application might deviate from the result annotated by the benchmark designers. We thus manually analyzed each of the benchmark applications, extracting all flows expected under our configuration setup.

All benchmarks whose Number of designer’s expected flow(s) are different from Number of our expected flow(s) are highlighted in blue.

Category	AppID	AppName	Target CriteriaID	TargetCriteriaDescription	# designer's expected flow(s)	# our expected flow(s)
DroidBench
1. Aliasing	1	FlowSensitivity1	DB1.1	Aliases must be computed in a flow-sensitive fashion, otherwise a false positive is found.	0	0
	2	Merge1	DB1.2	Aliases must be computed precisely or a false positive will be found.	0	1
	3	SimpleAliasing1	DB1.3	Aliases must be computed soundly or the leak will be missed.	1	1
	4	StrongUpdate1	DB1.4	The alias analysis must support strong updates for not causing a false positive.	1	0
2. Android Specific	5	ApplicationModeling1	DB2.1	Aliases must be computed precisely or a false positive will be found.	1	0
	6	DirectLeak1	DB2.2	The value of a source is directly sent to a sink.	1	1
	7	InactiveActivity	DB2.3	The analysis has to be aware that the activity is set to inactive in manifest file.	0	0
	8	Library2	DB2.4	The analysis must correctly handle custom libraries.	1	1
	9	LogNoLeak	DB2.5	The analysis has to be aware that no tainted value can reach the sink.	0	0
	10	Obfuscation1	DB2.6	The analysis must not be fooled by fake implementations of system classes contained in the APK file.	1	1
	11	Parcel1	DB2.7	Parcel marshall and unmarshalling.	1	1
	12	PrivateDataLeak3	DB2.8	The analysis must propagate taints across file system accesses.	2	1
	13	PublicAPIField1	DB2.9	Must have accurate modeling for API classes that expose fields.	1	1
	14	PublicAPIField2	DB2.10	Analysis must have a model of Intent implementation to setter / getter of Intent fields.	1	1
	15	View1	DB2.11	The analysis needs to properly include custom views into the lifecycle.	1	1
3. Arrays And Lists	16	ArrayAccess1	DB3.1	The analysis must distinguish between different array positions to recognize that the tainted data does not get leaked.	0	0
	17	ArrayAccess2	DB3.2	The analysis must distinguish between different array positions and has to evaluate the function to recognize that the tainted data does not get leaked.	0	0
	18	ArrayAccess3	DB3.3	The analysis must support arrays.	1	1
	19	ArrayAccess4	DB3.4	The analysis must correctly handle fields of objects inside arrays.	0	0
	20	ArrayAccess5	DB3.5	The analysis must correctly distinguish between the size of an array and its elements.	0	0
	21	ArrayCopy1	DB3.6	The analysis tool must have a model for System.arraycopy().	1	1
	22	ArrayToString1	DB3.7	Have to model that Array.toString invokes toString() for each object of array.	1	1
	23	HashMapAccess1	DB3.8	The analysis must distinguish between different hash map entries to recognize that the tainted data does not get leaked.	0	0
	24	ListAccess1	DB3.9	The analysis must distinguish between different list positions to recognize that the tainted data does not get leaked.	0	0
	25	MultidimensionalArray1	DB3.10	The analysis tool has to be able to track an element within a multidimensional array.	1	1
4. Callbacks	26	AnonymousClass1	DB4.1	The analysis must handle callbacks, anonymous inner classes and static fields.	2	2
	27	Button1	DB4.2	The analysis must analyze the layout xml file and take the lifecycle into account (onCreate is executed before user interaction).	1	1
	28	Button2	DB4.3	The analysis must be able to analyze listeners, know that callback of button3 is ClickOnButton3 (defined in xml file) and has to handle the arbitrary order of the listener callbacks.	3	3
	29	Button3	DB4.4	The analysis must be able to detect callback handlers registered in other callback handlers.	1	1
	30	Button4	DB4.5	The analysis must analyze the layout xml file and correctly handle include directives.	1	1
	31	Button5	DB4.6	Must correctly model that a Button is represented by a single object in the runtime, and that object is delivered to multiple calls of onClick.	1	1
	32	LocationLeak1	DB4.7	The analysis must be able to emulate the Android activity lifecycle correctly, integrate the callback method onLocationChanged and detect the callback methods as source. This example contains a location information leakage in the onResume() callback method. The data source is placed into the onLocationChanged() callback method, especially the parameter loc.	2	2
	33	LocationLeak2	DB4.8	The analysis must be able to emulate the Android activity lifecycle correctly, integrate the callback method onLocationChanged and detect the callback methods as source. This example contains a location information leakage in the onResume() callback method. The data source is placed into the onLocationChanged() callback method, especially the parameter loc. In contrast to LocationLeak1 the activity implements the Listener directly (no inner class).	2	2
	34	LocationLeak3	DB4.9	The analysis must be able to emulate the Android activity lifecycle correctly, integrate the callback method onLocationChanged, detect the callback methods as source and connect the callback class to the activity via the interface.	1	2
	35	MethodOverride1	DB4.10	The analysis must be able to detect the leak in an internal Android method that is overwritten.	1	1
	36	MultiHandlers1	DB4.11	The analysis must be able to correctly associate callback handlers with the respective activities.	0	0
	37	Ordering1	DB4.12	The analysis must take the order of callback registration and sink call into account.	0	0
	38	RegisterGlobal1	DB4.13	The analysis must support globally-registered callback handlers.	1	1
	39	RegisterGlobal2			1	1
	40	Unregister1	DB4.14	The analysis must take into accounts that callbacks can be unregistered again.	0	0
5. Emulator Detection	41	Battery1	DB5.1	The (dynamic) analysis must avoid being detected and circumvented.	1	1
	42	Bluetooth1			1	1
	43	Build1			1	1
	44	Contacts1			1	1
	46	DeviceId1			1	1
	47	File1			1	1
	49	IP1			1	1
	50	PI1			1	1
	52	PlayStore2			1	1
	53	Sensors1			1	1
	54	SubscriberId1			1	1
	55	VoiceMail1			1	1
	45	ContentProvider1	DB5.2	The (dynamic) analysis must avoid being detected and circumvented. Sends the IMEI as an SMS message and writes it to the log file. Emulator detection is performed on the IMEI and the app is killed if the IMEI is a 00..00 pattern.	2	2
	48	IMEI1	DB5.3	The (dynamic) analysis must avoid being detected and circumvented. Sends the IMEI as an SMS message and writes it to the log file. Emulator detection is performed by cutting the secret message at an index computed on the IMEI which is known to always be 000..0 on an emulator.	2	0
	51	PlayStore1	DB5.4	The (dynamic) analysis must avoid being detected and circumvented. Sends the IMEI as an SMS message and writes it to the log file. Emulator detection is performed by checking whether the Play Store app is installed on the system.	2	2
6. Field And Object Sensitivity	56	FieldSensitivity1	DB6.1	The analysis must be able to distinguish between different fields of an object.	0	0
	57	FieldSensitivity2	DB6.2	The analysis must be able to distinguish between different fields of an object. An object has two fields, the one that gets tainted is sent to a sink.	0	0
	58	FieldSensitivity3	DB6.3	The analysis must be able to distinguish between different fields of an object. This is an easier version of FieldSensitivity1.	1	1
	59	FieldSensitivity4	DB6.4	The analysis has to consider the order of the statements for tainted fields.	0	0
	60	InheritedObjects1	DB6.5	The analysis must be able to decide on the subtype of a variable based on a condition.	1	1
	61	ObjectSensitivity1	DB6.6	The analysis must be able to distinguish between two objects of the same type that are initialized by the same constructor.	0	0
	62	ObjectSensitivity2	DB6.7	The analysis must be able to remove taints from variables and fields.	0	0
7. General Java	63	Clone1	DB7.1	Must model clone of list.	1	1
	64	Exceptions1	DB7.2	The analysis must handle exceptions.	1	1
	65	Exceptions2	DB7.3	The analysis must handle implicit exceptions.	1	1
	66	Exceptions3	DB7.4	The analysis must precisely model which exceptions can occur and which ones can't.	0	0
	67	Exceptions4	DB7.5	The analysis must handle exception data.	1	1
	68	Exceptions5	DB7.6	The analysis must handle exception data across method calls.	1	1
	69	Exceptions6			1	1
	70	Exceptions7	DB7.7	The analysis must precisely handle exception types across method calls.	0	0
	71	FactoryMethods1	DB7.8	The analysis must be able to handle factory methods contained in the operating system.	2	2
	72	Loop1	DB7.9	The analysis must handle standard java constructs.	1	1
	73	Loop2			1	1
	74	Serialization1	DB7.10	Must model serialization.	1	1
	75	SourceCodeSpecific1	DB7.11	The analysis must handle standard java constructs. Tainted data is created in a condition branch and afterwards sent to a sink in a loop.	1	1
	76	StartProcessWithSecret1	DB7.12	The analysis tool has to be able to track taint flown to sink through a receiver.	1	1
	77	StaticInitialization1	DB7.13	The analysis has to consider static initialization. Tainted data is leaked to a sink during static initialization of a class.	1	1
	78	StaticInitialization2	DB7.14	The analysis has to consider static initialization. Sensitive data is obtained during static initialization of a class and leaked in non-static code.	1	1
	79	StaticInitialization3	DB7.15	The order of execution of static initializers is not defined in Java. This test stresses a particular order to link a flow.	1	1
	80	StringFormatter1	DB7.16	Modeling of StringBuffer and StringFormatter.	1	1
	81	StringPatternMatching1	DB7.17	Flows through multiple object allocated in API code.	1	1
	82	StringToCharArray1	DB7.18	The analysis tool has to be able to follow taint through character-string conversion.	1	1
	83	StringToOutputStream1	DB7.19	The analysis tool has to be able to track tainted value through different stream/memory operations.	1	1
	84	UnreachableCode	DB7.20	The analysis has to discover that the unit is not called.	0	0
	85	VirtualDispatch1	DB7.21	The analysis must be able to handle invoke-virtual statements. Additionally the clickButton() callback must be correctly considered as a callback.	1	1
	86	VirtualDispatch2	DB7.22	The analysis tool has to be able to differentiate the base and the derived class objects.	1	1
	87	VirtualDispatch3	DB7.23	The callgraph analysis must be able to deal with factory methods.	0	0
8. Inter Component Communication	88	ActivityCommunication1	DB8.1	The analysis must be able to model the lifecycle of Activities and allow arbitrary execution order of the Activities.	1	1
	89	ActivityCommunication2	DB8.2	The analysis tool has to be able to evaluate substring operation and track tainted value through another Activity.	1	2
	90	ActivityCommunication3	DB8.3	The analysis tool has to be able to resolve component from a name and track it through startActivity.	1	2
	91	ActivityCommunication4	DB8.4	The analysis tool needs to be able to analyze constant string with concatenation operation and able to resolve the Intent for the resulted string and follow tainted data to the next Activity.	1	2
	92	ActivityCommunication5	DB8.5	The analysis tool must be able to resolve Intent's component from a component name and follow the taint to another Activity.	1	2
	93	ActivityCommunication6	DB8.6	The analysis tool has to be able to track an Intent through a list operation.	1	2
	94	ActivityCommunication7	DB8.7	The analysis tool has to be able to resolve an Intent of a non-constant Activity class.	1	2
	95	ActivityCommunication8	DB8.8	The analysis tool has to be able to track a string through the list, then resolve Intent's action and follow the Intent to the Activity to be activated.	1	2
	96	BroadcastTaintAndLeak1	DB8.9	The analysis tool has to be able to recognize a broadcast receiver and models its IntentFilter.	1	2
	97	ComponentNotInManifest1	DB8.10	The analysis must recognize that activity is not startable if it is not in the AndroidManifest.xml	0	1
	98	EventOrdering1	DB8.11	The analysis tool has to be able to take into account different runs of the app. In this case, the end of one run is the source and the beginning of the next run is the sink.	1	1
	99	IntentSink1	DB8.12	The analysis must be able to track the taint in the intent and recognize the setResult sink.	1	1
	100	IntentSink2	DB8.13	The analysis must be able to parse the layout xml file correctly, track the taint in the intent and recognize the startActivity sink.	1	1
	101	IntentSource1	DB8.14	The analysis must be able to track the taint in the intent and recognize the onActivityResult(-1,-1,intent) source.	2	0
	102	ServiceCommunication1	DB8.15	The analysis tool has to be able through ICC message passing / Handler constructs.	1	1
	103	SharedPreferences1	DB8.16	Modeling of SharedPreferences.	1	1
	104	Singletons1	DB8.17	Analysis has to reason about all legal interleavings of separate components.	1	1
	105	UnresolvableIntent1	DB8.18	The analysis tool has to be able to identify unresolvable Intent and not associate the unresolved Intent with any Activity.	2	3
9. Lifecycle	106	ActivityEventSequence1	DB9.1	The analysis tool must be able to incorporate only those callback flows which can occur in real Android apps. onRestoreInstanceState() callback is not invoked in all cases.	1	1
	107	ActivityEventSequence2	DB9.2	The analysis tool must be able to detect data leaks which are triggered by different ordering of event sequences. Each event in the event sequence invokes a set of callbacks in a specific order. Following attack occurs when onUserLeaveHint() -> onUserLeaveHint()->onSaveInstanceState()->onRestoreInstanceState()->onSaveInstanceState() callbacks are invoked in the given order.	1	0
	108	ActivityEventSequence3	DB9.3	The analysis tool must be able to detect data leaks which are triggered by different ordering of event sequences. Each event in the event sequence invokes a set of callbacks in a specific order.	1	1
	109	ActivityLifecycle1	DB9.4	The analysis must be able to handle the activity lifecycle correctly and handle try/catch blocks.	1	1
	110	ActivityLifecycle2	DB9.5	The analysis must be able to handle the activity lifecycle correctly and detect the callback method that is inherited from a superclass.	1	1
	111	ActivityLifecycle3	DB9.6	The analysis must be able to handle the activity lifecycle correctly, in particular the less common instanceState callbacks.	1	1
	112	ActivityLifecycle4	DB9.7	The analysis must be able to handle the activity lifecycle correctly and deal with loops in the lifecycle because originally onPause is called before onResume.	1	1
	113	ActivitySavedState1	DB9.8	Event ordering and Activity saved state.	1	1
	114	ApplicationLifecycle1	DB9.9	Correct handling of the Application object in the component lifecycle.	1	1
	115	ApplicationLifecycle2	DB9.10	Correct handling of callbacks in the Application object.	1	1
	116	ApplicationLifecycle3	DB9.11	Correct handling of the Application object and the ContentProvider. Note that the ContentProvider.onCreate() method is called before Application.onCreate() is invoked.	1	1
	117	AsynchronousEventOrdering1	DB9.12	The analysis must account for all legal ordering of asynch events with respect to the activity lifecycle.	1	1
	118	BroadcastReceiverLifecycle1	DB9.13	The analysis must be able to handle the broadcast receiver lifecycle correctly and evaluate the condition.	1	1
	119	BroadcastReceiverLifecycle2	DB9.14	The analysis must be able to handle the dynamic registration of broadcast receivers.	1	1
	120	BroadcastReceiverLifecycle3			1	1
	121	EventOrdering1	DB9.15	Test case for considering all possible event orderings for event. There is a leak when onLowMemory is called twice without a call to onContentChanged().	1	1
	122	FragmentLifecycle1	DB9.16	The analysis has to be aware of the lifecycle methods of fragments. The fragment is not defined in xml but in the code.	1	1
	123	FragmentLifecycle2	DB9.17	The analysis tool must handle Fragment, its callbacks, and association with the parent Activity properly.	1	1
	124	ServiceEventSequence1	DB9.18	The analysis tool must be able to detect data leaks which are triggered by different ordering of events.	1	1
	125	ServiceEventSequence2	DB9.19	The analysis tool must be able to analyze all possible flows from onUnbind() callback, that is, onBind() and onRebind().	1	1
	126	ServiceEventSequence3			1	1
	127	ServiceLifecycle1	DB9.20	The analysis must be able to handle the service lifecycle correctly.	1	1
	128	ServiceLifecycle2	DB9.21	Same service object is used for each startService -> onStartCommand call.	1	1
	129	SharedPreferenceChanged1	DB9.22	In this case, the change listener has to be called with the shared preferences that are changed.	1	1
10. Reflection	130	Reflection1	DB10.1	The analysis must be able to handle reflective class instantiations.	1	1
	131	Reflection2	DB10.2	The analysis must be able to handle code implemented in classes loaded using reflection.	1	1
	132	Reflection3	DB10.3	The analysis must be able to reflective invocations of methods without type of information on the target class.	1	1
	133	Reflection4	DB10.4	The analysis must be able to correctly handle sources and sinks in classes used through reflection.	1	1
	134	Reflection5	DB10.5	The analysis must recognize that all type of reflections.	1	1
	135	Reflection6			1	1
	136	Reflection7			1	1
	137	Reflection8			1	1
	138	Reflection9			1	1
11. Reflection ICC	139	ActivityCommunication2	DB11.1	The analysis must be able to handle reflective method invocation and inter-component communication.	1	2
	140	AllReflection			1	2
	141	OnlyIntent			1	2
	142	OnlyIntentReceive			1	2
	143	OnlySMS			1	2
	144	OnlyTelephony			1	2
	145	OnlyTelephony_Dynamic			2	3
	146	OnlyTelephony_Reverse			1	2
	147	OnlyTelephony_Substring			1	2
	148	SharedPreferences1			1	2
12. Threading	149	AsyncTask1	DB12.1	The analysis must be able to correctly handle Android's AsyncTask mechanism.	1	1
	150	Executor1	DB12.2	The analysis must be able to correctly handle Java's Executor mechanism.	1	1
	151	JavaThread1	DB12.3	The analysis must be able to correctly handle Java threads. Sensitive data is read in onCreate() and send out in a dedicated thread started using Java's normal threading mechanism.	1	1
	152	JavaThread2	DB12.4	The analysis must be able to correctly handle Java threads. Sensitive data is read in onCreate() and send out in a dedicated thread started using Java's Runnable mechanism.	1	1
	153	Looper1	DB12.5	The analysis must be able to correctly handle Android's Looper infrastructure.	1	1
	154	TimeTask1	DB12.6	The analysis must be able to correctly handle Java's TimerTask infrastructure.	1	1
13. Unreachable Code	155	SimpleUnreachable1	DB13.1	The analysis should detect that some branches are unreachable.	0	0
	156	UnreachableBoth			0	0
	157	UnreachableSink1			0	0
	158	UnreachableSource1			0	0
ICCBench
1. Icc Handing	1	icc_explicit_nosrc_nosink	ICC1.1	The analysis must be able to resolve explicit ICC calls and handle data flow across different components. Insensitive v is sent to component FooActivity via explicit ICC.	0	0
	2	icc_explicit_nosrc_sink			0	0
	3	icc_explicit_src_nosink	ICC1.2	The analysis must be able to resolve explicit ICC calls and handle data flow across different components. The value v of a source is sent to component FooActivity via explicit ICC.	0	1
	4	icc_explicit_src_sink			1	2
	13	icc_explicit1			1	2
	5	icc_implicit_nosrc_nosink	ICC1.3	The analysis must be able to resolve implicit (Action) ICC calls and handle data flow across different components. Insensitive value v is sent to component FooActivity via implicit ICC.	0	0
	6	icc_implicit_nosrc_sink			0	0
	7	icc_implicit_src_nosink	ICC1.4	The analysis must be able to resolve implicit (Action) ICC calls and handle data flow across different components. The value v of a source is sent to component FooActivity via implicit ICC.	1	1
	8	icc_implicit_src_sink			2	2
	14	icc_implicit_action			2	2
	9	icc_intentservice	ICC1.5	The analysis must be able to resolve IntentService and handle data flow across different components.	1	1
	10	icc_stateful	ICC1.6	The analysis must be able to resolve stateful ICC call and handle data flow across different components.	3	2
2. Icc Target Finding	11	icc_dynregister1	ICC2.1	The analysis must be able to resolve dynamically registered component, implicit ICC calls and handle data flow across different components.	2	2
	12	icc_dynregister2	ICC2.2	The analysis must be able to handle string manipulation, resolve dynamically registered component, implicit ICC calls and handle data flow across different components.	2	2
	15	icc_implicit_category	ICC2.3	The analysis must be able to resolve implicit (Categories) ICC calls and handle data flow across different components.	2	2
	16	icc_implicit_data1	ICC2.4	The analysis must be able to resolve implicit (Data) ICC calls and handle data flow across different components.	2	2
	17	icc_implicit_data2	ICC2.5	The analysis must be able to resolve implicit (MType) ICC calls and handle data flow across different components.	2	2
	18	icc_implicit_mix1	ICC2.6	The analysis must be able to resolve implicit (Data & MType) ICC calls and handle data flow across different components.	3	3
	19	icc_implicit_mix2	ICC2.7	The analysis must be able to resolve implicit (Mix all) ICC calls and handle data flow across different components.	2	2
3. Mixed	20	icc_rpc_comprehensive	ICC3.1	The analysis must be able to resolve AIDL, stateful ICC and handle data flow across different components.	3	2
4. Rpc Handing	21	rpc_localservice	ICC4.1	The analysis must be able to resolve LocalService and handle data flow across different components.	1	1
	22	rpc_messengerservice	ICC4.2	The analysis must be able to resolve MessengerService and handle data flow across different components.	1	1
	23	rpc_remoteservice	ICC4.3	The analysis must be able to resolve AIDL and handle data flow across different components.	1	1
	24	rpc_returnsensitive	ICC4.4	The analysis must be able to resolve LocalService and handle data flow across different components.	1	1

Excluded benchmarks

As discussed in Section 3.3.1 of our paper, we excluded 31 applications from DroidBench that focus on testing configuration options that we disabled. We listed these applications in the following table.

The table consists of the following columns:

• Category: The folder name that contains the application.
• AppName: The name of the benchmark.
• Reason: The reason why we excluded an application.

For category Inter-App Communication, there are 10 applications in total. These 10 applications form two IAC test cases.

Category	AppName	Reason
Dynamic Code Loading	DynamicBoth1	Unsupported functionality: all three tools explicitly claim that they cannot handle dynamic code loading.
	DynamicSink1
	DynamicSource1
Inter-App Communication	Echoer+SendSMS+StartActivityForResult1	Disabled configuration option: IAC
	Collector+DeviceId_Broadcast1+DeviceId_ContentProvider1+ DeviceId_OrderedIntent1+DeviceId_Service1+Location1+Location_Broadcast1
General Java	VirtualDispatch4	Disabled configuration option: implicit flows
Android Specific	PrivateDataLeak1	Disabled configuration option: sensitive UI detection
	PrivateDataLeak2
Implicit Flows	ImplicitFlow1	Disabled configuration option: implicit flows
	ImplicitFlow2
	ImplicitFlow3
	ImplicitFlow4
	ImplicitFlow5
	ImplicitFlow6
Self-Modification	BytecodeTamper1	Disabled configuration option: native code analysis
	BytecodeTamper2
	BytecodeTamper3
	BytecodeTamper4
Native Code	JavaIDFunction	Disabled configuration option: native code analysis
	NativeIDFunction
	SinkInNativeCode
	SinkInNativeLibCode
	SourceInNativeCode

Sources of failures

Our analysis results can be downloaded here. In this section, we detailedly list our manual inspection results about sources of failures on a benchmark basis. The table below have multiple columns, we will describe each column and provide instructions on how to read the table.

The table consists of two main parts, which are:

• AppName and TargetCriteriaID: These two columns are information of benchmarks, matching AppID+AppName, and TargetCriteriaID in Selected benchmarks and expected results table.

• The left columns are all about results of studied tools. We list details about FP (false positives) and FN (false negatives) flows reported by each tool. For each benchmark, we list:

  • #FP/#FN: The number of FPs/FNs reported by a tool.

  • FailingCriteria: Detailed breakdown of all FPs(/FNs) got reported by a tool on the benchmark. The format is [Number of FPs(/FNs)]x[FailingCriteria of The FPs(/FNs)]. Note that we highlighted flows reported by tools which are not original designed for the tool. We also included flows affected by special semantics of Amandroid and DroidSafe in this column, denoted by the following:

    • SEM1: Amandroid and DroidSafe filter out intra-component flows for ICC-related sinks.
    • SEM2: DroidSafe reports the location of the entry method.

Let’s take the benchmark ActivityCommunication2 as an example, what we can learn from the table is:

• FlowDroid reported 1 FP due to criterion DB8.2. Details about the criterion can be found in table Selected benchmarks and expected results.
• Amandroid reported 2 FPs. One FP is due to criterion DB8.2, which is expected, and an FN is due to criterion UBC4, which is unexpected.
• DroidSafe successfully handled the benchmark, with no FP nor FN.

Details of our newly-identified sources of failures (i.e., UBCx) can be found here.

AppName	TargetCriteriaID	FlowDroid v2.7.1				DroidRA + FlowDroid v2.7.1				Amandroid v3.2.0				DroidRA + Amandroid v3.2.0				DroidSafe				DroidRA + DroidSafe
		FP		FN		FP		FN		FP		FN		FP		FN		FP		FN		FP		FN
		#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria
DroidBench
1. FlowSensitivity1	DB1.1	0		0		0		0		1	1xDB4.12	0		1	1xDB4.12	0		1	1xDB1.1	0		1	1xDB1.1	0
2. Merge1	DB1.2	0		0		0		0		0		0		0		0		0		0		0		0
3. SimpleAliasing1	DB1.3	0		0		0		0		0		0		0		0		0		0		0		0
4. StrongUpdate1	DB1.4	0		0		0		0		0		0		0		0		1	1xDB1.1	0		1	1xDB1.1	0
5. 5. ApplicationModeling1	DB2.1	1	1xDB2.1	0		1	1xDB2.1	0		0		0		0		0		1	1xDB2.1	0		1	1xDB2.1	0
6. DirectLeak1	DB2.2	0		0		0		0		0		0		0		0		0		0		0		0
7. InactiveActivity	DB2.3	0		0		0		0		0		0		0		0		0		0		0		0
8. Library2	DB2.4	0		0		0		0		0		0		0		0		0		0		0		0
9. LogNoLeak	DB2.5	0		0		0		0		0		0		0		0		0		0		0		0
10. Obfuscation1	DB2.6	0		0		0		0		0		0		0		0		0		1	1xDB2.6	0		1	1xDB2.6
11. Parcel1	DB2.7	0		1	1xBUG1	0		1	1xBUG1	0		1	1xDB2.7	0		1	1xDB2.7	0		0		0		0
12. PrivateDataLeak3	DB2.8	0		1	1xDB2.8	0		1	1xDB2.8	0		1	1xDB2.8	0		1	1xDB2.8	0		0		0		0
13. PublicAPIField1	DB2.9	0		1	1xBUG1	0		1	1xBUG1	0		0		0		0		0		0		0		0
14. PublicAPIField2	DB2.10	0		0		0		0		0		0		0		0		0		0		0		0
15. View1	DB2.11	0		0		0		0		0		1	1xDB2.11	0		1	1xDB2.11	0		0		0		0
16. ArrayAccess1	DB3.1	1	1xDB3.1	0		1	1xDB3.1	0		1	1xDB3.1	0		1	1xDB3.1	0		1	1xDB3.1	0		1	1xDB3.1	0
17. ArrayAccess2	DB3.2	1	1xDB3.2	0		1	1xDB3.2	0		1	1xDB3.2	0		1	1xDB3.2	0		1	1xDB3.2	0		1	1xDB3.2	0
18. ArrayAccess3	DB3.3	0		0		0		0		0		0		0		0		0		0		0		0
19. ArrayAccess4	DB3.4	0		0		0		0		0		0		0		0		0		0		0		0
20. ArrayAccess5	DB3.5	1	1xDB3.5	0		1	1xDB3.5	0		0		0		0		0		0		0		0		0
21. ArrayCopy1	DB3.6	0		0		0		0		0		1	1xDB3.6	0		1	1xDB3.6	0		0		0		0
22. ArrayToString1	DB3.7	0		0		0		0		0		1	1xDB3.7	0		1	1xDB3.7	0		0		0		0
23. HashMapAccess1	DB3.8	1	1xDB3.8	0		1	1xDB3.8	0		1	1xDB3.8	0		1	1xDB3.8	0		1	1xDB3.8	0		1	1xDB3.8	0
24. ListAccess1	DB3.9	1	1xDB3.9	0		1	1xDB3.9	0		1	1xDB3.9	0		1	1xDB3.9	0		1	1xDB3.9	0		1	1xDB3.9	0
25. MultidimensionalArray1	DB3.10	0		0		0		0		0		1	1xDB3.10	0		1	1xDB3.10	0		0		0		0
26. AnonymousClass1	DB4.1	0		0		0		0		2	1xU-DB2.12 1xU-DB4.15	2	2xU-DB4.15	2	1xU-DB2.12 1xU-DB4.15	2	2xU-DB4.15	0		0		0		0
27. Button1	DB4.2	0		0		0		0		0		0		0		0		0		0		0		0
28. Button2	DB4.3	1	1xDB6.7	0		1	1xDB6.7	0		0		0		0		0		1	1xDB1.1	0		1	1xDB1.1	0
29. Button3	DB4.4	0		0		0		0		0		1	1xDB4.4	0		1	1xDB4.4	0		0		0		0
30. Button4	DB4.5	0		0		0		0		0		1	1xDB4.5	0		1	1xDB4.5	0		0		0		0
31. Button5	DB4.6	0		1	1xU-DB7.24	0		1	1xU-DB7.24	0		1	1xU-DB4.15	0		1	1xU-DB4.15	0		0		0		0
32. LocationLeak1	DB4.7	0		0		0		0		3	1xU-DB2.12 2xU-DB4.15	2	2xU-DB4.15	3	1xU-DB2.12 2xU-DB4.15	2	2xU-DB4.15	0		0		0		0
33. LocationLeak2	DB4.8	0		0		0		0		3	1xU-DB2.12 2xU-DB4.15	2	2xU-DB4.15	3	1xU-DB2.12 2xU-DB4.15	2	2xU-DB4.15	0		0		0		0
34. LocationLeak3	DB4.9	0		0		0		0		0		2	2xU-DB4.15	0		2	2xU-DB4.15	0		0		0		0
35. MethodOverride1	DB4.10	0		0		0		0		0		0		0		0		0		0		0		0
36. MultiHandlers1	DB4.11	0		0		0		0		0		0		0		0		0		0		0		0
37. Ordering1	DB4.12	0		0		0		0		3	1xU-DB2.12 2xU-DB4.15	0		3	1xU-DB2.12 2xU-DB4.15	0		2	2xDB4.12	0		2	2xDB4.12	0
38. RegisterGlobal1	DB4.13	0		0		0		0		0		1	1xDB9.9	0		1	1xDB9.9	0		0		0		0
39. RegisterGlobal2		0		0		0		0		0		1	1xDB9.9	0		1	1xDB9.9	0		0		0		0
40. Unregister1	DB4.14	1	1xDB4.14	0		1	1xDB4.14	0		1	1xDB4.14	0		1	1xDB4.14	0		1	1xDB4.14	0		1	1xDB4.14	0
41. Battery1	DB5.1	0		0		0		0		0		0		0		0		0		0		0		0
42. Bluetooth1		0		0		0		0		0		0		0		0		0		0		0		0
43. Build1		0		0		0		0		0		0		0		0		0		0		0		0
44. Contacts1		0		0		0		0		0		0		0		0		0		0		0		0
46. DeviceId1		0		0		0		0		0		0		0		0		0		0		0		0
47. File1		0		0		0		0		0		0		0		0		0		0		0		0
49. IP1		0		0		0		0		0		0		0		0		0		0		0		0
51. PlayStore1		0		0		0		0		0		0		0		0		0		0		0		0
52. PlayStore2		0		0		0		0		0		0		0		0		0		1	1xTO	0		1	1xTO
53. Sensors1		0		0		0		0		0		0		0		0		0		0		0		0
54. SubscriberId1		0		0		0		0		0		0		0		0		0		0		0		0
55. VoiceMail1		0		0		0		0		0		0		0		0		0		0		0		0
45. ContentProvider1	DB5.2	0		0		0		0		0		0		0		0		0		0		0		0
48. IMEI1	DB5.3	0		0		0		0		0		0		0		0		0		0		0		0
50. PI1	DB5.4	0		0		0		0		0		0		0		0		0		0		0		0
56. FieldSensitivity1	DB6.1	0		0		0		0		0		0		0		0		0		0		0		0
57. FieldSensitivity2	DB6.2	0		0		0		0		0		0		0		0		0		0		0		0
58. FieldSensitivity3	DB6.3	0		0		0		0		0		0		0		0		0		0		0		0
59. FieldSensitivity4	DB6.4	0		0		0		0		0		0		0		0		1	1xDB1.1	0		1	1xDB1.1	0
60. InheritedObjects1	DB6.5	0		0		0		0		0		0		0		0		0		0		0		0
61. ObjectSensitivity1	DB6.6	0		0		0		0		0		0		0		0		0		0		0		0
62. ObjectSensitivity2	DB6.7	0		0		0		0		0		0		0		0		1	1xDB1.1	0		1	1xDB1.1	0
63. Clone1	DB7.1	0		0		0		0		0		0		0		0		0		0		0		0
64. Exceptions1	DB7.2	0		0		0		0		0		0		0		0		0		0		0		0
65. Exceptions2	DB7.3	0		0		0		0		3	3xU-DB2.12	0		3	3xU-DB2.12	0		0		0		0		0
66. Exceptions3	DB7.4	1	1xDB7.4	0		1	1xDB7.4	0		4	1xDB7.4 3xU-DB2.12	0		4	1xDB7.4 3xU-DB2.12	0		1	1xDB7.4	0		1	1xDB7.4	0
67. Exceptions4	DB7.5	0		0		0		0		0		1	1xDB7.5	0		1	1xDB7.5	0		0		0		0
68. Exceptions5	DB7.6	0		0		0		0		0		1	1xDB7.6	0		1	1xDB7.6	0		0		0		0
69. Exceptions6		0		0		0		0		3	3xU-DB2.12	0		3	3xU-DB2.12	0		0		0		0		0
70. Exceptions7	DB7.7	0		0		0		0		4	1xDB7.4 3xU-DB2.12	0		4	1xDB7.4 3xU-DB2.12	0		1	1xDB7.4	0		1	1xDB7.4	0
71. FactoryMethods1	DB7.8	0		0		0		0		1	1xU-DB2.12	2	2xU-DB4.15	1	1xU-DB2.12	2	2xU-DB4.15	0		0		0		0
72. Loop1	DB7.9	0		0		0		0		0		1	1xUN	0		1	1xUN	0		0		0		0
73. Loop2		0		0		0		0		0		1	1xUN	0		1	1xUN	0		0		0		0
74. Serialization1	DB7.10	0		1	1xDB7.10	0		1	1xDB7.10	0		1	1xDB7.10	0		1	1xDB7.10	0		0		0		0
75. SourceCodeSpecific1	DB7.11	0		0		0		0		4	4xU-DB2.12	0		4	4xU-DB2.12	0		0		0		0		0
76. StartProcessWithSecret1	DB7.12	0		0		0		0		0		1	1xDB7.12	0		1	1xDB7.12	0		0		0		0
77. StaticInitialization1	DB7.13	0		1	1xDB7.13	0		1	1xDB7.13	0		0		0		0		0		0		0		0
78. StaticInitialization2	DB7.14	0		0		0		0		0		0		0		0		0		0		0		0
79. StaticInitialization3	DB7.15	0		1	1xDB7.15	0		1	1xDB7.15	0		0		0		0		0		0		0		0
80. StringFormatter1	DB7.16	0		1	1xDB7.16	0		1	1xDB7.16	0		1	1xDB7.16	0		1	1xDB7.16	0		0		0		0
81. StringPatternMatching1	DB7.17	0		0		0		0		0		0		0		0		0		0		0		0
82. StringToCharArray1	DB7.18	0		0		0		0		0		0		0		0		0		0		0		0
83. StringToOutputStream1	DB7.19	0		0		0		0		0		0		0		0		0		0		0		0
84. UnreachableCode	DB7.20	0		0		0		0		0		0		0		0		0		0		0		0
85. VirtualDispatch1	DB7.21	1	1xDB7.21	0		1	1xDB7.21	0		1	1xDB7.21	0		1	1xDB7.21	0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
86. VirtualDispatch2	DB7.22	1	1xDB7.22	0		1	1xDB7.22	0		0		0		0		0		0		0		0		0
87. VirtualDispatch3	DB7.23	1	1xDB7.23	0		1	1xDB7.23	0		0		0		0		0		0		0		0		0
88. ActivityCommunication1	DB8.1	0		0		0		0		0		0		0		0		0		0		0		0
89. ActivityCommunication2	DB8.2	0		1	1xDB8.2	0		1	1xDB8.2	1	1xDB8.2	1	1xSEM1	1	1xDB8.2	1	1xSEM1	0		0		0		0
90. ActivityCommunication3	DB8.3	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1
91. ActivityCommunication4	DB8.4	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	0		0		0		0
92. ActivityCommunication5	DB8.5	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1
93. ActivityCommunication6	DB8.6	0		1	1xDB8.6	0		1	1xDB8.6	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1
94. ActivityCommunication7	DB8.7	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1
95. ActivityCommunication8	DB8.8	0		1	1xDB8.8	0		1	1xDB8.8	0		1	1xSEM1	0		1	1xSEM1	0		0		0		0
96. BroadcastTaintAndLeak1	DB8.9	0		1	1xBUG2	0		1	1xBUG2	0		1	1xSEM1	0		1	1xSEM1	0		1	1xUN	0		1	1xUN
97. ComponentNotInManifest1	DB8.10	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1	0		1	1xSEM1
98. EventOrdering1	DB8.11	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		0		0		0
99. IntentSink1	DB8.12	0		0		0		0		1	1xU-DB2.12	0		1	1xU-DB2.12	0		0		0		0		0
100. IntentSink2	DB8.13	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1	1	1xDB8.18	0		1	1xDB8.18	0
101. IntentSource1	DB8.14	0		0		0		0		1	1xU-DB4.16	0		1	1xU-DB4.16	0		0		0		0		0
102. ServiceCommunication1	DB8.15	0		1	1xEX	0		1	1xEX	0		0		0		0		0		0		0		0
103. SharedPreferences1	DB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		0		0		0
104. Singletons1	DB8.17	0		0		0		0		0		1	1xDB8.17	0		1	1xDB8.17	0		0		0		0
105. UnresolvableIntent1	DB8.18	0		2	2xDB8.18	0		2	2xDB8.18	0		1	1xSEM1	0		1	1xSEM1	0		0		0		0
106. ActivityEventSequence1	DB9.1	0		0		0		0		0		1	1xDB9.1	0		1	1xDB9.1	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
107. ActivityEventSequence2	DB9.2	0		0		0		0		0		0		0		0		1	1xDB9.2	0		1	1xDB9.2	0
108. ActivityEventSequence3	DB9.3	0		1	1xDB9.3	0		1	1xDB9.3	1	1xU-DB2.12	0		1	1xU-DB2.12	0		1	1xSEM2	0		1	1xSEM2	0
109. ActivityLifecycle1	DB9.4	0		0		0		0		0		0		0		0		0		0		0		0
110. ActivityLifecycle2	DB9.5	0		0		0		0		0		1	1xDB9.5	0		1	1xDB9.5	0		0		0		0
111. ActivityLifecycle3	DB9.6	0		0		0		0		0		0		0		0		0		0		0		0
112. ActivityLifecycle4	DB9.7	0		0		0		0		0		0		0		0		0		0		0		0
113. ActivitySavedState1	DB9.8	0		1	1xDB9.8	0		1	1xDB9.8	0		1	1xDB9.8	0		1	1xDB9.8	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
114. ApplicationLifecycle1	DB9.9	0		0		0		0		0		1	1xDB9.9	0		1	1xDB9.9	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
115. ApplicationLifecycle2	DB9.10	0		0		0		0		0		1	1xDB9.9	0		1	1xDB9.9	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
116. ApplicationLifecycle3	DB9.11	0		0		0		0		0		1	1xDB9.9	0		1	1xDB9.9	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
117. AsynchronousEventOrdering1	DB9.12	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
118. BroadcastReceiverLifecycle1	DB9.13	0		0		0		0		0		0		0		0		0		0		0		0
119. BroadcastReceiverLifecycle2	DB9.14	0		1	1xDB9.14	0		1	1xDB9.14	0		1	1xDB9.14	0		1	1xDB9.14	0		0		0		0
120. BroadcastReceiverLifecycle3		0		0		0		0		0		0		0		0		1	1xSEM2	0		1	1xSEM2	0
121. EventOrdering1	DB9.15	0		0		0		0		0		0		0		0		0		0		0		0
122. FragmentLifecycle1	DB9.16	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
123. FragmentLifecycle2	DB9.17	0		1	1xDB9.17	0		1	1xDB9.17	0		1	1xDB9.17	0		1	1xDB9.17	1	1xSEM2	0		1	1xSEM2	0
124. ServiceEventSequence1	DB9.18	0		1	1xDB9.18	0		1	1xDB9.18	0		0		0		0		1	1xSEM2	0		1	1xSEM2	0
125. ServiceEventSequence2	DB9.19	0		1	1xDB9.19	0		1	1xDB9.19	0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
126. ServiceEventSequence3		0		1	1xDB9.19	0		1	1xDB9.19	0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
127. ServiceLifecycle1	DB9.20	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
128. ServiceLifecycle2	DB9.21	0		0		0		0		0		0		0		0		1	1xSEM2	0		1	1xSEM2	0
129. SharedPreferenceChanged1	DB9.22	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	0		1	1xDB8.16	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
130. Reflection1	DB10.1	0		0		0		0		0		0		0		0		0		0		0		0
131. Reflection2	DB10.2	0		0		0		0		0		1	1xDB10.2	0		0		0		0		0		0
132. Reflection3	DB10.3	0		0		0		0		0		1	1xDB10.3	0		1	1xDB10.3	0		0		0		0
133. Reflection4	DB10.4	0		0		0		0		0		1	1xDB10.4	0		0		0		0		0		0
134. Reflection5	DB10.5	0		0		0		0		0		1	1xDB10.5	0		0		0		1	1xU-DB10.7	0		0
135. Reflection6		0		0		0		0		0		1	1xDB10.5	0		0		0		1	1xU-DB10.7	0		0
136. Reflection7		0		0		0		1	1xU-DB10.9	0		1	1xDB10.5	0		1	1xU-DB10.9	0		1	1xU-DB10.9	0		1	1xU-DB10.9
137. Reflection8		0		0		0		0		0		1	1xDB10.5	0		1	1xU-DB10.8	0		1	1xU-DB10.8	0		1	1xU-DB10.8
138. Reflection9		0		0		0		0		0		1	1xDB10.5	0		1	1xU-DB10.8	0		1	1xU-DB10.8	0		1	1xU-DB10.8
139. ActivityCommunication2	DB11.1	0		2	2xU-DB10.10	0		2	2xU-DB10.10	0		2	2xDB11.1	0		2	2xU-DB10.10	0		2	2xU-DB10.7	0		2	2xU-DB10.10
140. AllReflection		0		2	2xU-DB10.6	0		2	2xU-DB10.11	0		2	2xDB11.1	0		2	2xU-DB10.11	0		2	2xU-DB10.7	0		2	2xU-DB10.11
141. OnlyIntent		0		1	1xU-DB10.10	0		1	1xU-DB10.10	0		2	2xDB11.1	0		2	2xU-DB10.10	0		2	2xU-DB10.7	0		2	2xU-DB10.10
142. OnlyIntentReceive		0		2	2xU-DB10.6	0		2	2xU-DB10.11	0		2	2xDB11.1	0		2	2xU-DB10.11	0		2	1xU-DB10.7 1xSEM1	0		2	1xU-DB10.11 1xSEM1
143. OnlySMS		0		1	1xU-DB10.10	0		1	1xU-DB10.10	0		2	1xDB11.1 1xSEM1	0		2	1xU-DB10.10 1xSEM1	0		2	1xU-DB10.7 1xSEM1	0		2	1xU-DB10.10 1xSEM1
144. OnlyTelephony		0		2	2xU-DB10.6	0		2	2xU-DB10.11	0		2	2xDB11.1	0		2	2xU-DB10.11	0		0		1	1xSEM1	1	1xSEM1
145. OnlyTelephony_Dynamic		0		3	3xU-DB10.6	0		3	3xU-DB10.11	0		3	3xDB11.1	0		3	3xU-DB10.11	0		0		1	1xSEM1	1	1xSEM1
146. OnlyTelephony_Reverse		0		2	2xU-DB10.6	0		2	2xU-DB10.11	0		2	2xDB11.1	0		2	2xU-DB10.11	0		0		1	1xSEM1	1	1xSEM1
147. OnlyTelephony_Substring		0		2	2xU-DB10.6	0		2	2xU-DB10.11	0		2	2xDB11.1	0		2	2xU-DB10.11	0		0		1	1xSEM1	1	1xSEM1
148. SharedPreferences1		0		2	2xUN	0		2	2xU-DB10.10	0		2	2xDB11.1	0		2	2xU-DB10.10	0		2	2xU-DB10.7	0		2	2xU-DB10.10
149. AsyncTask1	DB12.1	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
150. Executor1	DB12.2	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
151. JavaThread1	DB12.3	0		0		0		0		0		0		0		0		1	1xSEM2	0		1	1xSEM2	0
152. JavaThread2	DB12.4	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
153. Looper1	DB12.5	0		0		0		0		0		0		0		0		1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
154. TimeTask1	DB12.6	0		1	1xDB12.6	0		1	1xDB12.6	0		1	1xDB12.6	0		1	1xDB12.6	1	1xSEM2	1	1xSEM2	1	1xSEM2	1	1xSEM2
155. SimpleUnreachable1	DB13.1	0		0		0		0		1	1xDB13.1	0		1	1xDB13.1	0		0		0		0		0
156. UnreachableBoth		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0
157. UnreachableSink1		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0
158. UnreachableSource1		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0		1	1xDB13.1	0
ICCBench
1. icc_explicit_nosrc_nosink	ICC1.1	0		0		0		0		0		0		0		0		NA
2. icc_explicit_nosrc_sink		0		0		0		0		0		0		0		0
3. icc_explicit_src_nosink	ICC1.2	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
4. icc_explicit_src_sink		0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
13. icc_explicit1		0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
5. icc_implicit_nosrc_nosink	ICC1.3	0		0		0		0		0		0		0		0
6. icc_implicit_nosrc_sink		0		0		0		0		0		0		0		0
7. icc_implicit_src_nosink	ICC1.4	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
8. icc_implicit_src_sink		0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
14. icc_implicit_action		0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
9. icc_intentservice	ICC1.5	0		1	1xBUG2	0		1	1xBUG2	0		0		0		0
10. icc_stateful	ICC1.6	0		1	1xICC1.6	0		1	1xICC1.6	2	1xU-DB2.12 1xU-DB4.16	0		2	1xU-DB2.12 1xU-DB4.16	0
11. icc_dynregister1	ICC2.1	0		1	1xBUG2	0		1	1xBUG2	0		1	1xSEM1	0		1	1xSEM1
12. icc_dynregister2	ICC2.2	0		1	1xBUG2	0		1	1xBUG2	1	1xICC2.2	1	1xSEM1	1	1xICC2.2	1	1xSEM1
15. icc_implicit_category	ICC2.3	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
16. icc_implicit_data1	ICC2.4	0		1	1xICC2.4	0		1	1xICC2.4	0		1	1xSEM1	0		1	1xSEM1
17. icc_implicit_data2	ICC2.5	0		1	1xICC2.5	0		1	1xICC2.5	0		1	1xSEM1	0		1	1xSEM1
18. icc_implicit_mix1	ICC2.6	0		0		0		0		0		2	2xSEM1	0		2	2xSEM1
19. icc_implicit_mix2	ICC2.7	0		0		0		0		0		1	1xSEM1	0		1	1xSEM1
20. icc_rpc_comprehensive	ICC3.1	0		2	2xEX	0		2	2xEX	2	1xU-DB2.12 1xU-DB4.16	0		2	1xU-DB2.12 1xU-DB4.16	0
21. rpc_localservice	ICC4.1	0		1	1xEX	0		1	1xEX	1	1xICC4.1	0		1	1xICC4.1	0
22. rpc_messengerservice	ICC4.2	0		1	1xEX	0		1	1xEX	0		0		0		0
23. rpc_remoteservice	ICC4.3	0		1	1xEX	0		1	1xEX	0		0		0		0
24. rpc_returnsensitive	ICC4.4	0		1	1xEX	0		1	1xEX	0		0		0		0

Execution time and memory consumption

In the following table, we detailedly list the execution time (in seconds) of each tool on DroidBench.

The table consists of two main parts, which are:

• AppName and TargetCriteriaID: These two columns are information of benchmarks, matching AppID+AppName, and TargetCriteriaID in Selected benchmarks and expected results table.
• The left columns are all about results of studied tools. We list details about the execution time of each tool.

AppName	TargetCriteriaID	FlowDroid v2.7.1	DroidRA + FlowDroid v2.7.1	Amandroid v3.2.0	DroidRA + Amandroid v3.2.0	DroidSafe	DroidRA + DroidSafe	DroidRA Instrumentation
1. FlowSensitivity1	DB1.1	15	15	21	18	197	290	72
2. Merge1	DB1.2	16	16	20	22	193	194	71
3. SimpleAliasing1	DB1.3	15	16	21	22	194	196	71
4. StrongUpdate1	DB1.4	16	16	20	17	195	194	69
5. 5. ApplicationModeling1	DB2.1	8	8	14	17	127	126	61
6. DirectLeak1	DB2.2	11	11	18	14	115	115	64
7. InactiveActivity	DB2.3	10	10	15	13	110	111	69
8. Library2	DB2.4	11	11	14	16	127	127	66
9. LogNoLeak	DB2.5	10	10	16	14	115	115	64
10. Obfuscation1	DB2.6	11	11	14	14	124	124	65
11. Parcel1	DB2.7	8	8	19	15	123	123	61
12. PrivateDataLeak3	DB2.8	11	11	26	16	129	130	65
13. PublicAPIField1	DB2.9	8	8	24	17	127	127	62
14. PublicAPIField2	DB2.10	8	8	15	14	123	122	61
15. View1	DB2.11	11	11	18	16	114	115	65
16. ArrayAccess1	DB3.1	11	11	18	14	115	115	64
17. ArrayAccess2	DB3.2	11	11	14	20	115	116	64
18. ArrayAccess3	DB3.3	11	11	15	15	114	116	66
19. ArrayAccess4	DB3.4	11	11	17	14	114	117	65
20. ArrayAccess5	DB3.5	11	11	14	15	115	115	64
21. ArrayCopy1	DB3.6	8	8	17	14	122	121	62
22. ArrayToString1	DB3.7	9	8	19	17	123	124	62
23. HashMapAccess1	DB3.8	12	11	14	14	125	124	65
24. ListAccess1	DB3.9	11	11	16	15	115	116	64
25. MultidimensionalArray1	DB3.10	8	8	14	18	122	122	62
26. AnonymousClass1	DB4.1	11	11	14	14	115	115	64
27. Button1	DB4.2	11	11	18	15	126	124	65
28. Button2	DB4.3	11	11	16	15	130	130	64
29. Button3	DB4.4	11	11	17	16	131	131	66
30. Button4	DB4.5	11	11	14	16	127	128	66
31. Button5	DB4.6	11	11	15	14	124	125	63
32. LocationLeak1	DB4.7	11	11	15	14	115	115	63
33. LocationLeak2	DB4.8	11	11	14	18	115	116	64
34. LocationLeak3	DB4.9	11	11	15	17	127	128	63
35. MethodOverride1	DB4.10	11	11	14	16	114	115	64
36. MultiHandlers1	DB4.11	11	11	15	16	130	131	64
37. Ordering1	DB4.12	11	11	16	15	129	128	64
38. RegisterGlobal1	DB4.13	11	11	17	15	129	131	64
39. RegisterGlobal2		11	11	15	18	126	129	65
40. Unregister1	DB4.14	11	11	14	16	129	130	65
41. Battery1	DB5.1	14	14	21	18	133	132	69
42. Bluetooth1		14	14	19	18	133	134	69
43. Build1		14	14	18	18	134	133	69
44. Contacts1		14	14	17	18	134	132	71
46. DeviceId1		14	14	20	17	134	132	69
47. File1		14	15	19	19	133	133	69
49. IP1		14	15	21	19	134	134	69
51. PlayStore1		16	16	19	18	195	194	67
52. PlayStore2		53	54	28	25	(TIMEOUT)	(TIMEOUT)	69
53. Sensors1		15	15	18	17	134	132	69
54. SubscriberId1		15	15	18	19	133	133	69
55. VoiceMail1		14	15	20	19	133	132	108
45. ContentProvider1	DB5.2	15	16	20	20	193	195	70
48. IMEI1	DB5.3	13	14	19	17	120	121	67
50. PI1	DB5.4	14	14	19	18	146	145	69
56. FieldSensitivity1	DB6.1	11	11	15	15	114	116	65
57. FieldSensitivity2	DB6.2	11	11	15	14	114	114	65
58. FieldSensitivity3	DB6.3	12	11	14	14	115	115	64
59. FieldSensitivity4	DB6.4	11	11	14	17	115	115	64
60. InheritedObjects1	DB6.5	12	11	16	15	115	116	65
61. ObjectSensitivity1	DB6.6	11	11	14	17	114	115	64
62. ObjectSensitivity2	DB6.7	11	11	17	14	115	116	65
63. Clone1	DB7.1	8	8	18	17	123	123	60
64. Exceptions1	DB7.2	11	11	15	15	126	128	64
65. Exceptions2	DB7.3	11	11	14	14	126	128	65
66. Exceptions3	DB7.4	11	11	16	16	125	126	65
67. Exceptions4	DB7.5	11	11	17	14	126	129	64
68. Exceptions5	DB7.6	11	11	17	15	125	128	64
69. Exceptions6		11	11	14	14	126	126	65
70. Exceptions7	DB7.7	11	11	15	16	126	126	65
71. FactoryMethods1	DB7.8	11	11	16	14	126	130	64
72. Loop1	DB7.9	11	11	14	14	115	116	64
73. Loop2		11	11	14	17	115	115	65
74. Serialization1	DB7.10	9	8	17	17	165	165	60
75. SourceCodeSpecific1	DB7.11	12	11	16	14	115	115	65
76. StartProcessWithSecret1	DB7.12	9	8	14	17	124	122	61
77. StaticInitialization1	DB7.13	11	11	16	14	115	115	64
78. StaticInitialization2	DB7.14	11	11	16	15	114	115	63
79. StaticInitialization3	DB7.15	9	8	21	14	122	123	62
80. StringFormatter1	DB7.16	9	8	14	18	129	129	61
81. StringPatternMatching1	DB7.17	8	8	14	15	121	122	62
82. StringToCharArray1	DB7.18	8	8	17	18	126	127	61
83. StringToOutputStream1	DB7.19	8	8	16	15	123	124	61
84. UnreachableCode	DB7.20	11	11	14	14	114	115	65
85. VirtualDispatch1	DB7.21	11	11	16	17	123	125	64
86. VirtualDispatch2	DB7.22	9	8	14	16	124	122	62
87. VirtualDispatch3	DB7.23	16	17	18	20	194	195	70
88. ActivityCommunication1	DB8.1	11	11	16	16	129	129	65
89. ActivityCommunication2	DB8.2	12	13	17	14	130	129	66
90. ActivityCommunication3	DB8.3	12	13	16	15	129	129	66
91. ActivityCommunication4	DB8.4	13	13	15	15	129	130	65
92. ActivityCommunication5	DB8.5	13	12	15	15	131	130	67
93. ActivityCommunication6	DB8.6	13	12	19	17	130	130	66
94. ActivityCommunication7	DB8.7	13	13	23	21	130	131	64
95. ActivityCommunication8	DB8.8	12	12	18	17	130	130	65
96. BroadcastTaintAndLeak1	DB8.9	9	8	17	24	111	110	61
97. ComponentNotInManifest1	DB8.10	12	12	15	20	127	128	64
98. EventOrdering1	DB8.11	12	12	17	22	129	130	65
99. IntentSink1	DB8.12	11	11	14	15	115	116	64
100. IntentSink2	DB8.13	11	11	19	14	125	128	62
101. IntentSource1	DB8.14	12	12	14	15	127	126	66
102. ServiceCommunication1	DB8.15	(EX)	(EX)	17	20	131	130	62
103. SharedPreferences1	DB8.16	8	8	14	18	124	125	62
104. Singletons1	DB8.17	8	8	15	15	125	122	62
105. UnresolvableIntent1	DB8.18	12	12	16	15	130	130	66
106. ActivityEventSequence1	DB9.1	11	11	15	14	129	131	63
107. ActivityEventSequence2	DB9.2	11	11	16	18	130	131	66
108. ActivityEventSequence3	DB9.3	11	11	15	14	129	130	66
109. ActivityLifecycle1	DB9.4	11	11	18	15	148	149	62
110. ActivityLifecycle2	DB9.5	11	11	17	14	114	115	64
111. ActivityLifecycle3	DB9.6	11	11	14	15	115	115	64
112. ActivityLifecycle4	DB9.7	11	11	16	17	114	115	64
113. ActivitySavedState1	DB9.8	9	8	14	16	122	122	61
114. ApplicationLifecycle1	DB9.9	11	11	14	16	114	115	65
115. ApplicationLifecycle2	DB9.10	11	11	14	15	115	115	64
116. ApplicationLifecycle3	DB9.11	12	11	17	15	124	125	68
117. AsynchronousEventOrdering1	DB9.12	9	8	17	16	121	121	86
118. BroadcastReceiverLifecycle1	DB9.13	11	11	14	14	107	108	102
119. BroadcastReceiverLifecycle2	DB9.14	13	13	15	15	128	128	67
120. BroadcastReceiverLifecycle3		13	12	17	16	128	129	85
121. EventOrdering1	DB9.15	8	8	14	16	123	122	105
122. FragmentLifecycle1	DB9.16	13	13	17	15	1443	1444	67
123. FragmentLifecycle2	DB9.17	10	10	17	16	131	132	64
124. ServiceEventSequence1	DB9.18	12	13	21	18	129	130	116
125. ServiceEventSequence2	DB9.19	12	12	17	15	131	132	71
126. ServiceEventSequence3		12	13	18	17	130	131	67
127. ServiceLifecycle1	DB9.20	11	11	15	16	108	108	72
128. ServiceLifecycle2	DB9.21	8	8	16	17	122	123	63
129. SharedPreferenceChanged1	DB9.22	8	8	16	15	122	122	154
130. Reflection1	DB10.1	12	13	16	15	129	130	106
131. Reflection2	DB10.2	11	12	22	17	135	136	127
132. Reflection3	DB10.3	11	12	19	19	131	131	72
133. Reflection4	DB10.4	11	11	17	18	136	137	76
134. Reflection5	DB10.5	11	12	17	16	135	134	116
135. Reflection6		11	11	16	16	135	136	70
136. Reflection7		11	11	18	15	136	135	71
137. Reflection8		11	11	15	18	135	136	128
138. Reflection9		11	11	16	17	136	135	73
139. ActivityCommunication2	DB11.1	9	9	17	16	127	129	111
140. AllReflection		11	11	17	17	128	127	79
141. OnlyIntent		11	12	14	16	128	128	71
142. OnlyIntentReceive		11	12	17	15	126	127	78
143. OnlySMS		11	12	15	16	127	129	112
144. OnlyTelephony		11	11	17	17	124	125	92
145. OnlyTelephony_Dynamic		8	8	17	14	123	123	89
146. OnlyTelephony_Reverse		11	12	15	18	124	133	186
147. OnlyTelephony_Substring		11	12	17	18	124	134	72
148. SharedPreferences1		11	12	15	15	126	127	66
149. AsyncTask1	DB12.1	12	12	14	17	128	127	65
150. Executor1	DB12.2	12	13	16	16	130	130	66
151. JavaThread1	DB12.3	12	12	14	17	128	129	66
152. JavaThread2	DB12.4	12	13	15	16	129	128	66
153. Looper1	DB12.5	16	16	19	21	194	195	70
154. TimeTask1	DB12.6	16	16	18	19	196	195	70
155. SimpleUnreachable1	DB13.1	21	22	29	19	113	113	79
156. UnreachableBoth		8	8	18	15	268	134	62
157. UnreachableSink1		8	8	15	14	200	154	62
158. UnreachableSource1		8	8	16	16	190	183	62
	Min	8	8	14	13	107	108	60
	Max	53	54	29	25	1443	1444	186
	Mean	12	12	17	16	139	139	71
	Median	11	11	16	16	126	128	65

In the following table, we detailedly list the memory consumption (in MBs) of each tool on DroidBench. The below table also consists of two main parts, similar to the above table for execution time, while the second part is with with memory consumption.

AppName	TargetCriteriaID	FlowDroid v2.7.1	DroidRA + FlowDroid v2.7.1	Amandroid v3.2.0	DroidRA + Amandroid v3.2.0	DroidSafe	DroidRA + DroidSafe	DroidRA Instrumentation
1. FlowSensitivity1	DB1.1	498	509	2443	2468	11161	10921	12714
2. Merge1	DB1.2	524	511	2463	2441	10548	10450	13089
3. SimpleAliasing1	DB1.3	507	515	2426	2435	10234	10447	13093
4. StrongUpdate1	DB1.4	518	519	2479	2459	10177	10410	12722
5. 5. ApplicationModeling1	DB2.1	348	355	2405	2412	6376	6722	12869
6. DirectLeak1	DB2.2	376	384	2400	2377	6684	6398	12705
7. InactiveActivity	DB2.3	246	251	2366	2361	6249	6150	12821
8. Library2	DB2.4	389	397	2401	2444	6382	6462	12702
9. LogNoLeak	DB2.5	272	295	2420	2387	6266	6249	12705
10. Obfuscation1	DB2.6	391	404	2411	2412	6539	6342	12700
11. Parcel1	DB2.7	359	363	2420	2440	6705	6811	12894
12. PrivateDataLeak3	DB2.8	400	409	2381	2425	6447	6321	12706
13. PublicAPIField1	DB2.9	344	349	2430	2399	6451	6473	12874
14. PublicAPIField2	DB2.10	349	342	2426	2433	6239	6537	12881
15. View1	DB2.11	383	390	2401	2423	6521	6554	12713
16. ArrayAccess1	DB3.1	393	394	2395	2384	6553	6408	12709
17. ArrayAccess2	DB3.2	395	382	2396	2390	6441	6231	12697
18. ArrayAccess3	DB3.3	388	380	2385	2399	6610	6406	12703
19. ArrayAccess4	DB3.4	393	389	2399	2406	6436	6482	12704
20. ArrayAccess5	DB3.5	393	389	2392	2388	6625	6513	12700
21. ArrayCopy1	DB3.6	337	341	2413	2409	6605	6608	12885
22. ArrayToString1	DB3.7	339	343	2409	2411	6560	6474	12889
23. HashMapAccess1	DB3.8	394	400	2434	2452	6414	6641	12706
24. ListAccess1	DB3.9	371	383	2420	2395	6445	6342	12704
25. MultidimensionalArray1	DB3.10	346	340	2412	2407	6585	6725	12887
26. AnonymousClass1	DB4.1	375	383	2383	2394	6505	6274	12702
27. Button1	DB4.2	378	385	2411	2416	6608	6444	12702
28. Button2	DB4.3	392	398	2399	2436	6579	6344	12702
29. Button3	DB4.4	396	399	2435	2431	6487	6468	12697
30. Button4	DB4.5	390	388	2392	2385	6343	6391	12707
31. Button5	DB4.6	397	401	2437	2422	6430	6611	12710
32. LocationLeak1	DB4.7	381	392	2416	2383	6612	6487	12706
33. LocationLeak2	DB4.8	398	399	2395	2388	6590	6539	12705
34. LocationLeak3	DB4.9	407	409	2423	2426	6477	6246	12700
35. MethodOverride1	DB4.10	377	376	2390	2377	6515	6378	12697
36. MultiHandlers1	DB4.11	408	399	2413	2420	6538	6436	12702
37. Ordering1	DB4.12	409	414	2447	2435	6603	6571	12701
38. RegisterGlobal1	DB4.13	400	389	2410	2438	6443	6462	12702
39. RegisterGlobal2		405	388	2413	2432	6434	6487	12698
40. Unregister1	DB4.14	404	387	2427	2408	6149	6497	12699
41. Battery1	DB5.1	532	534	2437	2450	6467	6295	13070
42. Bluetooth1		535	539	2434	2435	6324	6331	13077
43. Build1		546	548	2437	2440	6653	6284	12703
44. Contacts1		544	545	2462	2431	6378	6541	13625
46. DeviceId1		491	522	2442	2451	10407	10372	13117
47. File1		537	536	2452	2445	6590	6621	12819
49. IP1		535	541	2447	2439	6513	6470	12831
51. PlayStore1		514	515	2430	2429	6352	6160	13083
52. PlayStore2		544	550	2448	2449	6641	6341	13190
53. Sensors1		546	561	2448	2427	7809	7775	13194
54. SubscriberId1		511	519	2460	2469	10621	10391	12837
55. VoiceMail1		5228	5226	3677	3105	(TIMEOUT)	(TIMEOUT)	14851
45. ContentProvider1	DB5.2	535	537	2456	2440	6551	6335	13198
48. IMEI1	DB5.3	539	551	2450	2433	6480	6500	13367
50. PI1	DB5.4	538	536	2459	2443	6363	6500	12706
56. FieldSensitivity1	DB6.1	392	376	2375	2401	6657	6407	12710
57. FieldSensitivity2	DB6.2	384	388	2369	2399	6462	6456	12706
58. FieldSensitivity3	DB6.3	371	387	2394	2401	6415	6292	12703
59. FieldSensitivity4	DB6.4	385	385	2432	2384	6502	6476	12707
60. InheritedObjects1	DB6.5	382	385	2373	2391	6374	6504	12703
61. ObjectSensitivity1	DB6.6	388	381	2390	2409	6477	6316	12694
62. ObjectSensitivity2	DB6.7	374	387	2376	2390	6536	6550	12704
63. Clone1	DB7.1	345	346	2416	2408	6573	6421	12881
64. Exceptions1	DB7.2	401	394	2433	2432	6555	6257	12696
65. Exceptions2	DB7.3	398	404	2444	2431	6379	6274	12713
66. Exceptions3	DB7.4	389	399	2418	2435	6602	6420	12703
67. Exceptions4	DB7.5	397	402	2395	2413	6432	6076	12706
68. Exceptions5	DB7.6	397	411	2416	2413	6556	6392	12705
69. Exceptions6		407	413	2409	2421	6334	6470	12702
70. Exceptions7	DB7.7	399	406	2421	2416	6518	6314	12705
71. FactoryMethods1	DB7.8	404	407	2401	2415	6613	6294	12704
72. Loop1	DB7.9	396	399	2383	2376	6392	6573	12708
73. Loop2		382	386	2379	2410	6533	6308	12708
74. Serialization1	DB7.10	344	358	2428	2434	9290	9407	12873
75. SourceCodeSpecific1	DB7.11	383	383	2388	2392	6643	6563	12709
76. StartProcessWithSecret1	DB7.12	340	354	2406	2407	6338	6591	12880
77. StaticInitialization1	DB7.13	374	385	2415	2392	6229	6594	12701
78. StaticInitialization2	DB7.14	384	395	2395	2389	6367	6417	12703
79. StaticInitialization3	DB7.15	337	328	2412	2431	6380	6528	12872
80. StringFormatter1	DB7.16	346	345	2411	2402	6389	6535	12877
81. StringPatternMatching1	DB7.17	352	344	2421	2421	6581	6722	12879
82. StringToCharArray1	DB7.18	343	344	2406	2385	6487	6325	12880
83. StringToOutputStream1	DB7.19	347	333	2453	2413	6625	6527	12882
84. UnreachableCode	DB7.20	290	290	2373	2380	6559	6434	12699
85. VirtualDispatch1	DB7.21	382	403	2414	2418	6638	6642	12696
86. VirtualDispatch2	DB7.22	331	341	2408	2391	6374	6349	12866
87. VirtualDispatch3	DB7.23	529	526	2452	2462	10521	10423	12715
88. ActivityCommunication1	DB8.1	389	391	2425	2440	6532	6726	12693
89. ActivityCommunication2	DB8.2	457	446	2438	2427	6455	6395	12711
90. ActivityCommunication3	DB8.3	457	455	2417	2438	6494	6457	12708
91. ActivityCommunication4	DB8.4	433	434	2423	2439	6312	6610	12710
92. ActivityCommunication5	DB8.5	423	442	2418	2433	6424	6352	12712
93. ActivityCommunication6	DB8.6	436	444	2444	2415	6589	6470	12703
94. ActivityCommunication7	DB8.7	436	439	2425	2434	6489	6347	12711
95. ActivityCommunication8	DB8.8	448	446	2412	2403	6334	6443	12712
96. BroadcastTaintAndLeak1	DB8.9	321	328	2417	2388	6223	6227	12877
97. ComponentNotInManifest1	DB8.10	451	443	2410	2419	6446	6351	12701
98. EventOrdering1	DB8.11	438	426	2438	2437	6400	6358	12707
99. IntentSink1	DB8.12	391	386	2388	2379	6182	6541	12702
100. IntentSink2	DB8.13	387	383	2413	2408	6604	6323	12699
101. IntentSource1	DB8.14	324	317	2411	2421	6204	6208	12702
102. ServiceCommunication1	DB8.15	(EX)	(EX)	2434	2419	6445	6756	12880
103. SharedPreferences1	DB8.16	336	338	2421	2439	6637	6541	12875
104. Singletons1	DB8.17	354	346	2418	2442	6592	6684	12869
105. UnresolvableIntent1	DB8.18	455	432	2442	2427	6519	6518	13071
106. ActivityEventSequence1	DB9.1	397	396	2411	2389	6932	6501	12698
107. ActivityEventSequence2	DB9.2	393	401	2412	2436	6647	6376	12704
108. ActivityEventSequence3	DB9.3	413	425	2415	2435	6668	6544	12700
109. ActivityLifecycle1	DB9.4	395	385	2404	2391	8397	8307	12701
110. ActivityLifecycle2	DB9.5	377	379	2389	2379	6793	6389	12707
111. ActivityLifecycle3	DB9.6	377	386	2379	2392	6546	6381	12699
112. ActivityLifecycle4	DB9.7	389	383	2398	2381	6295	6506	12707
113. ActivitySavedState1	DB9.8	335	353	2405	2414	6673	6354	12871
114. ApplicationLifecycle1	DB9.9	380	388	2366	2383	6520	6296	12695
115. ApplicationLifecycle2	DB9.10	373	399	2391	2395	6383	6601	12706
116. ApplicationLifecycle3	DB9.11	390	390	2399	2441	6774	6491	12693
117. AsynchronousEventOrdering1	DB9.12	341	346	2423	2419	6623	6501	12879
118. BroadcastReceiverLifecycle1	DB9.13	386	375	2403	2376	6137	6332	12738
119. BroadcastReceiverLifecycle2	DB9.14	426	438	2417	2428	6585	6491	12709
120. BroadcastReceiverLifecycle3		412	418	2414	2432	6407	6593	12711
121. EventOrdering1	DB9.15	345	325	2410	2430	6314	6656	12879
122. FragmentLifecycle1	DB9.16	424	417	2437	2457	27488	27604	12719
123. FragmentLifecycle2	DB9.17	279	286	2452	2426	6593	6421	12710
124. ServiceEventSequence1	DB9.18	462	442	2427	2413	6491	6396	12713
125. ServiceEventSequence2	DB9.19	435	446	2431	2413	6658	6594	12706
126. ServiceEventSequence3		443	442	2395	2419	6523	6468	12713
127. ServiceLifecycle1	DB9.20	388	381	2362	2396	6216	6206	12685
128. ServiceLifecycle2	DB9.21	342	338	2420	2407	6717	6505	12866
129. SharedPreferenceChanged1	DB9.22	330	340	2429	2430	6623	6557	12879
130. Reflection1	DB10.1	448	461	2457	2487	6537	6313	12708
131. Reflection2	DB10.2	334	343	2433	2454	7829	7785	12702
132. Reflection3	DB10.3	421	422	2432	2417	7534	7602	12705
133. Reflection4	DB10.4	307	321	2430	2442	7585	7507	12704
134. Reflection5	DB10.5	412	412	2431	2447	7234	7554	12698
135. Reflection6		298	303	2445	2414	7799	7618	12705
136. Reflection7		301	295	2439	2430	7577	7488	12709
137. Reflection8		299	301	2418	2445	7649	7622	12698
138. Reflection9		292	315	2427	2416	7564	7501	12700
139. ActivityCommunication2	DB11.1	359	366	2431	2447	6693	6550	12862
140. AllReflection		396	405	2412	2425	6392	6574	12708
141. OnlyIntent		396	394	2465	2430	6483	6309	12708
142. OnlyIntentReceive		409	420	2415	2407	6547	6610	12702
143. OnlySMS		391	397	2437	2439	6424	6533	12706
144. OnlyTelephony		405	399	2415	2417	6562	6442	12697
145. OnlyTelephony_Dynamic		348	355	2417	2407	6532	6428	12882
146. OnlyTelephony_Reverse		406	410	2420	2416	6593	6520	12692
147. OnlyTelephony_Substring		396	410	2413	2443	6578	6277	12702
148. SharedPreferences1		417	406	2412	2427	6234	6491	12704
149. AsyncTask1	DB12.1	431	429	2432	2423	6627	6396	12709
150. Executor1	DB12.2	422	433	2416	2415	6336	6498	12710
151. JavaThread1	DB12.3	430	436	2403	2416	6425	6412	12708
152. JavaThread2	DB12.4	437	425	2414	2420	6271	6413	12714
153. Looper1	DB12.5	510	520	2446	2459	10377	10361	12723
154. TimeTask1	DB12.6	520	511	2449	2470	10436	10536	13088
155. SimpleUnreachable1	DB13.1	512	516	2479	2467	6399	6402	13950
156. UnreachableBoth		317	327	2409	2424	7322	6608	12876
157. UnreachableSink1		339	336	2419	2405	7493	6834	12883
158. UnreachableSource1		329	346	2423	2403	7138	6874	12893
	Min	432	434	2426	2423	6962	6917	12801
	Max	5228	5226	3677	3105	27488	27604	14851
	Mean	246	251	2362	2361	6137	6076	12685
	Median	393	394	2417	2419.5	6536	6487	12708

Comparison to our ISSTA’18 paper

We compare our results to our ISSTA’18 paper. There are two differences in our experiments compared to the ISSTA’18 paper:

• For selected Android static taint analyzers, we used FlowDroid v2.7.1 in this work vs. v2.0 in the earlier work, and Amandroid v3.2.0 vs. v3.1.1 in the earlier work. DroidSafe did not have any updates from the time of ISSTA’18.
• For selected benchmark suites, we used DroidBench v3.0 instead of v2.0 as in the earlier work. We included 49 new benchmark apps from DroidBench v3.0 in out test suite. ICCBench did not have any updates from the time of ISSTA’18.

In the following two tables, we list benchmark apps on which different versions of FlowDroid and Amandroid reported inconsistent results. As we only ran benchmark apps in DroidBench v2.0 in our ISSTA’18 paper, we only present results for benchmark apps in DroidBench v2.0.

Specifically, these tables consists of the following two parts:

• App and TargetCriteriaID: These two columns are information of benchmarks, matching AppID+AppName, and TargetCriteriaID in Selected benchmarks and expected results table.
• Results of FlowDroid or Amandroid under different version. We list details about FP (false positives) and FN (false negatives) flows reported by FlowDroid v2.7.1 and v2.0 (in the first table) as well as Amandroid v3.2.0 and v3.1.1 (in the second table). We highlighted differences between two versions of each tool.

Details of our newly-identified sources of failures (i.e., UBCx) can be found here.

AppName	TargetCriteriaID	FlowDroid v2.7.1				FlowDroid v2.0
		FP		FN		FP		FN
		#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria
DroidBench
11. Parcel1	DB2.7	0		1	1xBUG1	0		0
13. PublicAPIField1	DB2.9	0		1	1xBUG1	0		0
14. PublicAPIField2	DB2.10	0		0		0		1	1xDB2.10
96. BroadcastTaintAndLeak1	DB8.9	0		1	1xBUG2	0		0
132. Reflection3	DB10.3	0		1	1xBUG1	0		0
ICCBench
9. icc_intentservice	ICC1.5	0		1	1xBUG2	0		0
10. icc_stateful	ICC1.6	0		1	1xICC1.6	0		0
11. icc_dynregister1	ICC2.1	0		1	1xBUG2	0		0
12. icc_dynregister2	ICC2.2	0		1	1xBUG2	0		0

AppName	TargetCriteriaID	Amandroid v3.2.0				Amandroid v3.1.1
		FP		FN		FP		FN
		#FP	Criteria	#FN	Criteria	#FP	Criteria	#FN	Criteria
DroidBench
25. MultidimensionalArray1	DB3.10	0		1	1xDB3.10	0		0
34. LocationLeak3	DB4.9	0		2	2xUBC4	2	1xDB6.1 1xUBC4	2	2xUBC4
36. MultiHandlers1	DB4.11	0		0		4	4xDB4.11	0
65. Exceptions2	DB7.3	3	3xUBC3	0		0		0
66. Exceptions3	DB7.4	4	1xDB7.4 3xUBC3	0		1	1xDB7.4	0
72. Loop1	DB7.9	0		1	1xUN	0		0
73. Loop2		0		1	1xUN	0		0
89. ActivityCommunication2	DB8.2	1	1xDB8.2	1	1xSEM1	1	1xDB8.2	0
90. ActivityCommunication3	DB8.3	0		1	1xSEM1	0		0
91. ActivityCommunication4	DB8.4	0		1	1xSEM1	0		0
92. ActivityCommunication5	DB8.5	0		1	1xSEM1	0		0
93. ActivityCommunication6	DB8.6	0		1	1xSEM1	0		0
94. ActivityCommunication7	DB8.7	0		1	1xSEM1	0		0
95. ActivityCommunication8	DB8.8	0		1	1xSEM1	0		0
96. BroadcastTaintAndLeak1	DB8.9	0		1	1xSEM1	1	1xDB6.1	0
97. ComponentNotInManifest1	DB8.10	0		1	1xSEM1	0		0
100. IntentSink2	DB8.13	0		1	1xSEM1	1	1xDB6.1	0
101. IntentSource1	DB8.14	1	1xUBC5	0		3	1xDB6.1 2xUBC5	0
105. UnresolvableIntent1	DB8.18	0		1	1xSEM1	0		0
110. ActivityLifecycle2	DB9.5	0		1	1xDB9.5	0		0
131. Reflection2	DB10.2	0		1	1xDB10.2	0		0
133. Reflection4	DB10.4	0		1	1xDB10.4	0		0
ICCBench
3. icc_explicit_src_nosink	ICC1.2	0		1	1xSEM1	0		0
4. icc_explicit_src_sink		0		1	1xSEM1	0		0
13. icc_explicit1		0		1	1xSEM1	0		0
7. icc_implicit_src_nosink	ICC1.4	0		1	1xSEM1	0		0
8. icc_implicit_src_sink		0		1	1xSEM1	0		0
14. icc_implicit_action		0		1	1xSEM1	0		0
11. icc_dynregister1	ICC2.1	0		1	1xSEM1	0		0
12. icc_dynregister2	ICC2.2	1	1xICC2.2	1	1xSEM1	1	1xICC2.2	0
15. icc_implicit_category	ICC2.3	0		1	1xSEM1	0		0
16. icc_implicit_data1	ICC2.4	0		1	1xSEM1	0		0
17. icc_implicit_data2	ICC2.5	0		1	1xSEM1	1	1xDB6.1	0
18. icc_implicit_mix1	ICC2.6	0		2	2xSEM1	0		0
19. icc_implicit_mix2	ICC2.7	0		1	1xSEM1	0		0
21. rpc_localservice	ICC4.1	1	1xICC4.1	0		0		0

Other results

For comparative purposes, we provided results of the following versions of tools, running on selected 158 apps of DroidBench v3.0 and 24 apps of ICCBench v2.0.

• FlowDroid v2.7.1 without IccTA.
• FlowDroid v2.0.
• FlowDroid v1.5.
• Amandroid v3.1.1.

The following table consists of three parts:

• Tool: The name of the tool and its version.
• #FP and #FN: Number of FP (false positives) and FN (false negatives) flows reported by each version of tool.
• Precision, Recall, and F-Measure: The calculated accuracy of each version of tool.

Tool	#FP	#FN	Precision	Recall	F-Measure
DroidBench
FlowDroid v2.7.1 (without IccTA)	15	56	87	65	74
FlowDroid v2.0	15	44	88	72	79
FlowDroid v1.5	16	65	85	59	70
Amandroid v3.1.1	43	59	70	63	66
ICCBench
FlowDroid v2.7.1 (without IccTA)	0	20	100	41	58
FlowDroid v2.0	0	8	100	76	87
FlowDroid v1.5	0	17	100	50	67
Amandroid v3.1.1	6	0	85	100	92