Benchmarking
Benchmark runs across different scopes of Java bytecode:
- runtime — Java runtime with no additional dependencies
- runtime + guava — Java runtime with one JAR file for
guava - runtime + project classpath — Java runtime with all visible dependencies of the JacoDB project
- runtime + Idea community — Java runtime with all visible dependencies of an IntelliJ IDEA Community project
JacoDB benchmarks also include a scheme with the Usages feature installed.
Environment
| OS | Windows 10 Pro |
| Processor | 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80 GHz |
| RAM | 16 GB |
| Storage | SSD |
JacoDB benchmarking results
Run command:
./gradlew jcdbBenchmark
| Benchmark | Repeats | Avg time per operation |
|---|---|---|
| runtime | 5 | 1064 ms |
| runtime + guava | 5 | 1090 ms |
| runtime + project dependencies | 5 | 1551 ms |
| runtime + IDEA community dependecies | 5 | 9054 ms |
| runtime + Usages | 5 | 1790 ms |
| runtime + guava + Usages | 5 | 2590 ms |
| runtime + project dependencies + Usages | 5 | 2915 ms |
| runtime + IDEA community + Usages | 5 | 9798 ms |
Soot benchmarking results
Run command:
./gradlew sootBenchmark
| Benchmark | Repeats | Avg time per operation |
|---|---|---|
| runtime | 5 | 20985 ms |
| runtime + guava | 5 | 23323 ms |
| runtime + project dependencies | 5 | 24027 ms |
| runtime + IDEA community dependecies | 5 | 30286 ms |
SootUp benchmarking results
Runs with createInDemandView
Run command:
./gradlew sootupBenchmark
| Benchmark | Repeats | Avg time per operation |
|---|---|---|
| runtime | 5 | 0.23 ms |
| runtime + guava | 5 | 1 ms |
| runtime + project dependencies | 5 | 60 ms |
| runtime + IDEA community dependecies | 5 | 167 ms |
Runs with createFullView
Consumed almost all of 16 GB RAM memory (failed with the OOM error on 12 GB).
| Benchmark | Repeats | Avg time per operation |
|---|---|---|
| runtime | 5 | 0.23 ms |
| runtime + guava | 5 | 89 ms |
| runtime + project dependencies | 5 | 4247 ms |
| runtime + IDEA community dependecies | 5 | 114 723 ms (fails on 12Gb) |
Comparison
Soot
The Soot framework just reads all the JAR files and stores bytecode in memory.
Unlike Soot, JacoDB reads the available .class files from the JAR files and builds folders in parallel. As a result,
it is faster when
dealing with multiple JAR files (due to parallel execution). Note, that JacoDB performs a lot of tasks in
background while almost all the API is ready for use (Hierarchy and Usages require the background activity to be
finished).
SootUp
The SootUp framework ignores runtime libraries. That is why the startup time for the “runtime only” condition is almost equal to zero.
JacoDB
JacoDB performs tasks in background while almost all the API is ready for use. Here are the results for performing
the background activities without the Usages feature:
./gradlew awaitBackgroundBenchmark
| Benchmark | Repeats | Avg time per operation |
|---|---|---|
| runtime: wait for background jobs | 5 | 3182 ms |
| runtime + project dependencies: wait for background jobs | 5 | 14737 ms |
| runtime + IDEA community dependencies: wait for background jobs | 2 | 137528 ms |
For the IntelliJ IDEA Community code base, the size of the resulting SQLite database file is ~3.5 GB.