Make the benchmark build the app itself, and record what the runner does to it

A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back.

Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it.

coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates.

Gate green, map current.
This commit is contained in:
2026-10-17 10:31:56 +02:00
parent 94f85bd287
commit cb2db9ab63
6 changed files with 65 additions and 42 deletions
+26 -13
View File
@@ -96,22 +96,31 @@ android {
)
}
/*
* What macrobenchmark measures: release rules (R8 on, not debuggable) but signed with the
* debug key so it can be installed on a test device. The manifest in src/benchmark adds the
* one thing a benchmark cannot work without — `<profileable android:shell="true"/>`.
* What macrobenchmark measures: release rules (R8 on, not debuggable) but measurable.
*
* The suffix is not cosmetic, and it is `.perf` rather than `.benchmark` on purpose: the
* benchmark module's own APK is `de.softwareapp_hb.filemanager.benchmark`, so an app under
* test with that id collides with the test APK and the install fails with
* INSTALL_FAILED_VERSION_DOWNGRADE. Without a suffix at all, this build would have the app's
* own package id and a debug signature — which cannot be installed beside a release build,
* and would replace a release install if the keys happened to match. A benchmark must never be
* able to take over the shipped app.
* **It keeps the app's own package id on purpose.** A separate id (`.perf`) was the first
* design, and it is the tidier one — a benchmark can never take over a release install — but
* on this MIUI device a *new* package id cannot be installed unattended at all: `adb install`
* is refused with INSTALL_FAILED_USER_RESTRICTED, and installing by tapping the APK left the
* package listed by `pm list packages` yet unknown to `pm enable` and invisible to
* PackageManager, which is exactly what macrobenchmark reports as "Unable to find target
* package". Replacing the installed app is the only path that works without a human in the
* loop for every run.
*
* So it is signed with the release key when that key is configured: same package, same
* signature, so it *updates* the installed app and keeps its data. Without `keystore.properties`
* (CI, a fresh clone) it falls back to the debug key and can then only be installed on a device
* that has no release build of the app.
*
* **This build replaces a release install.** Reinstalling the release APK puts it back.
*/
create("benchmark") {
initWith(getByName("release"))
applicationIdSuffix = ".perf"
signingConfig = signingConfigs.getByName("debug")
signingConfig = if (hasReleaseSigning) {
signingConfigs.getByName("release")
} else {
signingConfigs.getByName("debug")
}
matchingFallbacks += listOf("release")
isDebuggable = false
}
@@ -131,8 +140,12 @@ android {
* `testBenchmarkUnitTest`.
*
* ./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
*
* `hasProperty` rather than `providers.gradleProperty`: the latter is a Provider, and inside this
* block it read as absent even when the flag was passed — which showed up as the benchmark
* installing the *debug* APK and failing on it.
*/
if (providers.gradleProperty("benchmarkBuild").isPresent) {
if (project.hasProperty("benchmarkBuild")) {
testBuildType = "benchmark"
}