Make the benchmark build the app itself, and record what the runner does to it

A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back.

Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it.

coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates.

Gate green, map current.
This commit is contained in:
2026-10-17 10:31:56 +02:00
parent 94f85bd287
commit cb2db9ab63
6 changed files with 65 additions and 42 deletions
+2 -2
View File
@@ -2,7 +2,7 @@
Generated from the source by `tools/make-code-map.py` - do not edit by hand. Regenerate it after a change (`python3 tools/make-code-map.py`); `--check` exits non-zero when this file is stale.
77 source files, 12659 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification.
77 source files, 12660 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification.
Start at `AGENTS.md`. What is done, and why: `docs/ROADMAP.md` and `CHANGELOG.md`. What was verified by hand rather than by a test: `docs/EDGE_CASE_TESTING.md`.
@@ -170,7 +170,7 @@ The macrobenchmark test APK: four measurements over a fixture folder, run on a d
| file | lines | what it holds | tests |
|---|---|---|---|
| `BrowserBenchmark.kt` | 167 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - |
| `BrowserBenchmark.kt` | 168 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - |
## Behaviour index
+31 -22
View File
@@ -51,9 +51,14 @@ first items of the speed work.
## Measured on demand: the `:benchmark` module
`benchmark/` holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
build of the app: release rules (R8 on, not debuggable), debug-signed, and profileable — under its own
package id, `…filemanager.perf`, so it can never collide with the test APK or take over a release
install on the device.
build of the app: release rules (R8 on, not debuggable) and profileable, **under the app's own package id
and signed with the release key**, so installing it *updates* an installed release build and keeps its
data. A separate package id (`.perf`) was the first design and is the tidier one, but on this MIUI phone a
new package id cannot be installed without a human: `adb install` is refused with
`INSTALL_FAILED_USER_RESTRICTED`, and installing by tapping the APK left the package visible to
`pm list packages` yet invisible to `PackageManager`, which is what macrobenchmark reports as *"Unable to
find target package"*. The trade-off is written in `:app`'s build file: **this build replaces a release
install**, and reinstalling the release APK puts it back.
| benchmark | metric | what it answers |
|---|---|---|
@@ -69,30 +74,34 @@ baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up,
module.
**Two of these have produced a number** (`coldStart`, above); the three interaction ones have not, and
the reason is the device rather than the code. Getting them to run at all took four fixes, each of which
is written next to the code it belongs to: the app under test needs its own package id; it needs to be
installed by hand once, because MIUI refuses `adb install` of a new package id without a human
(`INSTALL_FAILED_USER_RESTRICTED`); the test APK needs package-visibility for the app it measures, or
macrobenchmark reports `NameNotFoundException` for an app that `pm list packages` shows installed; and
the fixture has to be created through adb, because `UiAutomation.executeShellCommand` runs with the
*test app's* uid and cannot write to `/sdcard`.
the reason is a device policy rather than the code. Getting this far took five fixes, each written next
to what it constrains: the app under test needs the app's own package id and release key (a new id cannot
be installed here); the test APK needs package visibility for it; the fixture has to be created through
adb, because `UiAutomation.executeShellCommand` runs with the *test app's* uid and cannot write `/sdcard`;
`testBuildType` has to sit behind `-PbenchmarkBuild`, or `:app:testDebugUnitTest` stops existing; and
`hasProperty` is needed to read that flag, because the Provider form read as absent and the runner then
installed the *debug* APK and failed on it.
The workflow that avoids all of it after the first install:
The workflow:
```bash
# once, by hand on the phone (Files → Download → tap), because MIUI gates adb installs:
# /sdcard/Download/PlainFiles-perf.apk (app under test, package …filemanager.perf)
# /sdcard/Download/PlainFiles-benchmark-test.apk
./tools/benchmark-fixture.sh # create the fixture and grant storage access
adb shell am instrument -w \
-e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \
de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner
./tools/benchmark-fixture.sh # the fixture, and the storage grant the benchmark build lacks
./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
```
`am instrument` is preferred over `./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest` on a
device with an install gate: the Gradle task installs *and uninstalls* the APKs around the run, so every
run needs a human again — and an uninstall between runs is what made the app under test disappear once.
**The Gradle runner is required, not preferred.** Macrobenchmark's rule uninstalls the app at the start
of a run and reinstalls it before measuring. On this device *that reinstall* is what MIUI refuses, so the
run leaves the app uninstalled and then fails with `NameNotFoundException` for an app that was there a
second earlier — seen twice through `am instrument` and once through `connectedDebugAndroidTest`, with
`Benchmark: Uninstalling de.softwareapp_hb.filemanager` in logcat and the package gone from
`pm list packages` afterwards.
The fix is a device setting rather than code: *Developer options → Install via USB* — and on MIUI usually
*MIUI optimization off* as well — so that an install raised by the benchmark is allowed without a prompt.
Until that is on, **the interaction benchmarks cannot run on this phone**, and the cold-start number above
stands alone. `tools/benchmark-fixture.sh` is still needed either way: the benchmark build has no storage
grant of its own, and `UiAutomation.executeShellCommand` (the only shell a test has) runs with the test
app's uid and cannot write `/sdcard`.
## Not measured yet, and why