Add a macrobenchmark module, and find out what the build was hiding

docs/PERFORMANCE.md has been honest that three of its four numbers do not exist. They exist now as
measurements, if not yet as results: benchmark/ is a macrobenchmark module with four tests - cold start
with StartupTimingMetric, then opening, scrolling and searching a folder of a thousand files with
FrameTimingMetric - sharing a fixture at /sdcard/AABenchmark that is created once per run and deleted in
@AfterClass. Frames rather than a named trace section on purpose: what a user calls slow is jank on
screen, and the alternative puts benchmarking calls into production code for the sake of a number.
CompilationMode.None() is the honest cold-install baseline; Partial() plus a baseline profile is the
follow-up and needs the same module. The storage grant is applied with appops in setup, which is what
lets the run be unattended - a permission dialog is the one thing a test cannot answer.

Three build facts had to be discovered rather than designed, and each is now written where it will be
read. The app under test cannot keep its own package id: it needs a debug signature to install on a test
device, so it cannot replace a release install, and an un-suffixed build is exactly that. It also cannot
use the obvious suffix: `.benchmark` is already the benchmark module's own APK id, and the test APK then
fails to install with INSTALL_FAILED_VERSION_DOWNGRADE - which is what the first attempt did. Hence
`.perf`. And `testBuildType = "benchmark"` cannot be set unconditionally, because it makes benchmark the
variant that *unit* tests run against and :app:testDebugUnitTest - the task the entire gate is built on -
stops existing; it is behind -PbenchmarkBuild instead. All three were measured, not guessed, and the
third would have broken CI silently.

Two smaller things came with it. tools/verify.sh --release now also compiles the app's benchmark variant
and the test APK, because build wiring nobody runs is wiring that rots; no device is involved. And the
code map grew an EXTRA_ROOTS section so it indexes benchmark/ too - a map that silently omits a module
lies by omission, which is worse than the one line it costs.

The numbers themselves are still missing, and for a device reason rather than a code one: installing the
new package id on this MIUI phone fails with INSTALL_FAILED_USER_RESTRICTED (the "Install via USB" gate,
which wants a human and answered too late). Both APKs are pushed to /sdcard/Download/ for a manual
install, and the run is one command away:
`adb shell am instrument -w -e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner`.

Gate green with --release, including the new compile step; map current; 386 tests unchanged.
This commit is contained in:
2026-10-17 09:45:36 +02:00
parent 0bbee25ab3
commit 1f94d0a38a
13 changed files with 406 additions and 4 deletions
+43
View File
@@ -39,6 +39,49 @@ ours are mechanical to move and neither is fixed yet: the cache directory is onl
copies something, and the free-space line on the roots screen can arrive a frame later. They are the
first items of the speed work.
## Measured on demand: the `:benchmark` module
`benchmark/` holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
build of the app: release rules (R8 on, not debuggable), debug-signed, and profileable — under its own
package id, `…filemanager.perf`, so it can never collide with the test APK or take over a release
install on the device.
| benchmark | metric | what it answers |
|---|---|---|
| `coldStart` | `StartupTimingMetric` | tap → first frame |
| `openAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while a 1 000-file folder fills in |
| `scrollAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while scrolling that folder |
| `searchInsideAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while typing into search |
They share a fixture — `/sdcard/AABenchmark`, a thousand empty files, created once per run and deleted
in `@AfterClass` — and the storage grant is applied with `appops`, so an unattended run is possible.
`CompilationMode.None()` is deliberate: no profile is applied, which is the honest cold-install
baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up, and needs the same
module.
**They have not produced numbers yet.** On this phone `adb install` of the new package id is refused with
`INSTALL_FAILED_USER_RESTRICTED` — MIUI's "Install via USB" gate, which wants a human — so the two APKs
were pushed to `/sdcard/Download/` (`PlainFiles-perf.apk`, `PlainFiles-benchmark-test.apk`) to be
installed by hand. Once they are on the device:
```bash
adb shell am instrument -w \
-e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \
de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner
```
With a device that accepts installs, the Gradle path does the same and writes the numbers next to the
build output:
```bash
./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
```
`-PbenchmarkBuild` is not decoration: it is what points the app's instrumentation at the `benchmark`
build type. Setting that unconditionally in the build file makes `benchmark` the variant *unit* tests
run against, and `:app:testDebugUnitTest` — the task the gate is built on — disappears. Measured, not
guessed.
## Not measured yet, and why
| number | how it would be taken | why it is not here |