A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back.
Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it.
coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates.
Gate green, map current.
coldStart measures 532 ms median (min 499, max 805) on the benchmark build - R8, not debuggable, profileable, CompilationMode.None - against 1814 ms on the debug build, which is the honest illustration of why a debug number must never be published as the app's performance. The tool also warns that it cannot clear the runtime image on this Android version, so the figure is optimistic by an unmeasured amount; that is in PERFORMANCE.md next to the number rather than in a footnote.
The three interaction benchmarks are written and compile but have not produced a number, and getting that far took four fixes worth naming. The app under test cannot keep the app's package id (signature) and cannot use .benchmark (the test APK owns that id, INSTALL_FAILED_VERSION_DOWNGRADE); macrobenchmark needs package visibility for it or it reports NameNotFoundException for an app that pm list packages shows installed; UiAutomation.executeShellCommand runs as the test app's uid, so the fixture has to be made through adb instead - that one cost the first attempt a hang; and testBuildType has to be behind a flag or :app:testDebugUnitTest stops existing.
The workflow that works on a device with an install gate is now documented: install the two APKs by hand once, run tools/benchmark-fixture.sh, then am instrument. The Gradle task is documented as the alternative and as the reason the app under test disappeared once - it uninstalls around the run.
Gate green, map current, 386 tests unchanged.
docs/PERFORMANCE.md has been honest that three of its four numbers do not exist. They exist now as
measurements, if not yet as results: benchmark/ is a macrobenchmark module with four tests - cold start
with StartupTimingMetric, then opening, scrolling and searching a folder of a thousand files with
FrameTimingMetric - sharing a fixture at /sdcard/AABenchmark that is created once per run and deleted in
@AfterClass. Frames rather than a named trace section on purpose: what a user calls slow is jank on
screen, and the alternative puts benchmarking calls into production code for the sake of a number.
CompilationMode.None() is the honest cold-install baseline; Partial() plus a baseline profile is the
follow-up and needs the same module. The storage grant is applied with appops in setup, which is what
lets the run be unattended - a permission dialog is the one thing a test cannot answer.
Three build facts had to be discovered rather than designed, and each is now written where it will be
read. The app under test cannot keep its own package id: it needs a debug signature to install on a test
device, so it cannot replace a release install, and an un-suffixed build is exactly that. It also cannot
use the obvious suffix: `.benchmark` is already the benchmark module's own APK id, and the test APK then
fails to install with INSTALL_FAILED_VERSION_DOWNGRADE - which is what the first attempt did. Hence
`.perf`. And `testBuildType = "benchmark"` cannot be set unconditionally, because it makes benchmark the
variant that *unit* tests run against and :app:testDebugUnitTest - the task the entire gate is built on -
stops existing; it is behind -PbenchmarkBuild instead. All three were measured, not guessed, and the
third would have broken CI silently.
Two smaller things came with it. tools/verify.sh --release now also compiles the app's benchmark variant
and the test APK, because build wiring nobody runs is wiring that rots; no device is involved. And the
code map grew an EXTRA_ROOTS section so it indexes benchmark/ too - a map that silently omits a module
lies by omission, which is worse than the one line it costs.
The numbers themselves are still missing, and for a device reason rather than a code one: installing the
new package id on this MIUI phone fails with INSTALL_FAILED_USER_RESTRICTED (the "Install via USB" gate,
which wants a human and answered too late). Both APKs are pushed to /sdcard/Download/ for a manual
install, and the run is one command away:
`adb shell am instrument -w -e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner`.
Gate green with --release, including the new compile step; map current; 386 tests unchanged.
SoftwareApp