Make the benchmark build the app itself, and record what the runner does to it
A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back. Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it. coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates. Gate green, map current.
This commit is contained in:
+2
-2
@@ -2,7 +2,7 @@
|
||||
|
||||
Generated from the source by `tools/make-code-map.py` - do not edit by hand. Regenerate it after a change (`python3 tools/make-code-map.py`); `--check` exits non-zero when this file is stale.
|
||||
|
||||
77 source files, 12659 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification.
|
||||
77 source files, 12660 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification.
|
||||
|
||||
Start at `AGENTS.md`. What is done, and why: `docs/ROADMAP.md` and `CHANGELOG.md`. What was verified by hand rather than by a test: `docs/EDGE_CASE_TESTING.md`.
|
||||
|
||||
@@ -170,7 +170,7 @@ The macrobenchmark test APK: four measurements over a fixture folder, run on a d
|
||||
|
||||
| file | lines | what it holds | tests |
|
||||
|---|---|---|---|
|
||||
| `BrowserBenchmark.kt` | 167 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - |
|
||||
| `BrowserBenchmark.kt` | 168 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - |
|
||||
|
||||
## Behaviour index
|
||||
|
||||
|
||||
+31
-22
@@ -51,9 +51,14 @@ first items of the speed work.
|
||||
## Measured on demand: the `:benchmark` module
|
||||
|
||||
`benchmark/` holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
|
||||
build of the app: release rules (R8 on, not debuggable), debug-signed, and profileable — under its own
|
||||
package id, `…filemanager.perf`, so it can never collide with the test APK or take over a release
|
||||
install on the device.
|
||||
build of the app: release rules (R8 on, not debuggable) and profileable, **under the app's own package id
|
||||
and signed with the release key**, so installing it *updates* an installed release build and keeps its
|
||||
data. A separate package id (`.perf`) was the first design and is the tidier one, but on this MIUI phone a
|
||||
new package id cannot be installed without a human: `adb install` is refused with
|
||||
`INSTALL_FAILED_USER_RESTRICTED`, and installing by tapping the APK left the package visible to
|
||||
`pm list packages` yet invisible to `PackageManager`, which is what macrobenchmark reports as *"Unable to
|
||||
find target package"*. The trade-off is written in `:app`'s build file: **this build replaces a release
|
||||
install**, and reinstalling the release APK puts it back.
|
||||
|
||||
| benchmark | metric | what it answers |
|
||||
|---|---|---|
|
||||
@@ -69,30 +74,34 @@ baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up,
|
||||
module.
|
||||
|
||||
**Two of these have produced a number** (`coldStart`, above); the three interaction ones have not, and
|
||||
the reason is the device rather than the code. Getting them to run at all took four fixes, each of which
|
||||
is written next to the code it belongs to: the app under test needs its own package id; it needs to be
|
||||
installed by hand once, because MIUI refuses `adb install` of a new package id without a human
|
||||
(`INSTALL_FAILED_USER_RESTRICTED`); the test APK needs package-visibility for the app it measures, or
|
||||
macrobenchmark reports `NameNotFoundException` for an app that `pm list packages` shows installed; and
|
||||
the fixture has to be created through adb, because `UiAutomation.executeShellCommand` runs with the
|
||||
*test app's* uid and cannot write to `/sdcard`.
|
||||
the reason is a device policy rather than the code. Getting this far took five fixes, each written next
|
||||
to what it constrains: the app under test needs the app's own package id and release key (a new id cannot
|
||||
be installed here); the test APK needs package visibility for it; the fixture has to be created through
|
||||
adb, because `UiAutomation.executeShellCommand` runs with the *test app's* uid and cannot write `/sdcard`;
|
||||
`testBuildType` has to sit behind `-PbenchmarkBuild`, or `:app:testDebugUnitTest` stops existing; and
|
||||
`hasProperty` is needed to read that flag, because the Provider form read as absent and the runner then
|
||||
installed the *debug* APK and failed on it.
|
||||
|
||||
The workflow that avoids all of it after the first install:
|
||||
The workflow:
|
||||
|
||||
```bash
|
||||
# once, by hand on the phone (Files → Download → tap), because MIUI gates adb installs:
|
||||
# /sdcard/Download/PlainFiles-perf.apk (app under test, package …filemanager.perf)
|
||||
# /sdcard/Download/PlainFiles-benchmark-test.apk
|
||||
|
||||
./tools/benchmark-fixture.sh # create the fixture and grant storage access
|
||||
adb shell am instrument -w \
|
||||
-e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \
|
||||
de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner
|
||||
./tools/benchmark-fixture.sh # the fixture, and the storage grant the benchmark build lacks
|
||||
./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
|
||||
```
|
||||
|
||||
`am instrument` is preferred over `./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest` on a
|
||||
device with an install gate: the Gradle task installs *and uninstalls* the APKs around the run, so every
|
||||
run needs a human again — and an uninstall between runs is what made the app under test disappear once.
|
||||
**The Gradle runner is required, not preferred.** Macrobenchmark's rule uninstalls the app at the start
|
||||
of a run and reinstalls it before measuring. On this device *that reinstall* is what MIUI refuses, so the
|
||||
run leaves the app uninstalled and then fails with `NameNotFoundException` for an app that was there a
|
||||
second earlier — seen twice through `am instrument` and once through `connectedDebugAndroidTest`, with
|
||||
`Benchmark: Uninstalling de.softwareapp_hb.filemanager` in logcat and the package gone from
|
||||
`pm list packages` afterwards.
|
||||
|
||||
The fix is a device setting rather than code: *Developer options → Install via USB* — and on MIUI usually
|
||||
*MIUI optimization off* as well — so that an install raised by the benchmark is allowed without a prompt.
|
||||
Until that is on, **the interaction benchmarks cannot run on this phone**, and the cold-start number above
|
||||
stands alone. `tools/benchmark-fixture.sh` is still needed either way: the benchmark build has no storage
|
||||
grant of its own, and `UiAutomation.executeShellCommand` (the only shell a test has) runs with the test
|
||||
app's uid and cannot write `/sdcard`.
|
||||
|
||||
## Not measured yet, and why
|
||||
|
||||
|
||||
Reference in New Issue
Block a user