Record what actually blocks the interaction benchmarks: the reinstall clears the grant

The diagnosis is now precise and it is not the installer dialog. Macrobenchmark's rule uninstalls the app and reinstalls it before measuring; an uninstall clears app-ops, so the MANAGE_EXTERNAL_STORAGE grant that tools/benchmark-fixture.sh applies is gone by the time the test navigates. appops get reads No operations after a run, the app correctly shows its choose-a-folder fallback, and the volume row the test looks for is not on screen - which is exactly the assertion that fired.

No test can work around it: granting an app-op needs the shell uid, and UiAutomation.executeShellCommand runs as the test app. So the three interaction benchmarks cannot run as macrobenchmark tests on any device where this app needs the grant. PERFORMANCE.md now says that, names the two paths that would give the numbers (a plain instrumented test that avoids the rule, or a benchmark flow that needs no grant), and says what each costs. The cold-start number stands.
This commit is contained in:
2026-10-17 10:40:56 +02:00
parent cb2db9ab63
commit 5a6ae3f71d
+21 -11
View File
@@ -90,18 +90,28 @@ The workflow:
``` ```
**The Gradle runner is required, not preferred.** Macrobenchmark's rule uninstalls the app at the start **The Gradle runner is required, not preferred.** Macrobenchmark's rule uninstalls the app at the start
of a run and reinstalls it before measuring. On this device *that reinstall* is what MIUI refuses, so the of a run and reinstalls it before measuring. That alone is survivable; what is not is the side effect:
run leaves the app uninstalled and then fails with `NameNotFoundException` for an app that was there a **an uninstall clears the app's app-ops**, so the `MANAGE_EXTERNAL_STORAGE` grant from
second earlier — seen twice through `am instrument` and once through `connectedDebugAndroidTest`, with `tools/benchmark-fixture.sh` is gone by the time the test navigates. The app then correctly shows its
`Benchmark: Uninstalling de.softwareapp_hb.filemanager` in logcat and the package gone from "choose a folder" fallback, the volume row is not on the screen, and the assertion says so. Checked
`pm list packages` afterwards. directly — `appops get … MANAGE_EXTERNAL_STORAGE` reads `No operations` after a run that passed through
the reinstall.
The fix is a device setting rather than code: *Developer options → Install via USB* — and on MIUI usually There is no way around that from inside the test: granting an app-op needs the shell's uid, and
*MIUI optimization off* as well — so that an install raised by the benchmark is allowed without a prompt. `UiAutomation.executeShellCommand` runs with the *test app's* uid. So on a device where this app needs
Until that is on, **the interaction benchmarks cannot run on this phone**, and the cold-start number above that grant — which is every device, for the browser screen — **the three interaction benchmarks cannot run
stands alone. `tools/benchmark-fixture.sh` is still needed either way: the benchmark build has no storage as macrobenchmark tests here**, and the cold-start number above stands alone.
grant of its own, and `UiAutomation.executeShellCommand` (the only shell a test has) runs with the test
app's uid and cannot write `/sdcard`. Two paths would give the numbers, and neither is a small change:
- **A plain instrumented test instead of a macrobenchmark one** (no rule, so no reinstall): install the
benchmark build once, grant, then navigate with UiAutomator and read frame durations from
`FrameMetrics`. It measures the release-like build, but it is our own measurement code rather than a
maintained library, and `FrameTimingMetric` exists precisely so nobody has to write that.
- **An app that does not need the grant for the flow being measured** — for instance a benchmark
build that browses its own external files directory, which needs no permission. That measures a
different screen than the one users open, so it answers a narrower question than "how fast does
opening a 1 000-file folder feel".
## Not measured yet, and why ## Not measured yet, and why