fishfacegit/file-manager
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Make the benchmark actually measure the benchmark build, and record what the phone will and will not answer

Browse Source 
The run was installing app-debug.apk the whole time, which would have published debug numbers as the release-like ones. AGP 9's com.android.test extension has no testBuildType and no targetVariant - it exposes only targetProjectPath - and the target variant is matched by name against the test module's own build types. So the module now declares a benchmark build type, and inherits the debug signing config with initWith: a declared type starts unsigned, and an unsigned test APK fails with INSTALL_PARSE_FAILED_NO_CERTIFICATES. The command is now :benchmark:connectedBenchmarkAndroidTest, and -PbenchmarkBuild is gone from the workflow; the conditional testBuildType it used to set in :app only ever moved the unit-test variant, which is why it made :app:testDebugUnitTest disappear.

The storage grant also had to move into the test. The rule uninstalls and reinstalls the app before measuring, an uninstall clears app-ops, and appops set from inside the test does not work even under adoptShellPermissionIdentity - the command fails to stderr, which executeShellCommand does not return, and appops get still reads No operations. What does work is tapping what a user taps: Grant full access, then the platform consent, then back out of the all-files page. grantStorageAccess does that in setupBlock, unmeasured, and it is what turned three failing tests into three passing ones.

The search test now waits for the app bar's search action in setup instead of looking it up immediately inside the measured block, where it came back null one frame after a thousand rows rendered.

What the device answers: coldStart passes at 560.6 ms P50 (a third measurement of the 532 ms figure), scroll passes but reports only frameCount and no frame durations at all, open dies on INSTALL_FAILED_USER_RESTRICTED because the rule's reinstall wants a human, and search still cannot see the search action. PERFORMANCE.md now states that per benchmark instead of the earlier guess, names the four switches that make an unattended run possible, and says that three of the four numbers need a device that installs unattended and reports frame timing.
This commit is contained in:
Hadrian Burkhardt
2026-10-17 11:23:36 +02:00
parent 5a6ae3f71d
commit ec76acfdd6
6 changed files with 150 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PERFORMANCE.md
+60 -47
View File
@@ -18,8 +18,10 @@ Until this file existed, the app had **no performance number of any kind**.
The first is `./tools/measure-startup.sh 5` (`am start -W`, one warm-up start discarded because the first
launch after an install pays for dexopt). The second is `BrowserBenchmark#coldStart` in the `:benchmark`
module: `StartupTimingMetric`, five iterations, `CompilationMode.None()`, measured as time to initial
display — min 499 ms, median 532 ms, max 805 ms. Measured twice: 615 ms before the storage grant was in
place (the app was showing the permission screen) and 532 ms after, which is the figure above.
display — min 499 ms, median 532 ms, max 805 ms. Measured three times: 615 ms before the storage grant was
in place (the app was showing the permission screen), 532 ms after it, and 560 ms (min 539, max 698) on a
later run that also exercised the full benchmark suite. The three runs agree within a phone's patience;
the 532 ms figure above is the middle one.
Two caveats belong to that number rather than to a footnote. The tool itself warns *"Unable to clear
Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to
@@ -67,63 +69,74 @@ install**, and reinstalling the release APK puts it back.
| `scrollAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while scrolling that folder |
| `searchInsideAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while typing into search |
They share a fixture — `/sdcard/AABenchmark`, a thousand empty files, created once per run and deleted
in `@AfterClass` — and the storage grant is applied with `appops`, so an unattended run is possible.
`CompilationMode.None()` is deliberate: no profile is applied, which is the honest cold-install
baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up, and needs the same
module.
They share a fixture — `/sdcard/AABenchmark`, a thousand empty files, created by `tools/benchmark-fixture.sh`
and removed with `--remove`. `CompilationMode.None()` is deliberate: no profile is applied, which is the
honest cold-install baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up, and
needs the same module.
**Two of these have produced a number** (`coldStart`, above); the three interaction ones have not, and
the reason is a device policy rather than the code. Getting this far took five fixes, each written next
to what it constrains: the app under test needs the app's own package id and release key (a new id cannot
be installed here); the test APK needs package visibility for it; the fixture has to be created through
adb, because `UiAutomation.executeShellCommand` runs with the *test app's* uid and cannot write `/sdcard`;
`testBuildType` has to sit behind `-PbenchmarkBuild`, or `:app:testDebugUnitTest` stops existing; and
`hasProperty` is needed to read that flag, because the Provider form read as absent and the runner then
installed the *debug* APK and failed on it.
### What the module has actually measured
The workflow:
| benchmark | outcome on the Xiaomi phone (Android 12, MIUI) |
|---|---|
| `coldStart` | **passes** — 560.6 ms P50, min 539, max 698 over five iterations. A second measurement of the 532 ms above, and the only number this toolchain reports here |
| `scrollAFolderOfAThousandFiles` | passes, but reports **only `frameCount`** (46–100 frames per iteration): this device hands macrobenchmark no `frameDurationCpuMs`/`frameOverrunMs` at all |
| `openAFolderOfAThousandFiles` | `INSTALL_FAILED_USER_RESTRICTED: Install canceled by user` — the rule reinstalls the app mid-run, and MIUI wants a tap for that |
| `searchInsideAFolderOfAThousandFiles` | the app bar's search action is not in the accessibility tree within ten seconds of the folder opening, though it is there when the same taps are made by hand |
So the interaction numbers are not written down here because **the device will not produce them**, not
because the code cannot ask. Two separate device behaviours are in the way, and both are outside the app:
a reinstall that needs a human, and a platform that reports frame counts but not frame durations.
### The four switches that make an unattended run possible at all
1. **The app under test is the app itself** — same package id, release key — because a new package id
cannot be installed on this phone (`INSTALL_FAILED_USER_RESTRICTED`, and an APK installed by tapping
ended up visible to `pm list packages` but invisible to `PackageManager`). The cost is recorded in
`:app`'s build file: **this build replaces a release install**; reinstalling the release APK puts it
back.
2. **The test module declares a `benchmark` build type.** AGP 9's `com.android.test` extension has no
`testBuildType` and no `targetVariant` — only `targetProjectPath` — and the target variant is matched
*by name* against the test module's own build types. Without that type every run installed
`app-debug.apk`, so the numbers would have been debug numbers. The type also has to
`initWith(getByName("debug"))`: a declared type starts unsigned, and an unsigned test APK fails with
`INSTALL_PARSE_FAILED_NO_CERTIFICATES`.
3. **The storage grant is walked through the UI, not set with `appops`.** The rule uninstalls and
reinstalls the app, and an uninstall clears app-ops, so a grant applied by the fixture script is gone
by the time the app starts. `appops set` from inside the test does not work either — even under
`adoptShellPermissionIdentity`, the command fails to stderr (which `executeShellCommand` does not
return) and `appops get` still reads `No operations`. What does work is tapping what a user taps:
the app's "Grant full access" button, then the platform's consent, then leaving the all-files page.
`grantStorageAccess` in `BrowserBenchmark` does exactly that, in `setupBlock`, unmeasured.
4. **The fixture is created through adb**, because `UiAutomation.executeShellCommand` runs with the *test
app's* uid and cannot write `/sdcard`.
### The workflow
```bash
./tools/benchmark-fixture.sh # the fixture, and the storage grant the benchmark build lacks
./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files
./gradlew :benchmark:connectedBenchmarkAndroidTest
```
**The Gradle runner is required, not preferred.** Macrobenchmark's rule uninstalls the app at the start
of a run and reinstalls it before measuring. That alone is survivable; what is not is the side effect:
**an uninstall clears the app's app-ops**, so the `MANAGE_EXTERNAL_STORAGE` grant from
`tools/benchmark-fixture.sh` is gone by the time the test navigates. The app then correctly shows its
"choose a folder" fallback, the volume row is not on the screen, and the assertion says so. Checked
directly — `appops get … MANAGE_EXTERNAL_STORAGE` reads `No operations` after a run that passed through
the reinstall.
There is no way around that from inside the test: granting an app-op needs the shell's uid, and
`UiAutomation.executeShellCommand` runs with the *test app's* uid. So on a device where this app needs
that grant — which is every device, for the browser screen — **the three interaction benchmarks cannot run
as macrobenchmark tests here**, and the cold-start number above stands alone.
Two paths would give the numbers, and neither is a small change:
- **A plain instrumented test instead of a macrobenchmark one** (no rule, so no reinstall): install the
benchmark build once, grant, then navigate with UiAutomator and read frame durations from
`FrameMetrics`. It measures the release-like build, but it is our own measurement code rather than a
maintained library, and `FrameTimingMetric` exists precisely so nobody has to write that.
- **An app that does not need the grant for the flow being measured** — for instance a benchmark
build that browses its own external files directory, which needs no permission. That measures a
different screen than the one users open, so it answers a narrower question than "how fast does
opening a 1 000-file folder feel".
`coldStart` alone can be run with `-Pandroid.testInstrumentationRunnerArguments.class=…BrowserBenchmark#coldStart`.
The results land in `benchmark/build/outputs/androidTest-results/connected/benchmark/`, and — worth
knowing, because it cost an hour — the per-test JSON that holds the numbers is written to
`/sdcard/Android/media/<test-pkg>/additional_test_output/` and **deleted together with the test app when
the run ends**, often before Gradle has copied it. Copying it in a loop while the run is in progress is
what recovered the numbers above.
## Not measured yet, and why
| number | how it would be taken | why it is not here |
|---|---|---|
| Folder open, 1 000 / 5 000 / 20 000 files | macrobenchmark `StartupTimingMetric` over a fixture volume, or `TraceSectionMetric` around the listing | needs a `:benchmark` module and a profileable build installed under the **release package id**; on this MIUI phone the first install of a new package id waits for a confirmation dialog, so it cannot run unattended |
| Thumbnail paint while scrolling a photo grid | `FrameTimingMetric` | same |
| Keystroke → search result | `TraceSectionMetric` around the query and the index read | same |
| Folder open, 1 000 / 5 000 / 20 000 files | `FrameTimingMetric` around the folder tap (`openAFolderOfAThousandFiles` exists) | the rule reinstalls the app mid-run, and MIUI answers that install with **"Install canceled by user"** unless someone is there to tap |
| Thumbnail paint while scrolling a photo grid | `FrameTimingMetric` | same; and the scroll variant that *did* run reported only `frameCount`, because this device gives macrobenchmark no frame durations |
| Keystroke → search result | `FrameTimingMetric` around typing into search (`searchInsideAFolderOfAThousandFiles` exists) | the app bar's search action does not appear in the accessibility tree within ten seconds of the folder opening in the automated run, although the same taps by hand find it immediately |
The module does not exist yet, deliberately: half-configured build wiring that nobody can run is worse
than an honest gap. It is the next measurement step, and it belongs with the baseline-profile work,
which needs the same module.
The module exists and three of its four tests pass; what is missing is a device that lets macrobenchmark
install unattended *and* reports frame timing. A plain emulator image (or any non-MIUI phone) with the
system language set to English would answer all three — the fixture is a folder of empty files, so nothing
device-specific is being measured. Until then this file says what was measured and what was not, and the
cold-start number stands alone.
## What is measured instead, meanwhile