904b74773d
Three things went wrong in this script and are fixed here, each with the run that showed it. It remembered a button position and tapped it without re-checking the dialog was still there, so a tap landed on the Telegram icon on the home screen behind a dialog that had already closed; it refused to answer a second dialog because it compared window tokens, and every dialog gets a new one, so the third install in a run was skipped; and its idle poll was so slow that a pair of installs could not both be caught. Now: the position is only used while an installer window is focused, re-checked immediately before the tap; a newer dialog with the same installer activity is answered rather than skipped; and after every dialog the script watches at 0.4 s for the follow-up install. Verified with two consecutive uninstall/install cycles, the second answered from the measured position five seconds after the first, and by installing the release APK with only the watcher running. The benchmark's own reinstall is still refused - two dialogs answered during one run, both with the installer focused, and pm install still returned Install canceled by user, because MIUI closes the dialog when the session is over.
185 lines
13 KiB
Markdown
185 lines
13 KiB
Markdown
# Performance
|
||
|
||
What has been measured, on what, and — explicitly — what has **not** been measured at all. The rule
|
||
this file exists to enforce comes from the competitive survey: the market leader's top complaint is
|
||
thumbnails that load late, its own support reply to a speed complaint was *"your SD card is slow"*,
|
||
and the lesson recorded there was to measure what was measured and never close a performance
|
||
complaint by blaming the hardware.
|
||
|
||
Until this file existed, the app had **no performance number of any kind**.
|
||
|
||
## Cold start
|
||
|
||
| build | device | runs (ms) | median |
|
||
|---|---|---|---|
|
||
| debug, `1.0.0-debug` | Xiaomi phone, Android 15 | 1 758 · 1 773 · 1 814 · 1 926 · 2 043 | **1 814 ms** |
|
||
| `benchmark` (R8, not debuggable, profileable) | same | — | **532 ms** |
|
||
|
||
The first is `./tools/measure-startup.sh 5` (`am start -W`, one warm-up start discarded because the first
|
||
launch after an install pays for dexopt). The second is `BrowserBenchmark#coldStart` in the `:benchmark`
|
||
module: `StartupTimingMetric`, five iterations, `CompilationMode.None()`, measured as time to initial
|
||
display — min 499 ms, median 532 ms, max 805 ms. Measured three times: 615 ms before the storage grant was
|
||
in place (the app was showing the permission screen), 532 ms after it, and 560 ms (min 539, max 698) on a
|
||
later run that also exercised the full benchmark suite. The three runs agree within a phone's patience;
|
||
the 532 ms figure above is the middle one.
|
||
|
||
Two caveats belong to that number rather than to a footnote. The tool itself warns *"Unable to clear
|
||
Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to
|
||
accelerated class loading"*, so it is optimistic by an unmeasured amount on this Android version. And
|
||
`CompilationMode.None()` means no compilation profile is applied — what a fresh install does — so a
|
||
baseline profile would lower it further and is the obvious next measurement.
|
||
|
||
The debug build is deliberately the *slow* case — StrictMode on, LeakCanary attached, no R8 — which is
|
||
why the same screen takes it three times as long. That is the honest illustration of why a debug number
|
||
must never be published as the app's performance.
|
||
|
||
### What StrictMode found on the first cold start
|
||
|
||
The same debug build now logs StrictMode violations, and the first cold start produced five, against a
|
||
1 814 ms start:
|
||
|
||
| duration | where | ours? |
|
||
|---|---|---|
|
||
| 492 ms | `BrowserViewModel.Factory` → `app.cacheDir` | **yes** — `Context.getCacheDir()` checks and creates the private cache directory while the view model is being created, on the main thread |
|
||
| 397 ms | `BrowserViewModel.rootsOf` → `StorageProvider.space` → `VolumeRepository.stat` | **yes** — free space for every volume is read with `StatFs` inside `refresh()`, which the screen calls from a `LaunchedEffect` |
|
||
| 477 ms | `FileManagerTheme` → `typography = FileManagerTypography` | no — the platform loading its own typefaces on first composition |
|
||
| 54 ms × 2 | `miui.util.font.FontSettings` via `MainActivity.onCreate` | no — the OEM's font hook doing `File.exists` while the activity is created |
|
||
|
||
That is **~1.5 s of main-thread disk I/O inside a 1.8 s start**, of which **~0.9 s is ours**. Both of
|
||
ours are mechanical to move and neither is fixed yet: the cache directory is only needed when a preview
|
||
copies something, and the free-space line on the roots screen can arrive a frame later. They are the
|
||
first items of the speed work.
|
||
|
||
## Measured on demand: the `:benchmark` module
|
||
|
||
`benchmark/` holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
|
||
build of the app: release rules (R8 on, not debuggable) and profileable, **under the app's own package id
|
||
and signed with the release key**, so installing it *updates* an installed release build and keeps its
|
||
data. A separate package id (`.perf`) was the first design and is the tidier one, but on this MIUI phone a
|
||
new package id cannot be installed without a human: `adb install` is refused with
|
||
`INSTALL_FAILED_USER_RESTRICTED`, and installing by tapping the APK left the package visible to
|
||
`pm list packages` yet invisible to `PackageManager`, which is what macrobenchmark reports as *"Unable to
|
||
find target package"*. The trade-off is written in `:app`'s build file: **this build replaces a release
|
||
install**, and reinstalling the release APK puts it back.
|
||
|
||
| benchmark | metric | what it answers |
|
||
|---|---|---|
|
||
| `coldStart` | `StartupTimingMetric` | tap → first frame |
|
||
| `openAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while a 1 000-file folder fills in |
|
||
| `scrollAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while scrolling that folder |
|
||
| `searchInsideAFolderOfAThousandFiles` | `FrameTimingMetric` | jank while typing into search |
|
||
|
||
They share a fixture — `/sdcard/AABenchmark`, a thousand empty files, created by `tools/benchmark-fixture.sh`
|
||
and removed with `--remove`. `CompilationMode.None()` is deliberate: no profile is applied, which is the
|
||
honest cold-install baseline. `CompilationMode.Partial()` plus a baseline profile is the follow-up, and
|
||
needs the same module.
|
||
|
||
### What the module has actually measured
|
||
|
||
| benchmark | outcome on the Xiaomi phone (Android 12, MIUI) |
|
||
|---|---|
|
||
| `coldStart` | **passes** — 560.6 ms P50, min 539, max 698 over five iterations. A second measurement of the 532 ms above, and the only number this toolchain reports here |
|
||
| `scrollAFolderOfAThousandFiles` | passes, but reports **only `frameCount`** (46–100 frames per iteration): this device hands macrobenchmark no `frameDurationCpuMs`/`frameOverrunMs` at all |
|
||
| `openAFolderOfAThousandFiles` | `INSTALL_FAILED_USER_RESTRICTED: Install canceled by user` — the rule reinstalls the app before measuring, and MIUI refuses that reinstall |
|
||
| `searchInsideAFolderOfAThousandFiles` | the app bar's search action is not in the accessibility tree within ten seconds of the folder opening, though it is there when the same taps are made by hand |
|
||
|
||
So the interaction numbers are not written down here because **the device will not produce them**, not
|
||
because the code cannot ask. Two separate device behaviours are in the way, and both are outside the app:
|
||
a reinstall MIUI refuses, and a platform that reports frame counts but not frame durations.
|
||
|
||
### Why the reinstall cannot be answered
|
||
|
||
`tools/benchmark-watch-install.sh` exists to answer MIUI's confirmation from the host, side by side with a
|
||
run: it polls every five seconds, reads the screen when an installer window has focus, and taps the
|
||
**Install** button's own coordinates — never a guessed position. It works for an install that stays up
|
||
(`adb install` of a new package: the dialog appears after 1.7 s, counts down from ten, and the script
|
||
reads the button and taps it), and it does **not** work for the one that matters. Measured:
|
||
|
||
- The dialog the rule's install raises is focused for **well under a second** —
|
||
`13:51:27.70 AdbInstallActivity` → `13:51:28.38 null` in a 0.1 s focus trace, and one sample in a 0.5 s
|
||
trace. One `uiautomator dump` costs 3–4 s, so the button cannot even be read before the window is gone.
|
||
- It is `FLAG_SECURE`: a `screencap` taken while `mCurrentFocus` was the installer shows the home screen
|
||
behind it, which is why screenshots of "the dialog" show nothing.
|
||
- A tap on it, when a run did stay up long enough for the script to read and tap `Installieren`, still
|
||
ended with `INSTALL_FAILED_USER_RESTRICTED: Install canceled by user`. The same happened later with the
|
||
script answering *two* dialogs in one run from a measured button position, each logged with the
|
||
installer window still focused: the install was refused anyway. MIUI is closing its own dialog when the
|
||
install session ends, so the tap arrives after the outcome has been decided.
|
||
- `pm install` started from an *app* uid does not even reach the gate: MIUI's `XSpaceManagerServiceImpl`
|
||
throws `SecurityException: You either need MANAGE_USERS or CREATE_USERS permission to: query users`.
|
||
So the rule's install runs through the shell, MIUI intercepts it as an "Install via USB" request, and
|
||
gates *new* packages while letting updates through — which is why `adb install -r` of an
|
||
already-installed build returns `Success` instantly and the rule's reinstall never does.
|
||
|
||
The switch that governs this is MIUI's own **Developer options → Install via USB**, whose name the dialog
|
||
carries. It is not in any Android settings namespace, so it cannot be read or set away from the phone's
|
||
own UI.
|
||
|
||
### The four switches that make an unattended run possible at all
|
||
|
||
1. **The app under test is the app itself** — same package id, release key — because a new package id
|
||
cannot be installed on this phone (`INSTALL_FAILED_USER_RESTRICTED`, and an APK installed by tapping
|
||
ended up visible to `pm list packages` but invisible to `PackageManager`). The cost is recorded in
|
||
`:app`'s build file: **this build replaces a release install**; reinstalling the release APK puts it
|
||
back.
|
||
2. **The test module declares a `benchmark` build type.** AGP 9's `com.android.test` extension has no
|
||
`testBuildType` and no `targetVariant` — only `targetProjectPath` — and the target variant is matched
|
||
*by name* against the test module's own build types. Without that type every run installed
|
||
`app-debug.apk`, so the numbers would have been debug numbers. The type also has to
|
||
`initWith(getByName("debug"))`: a declared type starts unsigned, and an unsigned test APK fails with
|
||
`INSTALL_PARSE_FAILED_NO_CERTIFICATES`.
|
||
3. **The storage grant is walked through the UI, not set with `appops`.** The rule uninstalls and
|
||
reinstalls the app, and an uninstall clears app-ops, so a grant applied by the fixture script is gone
|
||
by the time the app starts. `appops set` from inside the test does not work either — even under
|
||
`adoptShellPermissionIdentity`, the command fails to stderr (which `executeShellCommand` does not
|
||
return) and `appops get` still reads `No operations`. What does work is tapping what a user taps:
|
||
the app's "Grant full access" button, then the platform's consent, then leaving the all-files page.
|
||
`grantStorageAccess` in `BrowserBenchmark` does exactly that, in `setupBlock`, unmeasured.
|
||
4. **The fixture is created through adb**, because `UiAutomation.executeShellCommand` runs with the *test
|
||
app's* uid and cannot write `/sdcard`.
|
||
|
||
### The workflow
|
||
|
||
```bash
|
||
./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files
|
||
./tools/benchmark-watch-install.sh --seconds 900 & # answers MIUI's install dialog if it stays up
|
||
./gradlew :benchmark:connectedBenchmarkAndroidTest
|
||
```
|
||
|
||
`coldStart` alone can be run with `-Pandroid.testInstrumentationRunnerArguments.class=…BrowserBenchmark#coldStart`.
|
||
The results land in `benchmark/build/outputs/androidTest-results/connected/benchmark/`, and — worth
|
||
knowing, because it cost an hour — the per-test JSON that holds the numbers is written to
|
||
`/sdcard/Android/media/<test-pkg>/additional_test_output/` and **deleted together with the test app when
|
||
the run ends**, often before Gradle has copied it. Copying it in a loop while the run is in progress is
|
||
what recovered the numbers above.
|
||
|
||
## Not measured yet, and why
|
||
|
||
| number | how it would be taken | why it is not here |
|
||
|---|---|---|
|
||
| Folder open, 1 000 / 5 000 / 20 000 files | `FrameTimingMetric` around the folder tap (`openAFolderOfAThousandFiles` exists) | the rule reinstalls the app mid-run, and MIUI answers that install with **"Install canceled by user"** unless someone is there to tap |
|
||
| Thumbnail paint while scrolling a photo grid | `FrameTimingMetric` | same; and the scroll variant that *did* run reported only `frameCount`, because this device gives macrobenchmark no frame durations |
|
||
| Keystroke → search result | `FrameTimingMetric` around typing into search (`searchInsideAFolderOfAThousandFiles` exists) | the app bar's search action does not appear in the accessibility tree within ten seconds of the folder opening in the automated run, although the same taps by hand find it immediately |
|
||
|
||
The module exists and three of its four tests pass; what is missing is a device that lets macrobenchmark
|
||
install unattended *and* reports frame timing. A plain emulator image (or any non-MIUI phone) with the
|
||
system language set to English would answer all three — the fixture is a folder of empty files, so nothing
|
||
device-specific is being measured. Until then this file says what was measured and what was not, and the
|
||
cold-start number stands alone.
|
||
|
||
## What is measured instead, meanwhile
|
||
|
||
The gate measures things that are not latency but do not rot: the test count, lint, detekt against its
|
||
baseline, and whether `docs/CODE_MAP.md` still matches the tree. Those are the numbers that can be
|
||
kept true by automation; the three above need a device and a decision about which device.
|
||
|
||
## Two rules
|
||
|
||
- **No number that was not taken from a real build on a real device.** Not from a debug build
|
||
pretending to be release, and not from reading the code.
|
||
- **No performance change judged by reading the code.** The bigger-thumbnail change (28 → 48 dp in a
|
||
row, 40 → 96 dp in a grid) is the standing example: it may be exactly right, and nobody measured
|
||
either side of it. `docs/COMPETITIVE_ANALYSIS.md` had already written "a thumbnail that is bigger and
|
||
slower is strictly worse than one that is smaller and instant" — which is a hypothesis until it is a
|
||
number.
|