The run was installing app-debug.apk the whole time, which would have published debug numbers as the release-like ones. AGP 9's com.android.test extension has no testBuildType and no targetVariant - it exposes only targetProjectPath - and the target variant is matched by name against the test module's own build types. So the module now declares a benchmark build type, and inherits the debug signing config with initWith: a declared type starts unsigned, and an unsigned test APK fails with INSTALL_PARSE_FAILED_NO_CERTIFICATES. The command is now :benchmark:connectedBenchmarkAndroidTest, and -PbenchmarkBuild is gone from the workflow; the conditional testBuildType it used to set in :app only ever moved the unit-test variant, which is why it made :app:testDebugUnitTest disappear. The storage grant also had to move into the test. The rule uninstalls and reinstalls the app before measuring, an uninstall clears app-ops, and appops set from inside the test does not work even under adoptShellPermissionIdentity - the command fails to stderr, which executeShellCommand does not return, and appops get still reads No operations. What does work is tapping what a user taps: Grant full access, then the platform consent, then back out of the all-files page. grantStorageAccess does that in setupBlock, unmeasured, and it is what turned three failing tests into three passing ones. The search test now waits for the app bar's search action in setup instead of looking it up immediately inside the measured block, where it came back null one frame after a thousand rows rendered. What the device answers: coldStart passes at 560.6 ms P50 (a third measurement of the 532 ms figure), scroll passes but reports only frameCount and no frame durations at all, open dies on INSTALL_FAILED_USER_RESTRICTED because the rule's reinstall wants a human, and search still cannot see the search action. PERFORMANCE.md now states that per benchmark instead of the earlier guess, names the four switches that make an unattended run possible, and says that three of the four numbers need a device that installs unattended and reports frame timing.
11 KiB
Performance
What has been measured, on what, and — explicitly — what has not been measured at all. The rule this file exists to enforce comes from the competitive survey: the market leader's top complaint is thumbnails that load late, its own support reply to a speed complaint was "your SD card is slow", and the lesson recorded there was to measure what was measured and never close a performance complaint by blaming the hardware.
Until this file existed, the app had no performance number of any kind.
Cold start
| build | device | runs (ms) | median |
|---|---|---|---|
debug, 1.0.0-debug |
Xiaomi phone, Android 15 | 1 758 · 1 773 · 1 814 · 1 926 · 2 043 | 1 814 ms |
benchmark (R8, not debuggable, profileable) |
same | — | 532 ms |
The first is ./tools/measure-startup.sh 5 (am start -W, one warm-up start discarded because the first
launch after an install pays for dexopt). The second is BrowserBenchmark#coldStart in the :benchmark
module: StartupTimingMetric, five iterations, CompilationMode.None(), measured as time to initial
display — min 499 ms, median 532 ms, max 805 ms. Measured three times: 615 ms before the storage grant was
in place (the app was showing the permission screen), 532 ms after it, and 560 ms (min 539, max 698) on a
later run that also exercised the full benchmark suite. The three runs agree within a phone's patience;
the 532 ms figure above is the middle one.
Two caveats belong to that number rather than to a footnote. The tool itself warns "Unable to clear
Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to
accelerated class loading", so it is optimistic by an unmeasured amount on this Android version. And
CompilationMode.None() means no compilation profile is applied — what a fresh install does — so a
baseline profile would lower it further and is the obvious next measurement.
The debug build is deliberately the slow case — StrictMode on, LeakCanary attached, no R8 — which is why the same screen takes it three times as long. That is the honest illustration of why a debug number must never be published as the app's performance.
What StrictMode found on the first cold start
The same debug build now logs StrictMode violations, and the first cold start produced five, against a 1 814 ms start:
| duration | where | ours? |
|---|---|---|
| 492 ms | BrowserViewModel.Factory → app.cacheDir |
yes — Context.getCacheDir() checks and creates the private cache directory while the view model is being created, on the main thread |
| 397 ms | BrowserViewModel.rootsOf → StorageProvider.space → VolumeRepository.stat |
yes — free space for every volume is read with StatFs inside refresh(), which the screen calls from a LaunchedEffect |
| 477 ms | FileManagerTheme → typography = FileManagerTypography |
no — the platform loading its own typefaces on first composition |
| 54 ms × 2 | miui.util.font.FontSettings via MainActivity.onCreate |
no — the OEM's font hook doing File.exists while the activity is created |
That is ~1.5 s of main-thread disk I/O inside a 1.8 s start, of which ~0.9 s is ours. Both of ours are mechanical to move and neither is fixed yet: the cache directory is only needed when a preview copies something, and the free-space line on the roots screen can arrive a frame later. They are the first items of the speed work.
Measured on demand: the :benchmark module
benchmark/ holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
build of the app: release rules (R8 on, not debuggable) and profileable, under the app's own package id
and signed with the release key, so installing it updates an installed release build and keeps its
data. A separate package id (.perf) was the first design and is the tidier one, but on this MIUI phone a
new package id cannot be installed without a human: adb install is refused with
INSTALL_FAILED_USER_RESTRICTED, and installing by tapping the APK left the package visible to
pm list packages yet invisible to PackageManager, which is what macrobenchmark reports as "Unable to
find target package". The trade-off is written in :app's build file: this build replaces a release
install, and reinstalling the release APK puts it back.
| benchmark | metric | what it answers |
|---|---|---|
coldStart |
StartupTimingMetric |
tap → first frame |
openAFolderOfAThousandFiles |
FrameTimingMetric |
jank while a 1 000-file folder fills in |
scrollAFolderOfAThousandFiles |
FrameTimingMetric |
jank while scrolling that folder |
searchInsideAFolderOfAThousandFiles |
FrameTimingMetric |
jank while typing into search |
They share a fixture — /sdcard/AABenchmark, a thousand empty files, created by tools/benchmark-fixture.sh
and removed with --remove. CompilationMode.None() is deliberate: no profile is applied, which is the
honest cold-install baseline. CompilationMode.Partial() plus a baseline profile is the follow-up, and
needs the same module.
What the module has actually measured
| benchmark | outcome on the Xiaomi phone (Android 12, MIUI) |
|---|---|
coldStart |
passes — 560.6 ms P50, min 539, max 698 over five iterations. A second measurement of the 532 ms above, and the only number this toolchain reports here |
scrollAFolderOfAThousandFiles |
passes, but reports only frameCount (46–100 frames per iteration): this device hands macrobenchmark no frameDurationCpuMs/frameOverrunMs at all |
openAFolderOfAThousandFiles |
INSTALL_FAILED_USER_RESTRICTED: Install canceled by user — the rule reinstalls the app mid-run, and MIUI wants a tap for that |
searchInsideAFolderOfAThousandFiles |
the app bar's search action is not in the accessibility tree within ten seconds of the folder opening, though it is there when the same taps are made by hand |
So the interaction numbers are not written down here because the device will not produce them, not because the code cannot ask. Two separate device behaviours are in the way, and both are outside the app: a reinstall that needs a human, and a platform that reports frame counts but not frame durations.
The four switches that make an unattended run possible at all
- The app under test is the app itself — same package id, release key — because a new package id
cannot be installed on this phone (
INSTALL_FAILED_USER_RESTRICTED, and an APK installed by tapping ended up visible topm list packagesbut invisible toPackageManager). The cost is recorded in:app's build file: this build replaces a release install; reinstalling the release APK puts it back. - The test module declares a
benchmarkbuild type. AGP 9'scom.android.testextension has notestBuildTypeand notargetVariant— onlytargetProjectPath— and the target variant is matched by name against the test module's own build types. Without that type every run installedapp-debug.apk, so the numbers would have been debug numbers. The type also has toinitWith(getByName("debug")): a declared type starts unsigned, and an unsigned test APK fails withINSTALL_PARSE_FAILED_NO_CERTIFICATES. - The storage grant is walked through the UI, not set with
appops. The rule uninstalls and reinstalls the app, and an uninstall clears app-ops, so a grant applied by the fixture script is gone by the time the app starts.appops setfrom inside the test does not work either — even underadoptShellPermissionIdentity, the command fails to stderr (whichexecuteShellCommanddoes not return) andappops getstill readsNo operations. What does work is tapping what a user taps: the app's "Grant full access" button, then the platform's consent, then leaving the all-files page.grantStorageAccessinBrowserBenchmarkdoes exactly that, insetupBlock, unmeasured. - The fixture is created through adb, because
UiAutomation.executeShellCommandruns with the test app's uid and cannot write/sdcard.
The workflow
./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files
./gradlew :benchmark:connectedBenchmarkAndroidTest
coldStart alone can be run with -Pandroid.testInstrumentationRunnerArguments.class=…BrowserBenchmark#coldStart.
The results land in benchmark/build/outputs/androidTest-results/connected/benchmark/, and — worth
knowing, because it cost an hour — the per-test JSON that holds the numbers is written to
/sdcard/Android/media/<test-pkg>/additional_test_output/ and deleted together with the test app when
the run ends, often before Gradle has copied it. Copying it in a loop while the run is in progress is
what recovered the numbers above.
Not measured yet, and why
| number | how it would be taken | why it is not here |
|---|---|---|
| Folder open, 1 000 / 5 000 / 20 000 files | FrameTimingMetric around the folder tap (openAFolderOfAThousandFiles exists) |
the rule reinstalls the app mid-run, and MIUI answers that install with "Install canceled by user" unless someone is there to tap |
| Thumbnail paint while scrolling a photo grid | FrameTimingMetric |
same; and the scroll variant that did run reported only frameCount, because this device gives macrobenchmark no frame durations |
| Keystroke → search result | FrameTimingMetric around typing into search (searchInsideAFolderOfAThousandFiles exists) |
the app bar's search action does not appear in the accessibility tree within ten seconds of the folder opening in the automated run, although the same taps by hand find it immediately |
The module exists and three of its four tests pass; what is missing is a device that lets macrobenchmark install unattended and reports frame timing. A plain emulator image (or any non-MIUI phone) with the system language set to English would answer all three — the fixture is a folder of empty files, so nothing device-specific is being measured. Until then this file says what was measured and what was not, and the cold-start number stands alone.
What is measured instead, meanwhile
The gate measures things that are not latency but do not rot: the test count, lint, detekt against its
baseline, and whether docs/CODE_MAP.md still matches the tree. Those are the numbers that can be
kept true by automation; the three above need a device and a decision about which device.
Two rules
- No number that was not taken from a real build on a real device. Not from a debug build pretending to be release, and not from reading the code.
- No performance change judged by reading the code. The bigger-thumbnail change (28 → 48 dp in a
row, 40 → 96 dp in a grid) is the standing example: it may be exactly right, and nobody measured
either side of it.
docs/COMPETITIVE_ANALYSIS.mdhad already written "a thumbnail that is bigger and slower is strictly worse than one that is smaller and instant" — which is a hypothesis until it is a number.