A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back. Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it. coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates. Gate green, map current.
8.7 KiB
Performance
What has been measured, on what, and — explicitly — what has not been measured at all. The rule this file exists to enforce comes from the competitive survey: the market leader's top complaint is thumbnails that load late, its own support reply to a speed complaint was "your SD card is slow", and the lesson recorded there was to measure what was measured and never close a performance complaint by blaming the hardware.
Until this file existed, the app had no performance number of any kind.
Cold start
| build | device | runs (ms) | median |
|---|---|---|---|
debug, 1.0.0-debug |
Xiaomi phone, Android 15 | 1 758 · 1 773 · 1 814 · 1 926 · 2 043 | 1 814 ms |
benchmark (R8, not debuggable, profileable) |
same | — | 532 ms |
The first is ./tools/measure-startup.sh 5 (am start -W, one warm-up start discarded because the first
launch after an install pays for dexopt). The second is BrowserBenchmark#coldStart in the :benchmark
module: StartupTimingMetric, five iterations, CompilationMode.None(), measured as time to initial
display — min 499 ms, median 532 ms, max 805 ms. Measured twice: 615 ms before the storage grant was in
place (the app was showing the permission screen) and 532 ms after, which is the figure above.
Two caveats belong to that number rather than to a footnote. The tool itself warns "Unable to clear
Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to
accelerated class loading", so it is optimistic by an unmeasured amount on this Android version. And
CompilationMode.None() means no compilation profile is applied — what a fresh install does — so a
baseline profile would lower it further and is the obvious next measurement.
The debug build is deliberately the slow case — StrictMode on, LeakCanary attached, no R8 — which is why the same screen takes it three times as long. That is the honest illustration of why a debug number must never be published as the app's performance.
What StrictMode found on the first cold start
The same debug build now logs StrictMode violations, and the first cold start produced five, against a 1 814 ms start:
| duration | where | ours? |
|---|---|---|
| 492 ms | BrowserViewModel.Factory → app.cacheDir |
yes — Context.getCacheDir() checks and creates the private cache directory while the view model is being created, on the main thread |
| 397 ms | BrowserViewModel.rootsOf → StorageProvider.space → VolumeRepository.stat |
yes — free space for every volume is read with StatFs inside refresh(), which the screen calls from a LaunchedEffect |
| 477 ms | FileManagerTheme → typography = FileManagerTypography |
no — the platform loading its own typefaces on first composition |
| 54 ms × 2 | miui.util.font.FontSettings via MainActivity.onCreate |
no — the OEM's font hook doing File.exists while the activity is created |
That is ~1.5 s of main-thread disk I/O inside a 1.8 s start, of which ~0.9 s is ours. Both of ours are mechanical to move and neither is fixed yet: the cache directory is only needed when a preview copies something, and the free-space line on the roots screen can arrive a frame later. They are the first items of the speed work.
Measured on demand: the :benchmark module
benchmark/ holds four macrobenchmark tests that measure the interaction numbers above on a benchmark
build of the app: release rules (R8 on, not debuggable) and profileable, under the app's own package id
and signed with the release key, so installing it updates an installed release build and keeps its
data. A separate package id (.perf) was the first design and is the tidier one, but on this MIUI phone a
new package id cannot be installed without a human: adb install is refused with
INSTALL_FAILED_USER_RESTRICTED, and installing by tapping the APK left the package visible to
pm list packages yet invisible to PackageManager, which is what macrobenchmark reports as "Unable to
find target package". The trade-off is written in :app's build file: this build replaces a release
install, and reinstalling the release APK puts it back.
| benchmark | metric | what it answers |
|---|---|---|
coldStart |
StartupTimingMetric |
tap → first frame |
openAFolderOfAThousandFiles |
FrameTimingMetric |
jank while a 1 000-file folder fills in |
scrollAFolderOfAThousandFiles |
FrameTimingMetric |
jank while scrolling that folder |
searchInsideAFolderOfAThousandFiles |
FrameTimingMetric |
jank while typing into search |
They share a fixture — /sdcard/AABenchmark, a thousand empty files, created once per run and deleted
in @AfterClass — and the storage grant is applied with appops, so an unattended run is possible.
CompilationMode.None() is deliberate: no profile is applied, which is the honest cold-install
baseline. CompilationMode.Partial() plus a baseline profile is the follow-up, and needs the same
module.
Two of these have produced a number (coldStart, above); the three interaction ones have not, and
the reason is a device policy rather than the code. Getting this far took five fixes, each written next
to what it constrains: the app under test needs the app's own package id and release key (a new id cannot
be installed here); the test APK needs package visibility for it; the fixture has to be created through
adb, because UiAutomation.executeShellCommand runs with the test app's uid and cannot write /sdcard;
testBuildType has to sit behind -PbenchmarkBuild, or :app:testDebugUnitTest stops existing; and
hasProperty is needed to read that flag, because the Provider form read as absent and the runner then
installed the debug APK and failed on it.
The workflow:
./tools/benchmark-fixture.sh # the fixture, and the storage grant the benchmark build lacks
./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
The Gradle runner is required, not preferred. Macrobenchmark's rule uninstalls the app at the start
of a run and reinstalls it before measuring. On this device that reinstall is what MIUI refuses, so the
run leaves the app uninstalled and then fails with NameNotFoundException for an app that was there a
second earlier — seen twice through am instrument and once through connectedDebugAndroidTest, with
Benchmark: Uninstalling de.softwareapp_hb.filemanager in logcat and the package gone from
pm list packages afterwards.
The fix is a device setting rather than code: Developer options → Install via USB — and on MIUI usually
MIUI optimization off as well — so that an install raised by the benchmark is allowed without a prompt.
Until that is on, the interaction benchmarks cannot run on this phone, and the cold-start number above
stands alone. tools/benchmark-fixture.sh is still needed either way: the benchmark build has no storage
grant of its own, and UiAutomation.executeShellCommand (the only shell a test has) runs with the test
app's uid and cannot write /sdcard.
Not measured yet, and why
| number | how it would be taken | why it is not here |
|---|---|---|
| Folder open, 1 000 / 5 000 / 20 000 files | macrobenchmark StartupTimingMetric over a fixture volume, or TraceSectionMetric around the listing |
needs a :benchmark module and a profileable build installed under the release package id; on this MIUI phone the first install of a new package id waits for a confirmation dialog, so it cannot run unattended |
| Thumbnail paint while scrolling a photo grid | FrameTimingMetric |
same |
| Keystroke → search result | TraceSectionMetric around the query and the index read |
same |
The module does not exist yet, deliberately: half-configured build wiring that nobody can run is worse than an honest gap. It is the next measurement step, and it belongs with the baseline-profile work, which needs the same module.
What is measured instead, meanwhile
The gate measures things that are not latency but do not rot: the test count, lint, detekt against its
baseline, and whether docs/CODE_MAP.md still matches the tree. Those are the numbers that can be
kept true by automation; the three above need a device and a decision about which device.
Two rules
- No number that was not taken from a real build on a real device. Not from a debug build pretending to be release, and not from reading the code.
- No performance change judged by reading the code. The bigger-thumbnail change (28 → 48 dp in a
row, 40 → 96 dp in a grid) is the standing example: it may be exactly right, and nobody measured
either side of it.
docs/COMPETITIVE_ANALYSIS.mdhad already written "a thumbnail that is bigger and slower is strictly worse than one that is smaller and instant" — which is a hypothesis until it is a number.