diff --git a/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt b/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt index 12d7fa6..7034bca 100644 --- a/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt +++ b/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt @@ -1,6 +1,7 @@ package de.softwareapp_hb.filemanager.benchmark import androidx.benchmark.macro.CompilationMode +import androidx.benchmark.macro.ExperimentalMacrobenchmarkApi import androidx.benchmark.macro.FrameTimingMetric import androidx.benchmark.macro.MacrobenchmarkScope import androidx.benchmark.macro.StartupMode @@ -22,10 +23,24 @@ import org.junit.runner.RunWith * * Three decisions are worth knowing before reading the output. * - * **`CompilationMode.None()`**: no compilation profile is applied, so these are the numbers of code - * that has never run. That is the honest baseline for "is this fast on a cold install", and it is - * also the mode that does not need a profile file to exist. `CompilationMode.Partial()` — what a - * baseline profile would buy — is the follow-up, and it needs the profile generated first. + * **`CompilationMode.Ignore()`, not `None()`.** Both apply no compilation profile, but `None()` reports + * `shouldReset = true`, and that is what makes macrobenchmark uninstall the app and reinstall it with + * `pm install -t --skip-verification` before *every* test. On this MIUI phone that install is + * intercepted by `com.miui.securitycenter/.permcenter.install.AdbInstallActivity`, which is + * `FLAG_SECURE`, lives a variable amount of time, and denies the install when nobody answers — and a + * denied install leaves the app uninstalled, so one refusal made every later test fail with + * `NameNotFoundException`. `Ignore()` reports `shouldReset = false`: the library installs nothing and + * measures what is there. So the APK is installed once, from the host: + * + * adb install -r app/build/outputs/apk/benchmark/app-benchmark.apk + * adb shell cmd package compile --reset de.softwareapp_hb.filemanager # the "never run" baseline + * ./gradlew :benchmark:connectedBenchmarkAndroidTest + * + * The `compile --reset` is what preserves the meaning of the old `None()` numbers: no profile, so the + * code being measured has never run. What changes is only *who* installs the APK — and it has to be the + * host, because an install started inside the test cannot be answered while it blocks. + * `CompilationMode.Partial()` — what a baseline profile would buy — is the follow-up, and it needs the + * profile generated first. * * **Frames, not milliseconds.** Three of the four use [FrameTimingMetric], because what a user calls * slow is jank on screen, not a state transition finishing. The alternative is a named trace section @@ -36,6 +51,7 @@ import org.junit.runner.RunWith * "Internal storage", "Search this folder". A localised device fails with a message saying so rather * than silently measuring something else. */ +@OptIn(ExperimentalMacrobenchmarkApi::class) @RunWith(AndroidJUnit4::class) class BrowserBenchmark { @@ -89,7 +105,7 @@ class BrowserBenchmark { fun coldStart() = benchmark.measureRepeated( packageName = app, metrics = listOf(StartupTimingMetric()), - compilationMode = CompilationMode.None(), + compilationMode = CompilationMode.Ignore(), iterations = 5, startupMode = StartupMode.COLD, ) { @@ -101,7 +117,7 @@ class BrowserBenchmark { fun openAFolderOfAThousandFiles() = benchmark.measureRepeated( packageName = app, metrics = listOf(FrameTimingMetric()), - compilationMode = CompilationMode.None(), + compilationMode = CompilationMode.Ignore(), iterations = 5, setupBlock = { openFixture(device) }, ) { @@ -113,7 +129,7 @@ class BrowserBenchmark { fun scrollAFolderOfAThousandFiles() = benchmark.measureRepeated( packageName = app, metrics = listOf(FrameTimingMetric()), - compilationMode = CompilationMode.None(), + compilationMode = CompilationMode.Ignore(), iterations = 5, setupBlock = { openFixture(device) @@ -136,7 +152,7 @@ class BrowserBenchmark { fun searchInsideAFolderOfAThousandFiles() = benchmark.measureRepeated( packageName = app, metrics = listOf(FrameTimingMetric()), - compilationMode = CompilationMode.None(), + compilationMode = CompilationMode.Ignore(), iterations = 5, setupBlock = { openFixture(device) diff --git a/docs/CODE_MAP.md b/docs/CODE_MAP.md index 9f4cd57..9ca5dde 100644 --- a/docs/CODE_MAP.md +++ b/docs/CODE_MAP.md @@ -2,7 +2,7 @@ Generated from the source by `tools/make-code-map.py` - do not edit by hand. Regenerate it after a change (`python3 tools/make-code-map.py`); `--check` exits non-zero when this file is stale. -77 source files, 12707 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification. +77 source files, 12723 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification. Start at `AGENTS.md`. What is done, and why: `docs/ROADMAP.md` and `CHANGELOG.md`. What was verified by hand rather than by a test: `docs/EDGE_CASE_TESTING.md`. @@ -170,7 +170,7 @@ The macrobenchmark test APK: four measurements over a fixture folder, run on a d | file | lines | what it holds | tests | |---|---|---|---| -| `BrowserBenchmark.kt` | 215 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - | +| `BrowserBenchmark.kt` | 231 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - | ## Behaviour index diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index bd492bf..7299b19 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -18,10 +18,11 @@ Until this file existed, the app had **no performance number of any kind**. The first is `./tools/measure-startup.sh 5` (`am start -W`, one warm-up start discarded because the first launch after an install pays for dexopt). The second is `BrowserBenchmark#coldStart` in the `:benchmark` module: `StartupTimingMetric`, five iterations, `CompilationMode.None()`, measured as time to initial -display — min 499 ms, median 532 ms, max 805 ms. Measured four times: 615 ms before the storage grant was -in place (the app was showing the permission screen), 532 ms after it, 560 ms (min 539, max 698) on an -earlier full-suite run, and 574 ms (min 554, max 730) on the run that first completed all four benchmarks. -The four runs agree within a phone's patience; the 532 ms figure above is the middle one. +display — min 499 ms, median 532 ms, max 805 ms. Measured five times, on four different runs: 615 ms +before the storage grant was in place (the app was showing the permission screen), 532 ms after it, 560 ms +(min 539, max 698), 574 ms (min 554, max 730), and 519.5 ms (min 503.6, max 544.7) on the first run that +used `CompilationMode.Ignore()`. They agree within a phone's patience; the 532 ms figure above is the middle +one, and the later runs are the ones whose method is worth copying. Two caveats belong to that number rather than to a footnote. The tool itself warns *"Unable to clear Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to @@ -81,9 +82,9 @@ The run that first completed all four tests (MIUI's install dialog answered by | benchmark | result | what it is | |---|---|---| -| `coldStart` | **574.17 ms** P50 (min 553.95, max 730.19) | time to initial display — the fourth measurement of the same thing, alongside 615 / 532 / 560 ms | -| `openAFolderOfAThousandFiles` | **14 frames** P50 (min 13, max 17) | frames drawn from the folder tap until `file-0.txt` is on screen | -| `scrollAFolderOfAThousandFiles` | **98 frames** P50 (min 93, max 100) | frames drawn across four swipes through the folder | +| `coldStart` | **519.5 ms** median (min 503.6, max 544.7) | time to initial display — the fifth measurement of the same thing, alongside 615 / 532 / 560 / 574 ms | +| `openAFolderOfAThousandFiles` | **14 frames** median (min 11, max 17) | frames drawn from the folder tap until `file-0.txt` is on screen | +| `scrollAFolderOfAThousandFiles` | **94 frames** median (min 48, max 96) | frames drawn across four swipes through the folder | | `searchInsideAFolderOfAThousandFiles` | **failed** | the app bar's search action is not in the accessibility tree within 20 s of the folder opening, although the same taps by hand find it immediately | **Those frame counts are not a jank metric, and the difference matters.** `FrameTimingMetric` normally @@ -99,33 +100,48 @@ could be copied a second time, so the figures above come from parsing it once, i re-running on a device without an install gate is the way to confirm them. And the run still ends with a failing `search` test, so its number is missing rather than zero. -### Why the reinstall cannot be answered by the app +### The install moved out of the test — that is the fix -macrobenchmark installs its target before measuring it, and on this phone MIUI puts that install behind -`com.miui.securitycenter/.permcenter.install.AdbInstallActivity`. The test cannot answer it itself: -`CompilationMode.installPackageFromPaths` builds `pm install -t …` and runs it through -`Shell.executeScriptCaptureStdout`, so the call blocks until the child exits and no test code is running -while the dialog is up. +The problem was never the dialog; it was *who* installs. `CompilationMode.None()` reports +`shouldReset = true`, and that is what makes `resetAndCompile` uninstall the app and reinstall it with +`pm install -t --skip-verification` before every single test. On this phone that install is intercepted by +`com.miui.securitycenter/.permcenter.install.AdbInstallActivity`, which the test cannot answer: the install +call blocks until `pm` exits, so no test code is running while the dialog is up. -`tools/benchmark-watch-install.sh` answers it from the host instead — but it is a workaround with a cost, -and it is not part of the benchmark workflow for that reason: MIUI keeps `mCurrentFocus` pointing at the -installer for a moment after the window has been removed, so a tap sent on that signal can land on the -home screen underneath (observed: it opened Telegram). The script now reads the button off the screen -before every tap and refuses to tap when the installer is not focused, which narrows the race to the -handful of milliseconds between that check and the tap — it does not close it. MIUI's own **Install via -USB** setting is what removes the dialog; with it on, installs need no tap at all. +`CompilationMode.Ignore()` reports `shouldReset = **false**` — the library installs nothing and measures +what is there. So the APK is installed once, from the host, and the run contains no install at all: -Three further measurements about that dialog, kept because they explain the workaround's shape: +```bash +./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files +adb install -r app/build/outputs/apk/benchmark/app-benchmark.apk +adb install -r benchmark/build/outputs/apk/benchmark/benchmark-benchmark.apk +adb shell am instrument -w -e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \ + de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner +``` -- It is `FLAG_SECURE`: a `screencap` taken while `mCurrentFocus` was the installer shows the home screen - behind it, which is why screenshots of "the dialog" show nothing. +Measured: with that command the four tests run with **zero installs** — `tools/benchmark-watch-install.sh` +logs heartbeats and nothing else — and three of them pass. `Ignore()` applies no compilation profile either, +so the numbers still describe code that has never run; a freshly installed APK has no profile to begin with. +Running through Gradle still works, but Gradle's own install steps happen before the tests and are gated +too, which is why `am instrument` is the documented path. + +`tools/benchmark-watch-install.sh` is kept for the one thing it is good at: the two host-side `adb install` +commands above. It reads the Install button off the screen and taps only while the installer is focused. It +is *not* a solution for a run — MIUI keeps `mCurrentFocus` on the installer for a moment after the window is +gone, so a tap sent on that signal can land on the home screen behind it (observed: it opened Telegram). + +Why the dialog could not simply be answered from the test, kept because it explains the shape of all this: + +- The dialog is `FLAG_SECURE`: a `screencap` taken while `mCurrentFocus` was the installer shows the home + screen behind it, which is why screenshots of "the dialog" show nothing. - Its lifetime is not stable. One 0.1 s focus trace caught it as - `13:51:27.70 AdbInstallActivity` → `13:51:28.38 null` — too short to read, let alone tap — while a later - trace showed the same window focused for 8.5 s, long enough for a dump and a tap. Presumably the - difference is whether MIUI is waiting for an answer or merely announcing the install. + `13:51:27.70 AdbInstallActivity` → `13:51:28.38 null` — too short to read, let alone tap — while another + showed the same window focused for 8.5 s. - `pm install` started from an *app* uid does not even reach the gate: MIUI's `XSpaceManagerServiceImpl` throws `SecurityException: You either need MANAGE_USERS or CREATE_USERS permission to: query users`. - The rule's install therefore runs through the shell, which is the path MIUI intercepts. + The rule's install therefore runs through the shell, the path MIUI intercepts. +- `MIUI`'s SecurityCenter cannot be disabled to get out of the way: `pm disable-user` answers + `SecurityException: Cannot disable miui core packages`. ### The four switches that make an unattended run possible at all @@ -140,38 +156,38 @@ Three further measurements about that dialog, kept because they explain the work `app-debug.apk`, so the numbers would have been debug numbers. The type also has to `initWith(getByName("debug"))`: a declared type starts unsigned, and an unsigned test APK fails with `INSTALL_PARSE_FAILED_NO_CERTIFICATES`. -3. **The storage grant is walked through the UI, not set with `appops`.** The rule uninstalls and - reinstalls the app, and an uninstall clears app-ops, so a grant applied by the fixture script is gone - by the time the app starts. `appops set` from inside the test does not work either — even under - `adoptShellPermissionIdentity`, the command fails to stderr (which `executeShellCommand` does not - return) and `appops get` still reads `No operations`. What does work is tapping what a user taps: - the app's "Grant full access" button, then the platform's consent, then leaving the all-files page. - `grantStorageAccess` in `BrowserBenchmark` does exactly that, in `setupBlock`, unmeasured. +3. **The storage grant survives because nothing uninstalls any more.** An uninstall clears app-ops, so + while the rule was reinstalling the app, a grant applied by the fixture script was gone by the time the + app started — and `appops set` from inside the test does not work either (even under + `adoptShellPermissionIdentity` it fails to stderr, which `executeShellCommand` does not return, and + `appops get` still reads `No operations`). With `Ignore()` the grant lands once and stays. + `grantStorageAccess` in `BrowserBenchmark` still walks the app's own "Grant full access" flow when the + app comes up without access, which costs nothing when it is not needed. 4. **The fixture is created through adb**, because `UiAutomation.executeShellCommand` runs with the *test app's* uid and cannot write `/sdcard`. ### The workflow ```bash -./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files -./tools/benchmark-watch-install.sh --seconds 900 & # answers MIUI's install dialog if it stays up -./gradlew :benchmark:connectedBenchmarkAndroidTest +./tools/benchmark-fixture.sh # /sdcard/AABenchmark, 1000 empty files +adb install -r app/build/outputs/apk/benchmark/app-benchmark.apk # the build under measure +adb install -r benchmark/build/outputs/apk/benchmark/benchmark-benchmark.apk +adb shell am instrument -w -e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \ + de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner ``` -`coldStart` alone can be run with `-Pandroid.testInstrumentationRunnerArguments.class=…BrowserBenchmark#coldStart`. -The results land in `benchmark/build/outputs/androidTest-results/connected/benchmark/`, and — worth -knowing, because it cost an hour — the per-test JSON that holds the numbers is written to -`/sdcard/Android/media//additional_test_output/` and **deleted together with the test app when -the run ends**, often before Gradle has copied it. Copying it in a loop while the run is in progress is -what recovered the numbers above. +One test can be run by naming it: `-e class …BrowserBenchmark#coldStart`. The numbers are printed in the +instrumentation output itself, and the per-test JSON and Perfetto traces are left in +`/sdcard/Android/media/de.softwareapp_hb.filemanager.benchmark/` — they used to be deleted with the test app +when a run ended, which is why the earlier numbers had to be copied mid-run. ## Not measured yet, and why | number | status | |---|---| -| Folder open with 1 000 files | **14 frames** P50 — measured, but as a frame *count*, not a jank figure | -| Folder open with 5 000 / 20 000 files | not measured: the fixture script counts to 1 000, and every larger run needs another install cycle | -| Scrolling that folder | **98 frames** P50 over four swipes — again a frame count | +| Folder open with 1 000 files | **14 frames** median — measured, but as a frame *count*, not a jank figure | +| Folder open with 5 000 / 20 000 files | not measured: the fixture script counts to 1 000 | +| Scrolling that folder | **94 frames** median over four swipes — again a frame count | | Typing into search | **failed** in the automated run: the app bar's search action is not in the accessibility tree within 20 s of the folder opening, although the same taps by hand find it immediately | | Thumbnail paint while scrolling a photo grid | not measured: no test for it yet, and no frame durations on this device |