From cbb36abfce28885e0ffe31fc789bae611e53a1f3 Mon Sep 17 00:00:00 2001 From: Hadrian Burkhardt Date: Sat, 17 Oct 2026 15:58:03 +0200 Subject: [PATCH] Make the search benchmark start from a known screen, so all four measure The failure was not the app bar at all. Dumping the hierarchy at the point of failure showed a TextView whose text was 'Search this folder' and a desc='Clear search' beside it: the app was already in search mode, because the previous iteration left it there and CompilationMode.Ignore() reinstalls nothing between iterations. So the setup found a search field where it expected an action. openFixture now calls killProcess() before each iteration, and the test writes search-action-missing.xml into its external files directory when the action is missing, which is what turned this from a guess into a diagnosis. With that, one run gives OK (4 tests): coldStart 531.6 ms median (475.6-552.0), opening a 1000-file folder 12 frames (12-17), scrolling it with four swipes 97 frames (92-100), and typing into search 39 frames (36-40). The run before it, same method, gave 519.5 ms / 14 / 94 / 38, so the numbers repeat. PERFORMANCE.md now carries both runs side by side and only lists what is genuinely still missing: larger folders, the photo grid, and any jank figure at all, since this phone reports frame counts and no frame durations. --- .../filemanager/benchmark/BrowserBenchmark.kt | 20 ++++++- docs/CODE_MAP.md | 4 +- docs/PERFORMANCE.md | 56 ++++++++++--------- 3 files changed, 52 insertions(+), 28 deletions(-) diff --git a/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt b/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt index 7034bca..a6f8f03 100644 --- a/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt +++ b/benchmark/src/main/java/de/softwareapp_hb/filemanager/benchmark/BrowserBenchmark.kt @@ -8,9 +8,11 @@ import androidx.benchmark.macro.StartupMode import androidx.benchmark.macro.StartupTimingMetric import androidx.benchmark.macro.junit4.MacrobenchmarkRule import androidx.test.ext.junit.runners.AndroidJUnit4 +import androidx.test.platform.app.InstrumentationRegistry import androidx.test.uiautomator.By import androidx.test.uiautomator.UiDevice import androidx.test.uiautomator.Until +import java.io.File import org.junit.Assert.assertNotNull import org.junit.Rule import org.junit.Test @@ -161,9 +163,21 @@ class BrowserBenchmark { // The measured block clicks this, so it has to exist before the clock starts: right after a // thousand rows render, the app bar is a frame behind the list. Ten seconds was not enough on // this phone, so this one step gets twice that — it is setup, not measurement. + val searchAction = device.wait(Until.findObject(By.desc(SEARCH_ACTION)), SLOW_TIMEOUT_MS) + if (searchAction == null) { + // Leave the screen behind for whoever reads the failure: guessing what "the app bar is + // missing" looked like cost an afternoon. `adb pull` this file from the test app's + // external files directory. + val dump = File( + InstrumentationRegistry.getInstrumentation().context.getExternalFilesDir(null), + "search-action-missing.xml", + ) + device.dumpWindowHierarchy(dump) + System.err.println("hierarchy dumped to ${dump.absolutePath}") + } assertNotNull( "the app bar's search action is missing; see browser_search_hint in strings.xml", - device.wait(Until.findObject(By.desc(SEARCH_ACTION)), SLOW_TIMEOUT_MS), + searchAction, ) }, ) { @@ -182,6 +196,10 @@ class BrowserBenchmark { * instruction that is missing if something is not where it should be. */ private fun MacrobenchmarkScope.openFixture(device: UiDevice) { + // Kill the app first. Each iteration must start from the roots screen, and the activity keeps its + // state otherwise: the search iteration left search mode open, so the next one found a search field + // where it expected an app bar — measured, by dumping the hierarchy at the point of failure. + killProcess() pressHome() startActivityAndWait() grantStorageAccess(device) diff --git a/docs/CODE_MAP.md b/docs/CODE_MAP.md index 9ca5dde..2d0731a 100644 --- a/docs/CODE_MAP.md +++ b/docs/CODE_MAP.md @@ -2,7 +2,7 @@ Generated from the source by `tools/make-code-map.py` - do not edit by hand. Regenerate it after a change (`python3 tools/make-code-map.py`); `--check` exits non-zero when this file is stale. -77 source files, 12723 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification. +77 source files, 12741 lines - 33 test files, 386 tests. The behaviour index at the end lists every test name, and in this project a test name is a sentence about what the app does, so it doubles as the specification. Start at `AGENTS.md`. What is done, and why: `docs/ROADMAP.md` and `CHANGELOG.md`. What was verified by hand rather than by a test: `docs/EDGE_CASE_TESTING.md`. @@ -170,7 +170,7 @@ The macrobenchmark test APK: four measurements over a fixture folder, run on a d | file | lines | what it holds | tests | |---|---|---|---| -| `BrowserBenchmark.kt` | 231 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - | +| `BrowserBenchmark.kt` | 249 | The four numbers docs/PERFORMANCE.md promises, measured on a benchmark build of the app. BrowserBenchmark, Fixture | - | ## Behaviour index diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 7299b19..8601421 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -18,11 +18,11 @@ Until this file existed, the app had **no performance number of any kind**. The first is `./tools/measure-startup.sh 5` (`am start -W`, one warm-up start discarded because the first launch after an install pays for dexopt). The second is `BrowserBenchmark#coldStart` in the `:benchmark` module: `StartupTimingMetric`, five iterations, `CompilationMode.None()`, measured as time to initial -display — min 499 ms, median 532 ms, max 805 ms. Measured five times, on four different runs: 615 ms +display — min 499 ms, median 532 ms, max 805 ms. Measured six times, on five different runs: 615 ms before the storage grant was in place (the app was showing the permission screen), 532 ms after it, 560 ms -(min 539, max 698), 574 ms (min 554, max 730), and 519.5 ms (min 503.6, max 544.7) on the first run that -used `CompilationMode.Ignore()`. They agree within a phone's patience; the 532 ms figure above is the middle -one, and the later runs are the ones whose method is worth copying. +(min 539, max 698), 574 ms (min 554, max 730), 519.5 ms (min 503.6, max 544.7) and 531.6 ms (min 475.6, +max 552.0) — the last two on the runs that used `CompilationMode.Ignore()` and measured all four +benchmarks. They agree within a phone's patience; the 532 ms figure above is the middle one. Two caveats belong to that number rather than to a footnote. The tool itself warns *"Unable to clear Runtime Image, subsequent launches/iterations may exhibit faster startup than production due to @@ -77,15 +77,18 @@ needs the same module. ### What the module has measured -The run that first completed all four tests (MIUI's install dialog answered by -`tools/benchmark-watch-install.sh`, 5 iterations each, `CompilationMode.None()`): +The two runs that completed all four tests — same APK, `CompilationMode.Ignore()`, 5 iterations each, no +install during either run: -| benchmark | result | what it is | -|---|---|---| -| `coldStart` | **519.5 ms** median (min 503.6, max 544.7) | time to initial display — the fifth measurement of the same thing, alongside 615 / 532 / 560 / 574 ms | -| `openAFolderOfAThousandFiles` | **14 frames** median (min 11, max 17) | frames drawn from the folder tap until `file-0.txt` is on screen | -| `scrollAFolderOfAThousandFiles` | **94 frames** median (min 48, max 96) | frames drawn across four swipes through the folder | -| `searchInsideAFolderOfAThousandFiles` | **failed** | the app bar's search action is not in the accessibility tree within 20 s of the folder opening, although the same taps by hand find it immediately | +| benchmark | run 1 | run 2 | what it is | +|---|---|---|---| +| `coldStart` | **519.5 ms** (503.6–544.7) | **531.6 ms** (475.6–552.0) | time to initial display | +| `openAFolderOfAThousandFiles` | **14 frames** (11–17) | **12 frames** (12–17) | frames from the folder tap until `file-0.txt` is on screen | +| `scrollAFolderOfAThousandFiles` | **94 frames** (48–96) | **97 frames** (92–100) | frames across four swipes through the folder | +| `searchInsideAFolderOfAThousandFiles` | **38 frames** (33–44) | **39 frames** (36–40) | frames from opening search until `file-99.txt` is the result | + +The two runs agree closely, which is the point of running them twice: the method is stable enough that a +change in the app would show up as a change in these numbers rather than as noise. **Those frame counts are not a jank metric, and the difference matters.** `FrameTimingMetric` normally publishes `frameDurationCpuMs` and `frameOverrunMs` — how long each frame took and how far past its @@ -95,10 +98,15 @@ lasted. It says the interaction is short (14 frames is about a quarter of a seco say whether any of those frames was slow. A device that reports frame durations — an emulator or any non-MIUI phone — is what turns this into a jank number. -Two notes on provenance. The results JSON from that run was deleted by the next run's cleanup before it -could be copied a second time, so the figures above come from parsing it once, immediately after the run; -re-running on a device without an install gate is the way to confirm them. And the run still ends with a -failing `search` test, so its number is missing rather than zero. +One bug had to be fixed before `search` could be measured, and it is worth knowing about because it looked +like a UI problem and was a state problem: the search iteration ends with search mode open, and with +`Ignore()` nothing reinstalls the app between iterations, so the *next* iteration found a search field +where it expected an app bar — the assertion fired every time. The hierarchy dump taken at the point of +failure showed a `TextView` whose *text* was "Search this folder" and a `desc='Clear search'` beside it: +the app was already searching. `openFixture` now calls `killProcess()` before every iteration, so each one +starts on the roots screen. The dump itself stays: on a missing action it writes +`search-action-missing.xml` into the test app's external files directory, which is what made this +diagnosable instead of a guess. ### The install moved out of the test — that is the fix @@ -181,20 +189,18 @@ instrumentation output itself, and the per-test JSON and Perfetto traces are lef `/sdcard/Android/media/de.softwareapp_hb.filemanager.benchmark/` — they used to be deleted with the test app when a run ended, which is why the earlier numbers had to be copied mid-run. -## Not measured yet, and why +## Still not measured | number | status | |---|---| -| Folder open with 1 000 files | **14 frames** median — measured, but as a frame *count*, not a jank figure | | Folder open with 5 000 / 20 000 files | not measured: the fixture script counts to 1 000 | -| Scrolling that folder | **94 frames** median over four swipes — again a frame count | -| Typing into search | **failed** in the automated run: the app bar's search action is not in the accessibility tree within 20 s of the folder opening, although the same taps by hand find it immediately | -| Thumbnail paint while scrolling a photo grid | not measured: no test for it yet, and no frame durations on this device | +| Thumbnail paint while scrolling a photo grid | not measured: no test for it yet | +| Any jank figure at all | not measurable here: this phone's `FrameTimingMetric` output is `frameCount` only, with no `frameDurationCpuMs` or `frameOverrunMs` | -What the interaction numbers need is a device that reports frame durations; what the *repeated* runs need -is a device that installs unattended. This phone gives neither, and the two are independent: a -non-MIUI phone or an emulator with the system language set to English would answer both, and the fixture -is a folder of empty files, so nothing device-specific is being measured. +What the interaction numbers still need is a device that reports frame durations — an emulator or any +non-MIUI phone would do, and the fixture is a folder of empty files, so nothing device-specific is being +measured. The install side is solved: `CompilationMode.Ignore()` plus installing from the host means a run +touches the package manager zero times. ## What is measured instead, meanwhile