0a3ff882472de73123f1ac384c0929ed6b9beca2
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d243e1ba7b |
Let the benchmark fixture be any size, so the scaling question can be asked
count is now a parameter: ./tools/benchmark-fixture.sh --count 20000 builds the folder the performance doc has been promising 5000 and 20000-file numbers for. The script validates the argument rather than quietly building the default folder, which is the failure mode that would otherwise waste a run. Verified with 1500 files and then removed. |
||
|
|
14615f9842 |
Take the two main-thread disk reads off the startup path, and report what that bought
Both costs StrictMode found on a cold start are now off the main thread. scratchRoot is a Lazy<File>, so Context.getCacheDir - which checks and creates the private cache directory, measured at 492 ms - runs when a preview first copies something instead of while the view model is being created. The free-space probes moved into fillRootFreeSpace and fillFreeSpace, which run on Dispatchers.IO after the rows are already on screen and update the state when the filesystem answers; nothing waits for them, and fillFreeSpace checks the path is still the one on screen so a fast double-tap cannot leave a folder showing another one's number. Measured after: StrictMode no longer reports either violation, and the only ones left are MIUI's own font hook - FontSettings.loadFontSetting, TypefaceUtils.loadFontSettings, InterceptorProxy.create via MainActivity.onCreate, in 50-78 ms pairs. The debug cold start went from a 1814 ms median to 1803 ms and the benchmark coldStart from 519.5/531.6 ms to 485.6/568.9/600.0 ms. That is no improvement, and PERFORMANCE.md says so in those words: both violations were first-call costs (cache directory creation once per install, a cold StatFs page cache), the timed runs are warm, and the device's run-to-run spread of 115 ms is wider than anything the change could have bought. What it does change is the launch right after an install, which is the benchmark's first iteration: 648 ms max fresh against 550 ms warm. |
||
|
|
cbb36abfce |
Make the search benchmark start from a known screen, so all four measure
The failure was not the app bar at all. Dumping the hierarchy at the point of failure showed a TextView whose text was 'Search this folder' and a desc='Clear search' beside it: the app was already in search mode, because the previous iteration left it there and CompilationMode.Ignore() reinstalls nothing between iterations. So the setup found a search field where it expected an action. openFixture now calls killProcess() before each iteration, and the test writes search-action-missing.xml into its external files directory when the action is missing, which is what turned this from a guess into a diagnosis. With that, one run gives OK (4 tests): coldStart 531.6 ms median (475.6-552.0), opening a 1000-file folder 12 frames (12-17), scrolling it with four swipes 97 frames (92-100), and typing into search 39 frames (36-40). The run before it, same method, gave 519.5 ms / 14 / 94 / 38, so the numbers repeat. PERFORMANCE.md now carries both runs side by side and only lists what is genuinely still missing: larger folders, the photo grid, and any jank figure at all, since this phone reports frame counts and no frame durations. |
||
|
|
7514676384 |
Stop macrobenchmark from installing at all, and measure the app that is there
The MIUI dialog was never the problem to solve; the install was. CompilationMode.None() reports shouldReset = true, which is what makes resetAndCompile uninstall the app and run pm install -t --skip-verification before every test - the one command MIUI intercepts with a FLAG_SECURE dialog that nothing inside a blocking install call can answer. CompilationMode.Ignore() reports shouldReset = false: the library installs nothing and measures what is installed. So the APK is installed once from the host and the run contains no install at all. Measured on the phone: with Ignore() the four tests ran with zero installs - the watcher script logged heartbeats and no dialogs - and three passed. The numbers are the best yet and taken on a run with no install churn: coldStart 519.5 ms median (503.6-544.7), opening a 1000-file folder 14 frames median (11-17), scrolling it with four swipes 94 frames median (48-96). The search benchmark still fails on the app bar's search action not being in the accessibility tree. Also recorded: the storage grant now survives because nothing uninstalls, MIUI refuses to let its SecurityCenter be disabled (Cannot disable miui core packages), and the fix for the whole afternoon is five lines in the benchmark's KDoc plus a documented am instrument command, because Gradle's own install steps are gated too. |
||
|
|
b72847b822 |
Record the first complete benchmark run, and what its numbers are not
With the install watcher answering MIUI's dialog, all four tests ran and three produced numbers: coldStart 574.17 ms P50 (553.95-730.19), opening a 1000-file folder 14 frames P50 (13-17), scrolling it with four swipes 98 frames P50 (93-100). The search benchmark still fails, on the app bar's search action not being in the accessibility tree after twenty seconds - the setup wait is raised from ten to twenty now that the folder render is known to be the slow step. The frame counts are labelled as frame counts and not as jank, because this phone reports no frame durations at all: the same results file contains frameCount and nothing else, so 14 frames says the interaction is short and says nothing about whether a frame was slow. The doc also keeps the measurements that explain the watcher's shape - the dialog is FLAG_SECURE, its lifetime varies between under a second and 8.5 s, and pm install from an app uid fails earlier with MIUI's XSpace SecurityException - along with the honest cost: MIUI keeps mCurrentFocus on the installer after the window is gone, so a coordinate tap can land on the home screen, which it did. The watcher no longer reuses a saved position at all and refuses to tap within two seconds of the previous tap. |
||
|
|
904b74773d |
Answer install dialogs without ever tapping a closed one
Three things went wrong in this script and are fixed here, each with the run that showed it. It remembered a button position and tapped it without re-checking the dialog was still there, so a tap landed on the Telegram icon on the home screen behind a dialog that had already closed; it refused to answer a second dialog because it compared window tokens, and every dialog gets a new one, so the third install in a run was skipped; and its idle poll was so slow that a pair of installs could not both be caught. Now: the position is only used while an installer window is focused, re-checked immediately before the tap; a newer dialog with the same installer activity is answered rather than skipped; and after every dialog the script watches at 0.4 s for the follow-up install. Verified with two consecutive uninstall/install cycles, the second answered from the measured position five seconds after the first, and by installing the release APK with only the watcher running. The benchmark's own reinstall is still refused - two dialogs answered during one run, both with the installer focused, and pm install still returned Install canceled by user, because MIUI closes the dialog when the session is over. |
||
|
|
b91198fe6b |
Record why the reinstall cannot be answered, with the measurements that say so
The watcher now answers an install that stays up - it read and tapped Installieren during one run - and the run still ended with INSTALL_FAILED_USER_RESTRICTED, so the doc says what was measured instead of what was hoped. The dialog the rule's install raises is focused for under a second (13:51:27.70 AdbInstallActivity to 13:51:28.38 null) and is FLAG_SECURE, so a screencap of it shows the home screen behind it; one uiautomator dump takes 3-4 s, which is longer than the window exists. pm install from an app uid does not reach the gate at all - MIUI's XSpace service throws a SecurityException asking for MANAGE_USERS - so the rule installs through the shell, MIUI gates new packages and lets updates through, which is why adb install -r of an installed build is instant and the reinstall never is. The governing switch is MIUI's Install via USB, which is not in any Android settings namespace. |
||
|
|
ec76acfdd6 |
Make the benchmark actually measure the benchmark build, and record what the phone will and will not answer
The run was installing app-debug.apk the whole time, which would have published debug numbers as the release-like ones. AGP 9's com.android.test extension has no testBuildType and no targetVariant - it exposes only targetProjectPath - and the target variant is matched by name against the test module's own build types. So the module now declares a benchmark build type, and inherits the debug signing config with initWith: a declared type starts unsigned, and an unsigned test APK fails with INSTALL_PARSE_FAILED_NO_CERTIFICATES. The command is now :benchmark:connectedBenchmarkAndroidTest, and -PbenchmarkBuild is gone from the workflow; the conditional testBuildType it used to set in :app only ever moved the unit-test variant, which is why it made :app:testDebugUnitTest disappear. The storage grant also had to move into the test. The rule uninstalls and reinstalls the app before measuring, an uninstall clears app-ops, and appops set from inside the test does not work even under adoptShellPermissionIdentity - the command fails to stderr, which executeShellCommand does not return, and appops get still reads No operations. What does work is tapping what a user taps: Grant full access, then the platform consent, then back out of the all-files page. grantStorageAccess does that in setupBlock, unmeasured, and it is what turned three failing tests into three passing ones. The search test now waits for the app bar's search action in setup instead of looking it up immediately inside the measured block, where it came back null one frame after a thousand rows rendered. What the device answers: coldStart passes at 560.6 ms P50 (a third measurement of the 532 ms figure), scroll passes but reports only frameCount and no frame durations at all, open dies on INSTALL_FAILED_USER_RESTRICTED because the rule's reinstall wants a human, and search still cannot see the search action. PERFORMANCE.md now states that per benchmark instead of the earlier guess, names the four switches that make an unattended run possible, and says that three of the four numbers need a device that installs unattended and reports frame timing. |
||
|
|
5a6ae3f71d |
Record what actually blocks the interaction benchmarks: the reinstall clears the grant
The diagnosis is now precise and it is not the installer dialog. Macrobenchmark's rule uninstalls the app and reinstalls it before measuring; an uninstall clears app-ops, so the MANAGE_EXTERNAL_STORAGE grant that tools/benchmark-fixture.sh applies is gone by the time the test navigates. appops get reads No operations after a run, the app correctly shows its choose-a-folder fallback, and the volume row the test looks for is not on screen - which is exactly the assertion that fired. No test can work around it: granting an app-op needs the shell uid, and UiAutomation.executeShellCommand runs as the test app. So the three interaction benchmarks cannot run as macrobenchmark tests on any device where this app needs the grant. PERFORMANCE.md now says that, names the two paths that would give the numbers (a plain instrumented test that avoids the rule, or a benchmark flow that needs no grant), and says what each costs. The cold-start number stands. |
||
|
|
cb2db9ab63 |
Make the benchmark build the app itself, and record what the runner does to it
A separate package id for the app under test is the tidier design and cannot be used on this device: a new package id needs a human to approve the install every time, and installing by tapping the APK left the package listed by pm list packages yet unknown to pm enable and invisible to PackageManager. The benchmark build type therefore keeps the app's own id and takes the release key, so it updates an installed release build and keeps its data. The trade-off is stated where it happens: this build replaces a release install, and reinstalling the release APK puts it back. Two bugs surfaced on the way and both are now commented at the line they belong to. -PbenchmarkBuild was read with providers.gradleProperty, which reported it absent, so the runner installed the debug APK and failed on it; hasProperty is what works. And the reason am instrument cannot be used for this module is now recorded: MacrobenchmarkRule uninstalls the target at the start of a run and reinstalls it from the APK paths the Gradle runner passes in, so without those the app is uninstalled and left that way - which is what produced NameNotFoundException for an app that was installed a second earlier, twice, with Benchmark: Uninstalling in logcat to prove it. coldStart keeps its measured number (532 ms median). The three interaction benchmarks remain unmeasured for a reason that is now written down: each run needs one tap on the phone for the reinstall MIUI gates. Gate green, map current. |
||
|
|
94f85bd287 |
Get a real cold-start number out of the benchmark, and record what the device demanded
coldStart measures 532 ms median (min 499, max 805) on the benchmark build - R8, not debuggable, profileable, CompilationMode.None - against 1814 ms on the debug build, which is the honest illustration of why a debug number must never be published as the app's performance. The tool also warns that it cannot clear the runtime image on this Android version, so the figure is optimistic by an unmeasured amount; that is in PERFORMANCE.md next to the number rather than in a footnote. The three interaction benchmarks are written and compile but have not produced a number, and getting that far took four fixes worth naming. The app under test cannot keep the app's package id (signature) and cannot use .benchmark (the test APK owns that id, INSTALL_FAILED_VERSION_DOWNGRADE); macrobenchmark needs package visibility for it or it reports NameNotFoundException for an app that pm list packages shows installed; UiAutomation.executeShellCommand runs as the test app's uid, so the fixture has to be made through adb instead - that one cost the first attempt a hang; and testBuildType has to be behind a flag or :app:testDebugUnitTest stops existing. The workflow that works on a device with an install gate is now documented: install the two APKs by hand once, run tools/benchmark-fixture.sh, then am instrument. The Gradle task is documented as the alternative and as the reason the app under test disappeared once - it uninstalls around the run. Gate green, map current, 386 tests unchanged. |
||
|
|
1f94d0a38a |
Add a macrobenchmark module, and find out what the build was hiding
docs/PERFORMANCE.md has been honest that three of its four numbers do not exist. They exist now as measurements, if not yet as results: benchmark/ is a macrobenchmark module with four tests - cold start with StartupTimingMetric, then opening, scrolling and searching a folder of a thousand files with FrameTimingMetric - sharing a fixture at /sdcard/AABenchmark that is created once per run and deleted in @AfterClass. Frames rather than a named trace section on purpose: what a user calls slow is jank on screen, and the alternative puts benchmarking calls into production code for the sake of a number. CompilationMode.None() is the honest cold-install baseline; Partial() plus a baseline profile is the follow-up and needs the same module. The storage grant is applied with appops in setup, which is what lets the run be unattended - a permission dialog is the one thing a test cannot answer. Three build facts had to be discovered rather than designed, and each is now written where it will be read. The app under test cannot keep its own package id: it needs a debug signature to install on a test device, so it cannot replace a release install, and an un-suffixed build is exactly that. It also cannot use the obvious suffix: `.benchmark` is already the benchmark module's own APK id, and the test APK then fails to install with INSTALL_FAILED_VERSION_DOWNGRADE - which is what the first attempt did. Hence `.perf`. And `testBuildType = "benchmark"` cannot be set unconditionally, because it makes benchmark the variant that *unit* tests run against and :app:testDebugUnitTest - the task the entire gate is built on - stops existing; it is behind -PbenchmarkBuild instead. All three were measured, not guessed, and the third would have broken CI silently. Two smaller things came with it. tools/verify.sh --release now also compiles the app's benchmark variant and the test APK, because build wiring nobody runs is wiring that rots; no device is involved. And the code map grew an EXTRA_ROOTS section so it indexes benchmark/ too - a map that silently omits a module lies by omission, which is worse than the one line it costs. The numbers themselves are still missing, and for a device reason rather than a code one: installing the new package id on this MIUI phone fails with INSTALL_FAILED_USER_RESTRICTED (the "Install via USB" gate, which wants a human and answered too late). Both APKs are pushed to /sdcard/Download/ for a manual install, and the run is one command away: `adb shell am instrument -w -e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner`. Gate green with --release, including the new compile step; map current; 386 tests unchanged. |
||
|
|
0bbee25ab3 |
Test the interface as a user sees it, and let a debug build tell on itself
Fifteen rendering tests across four files are the first tests in this project that compose a screen, and they cover the claims that used to be prose only: the app bar's star exists where there is a folder to star and says Add or Remove according to the state, the selection bar offers Rename to several files and Details only to one, the batch dialog spells out the names it will produce and refuses a collision with the file named, the shared name dialog refuses a slash while it is typed, and a favourite that has gone says "Folder not found" rather than showing a path that is not there. They run on the JVM through Robolectric - the Compose test artifacts plus the test manifest - so they belong to the same verify.sh gate as everything else instead of needing a device: 386 tests in total. Writing them turned up one behaviour worth recording. The batch dialog continues the folder's series by default, so a collision cannot be shown unless the start number is forced back to 1 - which is precisely why the feature works in practice, and now also why two of these tests set the field explicitly. The debug build now enables StrictMode (disk reads and writes, network, leaked closables; logged, not fatal) and attaches LeakCanary. A file manager is the worst possible app for a blocking disk read on the main thread, and a debug build is the last place where noticing it is still cheap. Logging rather than a crash is deliberate: turning a slow screen into a crash on someone's phone is the worse trade. The flag is read from the manifest, not from BuildConfig, which this build does not generate. Performance has a number for the first time, and it immediately paid for itself. tools/measure-startup.sh takes cold start with `am start -W` - median of several runs, one warm-up discarded because the first launch after an install pays for dexopt - and docs/PERFORMANCE.md records 1814 ms over five runs, in the debug build, which is the slow case by construction. StrictMode's first run then produced five violations against that number: 492 ms resolving `app.cacheDir` inside the view-model factory, 397 ms reading free space with StatFs for every volume inside refresh(), 477 ms of the platform loading its own typefaces, and two 54 ms OEM font hooks. That is about 1.5 s of main-thread disk I/O in a 1.8 s start, of which ~0.9 s is ours and mechanical to move. Neither of ours is fixed here on purpose: the numbers and the two fixes are recorded in PERFORMANCE.md, and they are the first items of the speed work. What is not measured is stated in the same file rather than left to be assumed: folder open at 1 000 / 5 000 / 20 000 files, thumbnail paint while scrolling, and keystroke-to-result all need a `:benchmark` module and a profileable build under the release package id, and this device waits for a confirmation dialog on the first install of a new package id. The module is deliberately absent instead of half-configured: build wiring nobody can run is worse than a recorded gap. Gate green (map current, detekt, 386 tests, lint), and CHANGELOG, README, AGENTS.md, ROADMAP, PERFORMANCE.md, EDGE_CASE_TESTING and the regenerated map carry it. |