fishfacegit/file-manager
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

The benchmark runs in CI, and fails when the jank metric is missing

Browse Source 
Item 5 of the queue: .github/workflows/benchmark.yml measures on an API 35 emulator (reactivecircus/android-emulator-runner, google_apis, x86_64, SwiftShader), on demand and every Monday. Not on pull requests: a run is minutes of measurement behind twenty of booting an emulator, and the gate is meant to answer in one or two.

It installs both APKs and instruments directly instead of using connectedBenchmarkAndroidTest, because that task installs the app itself - which undoes the compile reset that makes the measurement a never-run baseline. tools/benchmark-report.py turns the JSON into two tables, keeping frameCount (a length) apart from frameDurationCpuMs and frameOverrunMs (speeds), and --require-frame-duration fails the job when the second table would be empty: the metric is the reason the job exists.

Driving the workflow's own script against the phone first paid for itself three times. It found that cmd package compile --reset needs adbd as root - a user build refuses it with SecurityException, so the workflow now does adb root first (google_apis allows it) instead of silently measuring a compiled app. It produced the second run of the 20 000-file open (16 frames median, 32 max) and the first numbers for scroll, search and cold start at that size: search costs 319 frames, which is now the biggest number in the file. And it showed the phone reporting frameDurationCpuMs and frameOverrunMs after all, so PERFORMANCE.md's 'this phone reports counts only' was a property of those earlier runs, not of the handset - corrected, with the numbers.
This commit is contained in:
Hadrian Burkhardt
2026-10-17 23:00:22 +02:00
parent 1ec71381a0
commit 66183a8596
5 changed files with 357 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PERFORMANCE.md
+68 -15
View File
@@ -117,11 +117,19 @@ change in the app would show up as a change in these numbers rather than as nois
**Those frame counts are not a jank metric, and the difference matters.** `FrameTimingMetric` normally
publishes `frameDurationCpuMs` and `frameOverrunMs` — how long each frame took and how far past its
deadline it ran. This phone reports neither: the same run's JSON contains `frameCount` and nothing else,
so "14 frames to open the folder" is how many frames the interaction took, not how long any of them
lasted. It says the interaction is short (14 frames is about a quarter of a second at 60 Hz); it cannot
say whether any of those frames was slow. A device that reports frame durations — an emulator or any
non-MIUI phone — is what turns this into a jank number.
deadline it ran. `frameCount`, which is what the table above is made of, is how many frames an interaction
took: a length, not a speed. It says a folder opens in about a fifth of a second at 60 Hz; it cannot say
whether any of those frames was slow.
The runs those numbers come from reported `frameCount` only — their JSON had no sampled metrics at all,
which is what started the search for a device that supplies them. **A later run on the same phone did
supply them**, so the absence was a property of those runs, not of the handset. For the 1 000-file open,
measured after the listing was made progressive, `frameDurationCpuMs` is **P50 13.9 ms, P90 26.5 ms,
P99 86.5 ms** and `frameOverrunMs` **P50 19.0 ms, P90 32.2 ms, P99 80.6 ms**, next to `frameCount` median
12 (max 48) from the same run. Against a 60 Hz budget of 16.7 ms that is "the median frame fits, one in ten
takes 26 ms" — a statement the counts could never make. Why the earlier runs captured no frame timelines
is not known; what is known is that a run either does or does not, and that `.github/workflows/benchmark.yml`
now runs it on an emulator where it always does.
### How it scales with folder size
@@ -204,6 +212,31 @@ batches still take the ~2 s described above — they arrive behind the progress
already scrolling rather than into an empty screen. Both halves matter, and only the first one is in this
table.
### The whole suite at 20 000 files, twice
The script that the CI workflow runs was also run against the same 20 000-file fixture — it is
device-agnostic apart from the emulator runner — which gave a second, independent run of the open
benchmark and, for the first time, the other three at that size:
| 20 000 files | one value per iteration | min–max | `frameDurationCpuMs` P50 / P90 / P99 |
|---|---|---|---|
| `openAFolderOfAThousandFiles` | **16 frames** | 13–32 | 7.8 / 21.6 / 100.6 ms |
| `scrollAFolderOfAThousandFiles` | **162 frames** | 148–172 | 9.0 / 21.0 / 25.2 ms |
| `searchInsideAFolderOfAThousandFiles` | **319 frames** | 300–653 | 6.5 / 11.5 / 18.5 ms |
| `coldStart` | **599 ms** to first display | 593–604 | — |
Two things in it are worth keeping:
* **The batching result holds in a second run.** The table above has 12 median / 48 max from one run; this
one has 16 median / 32 max on a build that also carries the M16 picker. That is the same number within
the noise, and the 598-frame stall has not come back.
* **Search is the next outlier.** 319 frames is about five seconds from the query to `file-99.txt`,
against 38–39 frames at 1 000 files: twenty times the entries for eight times the frames, so sublinear —
but still long enough to notice, and the per-frame numbers say drawing is not the reason (P90 11.5 ms).
Handing twenty thousand rows to a list in one go is; that is where a "large folder" pass would start.
`coldStart`'s 599 ms and 162 frames of scrolling are recorded here as the baseline those two get
compared against, not as findings.
And the fixture now survives: `tools/benchmark-fixture.sh --count 20000` reuses a folder that already holds
at least that many files, and `--fresh` rebuilds it. Building it takes about half an hour through MIUI's
storage layer, which is the whole reason to keep it.
@@ -317,17 +350,37 @@ instrumentation output itself, and the per-test JSON and Perfetto traces are lef
`/sdcard/Android/media/de.softwareapp_hb.filemanager.benchmark/` — they used to be deleted with the test app
when a run ended, which is why the earlier numbers had to be copied mid-run.
### The same thing in CI
`.github/workflows/benchmark.yml` runs those four commands on an emulator
(`reactivecircus/android-emulator-runner`, API 35, `google_apis`, x86_64, SwiftShader), on demand and every
Monday morning. It is deliberately not part of the pull-request gate: a run is minutes of measurement behind
twenty minutes of booting an emulator, and the gate is meant to answer in one or two minutes.
Two details make the numbers meaningful rather than merely present:
* The app is installed and `cmd package compile --reset` **before** the instrumentation starts, and the run
is driven with `am instrument` rather than `connectedBenchmarkAndroidTest` — the Gradle task installs the
app itself, which would undo that reset and quietly measure a compiled app.
* `tools/benchmark-report.py --require-frame-duration` turns the results into a table on the run's page and
**fails the job** when `frameDurationCpuMs` is missing. That metric is the reason the job exists; a run
without it is not a green run with a footnote.
The emulator's milliseconds are not a phone's — software rendering, a different CPU, no thermal limits — so
they are read as shape and trend (which benchmark is slow relative to the others, whether a change moved it),
not as an absolute.
## Still not measured
| number | status |
|---|---|
| Folder open with 5 000 / 20 000 files | not measured yet: `./tools/benchmark-fixture.sh --count 20000` builds that fixture, and the open/scroll numbers follow from re-running the same command |
| Folder open with 5 000 / 20 000 files | **measured**: see the scaling table above (`--count N` builds the fixture) |
| Thumbnail paint while scrolling a photo grid | not measured: no test for it yet |
| Any jank figure at all | not measurable here: this phone's `FrameTimingMetric` output is `frameCount` only, with no `frameDurationCpuMs` or `frameOverrunMs` |
| Any jank figure at all | **measured**: `frameDurationCpuMs` and `frameOverrunMs` both appear in the phone's 1 000-file run above, and the CI emulator run reports them every time |
What the interaction numbers still need is a device that reports frame durations — an emulator or any
non-MIUI phone would do, and the fixture is a folder of empty files, so nothing device-specific is being
measured. The install side is solved: `CompilationMode.Ignore()` plus installing from the host means a run
What is left for the interaction numbers is a device class decision rather than a measurement: the phone
numbers are one handset's, the emulator's are software-rendered, and neither is a claim about a mid-range
device. The install side is solved: `CompilationMode.Ignore()` plus installing from the host means a run
touches the package manager zero times.
## What is measured instead, meanwhile
@@ -372,8 +425,8 @@ adb shell cmd package compile --reset de.softwareapp_hb.filemanager # n
adb shell cmd package compile -m speed-profile -f de.softwareapp_hb.filemanager # profile applied
```
The emulator did answer one open question, though: on that path `FrameTimingMetric` **does** report what
the phone cannot — `frameDurationCpuMs` and `frameOverrunMs` are both present (P50 233 ms and 341 ms for
opening the folder, which is software rendering and not a number anyone should quote). The metric exists
as soon as the device supports frame timelines, so a physical API 33+ phone gives both the profile
delta and the jank figures the interaction benchmarks have been missing.
The emulator did answer one open question, though: on that path `FrameTimingMetric` reports both
`frameDurationCpuMs` and `frameOverrunMs` (P50 233 ms and 341 ms for opening the folder, which is software
rendering and not a number anyone should quote). The metric exists as soon as the device captures frame
timelines — which is what `.github/workflows/benchmark.yml` now guarantees, and what a physical API 33+
phone would combine with the profile delta.