The benchmark runs in CI, and fails when the jank metric is missing
Item 5 of the queue: .github/workflows/benchmark.yml measures on an API 35 emulator (reactivecircus/android-emulator-runner, google_apis, x86_64, SwiftShader), on demand and every Monday. Not on pull requests: a run is minutes of measurement behind twenty of booting an emulator, and the gate is meant to answer in one or two. It installs both APKs and instruments directly instead of using connectedBenchmarkAndroidTest, because that task installs the app itself - which undoes the compile reset that makes the measurement a never-run baseline. tools/benchmark-report.py turns the JSON into two tables, keeping frameCount (a length) apart from frameDurationCpuMs and frameOverrunMs (speeds), and --require-frame-duration fails the job when the second table would be empty: the metric is the reason the job exists. Driving the workflow's own script against the phone first paid for itself three times. It found that cmd package compile --reset needs adbd as root - a user build refuses it with SecurityException, so the workflow now does adb root first (google_apis allows it) instead of silently measuring a compiled app. It produced the second run of the 20 000-file open (16 frames median, 32 max) and the first numbers for scroll, search and cold start at that size: search costs 319 frames, which is now the biggest number in the file. And it showed the phone reporting frameDurationCpuMs and frameOverrunMs after all, so PERFORMANCE.md's 'this phone reports counts only' was a property of those earlier runs, not of the handset - corrected, with the numbers.
This commit is contained in:
+8
-2
@@ -89,8 +89,14 @@ release should not go out without.
|
||||
the selection rule. Driven on the phone for both verbs (see `EDGE_CASE_TESTING.md`), which is what caught
|
||||
the picker naming internal storage by its raw name `0`. **Not done**: the "copied to X — Show" message
|
||||
with a jump to the destination; the copy reports itself the way a paste does.
|
||||
5. **A CI run of the benchmarks on an emulator** — removes the phone from the loop and reports
|
||||
`frameDurationCpuMs`, which this device cannot.
|
||||
5. **A CI run of the benchmarks on an emulator** — **done**: `.github/workflows/benchmark.yml` runs the four
|
||||
benchmarks on API 35 (`reactivecircus/android-emulator-runner`, `google_apis`, x86_64, SwiftShader) on
|
||||
demand and every Monday, installs the app and resets its compilation before instrumenting it, pulls the
|
||||
JSON, and publishes `tools/benchmark-report.py`'s table on the run page — failing the job if
|
||||
`frameDurationCpuMs` is missing, since that metric is why the job exists. While wiring it up, the
|
||||
phone's own JSON from the progressive-listing run turned out to contain `frameDurationCpuMs` and
|
||||
`frameOverrunMs` after all (P50 13.9 ms / 19.0 ms for the 1 000-file open), so `PERFORMANCE.md`'s "counts
|
||||
only" finding was a property of the earlier runs, not of the device; corrected there.
|
||||
|
||||
### M16 — Copy to… / Move to…
|
||||
|
||||
|
||||
Reference in New Issue
Block a user