SoftwareApp
66183a8596
Item 5 of the queue: .github/workflows/benchmark.yml measures on an API 35 emulator (reactivecircus/android-emulator-runner, google_apis, x86_64, SwiftShader), on demand and every Monday. Not on pull requests: a run is minutes of measurement behind twenty of booting an emulator, and the gate is meant to answer in one or two. It installs both APKs and instruments directly instead of using connectedBenchmarkAndroidTest, because that task installs the app itself - which undoes the compile reset that makes the measurement a never-run baseline. tools/benchmark-report.py turns the JSON into two tables, keeping frameCount (a length) apart from frameDurationCpuMs and frameOverrunMs (speeds), and --require-frame-duration fails the job when the second table would be empty: the metric is the reason the job exists. Driving the workflow's own script against the phone first paid for itself three times. It found that cmd package compile --reset needs adbd as root - a user build refuses it with SecurityException, so the workflow now does adb root first (google_apis allows it) instead of silently measuring a compiled app. It produced the second run of the 20 000-file open (16 frames median, 32 max) and the first numbers for scroll, search and cold start at that size: search costs 319 frames, which is now the biggest number in the file. And it showed the phone reporting frameDurationCpuMs and frameOverrunMs after all, so PERFORMANCE.md's 'this phone reports counts only' was a property of those earlier runs, not of the handset - corrected, with the numbers.
112 lines
5.3 KiB
YAML
112 lines
5.3 KiB
YAML
name: benchmark
|
||
|
||
# The macrobenchmark run, on an emulator rather than on a phone.
|
||
#
|
||
# Three reasons this exists as a workflow and not as a local step:
|
||
#
|
||
# * **It reports the jank metric.** `frameDurationCpuMs` and `frameOverrunMs` need frame timelines, which
|
||
# this project's phone sometimes supplies and sometimes does not — the run that does is the one worth
|
||
# looking at, and a CI emulator supplies it every time.
|
||
# * **It takes the phone out of the loop.** No MIUI install prompts, no AVD to keep alive, no cable.
|
||
# * **It is a second opinion.** The numbers in docs/PERFORMANCE.md come from one handset; an emulator run
|
||
# is a different device and build, so a regression that is the app's shows up in both.
|
||
#
|
||
# It is not on every pull request: one run is the four benchmarks × five iterations, which is minutes of
|
||
# measurement and considerably more of waiting for an emulator to boot.
|
||
on:
|
||
workflow_dispatch:
|
||
schedule:
|
||
# Monday, early: the week starts with a number, not with a guess.
|
||
- cron: '41 4 * * 1'
|
||
|
||
concurrency:
|
||
group: benchmark
|
||
cancel-in-progress: false
|
||
|
||
jobs:
|
||
emulator:
|
||
runs-on: ubuntu-latest
|
||
timeout-minutes: 75
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: actions/setup-java@v4
|
||
with:
|
||
distribution: temurin
|
||
# detekt 1.23 cannot parse JDK 25's version string, and the gate runs on 21 — keep them equal
|
||
# so a benchmark run cannot fail for a reason unrelated to the benchmark.
|
||
java-version: '21'
|
||
- uses: android-actions/setup-android@v3
|
||
- uses: gradle/actions/setup-gradle@v4
|
||
|
||
# Built before the emulator starts: Gradle is quick on the runner, emulator boot is not, and the
|
||
# two would otherwise queue behind each other.
|
||
- name: Build the APKs
|
||
run: ./gradlew :app:assembleBenchmark :benchmark:assembleBenchmark --console=plain
|
||
|
||
- name: Measure on an emulator
|
||
uses: reactivecircus/android-emulator-runner@v2
|
||
with:
|
||
api-level: 35
|
||
# google_apis, not google_apis_playstore: the benchmark toggles the app's all-files access in
|
||
# Settings, and a Play image has no such page for a sideloaded app.
|
||
target: google_apis
|
||
arch: x86_64
|
||
profile: pixel_6
|
||
disable-animations: true
|
||
emulator-boot-timeout: 1200
|
||
# SwiftShader because the runner has no GPU: software rendering makes the milliseconds
|
||
# incomparable with a phone's, which is stated where the numbers are read. What survives is the
|
||
# shape — which benchmark is slow relative to the others — and the existence of the metric.
|
||
emulator-options: >-
|
||
-no-snapshot -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim
|
||
-camera-back none -camera-front none
|
||
script: |
|
||
set -o pipefail
|
||
# The fixture is built through adb because UiAutomation's shell runs as the app, not the shell
|
||
# user, and cannot write to /sdcard (see tools/benchmark-fixture.sh).
|
||
./tools/benchmark-fixture.sh
|
||
adb install -r app/build/outputs/apk/benchmark/app-benchmark.apk
|
||
adb install -r benchmark/build/outputs/apk/benchmark/benchmark-benchmark.apk
|
||
# The "never run" baseline: no profile, so what is measured has not been compiled by use.
|
||
# Clearing profile data is a system operation, so it needs adbd as root — which a google_apis
|
||
# image allows and a user build (the phone) does not, where this silently stays best-effort.
|
||
adb root || true
|
||
adb wait-for-device
|
||
adb shell cmd package compile --reset de.softwareapp_hb.filemanager || true
|
||
# Instrumented directly rather than through connectedBenchmarkAndroidTest, because that task
|
||
# installs the app itself — which would undo the compile reset above. `CompilationMode.Ignore()`
|
||
# in the test means nothing else installs either.
|
||
adb shell am instrument -w -r \
|
||
-e class de.softwareapp_hb.filemanager.benchmark.BrowserBenchmark \
|
||
de.softwareapp_hb.filemanager.benchmark/androidx.test.runner.AndroidJUnitRunner \
|
||
| tee instrument.log
|
||
status=${PIPESTATUS[0]}
|
||
mkdir -p benchmark-results
|
||
adb pull /sdcard/Android/media/de.softwareapp_hb.filemanager.benchmark benchmark-results/ || true
|
||
# The jank metric is the reason this job exists, so its absence fails the run rather than
|
||
# becoming a footnote in a log nobody opens.
|
||
python3 tools/benchmark-report.py --require-frame-duration benchmark-results > summary.md || status=1
|
||
cat summary.md
|
||
exit $status
|
||
|
||
- name: Keep the numbers
|
||
if: always()
|
||
uses: actions/upload-artifact@v4
|
||
with:
|
||
name: benchmark-results
|
||
path: |
|
||
benchmark-results
|
||
instrument.log
|
||
summary.md
|
||
if-no-files-found: warn
|
||
|
||
- name: Put the table on the run page
|
||
if: always()
|
||
run: |
|
||
if [ -f summary.md ]; then
|
||
cat summary.md >> "$GITHUB_STEP_SUMMARY"
|
||
else
|
||
echo "No summary: the benchmark did not get far enough to write one. See instrument.log." \
|
||
>> "$GITHUB_STEP_SUMMARY"
|
||
fi
|