Make the benchmark actually measure the benchmark build, and record what the phone will and will not answer
The run was installing app-debug.apk the whole time, which would have published debug numbers as the release-like ones. AGP 9's com.android.test extension has no testBuildType and no targetVariant - it exposes only targetProjectPath - and the target variant is matched by name against the test module's own build types. So the module now declares a benchmark build type, and inherits the debug signing config with initWith: a declared type starts unsigned, and an unsigned test APK fails with INSTALL_PARSE_FAILED_NO_CERTIFICATES. The command is now :benchmark:connectedBenchmarkAndroidTest, and -PbenchmarkBuild is gone from the workflow; the conditional testBuildType it used to set in :app only ever moved the unit-test variant, which is why it made :app:testDebugUnitTest disappear. The storage grant also had to move into the test. The rule uninstalls and reinstalls the app before measuring, an uninstall clears app-ops, and appops set from inside the test does not work even under adoptShellPermissionIdentity - the command fails to stderr, which executeShellCommand does not return, and appops get still reads No operations. What does work is tapping what a user taps: Grant full access, then the platform consent, then back out of the all-files page. grantStorageAccess does that in setupBlock, unmeasured, and it is what turned three failing tests into three passing ones. The search test now waits for the app bar's search action in setup instead of looking it up immediately inside the measured block, where it came back null one frame after a thousand rows rendered. What the device answers: coldStart passes at 560.6 ms P50 (a third measurement of the 532 ms figure), scroll passes but reports only frameCount and no frame durations at all, open dies on INSTALL_FAILED_USER_RESTRICTED because the rule's reinstall wants a human, and search still cannot see the search action. PERFORMANCE.md now states that per benchmark instead of the earlier guess, names the four switches that make an unattended run possible, and says that three of the four numbers need a device that installs unattended and reports frame timing.
This commit is contained in:
+5
-15
@@ -132,22 +132,12 @@ android {
|
||||
}
|
||||
|
||||
/*
|
||||
* Which build type the instrumentation tests run against — only when asked for.
|
||||
*
|
||||
* Setting this unconditionally also makes `benchmark` the variant that *unit* tests run against,
|
||||
* and `:app:testDebugUnitTest` — the task the gate and CI are built on — stops existing. That is
|
||||
* measured rather than guessed: with it set, the only unit-test task left was
|
||||
* `testBenchmarkUnitTest`.
|
||||
*
|
||||
* ./gradlew -PbenchmarkBuild :benchmark:connectedDebugAndroidTest
|
||||
*
|
||||
* `hasProperty` rather than `providers.gradleProperty`: the latter is a Provider, and inside this
|
||||
* block it read as absent even when the flag was passed — which showed up as the benchmark
|
||||
* installing the *debug* APK and failing on it.
|
||||
* No `testBuildType` here on purpose. It reads like "which build the benchmarks run against", but
|
||||
* in :app it only moves the variant that this module's own *unit* tests use — with it set, the only
|
||||
* unit-test task left was `testBenchmarkUnitTest` and `:app:testDebugUnitTest`, the task the gate
|
||||
* and CI are built on, stopped existing. The switch that matters lives in :benchmark, where
|
||||
* `targetVariant = "benchmark"` makes the run install `app-benchmark.apk` instead of `app-debug.apk`.
|
||||
*/
|
||||
if (project.hasProperty("benchmarkBuild")) {
|
||||
testBuildType = "benchmark"
|
||||
}
|
||||
|
||||
buildFeatures {
|
||||
compose = true
|
||||
|
||||
Reference in New Issue
Block a user