Make the benchmark actually measure the benchmark build, and record what the phone will and will not answer
The run was installing app-debug.apk the whole time, which would have published debug numbers as the release-like ones. AGP 9's com.android.test extension has no testBuildType and no targetVariant - it exposes only targetProjectPath - and the target variant is matched by name against the test module's own build types. So the module now declares a benchmark build type, and inherits the debug signing config with initWith: a declared type starts unsigned, and an unsigned test APK fails with INSTALL_PARSE_FAILED_NO_CERTIFICATES. The command is now :benchmark:connectedBenchmarkAndroidTest, and -PbenchmarkBuild is gone from the workflow; the conditional testBuildType it used to set in :app only ever moved the unit-test variant, which is why it made :app:testDebugUnitTest disappear. The storage grant also had to move into the test. The rule uninstalls and reinstalls the app before measuring, an uninstall clears app-ops, and appops set from inside the test does not work even under adoptShellPermissionIdentity - the command fails to stderr, which executeShellCommand does not return, and appops get still reads No operations. What does work is tapping what a user taps: Grant full access, then the platform consent, then back out of the all-files page. grantStorageAccess does that in setupBlock, unmeasured, and it is what turned three failing tests into three passing ones. The search test now waits for the app bar's search action in setup instead of looking it up immediately inside the measured block, where it came back null one frame after a thousand rows rendered. What the device answers: coldStart passes at 560.6 ms P50 (a third measurement of the 532 ms figure), scroll passes but reports only frameCount and no frame durations at all, open dies on INSTALL_FAILED_USER_RESTRICTED because the rule's reinstall wants a human, and search still cannot see the search action. PERFORMANCE.md now states that per benchmark instead of the earlier guess, names the four switches that make an unattended run possible, and says that three of the four numbers need a device that installs unattended and reports frame timing.
This commit is contained in:
@@ -16,6 +16,13 @@
|
||||
<uses-permission
|
||||
android:name="android.permission.QUERY_ALL_PACKAGES"
|
||||
tools:ignore="QueryAllPackagesPermission" />
|
||||
<!--
|
||||
No app-ops permission here: granting MANAGE_EXTERNAL_STORAGE with `appops set` from the test does
|
||||
not work even under adoptShellPermissionIdentity — the command fails to stderr, which
|
||||
executeShellCommand does not return, and `appops get` still reads `No operations` afterwards.
|
||||
The tests walk the app's own "Grant full access" flow instead; see
|
||||
BrowserBenchmark.grantStorageAccess.
|
||||
-->
|
||||
<queries>
|
||||
<package android:name="de.softwareapp_hb.filemanager" />
|
||||
</queries>
|
||||
|
||||
+54
-10
@@ -49,11 +49,41 @@ class BrowserBenchmark {
|
||||
*/
|
||||
private val app = "de.softwareapp_hb.filemanager"
|
||||
|
||||
// No setup by shell command here on purpose. UiAutomation's executeShellCommand runs with *this
|
||||
// app's* uid, not the shell user's, so it cannot create files in /sdcard and cannot grant itself
|
||||
// an app-op — the first attempt failed exactly that way, with a RuntimeException from
|
||||
// UiAutomationConnection and no fixture on disk. `tools/benchmark-fixture.sh` prepares the device
|
||||
// instead: it runs through adb, so it has the shell's rights.
|
||||
// Files still have to be created through adb — `tools/benchmark-fixture.sh` — because
|
||||
// UiAutomation's executeShellCommand runs with *this app's* uid, not the shell user's, and cannot
|
||||
// write to /sdcard. The storage *grant* is a different matter: it is walked through the app's own
|
||||
// "Grant full access" button, because `appops set` from the test does not take effect — see
|
||||
// [grantStorageAccess].
|
||||
|
||||
/**
|
||||
* Gives the app all-files access the way a user would, when it came up without it.
|
||||
*
|
||||
* This is needed on every run because macrobenchmark uninstalls the app before measuring it, and an
|
||||
* uninstall clears app-ops: the grant that `tools/benchmark-fixture.sh` applies beforehand is gone
|
||||
* by the time the app starts, which is why the app legitimately shows its "choose a folder"
|
||||
* fallback and the volume row the tests navigate to is not on the screen.
|
||||
*
|
||||
* Trying to grant it directly — `UiAutomation.executeShellCommand("appops set …")` under
|
||||
* `adoptShellPermissionIdentity` — does not work: the command fails and prints its complaint to
|
||||
* stderr, which `executeShellCommand` does not return, so the test continues as if it had been
|
||||
* granted and fails later looking for the row. Measured; `appops get` still read `No operations`.
|
||||
*
|
||||
* The flow below therefore taps what a user taps: the app's button, then whatever consent the
|
||||
* platform puts in front of the all-files page. MIUI adds its own — "… erlauben, auf Fotos und
|
||||
* Medien … zuzugreifen?" — whose allow button is the dialog's *first* button; other builds go
|
||||
* straight to the toggle page. Both end with the toggle switched on and the app in front again.
|
||||
*/
|
||||
private fun grantStorageAccess(device: UiDevice) {
|
||||
val button = device.wait(Until.findObject(By.text(GRANT_ACTION)), SHORT_MS) ?: return
|
||||
button.click()
|
||||
device.wait(Until.findObject(By.clazz("android.widget.Button")), SHORT_MS)?.click()
|
||||
val toggle = device.wait(Until.findObject(By.clazz("android.widget.Switch")), SHORT_MS)
|
||||
if (toggle != null) {
|
||||
if (!toggle.isChecked) toggle.click()
|
||||
device.pressBack()
|
||||
}
|
||||
device.waitForIdle()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun coldStart() = benchmark.measureRepeated(
|
||||
@@ -112,6 +142,13 @@ class BrowserBenchmark {
|
||||
openFixture(device)
|
||||
device.findObject(By.text(Fixture.NAME))!!.click()
|
||||
device.wait(Until.hasObject(By.text(Fixture.FIRST_FILE)), TIMEOUT_MS)
|
||||
// The measured block clicks this, so it has to exist before the clock starts: right after a
|
||||
// thousand rows render, the app bar is a frame behind the list and an immediate lookup
|
||||
// comes back null.
|
||||
assertNotNull(
|
||||
"the app bar's search action is missing; see browser_search_hint in strings.xml",
|
||||
device.wait(Until.findObject(By.desc(SEARCH_ACTION)), TIMEOUT_MS),
|
||||
)
|
||||
},
|
||||
) {
|
||||
device.findObject(By.desc(SEARCH_ACTION))!!.click()
|
||||
@@ -124,17 +161,18 @@ class BrowserBenchmark {
|
||||
// ------------------------------------------------------------------ setup
|
||||
|
||||
/**
|
||||
* The device is expected to be prepared already: `tools/benchmark-fixture.sh` creates the fixture
|
||||
* and grants the app its storage access. This only navigates to it, and fails with that
|
||||
* instruction if the row is not where it should be.
|
||||
* The device is expected to be prepared: `tools/benchmark-fixture.sh` creates the fixture folder.
|
||||
* This grants storage access if the reinstall took it away, then navigates, and fails with the
|
||||
* instruction that is missing if something is not where it should be.
|
||||
*/
|
||||
private fun MacrobenchmarkScope.openFixture(device: UiDevice) {
|
||||
pressHome()
|
||||
startActivityAndWait()
|
||||
grantStorageAccess(device)
|
||||
val volume = device.wait(Until.findObject(By.text("Internal storage")), TIMEOUT_MS)
|
||||
assertNotNull(
|
||||
"the volume row is not on screen. Is the device in English, and did the storage grant " +
|
||||
"land? The benchmark grants it with `appops set` before navigating.",
|
||||
"the volume row is not on screen, so the storage grant did not land. Is the device in " +
|
||||
"English? The app has to come up on its roots screen for this benchmark to navigate.",
|
||||
volume,
|
||||
)
|
||||
volume!!.click()
|
||||
@@ -148,8 +186,14 @@ class BrowserBenchmark {
|
||||
companion object {
|
||||
private const val TIMEOUT_MS = 10_000L
|
||||
|
||||
/** How long to wait for something that should already be on screen. */
|
||||
private const val SHORT_MS = 2_000L
|
||||
|
||||
/** The app bar's own description for the search action; see strings.xml. */
|
||||
private const val SEARCH_ACTION = "Search this folder"
|
||||
|
||||
/** The fallback screen's button, shown only when the app has no all-files access. */
|
||||
private const val GRANT_ACTION = "Grant full access"
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user