Hadrian Burkhardt
4706e2669c
An Android duplicate-photo finder. Pure-Kotlin core (SHA-256 + dHash hashing, exact and perceptual union-find grouping, three sensitivity presets) with MediaStore access, a persistent SQLite hash index and a tiered ScanEngine behind a Jetpack Compose UI. No INTERNET permission, no ads, no analytics, no account. Ships versionCode 2 / versionName 1.0.1, carrying the scan-stall hardening (BoundedCall breaker) and verified against a 5,363-photo device library. 80 JVM unit tests, lint clean. Release paperwork is in docs/ (Play checklist, store listing, edge-case testing), the upload-ready Play graphics and screenshots in store-assets/, and the generators for those assets in tools/.
213 lines
11 KiB
Markdown
213 lines
11 KiB
Markdown
# Edge-case testing
|
||
|
||
Fixtures are generated, not committed:
|
||
|
||
```bash
|
||
# text/PNG/JPEG fixtures
|
||
python3 tools/make-edge-case-fixtures.py
|
||
|
||
# ...plus HEIC (needs an encoder; ImageMagick's HEIC support is read-only)
|
||
python3 -m venv /tmp/heicvenv && /tmp/heicvenv/bin/pip install pillow-heif
|
||
/tmp/heicvenv/bin/python tools/make-edge-case-fixtures.py
|
||
|
||
# push to a phone
|
||
adb push fixtures/foto-dup-edge /sdcard/DCIM/foto-dup-edge
|
||
```
|
||
|
||
Everything lands in `fixtures/foto-dup-edge/` **inside the repo**. The directory is git-ignored
|
||
— the fixtures are regenerated on demand, so no binary test data is committed.
|
||
|
||
## What each fixture is for
|
||
|
||
| Fixture | Expected behaviour |
|
||
|---|---|
|
||
| `valid.jpg`, `tiny-1px.jpg`, `grayscale.jpg`, `with-alpha.png` | read normally |
|
||
| `zero-bytes.jpg`, `truncated.jpg`, `garbage-body.jpg` | skipped, counted in "could not be read", **no crash** |
|
||
| `renamed-png.jpg` (PNG named `.jpg`) | depends on the MIME reported by MediaStore — must not crash |
|
||
| `no-extension` | skipped by MediaStore; must not crash if it appears |
|
||
| `decompression-bomb.png` (declares 60 000²) | skipped, never allocates; exercises `DecodeMath` |
|
||
| `dup-a.jpg` / `dup-b.jpg` | **one exact-duplicate group** |
|
||
| `solid-0..5.png` | Flat colour: every one hashes *identically*, so they all belong in **one** group |
|
||
| `exif-orientation-1..8.jpg` | Identical **pixels** with different orientation **tags**, so they *display differently*. Only the upright one groups with the untagged PNG copies; the other seven are correctly distinct. A small group here therefore **proves EXIF normalisation is applied** — if it were ignored, all eight share identical pixels and would collapse into one group with the PNGs |
|
||
| `heic-basic.heic` | decoded and hashed like any other photo |
|
||
| `heic-dup.heic` | byte-identical to `heic-basic` → one exact-duplicate group |
|
||
| `heic-large.heic` (2400×1800) | exercises the subsampled fallback decode |
|
||
| `heic-solid-0/1.heic` | ditto the PNG solids, but they should join them: flat images of *any* colour are indistinguishable to the current descriptors |
|
||
| `scene.jpg` + `scene.heic` (one scene, two codecs) | informative: is a codec conversion still reported? |
|
||
|
||
## Licence
|
||
|
||
All fixtures are **generated locally from synthetic images**, so they carry no third-party
|
||
licence and can be pushed, kept or discarded freely.
|
||
|
||
The obvious sources for ready-made HEIC samples are **not** clean:
|
||
|
||
- `nokiatech/heif_conformance` — the canonical HEIC test set, but the repository has **no
|
||
LICENSE file** and the README states no terms, so it is all-rights-reserved by default.
|
||
- `strukturag/libheif` `examples/example.heic` — the repo is LGPL (library) / MIT (example
|
||
apps), but the **image file itself has no stated licence**.
|
||
- Wikimedia Commons — **hosts no HEIC files** (not an accepted upload format).
|
||
|
||
That is why the HEIC fixtures are encoded here with `pillow-heif` instead.
|
||
|
||
## Known gap this fixture set exposed
|
||
|
||
**Flat-colour images are indistinguishable to the current descriptors.** A uniform image has no
|
||
structure, so its pHash is always `1` and its RGB dHash is always `0` — regardless of the colour.
|
||
A black screenshot, a white scan and a grey solid therefore all hash *identically* and get reported
|
||
as mutual duplicates.
|
||
|
||
This is a genuine false-positive risk on real libraries (dark UI screenshots, blank scans). The fix
|
||
is to add the colour dimension the descriptors deliberately discard — a small colour histogram is
|
||
orthogonal to a grayscale structural hash and would separate them (see the benchmark harness, which
|
||
already has a colour-histogram implementation).
|
||
|
||
## Running the grouping logic on a desktop
|
||
|
||
The duplicate engine is pure Kotlin, so the algorithm can be exercised without a phone. Only
|
||
MediaStore, the SQLite index and the notification need Android.
|
||
|
||
```bash
|
||
# normalise EXIF orientation with Pillow, then fingerprint and group in Kotlin
|
||
python3 tools/normalise-for-local-grouping.py fixtures/foto-dup-edge build/fotonorm
|
||
./gradlew :app:testDebugUnitTest --tests '*LocalGroupingTest*' \
|
||
-DfotoLocal.dir=/tmp/fotonorm --rerun-tasks
|
||
```
|
||
|
||
(`--rerun-tasks` because Gradle does not always notice a changed `-D` value on the test task.)
|
||
|
||
Faithful: hashing, the adaptive threshold, banded candidate generation, the pHash∧RGB conjunction,
|
||
union-find clustering and the resulting groups.
|
||
Not faithful: MediaStore's view of what is indexed, the provider's thumbnail decode, and the
|
||
SQLite hash cache.
|
||
|
||
The last two matter — see below.
|
||
|
||
### Why the device and a desktop run can disagree on flat images
|
||
|
||
Local run of the fixture set: **all eight solids in one group**, as the algorithm demands.
|
||
On a real device the same eight came back as 3 + 2 + 2.
|
||
|
||
Reproduced locally by feeding the harness *thumbnail-sized JPEGs* (96×72, q75) instead of the
|
||
lossless solids: the result becomes **3 + 2 + 2**, matching the device.
|
||
|
||
Cause: JPEG quantises the DC coefficient, so a one-unit colour difference between neighbouring
|
||
solids falls below the quantisation step and they become effectively identical. On a device the
|
||
perceptual hash is taken from the provider's **lossy thumbnail**, so on a flat image it is
|
||
hash-of-quantisation-noise — there is no signal left to group by.
|
||
|
||
Two consequences:
|
||
|
||
1. Flat images (blank scans, dark UI screenshots) are grouped arbitrarily. **A colour signature is
|
||
the fix** — it is the one dimension the current descriptors discard.
|
||
2. Because the hash comes from the provider's thumbnail, it depends on the thumbnail's
|
||
re-encoding, not purely on the file. Real near-duplicates survive this (they are close
|
||
anyway), but the margin is thinner than a self-decoded hash would give.
|
||
|
||
## Running the scan pipeline on the JVM (Robolectric)
|
||
|
||
`ScanEngineEndToEndTest` drives the **real** pipeline — MediaStore query, head/tail signature,
|
||
SHA-256, SQLite index and grouper — against `FakeMediaStoreProvider`, a stand-in ContentProvider
|
||
registered under the MediaStore authority and backed by real files on disk.
|
||
|
||
```bash
|
||
./gradlew :app:testDebugUnitTest --tests '*ScanEngineEndToEndTest*'
|
||
```
|
||
|
||
**What it covers:** query and projection handling, the head/tail read, the exact-duplicate path over
|
||
genuine bytes, the index across two runs, unreadable-file reporting, and that the scan **runs to
|
||
completion instead of stalling** — the failure that froze on a device.
|
||
|
||
**What it cannot cover:** perceptual hashing. Robolectric's `ShadowBitmapFactory` does not decode
|
||
real pixel data, so every fingerprint is `null` and all files are reported as unreadable. The test
|
||
asserts that reality explicitly rather than pretending otherwise; near-duplicate behaviour has to
|
||
come from `bench/` or from a device.
|
||
|
||
### JDK 17+ needs two different module flags
|
||
|
||
Robolectric reflects into JDK internals, and on a modern JDK that fails unless the test JVM is
|
||
opened up. Two *distinct* flags are needed, and the second is easy to miss:
|
||
|
||
```
|
||
--add-opens=java.base/java.io=ALL-UNNAMED # FileDescriptor fields
|
||
--add-exports=java.base/jdk.internal.access=ALL-UNNAMED # SharedSecrets accessor
|
||
```
|
||
|
||
The second must be an **export**, not an open: Robolectric calls
|
||
`jdk.internal.access.SharedSecrets#getJavaIOFileDescriptorAccess` reflectively, and the failure
|
||
otherwise surfaces as an opaque `IllegalAccessException` at `Reflection.java:400`, wrapped in
|
||
*"Failed to interact with raw FileDescriptor internals; perhaps JRE has changed?"*.
|
||
|
||
## What cannot be generated locally
|
||
|
||
- **HEIC with an EXIF orientation tag.** `pillow-heif` accepts `exif=` on save but the
|
||
orientation tag reads back as `1` regardless, so a generated fixture would misrepresent what
|
||
it tests. This case needs a **real camera/iPhone HEIC** with a non-upright orientation.
|
||
- **HEIC from a real encoder** in general — worth spot-checking one real iPhone file.
|
||
- **Burst sequences / same-scene shots.** The one behaviour the benchmark cannot model, and the
|
||
most likely source of false groups. Needs real photos.
|
||
- **Permission revoked mid-scan**, and the Android 14 "Selected photos" flow: device only.
|
||
|
||
## Driving a scan on a device without touch input
|
||
|
||
The test device (Xiaomi M2101K7AG, Android 12 / MIUI, serial `2a2ae895`) denies the usual
|
||
automation, so a scan has to be driven from the app side instead:
|
||
|
||
| Usual approach | Result on this device |
|
||
|---|---|
|
||
| `adb shell input tap/keyevent` | `SecurityException: INJECT_EVENTS` — MIUI blocks it, and the toggle that would allow it has no adb-reachable switch |
|
||
| `adb shell pm grant <pkg> READ_EXTERNAL_STORAGE` | `SecurityException: Neither user 2000 nor current process has GRANT_RUNTIME_PERMISSIONS` |
|
||
| `adb install -g` | same refusal |
|
||
|
||
Two **debug-only** hooks exist for this, under `app/src/debug/`. Both are absent from the
|
||
release build — re-check with the audit command in `PLAY_RELEASE_CHECKLIST.md` § 5.
|
||
|
||
### Starting a scan
|
||
|
||
```bash
|
||
adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.START_SCAN \
|
||
-n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger
|
||
```
|
||
|
||
Add `--es policy exact|balanced|aggressive` to override the default (`balanced`). The receiver
|
||
logs what it did and the resulting `ScanState` under the `FotoDupDebug` tag, so progress **and**
|
||
the final timings are readable without the screen:
|
||
|
||
```bash
|
||
adb logcat -s FotoDupDebug
|
||
# poll the current state at any time:
|
||
adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.REPORT_STATE \
|
||
-n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger
|
||
```
|
||
|
||
### Granting photo access
|
||
|
||
`MANAGE_EXTERNAL_STORAGE` is the one storage permission that *can* be granted from adb, which is
|
||
why the **debug** manifest declares it:
|
||
|
||
```bash
|
||
adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE allow
|
||
# ...run the scan, then put it back:
|
||
adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE default
|
||
```
|
||
|
||
The app's own banner still reads *“Photo access needed”*, because `checkSelfPermission` is
|
||
unaffected by an appops grant. That is expected: it only affects the on-screen prompt, not the scan.
|
||
|
||
### Attaching a debugger for thread stacks
|
||
|
||
`jdb` attaches over JDWP to the debug build:
|
||
|
||
```bash
|
||
adb forward tcp:8700 jdwp:$(adb shell pidof de.softwareapp_hb.fotodup.finder.debug | tr -d '\r')
|
||
printf 'suspend\nwhere all\nquit\n' | jdb -attach localhost:8700
|
||
```
|
||
|
||
Two things learned the hard way:
|
||
|
||
- `print` needs a **selected frame**. With no current frame even `java.lang.Class` reports
|
||
"Name unknown", so expressions cannot be evaluated until a breakpoint has been reached.
|
||
- Leaving the main thread suspended for more than a few seconds **during an activity launch**
|
||
gets the process killed by the launch timeout. Evaluate and `cont` quickly.
|
||
|