# Edge-case testing

Fixtures are generated, not committed:

```bash
# text/PNG/JPEG fixtures
python3 tools/make-edge-case-fixtures.py

# ...plus HEIC (needs an encoder; ImageMagick's HEIC support is read-only)
python3 -m venv /tmp/heicvenv && /tmp/heicvenv/bin/pip install pillow-heif
/tmp/heicvenv/bin/python tools/make-edge-case-fixtures.py

# push to a phone
adb push fixtures/foto-dup-edge /sdcard/DCIM/foto-dup-edge
```

Everything lands in `fixtures/foto-dup-edge/` **inside the repo**. The directory is git-ignored
— the fixtures are regenerated on demand, so no binary test data is committed.

## What each fixture is for

| Fixture | Expected behaviour |
|---|---|
| `valid.jpg`, `tiny-1px.jpg`, `grayscale.jpg`, `with-alpha.png` | read normally |
| `zero-bytes.jpg`, `truncated.jpg`, `garbage-body.jpg` | skipped, counted in "could not be read", **no crash** |
| `renamed-png.jpg` (PNG named `.jpg`) | depends on the MIME reported by MediaStore — must not crash |
| `no-extension` | skipped by MediaStore; must not crash if it appears |
| `decompression-bomb.png` (declares 60 000²) | skipped, never allocates; exercises `DecodeMath` |
| `dup-a.jpg` / `dup-b.jpg` | **one exact-duplicate group** |
| `solid-0..5.png` | Flat colour: every one hashes *identically*, so they all belong in **one** group |
| `exif-orientation-1..8.jpg` | Identical **pixels** with different orientation **tags**, so they *display differently*. Only the upright one groups with the untagged PNG copies; the other seven are correctly distinct. A small group here therefore **proves EXIF normalisation is applied** — if it were ignored, all eight share identical pixels and would collapse into one group with the PNGs |
| `heic-basic.heic` | decoded and hashed like any other photo |
| `heic-dup.heic` | byte-identical to `heic-basic` → one exact-duplicate group |
| `heic-large.heic` (2400×1800) | exercises the subsampled fallback decode |
| `heic-solid-0/1.heic` | ditto the PNG solids, but they should join them: flat images of *any* colour are indistinguishable to the current descriptors |
| `scene.jpg` + `scene.heic` (one scene, two codecs) | informative: is a codec conversion still reported? |

## Licence

All fixtures are **generated locally from synthetic images**, so they carry no third-party
licence and can be pushed, kept or discarded freely.

The obvious sources for ready-made HEIC samples are **not** clean:

- `nokiatech/heif_conformance` — the canonical HEIC test set, but the repository has **no
  LICENSE file** and the README states no terms, so it is all-rights-reserved by default.
- `strukturag/libheif` `examples/example.heic` — the repo is LGPL (library) / MIT (example
  apps), but the **image file itself has no stated licence**.
- Wikimedia Commons — **hosts no HEIC files** (not an accepted upload format).

That is why the HEIC fixtures are encoded here with `pillow-heif` instead.

## Known gap this fixture set exposed

**Flat-colour images are indistinguishable to the current descriptors.** A uniform image has no
structure, so its pHash is always `1` and its RGB dHash is always `0` — regardless of the colour.
A black screenshot, a white scan and a grey solid therefore all hash *identically* and get reported
as mutual duplicates.

This is a genuine false-positive risk on real libraries (dark UI screenshots, blank scans). The fix
is to add the colour dimension the descriptors deliberately discard — a small colour histogram is
orthogonal to a grayscale structural hash and would separate them (see the benchmark harness, which
already has a colour-histogram implementation).

## Running the grouping logic on a desktop

The duplicate engine is pure Kotlin, so the algorithm can be exercised without a phone. Only
MediaStore, the SQLite index and the notification need Android.

```bash
# normalise EXIF orientation with Pillow, then fingerprint and group in Kotlin
python3 tools/normalise-for-local-grouping.py fixtures/foto-dup-edge build/fotonorm
./gradlew :app:testDebugUnitTest --tests '*LocalGroupingTest*' \
    -DfotoLocal.dir=/tmp/fotonorm --rerun-tasks
```

(`--rerun-tasks` because Gradle does not always notice a changed `-D` value on the test task.)

Faithful: hashing, the adaptive threshold, banded candidate generation, the pHash∧RGB conjunction,
union-find clustering and the resulting groups.
Not faithful: MediaStore's view of what is indexed, the provider's thumbnail decode, and the
SQLite hash cache.

The last two matter — see below.

### Why the device and a desktop run can disagree on flat images

Local run of the fixture set: **all eight solids in one group**, as the algorithm demands.
On a real device the same eight came back as 3 + 2 + 2.

Reproduced locally by feeding the harness *thumbnail-sized JPEGs* (96×72, q75) instead of the
lossless solids: the result becomes **3 + 2 + 2**, matching the device.

Cause: JPEG quantises the DC coefficient, so a one-unit colour difference between neighbouring
solids falls below the quantisation step and they become effectively identical. On a device the
perceptual hash is taken from the provider's **lossy thumbnail**, so on a flat image it is
hash-of-quantisation-noise — there is no signal left to group by.

Two consequences:

1. Flat images (blank scans, dark UI screenshots) are grouped arbitrarily. **A colour signature is
   the fix** — it is the one dimension the current descriptors discard.
2. Because the hash comes from the provider's thumbnail, it depends on the thumbnail's
   re-encoding, not purely on the file. Real near-duplicates survive this (they are close
   anyway), but the margin is thinner than a self-decoded hash would give.

## Running the scan pipeline on the JVM (Robolectric)

`ScanEngineEndToEndTest` drives the **real** pipeline — MediaStore query, head/tail signature,
SHA-256, SQLite index and grouper — against `FakeMediaStoreProvider`, a stand-in ContentProvider
registered under the MediaStore authority and backed by real files on disk.

```bash
./gradlew :app:testDebugUnitTest --tests '*ScanEngineEndToEndTest*'
```

**What it covers:** query and projection handling, the head/tail read, the exact-duplicate path over
genuine bytes, the index across two runs, unreadable-file reporting, and that the scan **runs to
completion instead of stalling** — the failure that froze on a device.

**What it cannot cover:** perceptual hashing. Robolectric's `ShadowBitmapFactory` does not decode
real pixel data, so every fingerprint is `null` and all files are reported as unreadable. The test
asserts that reality explicitly rather than pretending otherwise; near-duplicate behaviour has to
come from `bench/` or from a device.

### JDK 17+ needs two different module flags

Robolectric reflects into JDK internals, and on a modern JDK that fails unless the test JVM is
opened up. Two *distinct* flags are needed, and the second is easy to miss:

```
--add-opens=java.base/java.io=ALL-UNNAMED            # FileDescriptor fields
--add-exports=java.base/jdk.internal.access=ALL-UNNAMED   # SharedSecrets accessor
```

The second must be an **export**, not an open: Robolectric calls
`jdk.internal.access.SharedSecrets#getJavaIOFileDescriptorAccess` reflectively, and the failure
otherwise surfaces as an opaque `IllegalAccessException` at `Reflection.java:400`, wrapped in
*"Failed to interact with raw FileDescriptor internals; perhaps JRE has changed?"*.

## What cannot be generated locally

- **HEIC with an EXIF orientation tag.** `pillow-heif` accepts `exif=` on save but the
  orientation tag reads back as `1` regardless, so a generated fixture would misrepresent what
  it tests. This case needs a **real camera/iPhone HEIC** with a non-upright orientation.
- **HEIC from a real encoder** in general — worth spot-checking one real iPhone file.
- **Burst sequences / same-scene shots.** The one behaviour the benchmark cannot model, and the
  most likely source of false groups. Needs real photos.
- **Permission revoked mid-scan**, and the Android 14 "Selected photos" flow: device only.

## Driving a scan on a device without touch input

The test device (Xiaomi M2101K7AG, Android 12 / MIUI, serial `2a2ae895`) denies the usual
automation, so a scan has to be driven from the app side instead:

| Usual approach | Result on this device |
|---|---|
| `adb shell input tap/keyevent` | `SecurityException: INJECT_EVENTS` — MIUI blocks it, and the toggle that would allow it has no adb-reachable switch |
| `adb shell pm grant <pkg> READ_EXTERNAL_STORAGE` | `SecurityException: Neither user 2000 nor current process has GRANT_RUNTIME_PERMISSIONS` |
| `adb install -g` | same refusal |

Two **debug-only** hooks exist for this, under `app/src/debug/`. Both are absent from the
release build — re-check with the audit command in `PLAY_RELEASE_CHECKLIST.md` § 5.

### Starting a scan

```bash
adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.START_SCAN \
  -n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger
```

Add `--es policy exact|balanced|aggressive` to override the default (`balanced`). The receiver
logs what it did and the resulting `ScanState` under the `FotoDupDebug` tag, so progress **and**
the final timings are readable without the screen:

```bash
adb logcat -s FotoDupDebug
# poll the current state at any time:
adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.REPORT_STATE \
  -n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger
```

### Granting photo access

`MANAGE_EXTERNAL_STORAGE` is the one storage permission that *can* be granted from adb, which is
why the **debug** manifest declares it:

```bash
adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE allow
# ...run the scan, then put it back:
adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE default
```

The app's own banner still reads *“Photo access needed”*, because `checkSelfPermission` is
unaffected by an appops grant. That is expected: it only affects the on-screen prompt, not the scan.

### Attaching a debugger for thread stacks

`jdb` attaches over JDWP to the debug build:

```bash
adb forward tcp:8700 jdwp:$(adb shell pidof de.softwareapp_hb.fotodup.finder.debug | tr -d '\r')
printf 'suspend\nwhere all\nquit\n' | jdb -attach localhost:8700
```

Two things learned the hard way:

- `print` needs a **selected frame**. With no current frame even `java.lang.Class` reports
  "Name unknown", so expressions cannot be evaluated until a breakpoint has been reached.
- Leaving the main thread suspended for more than a few seconds **during an activity launch**
  gets the process killed by the launch timeout. Evaluate and `cont` quickly.