foto-duplicate-finder/README.md

# Foto Dup Finder

A fast, **ad-free**, **fully offline** Android app that finds and removes duplicate
and near-duplicate photos. Built as a direct answer to the slow, ad-laden cleanup
tools that ship with many phones.

Nothing ever leaves the device: the app declares no network permission, contains no
analytics and needs no account.

## Features

- **Two kinds of duplicates**
  - *Exact* — byte-identical copies, found with a cheap size/head/tail signature
    followed by a full SHA-256.
  - *Near* — resized, recompressed or lightly edited versions, found with a 64-bit
    perceptual difference hash (`dHash`) plus a Hamming-distance threshold.
- **Tiered, parallel pipeline**: only plausible candidates get a full hash, and the
  work is spread across all CPU cores.
- **Persistent index** so a re-scan only touches new or changed files.
- **Safe deletion** through the system consent dialog
  (`MediaStore.createDeleteRequest`) or the system trash (`createTrashRequest`).
  The app never requests `MANAGE_EXTERNAL_STORAGE`.
- **Two-step cleanup**: removing a row only *stages* it (the row disappears instantly,
  nothing is deleted); one **Apply** then performs the removal, so Android asks for a
  single confirmation for the whole batch instead of one per group.
- **Sensible defaults**: “select all but best” keeps the largest (then newest) copy
  of each group.
- **Android 14+ partial photo access** handled explicitly, with a clear prompt to
  upgrade to full access.

## Tech stack

| Area | Choice |
|---|---|
| Language | Kotlin 2.4.10 (using AGP 9 *built-in Kotlin*) |
| UI | Jetpack Compose, BOM 2026.06.01, Material 3 |
| Build | Gradle 9.3.1, Android Gradle Plugin 9.1.0 |
| Image loading | Coil 3.5.0 |
| Storage | MediaStore read + a small SQLite hash cache |
| SDK levels | `compileSdk`/`targetSdk` 36, `minSdk` 26 |

`targetSdk` is 36 because Google Play requires new apps and updates to target
Android 16 (API 36) from 31 August 2026.

> **Version note:** the Compose BOM is pinned to 2026.06.01 (Compose 1.11.4) and
> Coil to 3.5.0, because the next releases require `compileSdk 37`, which AGP 9.1.0
> does not support yet. Bump both together once a newer AGP lands.

## Project layout

```
app/src/main/java/de/softwareapp_hb/fotodup/finder/
├── core/      Pure-Kotlin duplicate engine (no Android imports!)
│   ├── Hashing.kt            SHA-256 + cheap size/head/tail signature
│   ├── PerceptualHash.kt     64-bit dHash + Hamming distance
│   ├── DuplicateGrouper.kt   exact + perceptual grouping (union-find)
│   ├── DuplicatePolicy.kt    Exact / Balanced / Aggressive thresholds
│   └── …
├── data/      MediaStore access, hashing, permissions, SQLite index, deletion
├── domain/    ScanEngine (tiered pipeline) + progress/result models
└── ui/        Compose theme, scanner screen, ViewModel, components
app/src/test/java/…            JVM unit tests for the core engine
```

`core/` deliberately has **no Android dependencies**, so the trickiest logic
(hashing, perceptual hashing, grouping) is covered by fast JVM tests.

## Building

Prerequisites:

- JDK 17+ (JDK 25 verified)
- Android SDK with `platforms;android-36` and `build-tools;36.0.0`

```bash
# Tell Gradle where the SDK lives (this file is git-ignored)
echo "sdk.dir=$HOME/Android/Sdk" > local.properties

./gradlew :app:assembleDebug        # debug APK
./gradlew :app:testDebugUnitTest    # JVM unit tests for the engine
./gradlew :app:lintDebug            # Android Lint
./gradlew :app:assembleRelease      # R8-minified release APK
./gradlew :app:bundleRelease        # Play-ready .aab
```

### Installing on a device

```bash
adb install -r app/build/outputs/apk/debug/app-debug.apk
```

## How detection works

```
MediaStore query
      │
      ▼
1. quick signature = SHA-256(size + first & last 32 KB)
      │   bucket by (size, signature)
      ▼
2. full SHA-256    only for buckets with 2+ members      → EXACT duplicates
      │
      ▼
3. perceptual fingerprint, both from ONE decode             → NEAR duplicates
      · pHash: 32x32 grayscale → DCT → 8x8 low-frequency block → 64 bits
      · RGB dHash: one 64-bit difference hash per colour channel
      │   candidates via 8-bit band index; a pair must pass BOTH tests
      │   the pHash cut is fitted per scan (see below), then union-find clusters
4. SQLite hash cache → the next scan only re-reads changed files
```

**Why two descriptors?** pHash sees luminance structure, the RGB hashes see colour
structure. A single Hamming threshold cannot hold out as a library grows, because
unrelated pairs accumulate with the square of the library size. Measured on the
benchmark harness, the widest pHash threshold with *zero* false pairs fell from 13
(120 images) to 5 (2 880 images) — so the app fits the cut to the library instead of
trusting a constant, and requires the colour structure to agree as well.

Sensitivity presets (both values are ceilings; the pHash one is adapted per scan):

| Policy | pHash ceiling | RGB ceiling | Notes |
|---|---|---|---|
| Exact copies only | – | – | byte-identical files, no decoding at all |
| Balanced (default) | 12 bits | 60 / 192 | the recommended setting |
| Aggressive | 18 bits | 84 / 192 | catches heavier re-edits |

Banding gives *exact* candidate recall for pHash distances up to 7 (see the comment in
`DuplicateGrouper`).

## Permissions

| Permission | Why it is needed |
|---|---|
| `READ_MEDIA_IMAGES` (API 33+) | read the photo library |
| `READ_EXTERNAL_STORAGE` (API ≤ 32) | same, on older devices |
| `READ_MEDIA_VISUAL_USER_SELECTED` (API 34+) | Android 14 "Selected photos" mode |
| `WRITE_EXTERNAL_STORAGE` (**API ≤ 29 only**) | legacy delete path on Android 10 |
| `POST_NOTIFICATIONS` (API 33+) | shows the scan-progress notification (optional — the scan runs without it) |
| `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_DATA_SYNC` | keeps a long scan alive while the app is backgrounded |

Deletion on Android 11+ always goes through the system consent dialog. The app
never requests `MANAGE_EXTERNAL_STORAGE`.

### Scanning in the background

A scan is owned by `ScanController` (process-scoped, created by the Application), not
by the screen's ViewModel. While one is running, `ScanService` holds a `dataSync`
foreground service with a progress notification that has a **Cancel** action. So:

- leaving the app does not kill the scan,
- the result survives screen rotation and re-entry,
- the user always has a visible way to stop it.

When `POST_NOTIFICATIONS` is denied the scan still works; the notification is simply
not shown.

## Testing

`app/src/test` holds JVM unit tests for `core/`: SHA-256 test vectors, dHash bit
patterns, Hamming distance, exact/perceptual grouping, aspect-ratio filtering and
the reclaimable-bytes maths.

```bash
./gradlew :app:testDebugUnitTest
```

There is no instrumentation test, so UI behaviour is verified by installing the debug build on a
device or an emulator (`adb install -r app/build/outputs/apk/debug/app-debug.apk`) and driving it by
hand or with `adb shell input tap`. That is how the store screenshots are taken; the tablet sets were
shot on AOSP AVDs running in **software emulation**, because this machine exposes no `/dev/kvm`
(see [docs/STORE_LISTING.md](docs/STORE_LISTING.md)).

## Releasing to Google Play

See [docs/PLAY_RELEASE_CHECKLIST.md](docs/PLAY_RELEASE_CHECKLIST.md) (signing, store
listing, Data Safety, release notes, rollout), [docs/STORE_LISTING.md](docs/STORE_LISTING.md)
(ready-to-paste listing copy plus the icon / feature graphic generators) and
[PRIVACY.md](PRIVACY.md).

The app is published as a **one-time paid app** — no ads, no subscription, no in-app purchases, and
therefore no Billing code or `com.android.vending.BILLING` permission in the APK: the price lives on
the listing, not in the build ([why](docs/STORE_LISTING.md#app-pricing)).

The upload-ready graphics and screenshots live in [`store-assets/`](store-assets/).
`tools/make-store-assets.sh` and `tools/make-store-screenshots.sh` regenerate them from the
app's own icon vectors and from `adb` captures — always inside the repo, never in `$HOME`.
Screenshots are kept per form factor (`store-assets/{raw-,}screenshots/{phone,tablet-7in,tablet-10in}/`)
because Play gives phones and each tablet size a slot of their own: `FORM_FACTOR=tablet-7in`
selects the 7-inch pipeline, and the tablet runs deliberately crop no system chrome because a
tablet frame is already inside Play's 2:1 cap. All three sets are captured — the phone set on a real
device, the tablet sets on AOSP AVDs; see [docs/STORE_LISTING.md](docs/STORE_LISTING.md).
`python3 tools/make-demo-photos.py` draws the synthetic photo library the results screenshot is
captured over, so the listing never shows a real one. The finished assets are **committed** —
including the foreground-service demonstration video that [the declaration](docs/PLAY_RELEASE_CHECKLIST.md)
links to — and Gitea serves them without a login (`…/raw/branch/main/store-assets/…`), so pushing
is the deployment step and that URL is the link itself.

User-visible changes are tracked in [CHANGELOG.md](CHANGELOG.md); the Play "What's new"
text you type per release is separate and lives only in the Play Console.

## Localisation

All user-visible text lives in `app/src/main/res/values/strings.xml`, with proper `<plurals>`
where a count can be one, and both the screen and the progress notification resolve their
labels through `text/TextRes.kt`. **English is currently the only locale.**

Adding a language is then a file, not a refactor:

```
app/src/main/res/values-de/strings.xml   # same keys, translated values
```

One caveat worth knowing: **Google Play's automatic translation translates your store
listing, not the app UI.** Translated UI text has to be supplied in `values-<locale>/`, by a
translator or by running the strings file through a translation service and reviewing it.

Byte units ("KB", "MB") and the duration formatter are intentionally left unlocalised.

## Known limitations

- Only **images** are scanned — videos are not supported yet.
- Near-duplicate detection ignores EXIF rotation, so an image saved physically
  rotated by 90° will not match its original.
- The Aggressive policy may miss some pairs because of the banded candidate search.
- The very first scan of a huge library (50k+ photos) takes a while; later scans
  reuse the hash index and are much faster.

## Roadmap

The project is tracked against a milestone plan (M0–M10). What is done, what remains,
the current release blockers and the candidate features all live in
[docs/ROADMAP.md](docs/ROADMAP.md).

## Licence

Proprietary — all rights reserved. Change this if you intend to open-source it.