SoftwareApp
fe28373433
The table said 12 and 18 bits; DuplicatePolicy says 15 and 22, and the device agrees: with a
22-photo library the fit has too little data and falls back to the ceiling, and the results screen
prints "similarity cut: pHash <= 15 bits (default)" - 15, not 12.
The RGB column of the same table already matched the enum (60 and 84), which is what a pHash-only
edit looks like: the ceilings were evidently retuned and this one column was not carried along.
The old 18 was not invented - it is the measured F1 optimum that AGGRESSIVE's doc comment calls out
("set above the measured F1 optimum (~18 on real photographs)"), i.e. the value the ceiling is
deliberately placed above rather than the ceiling itself. Saying so here keeps the corrected row
from reading as though a measured optimum had been changed.
239 lines
11 KiB
Markdown
239 lines
11 KiB
Markdown
# Foto Dup Finder
|
||
|
||
A fast, **ad-free**, **fully offline** Android app that finds and removes duplicate
|
||
and near-duplicate photos. Built as a direct answer to the slow, ad-laden cleanup
|
||
tools that ship with many phones.
|
||
|
||
Nothing ever leaves the device: the app declares no network permission, contains no
|
||
analytics and needs no account.
|
||
|
||
## Features
|
||
|
||
- **Two kinds of duplicates**
|
||
- *Exact* — byte-identical copies, found with a cheap size/head/tail signature
|
||
followed by a full SHA-256.
|
||
- *Near* — resized, recompressed or lightly edited versions, found with a 64-bit
|
||
perceptual difference hash (`dHash`) plus a Hamming-distance threshold.
|
||
- **Tiered, parallel pipeline**: only plausible candidates get a full hash, and the
|
||
work is spread across all CPU cores.
|
||
- **Persistent index** so a re-scan only touches new or changed files.
|
||
- **Safe deletion** through the system consent dialog
|
||
(`MediaStore.createDeleteRequest`) or the system trash (`createTrashRequest`).
|
||
The app never requests `MANAGE_EXTERNAL_STORAGE`.
|
||
- **Two-step cleanup**: removing a row only *stages* it (the row disappears instantly,
|
||
nothing is deleted); one **Apply** then performs the removal, so Android asks for a
|
||
single confirmation for the whole batch instead of one per group.
|
||
- **Sensible defaults**: “select all but best” keeps the largest (then newest) copy
|
||
of each group.
|
||
- **Android 14+ partial photo access** handled explicitly, with a clear prompt to
|
||
upgrade to full access.
|
||
|
||
## Tech stack
|
||
|
||
| Area | Choice |
|
||
|---|---|
|
||
| Language | Kotlin 2.4.10 (using AGP 9 *built-in Kotlin*) |
|
||
| UI | Jetpack Compose, BOM 2026.06.01, Material 3 |
|
||
| Build | Gradle 9.3.1, Android Gradle Plugin 9.1.0 |
|
||
| Image loading | Coil 3.5.0 |
|
||
| Storage | MediaStore read + a small SQLite hash cache |
|
||
| SDK levels | `compileSdk`/`targetSdk` 36, `minSdk` 26 |
|
||
|
||
`targetSdk` is 36 because Google Play requires new apps and updates to target
|
||
Android 16 (API 36) from 31 August 2026.
|
||
|
||
> **Version note:** the Compose BOM is pinned to 2026.06.01 (Compose 1.11.4) and
|
||
> Coil to 3.5.0, because the next releases require `compileSdk 37`, which AGP 9.1.0
|
||
> does not support yet. Bump both together once a newer AGP lands.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
app/src/main/java/de/softwareapp_hb/fotodup/finder/
|
||
├── core/ Pure-Kotlin duplicate engine (no Android imports!)
|
||
│ ├── Hashing.kt SHA-256 + cheap size/head/tail signature
|
||
│ ├── PerceptualHash.kt 64-bit dHash + Hamming distance
|
||
│ ├── DuplicateGrouper.kt exact + perceptual grouping (union-find)
|
||
│ ├── DuplicatePolicy.kt Exact / Balanced / Aggressive thresholds
|
||
│ └── …
|
||
├── data/ MediaStore access, hashing, permissions, SQLite index, deletion
|
||
├── domain/ ScanEngine (tiered pipeline) + progress/result models
|
||
└── ui/ Compose theme, scanner screen, ViewModel, components
|
||
app/src/test/java/… JVM unit tests for the core engine
|
||
```
|
||
|
||
`core/` deliberately has **no Android dependencies**, so the trickiest logic
|
||
(hashing, perceptual hashing, grouping) is covered by fast JVM tests.
|
||
|
||
## Building
|
||
|
||
Prerequisites:
|
||
|
||
- JDK 17+ (JDK 25 verified)
|
||
- Android SDK with `platforms;android-36` and `build-tools;36.0.0`
|
||
|
||
```bash
|
||
# Tell Gradle where the SDK lives (this file is git-ignored)
|
||
echo "sdk.dir=$HOME/Android/Sdk" > local.properties
|
||
|
||
./gradlew :app:assembleDebug # debug APK
|
||
./gradlew :app:testDebugUnitTest # JVM unit tests for the engine
|
||
./gradlew :app:lintDebug # Android Lint
|
||
./gradlew :app:assembleRelease # R8-minified release APK
|
||
./gradlew :app:bundleRelease # Play-ready .aab
|
||
```
|
||
|
||
### Installing on a device
|
||
|
||
```bash
|
||
adb install -r app/build/outputs/apk/debug/app-debug.apk
|
||
```
|
||
|
||
## How detection works
|
||
|
||
```
|
||
MediaStore query
|
||
│
|
||
▼
|
||
1. quick signature = SHA-256(size + first & last 32 KB)
|
||
│ bucket by (size, signature)
|
||
▼
|
||
2. full SHA-256 only for buckets with 2+ members → EXACT duplicates
|
||
│
|
||
▼
|
||
3. perceptual fingerprint, both from ONE decode → NEAR duplicates
|
||
· pHash: 32x32 grayscale → DCT → 8x8 low-frequency block → 64 bits
|
||
· RGB dHash: one 64-bit difference hash per colour channel
|
||
│ candidates via 8-bit band index; a pair must pass BOTH tests
|
||
│ the pHash cut is fitted per scan (see below), then union-find clusters
|
||
4. SQLite hash cache → the next scan only re-reads changed files
|
||
```
|
||
|
||
**Why two descriptors?** pHash sees luminance structure, the RGB hashes see colour
|
||
structure. A single Hamming threshold cannot hold out as a library grows, because
|
||
unrelated pairs accumulate with the square of the library size. Measured on the
|
||
benchmark harness, the widest pHash threshold with *zero* false pairs fell from 13
|
||
(120 images) to 5 (2 880 images) — so the app fits the cut to the library instead of
|
||
trusting a constant, and requires the colour structure to agree as well.
|
||
|
||
Sensitivity presets (both values are ceilings; the pHash one is adapted per scan):
|
||
|
||
| Policy | pHash ceiling | RGB ceiling | Notes |
|
||
|---|---|---|---|
|
||
| Exact copies only | – | – | byte-identical files, no decoding at all |
|
||
| Balanced (default) | 15 bits | 60 / 192 | the recommended setting |
|
||
| Aggressive | 22 bits | 84 / 192 | catches heavier re-edits |
|
||
|
||
Banding gives *exact* candidate recall for pHash distances up to 7 (see the comment in
|
||
`DuplicateGrouper`).
|
||
|
||
## Permissions
|
||
|
||
| Permission | Why it is needed |
|
||
|---|---|
|
||
| `READ_MEDIA_IMAGES` (API 33+) | read the photo library |
|
||
| `READ_EXTERNAL_STORAGE` (API ≤ 32) | same, on older devices |
|
||
| `READ_MEDIA_VISUAL_USER_SELECTED` (API 34+) | Android 14 "Selected photos" mode |
|
||
| `WRITE_EXTERNAL_STORAGE` (**API ≤ 29 only**) | legacy delete path on Android 10 |
|
||
| `POST_NOTIFICATIONS` (API 33+) | shows the scan-progress notification (optional — the scan runs without it) |
|
||
| `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_DATA_SYNC` | keeps a long scan alive while the app is backgrounded |
|
||
|
||
Deletion on Android 11+ always goes through the system consent dialog. The app
|
||
never requests `MANAGE_EXTERNAL_STORAGE`.
|
||
|
||
### Scanning in the background
|
||
|
||
A scan is owned by `ScanController` (process-scoped, created by the Application), not
|
||
by the screen's ViewModel. While one is running, `ScanService` holds a `dataSync`
|
||
foreground service with a progress notification that has a **Cancel** action. So:
|
||
|
||
- leaving the app does not kill the scan,
|
||
- the result survives screen rotation and re-entry,
|
||
- the user always has a visible way to stop it.
|
||
|
||
When `POST_NOTIFICATIONS` is denied the scan still works; the notification is simply
|
||
not shown.
|
||
|
||
## Testing
|
||
|
||
`app/src/test` holds JVM unit tests for `core/`: SHA-256 test vectors, dHash bit
|
||
patterns, Hamming distance, exact/perceptual grouping, aspect-ratio filtering and
|
||
the reclaimable-bytes maths.
|
||
|
||
```bash
|
||
./gradlew :app:testDebugUnitTest
|
||
```
|
||
|
||
There is no instrumentation test, so UI behaviour is verified by installing the debug build on a
|
||
device or an emulator (`adb install -r app/build/outputs/apk/debug/app-debug.apk`) and driving it by
|
||
hand or with `adb shell input tap`. That is how the store screenshots are taken; the tablet sets were
|
||
shot on AOSP AVDs running in **software emulation**, because this machine exposes no `/dev/kvm`
|
||
(see [docs/STORE_LISTING.md](docs/STORE_LISTING.md)).
|
||
|
||
## Releasing to Google Play
|
||
|
||
See [docs/PLAY_RELEASE_CHECKLIST.md](docs/PLAY_RELEASE_CHECKLIST.md) (signing, store
|
||
listing, Data Safety, release notes, rollout), [docs/STORE_LISTING.md](docs/STORE_LISTING.md)
|
||
(ready-to-paste listing copy plus the icon / feature graphic generators) and
|
||
[PRIVACY.md](PRIVACY.md).
|
||
|
||
The app is published as a **one-time paid app** — no ads, no subscription, no in-app purchases, and
|
||
therefore no Billing code or `com.android.vending.BILLING` permission in the APK: the price lives on
|
||
the listing, not in the build ([why](docs/STORE_LISTING.md#app-pricing)).
|
||
|
||
The upload-ready graphics and screenshots live in [`store-assets/`](store-assets/).
|
||
`tools/make-store-assets.sh` and `tools/make-store-screenshots.sh` regenerate them from the
|
||
app's own icon vectors and from `adb` captures — always inside the repo, never in `$HOME`.
|
||
Screenshots are kept per form factor (`store-assets/{raw-,}screenshots/{phone,tablet-7in,tablet-10in}/`)
|
||
because Play gives phones and each tablet size a slot of their own: `FORM_FACTOR=tablet-7in`
|
||
selects the 7-inch pipeline, and the tablet runs deliberately crop no system chrome because a
|
||
tablet frame is already inside Play's 2:1 cap. All three sets are captured — the phone set on a real
|
||
device, the tablet sets on AOSP AVDs; see [docs/STORE_LISTING.md](docs/STORE_LISTING.md).
|
||
`python3 tools/make-demo-photos.py` draws the synthetic photo library the results screenshot is
|
||
captured over, so the listing never shows a real one. The finished assets are **committed** —
|
||
including the foreground-service demonstration video that [the declaration](docs/PLAY_RELEASE_CHECKLIST.md)
|
||
links to — and Gitea serves them without a login, so pushing is the deployment step:
|
||
`…/src/branch/main/store-assets/…` is the page a browser opens (and the shape Play's video link
|
||
takes, because a `/raw/` URL carries a `default-src 'none'` CSP that stops a video playing inline),
|
||
`…/raw/branch/main/…` is the bytes.
|
||
|
||
User-visible changes are tracked in [CHANGELOG.md](CHANGELOG.md); the Play "What's new"
|
||
text you type per release is separate and lives only in the Play Console.
|
||
|
||
## Localisation
|
||
|
||
All user-visible text lives in `app/src/main/res/values/strings.xml`, with proper `<plurals>`
|
||
where a count can be one, and both the screen and the progress notification resolve their
|
||
labels through `text/TextRes.kt`. **English is currently the only locale.**
|
||
|
||
Adding a language is then a file, not a refactor:
|
||
|
||
```
|
||
app/src/main/res/values-de/strings.xml # same keys, translated values
|
||
```
|
||
|
||
One caveat worth knowing: **Google Play's automatic translation translates your store
|
||
listing, not the app UI.** Translated UI text has to be supplied in `values-<locale>/`, by a
|
||
translator or by running the strings file through a translation service and reviewing it.
|
||
|
||
Byte units ("KB", "MB") and the duration formatter are intentionally left unlocalised.
|
||
|
||
## Known limitations
|
||
|
||
- Only **images** are scanned — videos are not supported yet.
|
||
- Near-duplicate detection ignores EXIF rotation, so an image saved physically
|
||
rotated by 90° will not match its original.
|
||
- The Aggressive policy may miss some pairs because of the banded candidate search.
|
||
- The very first scan of a huge library (50k+ photos) takes a while; later scans
|
||
reuse the hash index and are much faster.
|
||
|
||
## Roadmap
|
||
|
||
The project is tracked against a milestone plan (M0–M10). What is done, what remains,
|
||
the current release blockers and the candidate features all live in
|
||
[docs/ROADMAP.md](docs/ROADMAP.md).
|
||
|
||
## Licence
|
||
|
||
Proprietary — all rights reserved. Change this if you intend to open-source it.
|