Initial commit: Foto Dup Finder 1.0.1

An Android duplicate-photo finder. Pure-Kotlin core (SHA-256 + dHash hashing, exact and
perceptual union-find grouping, three sensitivity presets) with MediaStore access, a
persistent SQLite hash index and a tiered ScanEngine behind a Jetpack Compose UI.
No INTERNET permission, no ads, no analytics, no account.

Ships versionCode 2 / versionName 1.0.1, carrying the scan-stall hardening (BoundedCall
breaker) and verified against a 5,363-photo device library. 80 JVM unit tests, lint clean.

Release paperwork is in docs/ (Play checklist, store listing, edge-case testing), the
upload-ready Play graphics and screenshots in store-assets/, and the generators for those
assets in tools/.
This commit is contained in:
Hadrian Burkhardt
2026-10-13 23:13:39 +02:00
commit 4706e2669c
100 changed files with 8998 additions and 0 deletions
+165
View File
@@ -0,0 +1,165 @@
# Changelog
All notable changes to Foto Dup Finder are documented in this file.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
`versionCode` and `versionName` are defined in `app/build.gradle.kts`. Each Play
upload needs a **higher `versionCode`** than every previous upload.
## [Unreleased]
## [1.0.1] — 2026-10-13 · versionCode 2
### Fixed
- **A stalled media provider could still make a scan look frozen.** The per-call deadline above
stops a *single* stuck file, but not a provider that has stopped answering altogether — on the
test device `MediaProvider` was restarting repeatedly during the scan. Every remaining photo then
waits out the full deadline, so the scan grinds at roughly one photo per timeout and looks hung,
while each abandoned call parks another thread. `data/BoundedCall` now counts abandoned calls:
after three in a row with nothing succeeding in between it treats the provider as **stalled** and
lets the remaining calls fail fast, so the phase reaches its end and the unreadable files are
reported instead of the scan appearing to freeze. The cooldown scales with the call deadline, and
a successful call clears it, so a brief `MediaProvider` restart is picked up again. Pinned by
`BoundedCallTest` (50 blocking calls park a bounded number of threads; the breaker clears once
the provider answers) and by an end-to-end test that a provider read which never returns cannot
freeze the scan.
- **One unresponsive thumbnail could freeze the whole scan.** The perceptual pass asks the media
provider for a thumbnail (`ContentResolver.loadThumbnail`). On the test device that call never
returned for a single, perfectly valid HEIC — the provider hung inside its own decode — and the
scan sat at 99 % of "Looking for visually similar photos…" for ever, at 0 % CPU. The per-photo
`withTimeoutOrNull` could not help: coroutine cancellation is **cooperative**, so it cannot
interrupt a thread parked in a binder transaction. Every provider call (signature read, content
hash, thumbnail/fallback decode) now runs under a hard deadline (`data/BoundedCall`), and a call
that overruns is abandoned — that one photo is skipped and counted among the unreadable ones
instead of stopping the scan. `BoundedCallTest` pins the behaviour.
## [1.0.0] — 2026-10-13 · versionCode 1
First public release.
### Added
- Exact duplicate detection: files are bucketed by size, refined with a cheap
`SHA-256(size + first & last 32 KB)` signature, then confirmed with a full
SHA-256 content hash.
- Near-duplicate detection via a 64-bit perceptual difference hash (`dHash`) on a
9×8 grayscale downscale, clustered by Hamming distance with union-find.
- Three sensitivity presets: Exact copies only / Balanced (default) / Aggressive.
- Tiered, CPU-parallel scan pipeline with a persistent SQLite hash index, so repeat
scans only re-read files that changed.
- Results UI grouped by duplicate set, with thumbnails, a "BEST" marker and
per-group All / None selection.
- "Select all but best" helper that keeps the largest (then newest) copy.
- Safe deletion through the system consent dialog
(`MediaStore.createDeleteRequest`) or the system trash
(`createTrashRequest`); legacy direct delete on Android 10.
- Scans run in a `dataSync` **foreground service**, so a long scan keeps going while
the app is backgrounded, with a progress notification that can cancel it.
- Android 13/14 permission handling, including the partial "Selected photos" mode.
- Material 3 UI with dynamic colour and dark-mode support.
- No ads, no accounts, no analytics and no network permission.
### Changed
- **Localisation groundwork (M8).** Every user-visible string moved out of Kotlin into
`res/values/strings.xml`, with real `<plurals>` where the count can be one ("1 duplicate"
vs "3 duplicates", "1 bit" vs "5 bits"). The screen and the progress notification share a
single enum-to-resource mapping (`text.labelRes()`) so their wording cannot drift apart.
English is currently the only locale — but note that **Play's automatic translation covers
the store listing, not the app UI**, so this is the prerequisite for translating the UI.
- Near-duplicate detection now uses a **DCT perceptual hash (pHash)** plus **per-channel
RGB difference hashes**. A pair must satisfy both. Measured on the benchmark harness,
pHash alone recovered 0.683 of duplicates at zero false positives versus 0.530 for the
previous dHash, and joint accuracy was 0.867 F1 versus 0.738.
- The pHash distance cut is **fitted to each library** (`AdaptiveThreshold`) instead of
being a fixed constant, because unrelated pairs grow with n² and a constant cannot hold
across library sizes.
### Fixed
- **Scan hang at the head/tail read (real root cause).** The byte-reading loops treated only a
negative result as end-of-stream. A non-blocking, pipe-backed file descriptor — which some media
providers hand back — reports **0** ("no bytes yet") instead, so `while (buffer.hasRemaining())`
spun forever and the scan froze mid-phase, pegging a CPU core. `PhotoHasher.readAt`,
`readAtMost` and the streaming SHA-256 now all treat a non-positive read as the end of the data.
Two watchdog tests (daemon thread + join timeout) make a regression fail rather than hang.
- Note on the per-photo timeout added just before: coroutine cancellation is cooperative, so it
bounds a *slow* file but cannot interrupt a call blocked in a native read. The loop guard above
is the actual fix for a stuck descriptor; the timeout stays as a bound on slow work.
- **A slow scan tail looked like a freeze.** Progress was only reported every `1/n` of a phase and
then at completion, so the final ~1 % of the library (up to ~50 photos) produced no update at
all — a stall of a minute looked identical to a hang. Progress now keeps updating through the
last interval.
- **One pathological file could stall a whole scan.** Each per-photo step now runs under a 15 s
budget, so a corrupt entry a provider blocks on, or an unusually expensive HEIC/RAW decode, is
abandoned and reported as unreadable instead of freezing the scan.
- **Hardened file handling (M7).** Every per-file step now sits behind a single failure
boundary, so a corrupt, truncated, zero-byte, RAW or outright hostile file is skipped
instead of aborting the scan or crashing the app. Coroutine cancellation still propagates
through it, a failed index write no longer discards a scan that already completed, and the
number of unreadable files is reported in the results rather than passing silently.
- Decode arithmetic moved to `core/DecodeMath` with division rather than multiplication, so a
crafted image declaring 100 000 x 100 000 cannot overflow the sample-size calculation.
- The per-group button now counts the copies **actually ticked in that row** rather than the
app's suggestion, and acts on exactly what it counts. With BEST + DUP1 + DUP2, unticking
DUP2 now reads "Remove 1 duplicate" instead of still claiming 2. When nothing in a row is
ticked the button says so and is disabled.
- **EXIF orientation** is now applied on the fallback decode path. `BitmapFactory` ignores
EXIF while the media provider's thumbnail does not, so the same photo could hash
differently depending on which path served it — and a thumbnail-cache eviction between
scans could leave a cached hash that no longer matched.
- Candidate-pair enumeration no longer keeps a global set of compared pairs, which for a
20 000-photo library meant millions of boxed `Long`s. Memory is now bounded by the
largest neighbour set.
- The bottom action bar no longer overlaps the system navigation bar (it now applies
`navigationBarsPadding()`); previously the trash/delete button sat underneath the
gesture handle or the 3-button bar on edge-to-edge devices.
- Deleting one group no longer clears a selection made in a different group.
- The *BEST* badge, "select all but best" and the per-group delete all use one rule
(`DuplicateGroup.suggestedKeep()`), so they can no longer disagree.
- Only one system confirmation dialog can be in flight; rapidly tapping a delete button
no longer queues up several approvals to dismiss one after another.
### Added (UI)
- **Two-step removal.** "Remove N duplicates" on a group row (or "Remove all selected" in
the action bar) only *stages* photos: the row disappears from the list immediately and
**nothing is deleted**. A separate **Apply** button performs the real removal, so on
Android 11+ there is **one** system confirmation for the whole batch instead of one
dialog per group. **Discard** undoes all staging.
- The list always shows what is *still* duplicated: staged copies are taken out of their
group, and a group left with fewer than two copies is dropped entirely.
- The copy marked *BEST* is always shown **first** in its group
(`DuplicateGroup.displayOrder()`), so it is visible as a reference even when the
group is collapsed. Previously it could be scrolled out of view.
### Performance
- Exact-duplicate hashing now runs **only** for photos that share a file size with
at least one other photo. In a typical library most photos have a unique size, so
for those no file is opened at all.
- The perceptual hash is taken from the media provider's **thumbnail cache**
(`loadThumbnail`) instead of decoding the full-resolution original, with a heavily
subsampled decode as a fallback.
- The head and tail bytes of a file are read through a **single** file descriptor.
- Hashing runs in small batches, reducing coroutine overhead from one coroutine per
photo to one per batch.
- The hash index is written with a compiled SQLite statement.
- Per-stage timings are shown in the results summary so slow stages are visible.
### Known limitations
- Images only — video files are not scanned.
- `dHash` ignores EXIF rotation, so a physically rotated copy will not match.
- The Aggressive preset may miss a few pairs because of the banded candidate search.
- The first scan of a very large library takes a while; later scans reuse the index.