commit 4706e2669c0c9316775bdc6c3e7ddf8ac3e2f463 Author: Hadrian Burkhardt Date: Tue Oct 13 23:13:39 2026 +0200 Initial commit: Foto Dup Finder 1.0.1 An Android duplicate-photo finder. Pure-Kotlin core (SHA-256 + dHash hashing, exact and perceptual union-find grouping, three sensitivity presets) with MediaStore access, a persistent SQLite hash index and a tiered ScanEngine behind a Jetpack Compose UI. No INTERNET permission, no ads, no analytics, no account. Ships versionCode 2 / versionName 1.0.1, carrying the scan-stall hardening (BoundedCall breaker) and verified against a 5,363-photo device library. 80 JVM unit tests, lint clean. Release paperwork is in docs/ (Play checklist, store listing, edge-case testing), the upload-ready Play graphics and screenshots in store-assets/, and the generators for those assets in tools/. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7c61607 --- /dev/null +++ b/.gitignore @@ -0,0 +1,27 @@ +# Android / Gradle +*.iml +.gradle/ +.kotlin/ +/local.properties +/.idea/ +.DS_Store +build/ +/captures +.externalNativeBuild +.cxx + +# Build outputs +*.apk +*.aab +*.ap_ +*.dex + +# Signing material — never commit keys! +*.keystore +*.jks +keystore.properties +!debug.keystore + +# Generated data — regenerated on demand, never committed +fixtures/ +store-assets/raw-screenshots/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..705f6bf --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,165 @@ +# Changelog + +All notable changes to Foto Dup Finder are documented in this file. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the +project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +`versionCode` and `versionName` are defined in `app/build.gradle.kts`. Each Play +upload needs a **higher `versionCode`** than every previous upload. + +## [Unreleased] + +## [1.0.1] — 2026-10-13 · versionCode 2 + +### Fixed + +- **A stalled media provider could still make a scan look frozen.** The per-call deadline above + stops a *single* stuck file, but not a provider that has stopped answering altogether — on the + test device `MediaProvider` was restarting repeatedly during the scan. Every remaining photo then + waits out the full deadline, so the scan grinds at roughly one photo per timeout and looks hung, + while each abandoned call parks another thread. `data/BoundedCall` now counts abandoned calls: + after three in a row with nothing succeeding in between it treats the provider as **stalled** and + lets the remaining calls fail fast, so the phase reaches its end and the unreadable files are + reported instead of the scan appearing to freeze. The cooldown scales with the call deadline, and + a successful call clears it, so a brief `MediaProvider` restart is picked up again. Pinned by + `BoundedCallTest` (50 blocking calls park a bounded number of threads; the breaker clears once + the provider answers) and by an end-to-end test that a provider read which never returns cannot + freeze the scan. + +- **One unresponsive thumbnail could freeze the whole scan.** The perceptual pass asks the media + provider for a thumbnail (`ContentResolver.loadThumbnail`). On the test device that call never + returned for a single, perfectly valid HEIC — the provider hung inside its own decode — and the + scan sat at 99 % of "Looking for visually similar photos…" for ever, at 0 % CPU. The per-photo + `withTimeoutOrNull` could not help: coroutine cancellation is **cooperative**, so it cannot + interrupt a thread parked in a binder transaction. Every provider call (signature read, content + hash, thumbnail/fallback decode) now runs under a hard deadline (`data/BoundedCall`), and a call + that overruns is abandoned — that one photo is skipped and counted among the unreadable ones + instead of stopping the scan. `BoundedCallTest` pins the behaviour. + +## [1.0.0] — 2026-10-13 · versionCode 1 + +First public release. + +### Added + +- Exact duplicate detection: files are bucketed by size, refined with a cheap + `SHA-256(size + first & last 32 KB)` signature, then confirmed with a full + SHA-256 content hash. +- Near-duplicate detection via a 64-bit perceptual difference hash (`dHash`) on a + 9×8 grayscale downscale, clustered by Hamming distance with union-find. +- Three sensitivity presets: Exact copies only / Balanced (default) / Aggressive. +- Tiered, CPU-parallel scan pipeline with a persistent SQLite hash index, so repeat + scans only re-read files that changed. +- Results UI grouped by duplicate set, with thumbnails, a "BEST" marker and + per-group All / None selection. +- "Select all but best" helper that keeps the largest (then newest) copy. +- Safe deletion through the system consent dialog + (`MediaStore.createDeleteRequest`) or the system trash + (`createTrashRequest`); legacy direct delete on Android 10. +- Scans run in a `dataSync` **foreground service**, so a long scan keeps going while + the app is backgrounded, with a progress notification that can cancel it. +- Android 13/14 permission handling, including the partial "Selected photos" mode. +- Material 3 UI with dynamic colour and dark-mode support. +- No ads, no accounts, no analytics and no network permission. + +### Changed + +- **Localisation groundwork (M8).** Every user-visible string moved out of Kotlin into + `res/values/strings.xml`, with real `` where the count can be one ("1 duplicate" + vs "3 duplicates", "1 bit" vs "5 bits"). The screen and the progress notification share a + single enum-to-resource mapping (`text.labelRes()`) so their wording cannot drift apart. + English is currently the only locale — but note that **Play's automatic translation covers + the store listing, not the app UI**, so this is the prerequisite for translating the UI. + +- Near-duplicate detection now uses a **DCT perceptual hash (pHash)** plus **per-channel + RGB difference hashes**. A pair must satisfy both. Measured on the benchmark harness, + pHash alone recovered 0.683 of duplicates at zero false positives versus 0.530 for the + previous dHash, and joint accuracy was 0.867 F1 versus 0.738. +- The pHash distance cut is **fitted to each library** (`AdaptiveThreshold`) instead of + being a fixed constant, because unrelated pairs grow with n² and a constant cannot hold + across library sizes. + +### Fixed + +- **Scan hang at the head/tail read (real root cause).** The byte-reading loops treated only a + negative result as end-of-stream. A non-blocking, pipe-backed file descriptor — which some media + providers hand back — reports **0** ("no bytes yet") instead, so `while (buffer.hasRemaining())` + spun forever and the scan froze mid-phase, pegging a CPU core. `PhotoHasher.readAt`, + `readAtMost` and the streaming SHA-256 now all treat a non-positive read as the end of the data. + Two watchdog tests (daemon thread + join timeout) make a regression fail rather than hang. +- Note on the per-photo timeout added just before: coroutine cancellation is cooperative, so it + bounds a *slow* file but cannot interrupt a call blocked in a native read. The loop guard above + is the actual fix for a stuck descriptor; the timeout stays as a bound on slow work. + +- **A slow scan tail looked like a freeze.** Progress was only reported every `1/n` of a phase and + then at completion, so the final ~1 % of the library (up to ~50 photos) produced no update at + all — a stall of a minute looked identical to a hang. Progress now keeps updating through the + last interval. +- **One pathological file could stall a whole scan.** Each per-photo step now runs under a 15 s + budget, so a corrupt entry a provider blocks on, or an unusually expensive HEIC/RAW decode, is + abandoned and reported as unreadable instead of freezing the scan. + +- **Hardened file handling (M7).** Every per-file step now sits behind a single failure + boundary, so a corrupt, truncated, zero-byte, RAW or outright hostile file is skipped + instead of aborting the scan or crashing the app. Coroutine cancellation still propagates + through it, a failed index write no longer discards a scan that already completed, and the + number of unreadable files is reported in the results rather than passing silently. +- Decode arithmetic moved to `core/DecodeMath` with division rather than multiplication, so a + crafted image declaring 100 000 x 100 000 cannot overflow the sample-size calculation. + +- The per-group button now counts the copies **actually ticked in that row** rather than the + app's suggestion, and acts on exactly what it counts. With BEST + DUP1 + DUP2, unticking + DUP2 now reads "Remove 1 duplicate" instead of still claiming 2. When nothing in a row is + ticked the button says so and is disabled. + +- **EXIF orientation** is now applied on the fallback decode path. `BitmapFactory` ignores + EXIF while the media provider's thumbnail does not, so the same photo could hash + differently depending on which path served it — and a thumbnail-cache eviction between + scans could leave a cached hash that no longer matched. +- Candidate-pair enumeration no longer keeps a global set of compared pairs, which for a + 20 000-photo library meant millions of boxed `Long`s. Memory is now bounded by the + largest neighbour set. + +- The bottom action bar no longer overlaps the system navigation bar (it now applies + `navigationBarsPadding()`); previously the trash/delete button sat underneath the + gesture handle or the 3-button bar on edge-to-edge devices. +- Deleting one group no longer clears a selection made in a different group. +- The *BEST* badge, "select all but best" and the per-group delete all use one rule + (`DuplicateGroup.suggestedKeep()`), so they can no longer disagree. +- Only one system confirmation dialog can be in flight; rapidly tapping a delete button + no longer queues up several approvals to dismiss one after another. + +### Added (UI) + +- **Two-step removal.** "Remove N duplicates" on a group row (or "Remove all selected" in + the action bar) only *stages* photos: the row disappears from the list immediately and + **nothing is deleted**. A separate **Apply** button performs the real removal, so on + Android 11+ there is **one** system confirmation for the whole batch instead of one + dialog per group. **Discard** undoes all staging. +- The list always shows what is *still* duplicated: staged copies are taken out of their + group, and a group left with fewer than two copies is dropped entirely. +- The copy marked *BEST* is always shown **first** in its group + (`DuplicateGroup.displayOrder()`), so it is visible as a reference even when the + group is collapsed. Previously it could be scrolled out of view. + +### Performance + +- Exact-duplicate hashing now runs **only** for photos that share a file size with + at least one other photo. In a typical library most photos have a unique size, so + for those no file is opened at all. +- The perceptual hash is taken from the media provider's **thumbnail cache** + (`loadThumbnail`) instead of decoding the full-resolution original, with a heavily + subsampled decode as a fallback. +- The head and tail bytes of a file are read through a **single** file descriptor. +- Hashing runs in small batches, reducing coroutine overhead from one coroutine per + photo to one per batch. +- The hash index is written with a compiled SQLite statement. +- Per-stage timings are shown in the results summary so slow stages are visible. + +### Known limitations + +- Images only — video files are not scanned. +- `dHash` ignores EXIF rotation, so a physically rotated copy will not match. +- The Aggressive preset may miss a few pairs because of the banded candidate search. +- The first scan of a very large library takes a while; later scans reuse the index. diff --git a/PRIVACY.md b/PRIVACY.md new file mode 100644 index 0000000..8c9caac --- /dev/null +++ b/PRIVACY.md @@ -0,0 +1,67 @@ +# Privacy Policy — Foto Dup Finder + +_Last updated: 2026-10-13_ + +## Summary + +Foto Dup Finder finds and removes duplicate photos **entirely on your device**. +The app does **not** collect, transmit, sell or share any of your data. It has no +network access at all. + +## What the app accesses + +| Data | Why | Where it goes | +|---|---|---| +| Your photos (images) | To compare them and detect duplicates | Never leaves your device | +| Photo metadata (file name, size, folder, dimensions, date) | To group and display results | Never leaves your device | +| A local cache of file hashes | To make repeat scans fast | Stored only in this app's private storage | + +## What the app does NOT do + +- It does **not** have the `INTERNET` permission, so it cannot send data anywhere. +- It does **not** collect analytics, crash reports, advertising identifiers or + location data. +- It does **not** contain advertising SDKs or third-party trackers. +- It does **not** create accounts and does **not** ask for your name, email or + phone number. +- It does **not** read photos other than the images you grant access to. + +## Permissions and what they mean + +| Permission | Purpose | +|---|---| +| Read photos (`READ_MEDIA_IMAGES` / `READ_EXTERNAL_STORAGE`) | Required to scan your photo library. You can deny this and the app simply will not scan. On Android 14+ you may grant access to *selected photos only*; the app then scans only those. | +| `READ_MEDIA_VISUAL_USER_SELECTED` | Supports Android 14's partial photo access. | +| `WRITE_EXTERNAL_STORAGE` (Android 10 only) | Used solely to delete photos you explicitly select on that older Android version. | + +## Deleting photos + +On Android 11 and newer, deletions are performed through the operating system's +own confirmation dialog (`MediaStore.createDeleteRequest` / `createTrashRequest`). +The app can never delete anything without your explicit confirmation, and it never +requests the broad `MANAGE_EXTERNAL_STORAGE` permission. + +On Android 10, deletions use the standard media API and still only affect the files +you selected. + +## Data retention and deletion + +The app stores a small local database of file hashes in its private app storage so +that repeat scans are fast. This database contains no image content and never leaves +your device. Uninstalling the app, or clearing its storage in Android Settings, +removes it completely. + +## Children + +The app is a general-purpose utility and is not directed at children. It collects no +personal data from anyone. + +## Changes + +If this policy changes, the updated version will be published at this address with a +new "last updated" date. + +## Contact + +Questions about this policy: **support@example.com** _(replace with your real +support address before publishing)_. diff --git a/README.md b/README.md new file mode 100644 index 0000000..33155db --- /dev/null +++ b/README.md @@ -0,0 +1,219 @@ +# Foto Dup Finder + +A fast, **ad-free**, **fully offline** Android app that finds and removes duplicate +and near-duplicate photos. Built as a direct answer to the slow, ad-laden cleanup +tools that ship with many phones. + +Nothing ever leaves the device: the app declares no network permission, contains no +analytics and needs no account. + +## Features + +- **Two kinds of duplicates** + - *Exact* — byte-identical copies, found with a cheap size/head/tail signature + followed by a full SHA-256. + - *Near* — resized, recompressed or lightly edited versions, found with a 64-bit + perceptual difference hash (`dHash`) plus a Hamming-distance threshold. +- **Tiered, parallel pipeline**: only plausible candidates get a full hash, and the + work is spread across all CPU cores. +- **Persistent index** so a re-scan only touches new or changed files. +- **Safe deletion** through the system consent dialog + (`MediaStore.createDeleteRequest`) or the system trash (`createTrashRequest`). + The app never requests `MANAGE_EXTERNAL_STORAGE`. +- **Two-step cleanup**: removing a row only *stages* it (the row disappears instantly, + nothing is deleted); one **Apply** then performs the removal, so Android asks for a + single confirmation for the whole batch instead of one per group. +- **Sensible defaults**: “select all but best” keeps the largest (then newest) copy + of each group. +- **Android 14+ partial photo access** handled explicitly, with a clear prompt to + upgrade to full access. + +## Tech stack + +| Area | Choice | +|---|---| +| Language | Kotlin 2.4.10 (using AGP 9 *built-in Kotlin*) | +| UI | Jetpack Compose, BOM 2026.06.01, Material 3 | +| Build | Gradle 9.3.1, Android Gradle Plugin 9.1.0 | +| Image loading | Coil 3.5.0 | +| Storage | MediaStore read + a small SQLite hash cache | +| SDK levels | `compileSdk`/`targetSdk` 36, `minSdk` 26 | + +`targetSdk` is 36 because Google Play requires new apps and updates to target +Android 16 (API 36) from 31 August 2026. + +> **Version note:** the Compose BOM is pinned to 2026.06.01 (Compose 1.11.4) and +> Coil to 3.5.0, because the next releases require `compileSdk 37`, which AGP 9.1.0 +> does not support yet. Bump both together once a newer AGP lands. + +## Project layout + +``` +app/src/main/java/de/softwareapp_hb/fotodup/finder/ +├── core/ Pure-Kotlin duplicate engine (no Android imports!) +│ ├── Hashing.kt SHA-256 + cheap size/head/tail signature +│ ├── PerceptualHash.kt 64-bit dHash + Hamming distance +│ ├── DuplicateGrouper.kt exact + perceptual grouping (union-find) +│ ├── DuplicatePolicy.kt Exact / Balanced / Aggressive thresholds +│ └── … +├── data/ MediaStore access, hashing, permissions, SQLite index, deletion +├── domain/ ScanEngine (tiered pipeline) + progress/result models +└── ui/ Compose theme, scanner screen, ViewModel, components +app/src/test/java/… JVM unit tests for the core engine +``` + +`core/` deliberately has **no Android dependencies**, so the trickiest logic +(hashing, perceptual hashing, grouping) is covered by fast JVM tests. + +## Building + +Prerequisites: + +- JDK 17+ (JDK 25 verified) +- Android SDK with `platforms;android-36` and `build-tools;36.0.0` + +```bash +# Tell Gradle where the SDK lives (this file is git-ignored) +echo "sdk.dir=$HOME/Android/Sdk" > local.properties + +./gradlew :app:assembleDebug # debug APK +./gradlew :app:testDebugUnitTest # JVM unit tests for the engine +./gradlew :app:lintDebug # Android Lint +./gradlew :app:assembleRelease # R8-minified release APK +./gradlew :app:bundleRelease # Play-ready .aab +``` + +### Installing on a device + +```bash +adb install -r app/build/outputs/apk/debug/app-debug.apk +``` + +## How detection works + +``` +MediaStore query + │ + ▼ +1. quick signature = SHA-256(size + first & last 32 KB) + │ bucket by (size, signature) + ▼ +2. full SHA-256 only for buckets with 2+ members → EXACT duplicates + │ + ▼ +3. perceptual fingerprint, both from ONE decode → NEAR duplicates + · pHash: 32x32 grayscale → DCT → 8x8 low-frequency block → 64 bits + · RGB dHash: one 64-bit difference hash per colour channel + │ candidates via 8-bit band index; a pair must pass BOTH tests + │ the pHash cut is fitted per scan (see below), then union-find clusters +4. SQLite hash cache → the next scan only re-reads changed files +``` + +**Why two descriptors?** pHash sees luminance structure, the RGB hashes see colour +structure. A single Hamming threshold cannot hold out as a library grows, because +unrelated pairs accumulate with the square of the library size. Measured on the +benchmark harness, the widest pHash threshold with *zero* false pairs fell from 13 +(120 images) to 5 (2 880 images) — so the app fits the cut to the library instead of +trusting a constant, and requires the colour structure to agree as well. + +Sensitivity presets (both values are ceilings; the pHash one is adapted per scan): + +| Policy | pHash ceiling | RGB ceiling | Notes | +|---|---|---|---| +| Exact copies only | – | – | byte-identical files, no decoding at all | +| Balanced (default) | 12 bits | 60 / 192 | the recommended setting | +| Aggressive | 18 bits | 84 / 192 | catches heavier re-edits | + +Banding gives *exact* candidate recall for pHash distances up to 7 (see the comment in +`DuplicateGrouper`). + +## Permissions + +| Permission | Why it is needed | +|---|---| +| `READ_MEDIA_IMAGES` (API 33+) | read the photo library | +| `READ_EXTERNAL_STORAGE` (API ≤ 32) | same, on older devices | +| `READ_MEDIA_VISUAL_USER_SELECTED` (API 34+) | Android 14 "Selected photos" mode | +| `WRITE_EXTERNAL_STORAGE` (**API ≤ 29 only**) | legacy delete path on Android 10 | +| `POST_NOTIFICATIONS` (API 33+) | shows the scan-progress notification (optional — the scan runs without it) | +| `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_DATA_SYNC` | keeps a long scan alive while the app is backgrounded | + +Deletion on Android 11+ always goes through the system consent dialog. The app +never requests `MANAGE_EXTERNAL_STORAGE`. + +### Scanning in the background + +A scan is owned by `ScanController` (process-scoped, created by the Application), not +by the screen's ViewModel. While one is running, `ScanService` holds a `dataSync` +foreground service with a progress notification that has a **Cancel** action. So: + +- leaving the app does not kill the scan, +- the result survives screen rotation and re-entry, +- the user always has a visible way to stop it. + +When `POST_NOTIFICATIONS` is denied the scan still works; the notification is simply +not shown. + +## Testing + +`app/src/test` holds JVM unit tests for `core/`: SHA-256 test vectors, dHash bit +patterns, Hamming distance, exact/perceptual grouping, aspect-ratio filtering and +the reclaimable-bytes maths. + +```bash +./gradlew :app:testDebugUnitTest +``` + +There is no instrumentation test and no emulator in this environment, so UI +behaviour is verified on a real device via `adb install` or Play internal testing. + +## Releasing to Google Play + +See [docs/PLAY_RELEASE_CHECKLIST.md](docs/PLAY_RELEASE_CHECKLIST.md) (signing, store +listing, Data Safety, release notes, rollout), [docs/STORE_LISTING.md](docs/STORE_LISTING.md) +(ready-to-paste listing copy plus the icon / feature graphic generators) and +[PRIVACY.md](PRIVACY.md). + +The upload-ready graphics and screenshots live in [`store-assets/`](store-assets/). +`tools/make-store-assets.sh` and `tools/make-store-screenshots.sh` regenerate them from the +app's own icon vectors and from `adb` captures — always inside the repo, never in `$HOME`. + +User-visible changes are tracked in [CHANGELOG.md](CHANGELOG.md); the Play "What's new" +text you type per release is separate and lives only in the Play Console. + +## Localisation + +All user-visible text lives in `app/src/main/res/values/strings.xml`, with proper `` +where a count can be one, and both the screen and the progress notification resolve their +labels through `text/TextRes.kt`. **English is currently the only locale.** + +Adding a language is then a file, not a refactor: + +``` +app/src/main/res/values-de/strings.xml # same keys, translated values +``` + +One caveat worth knowing: **Google Play's automatic translation translates your store +listing, not the app UI.** Translated UI text has to be supplied in `values-/`, by a +translator or by running the strings file through a translation service and reviewing it. + +Byte units ("KB", "MB") and the duration formatter are intentionally left unlocalised. + +## Known limitations + +- Only **images** are scanned — videos are not supported yet. +- Near-duplicate detection ignores EXIF rotation, so an image saved physically + rotated by 90° will not match its original. +- The Aggressive policy may miss some pairs because of the banded candidate search. +- The very first scan of a huge library (50k+ photos) takes a while; later scans + reuse the hash index and are much faster. + +## Roadmap + +The project is tracked against a milestone plan (M0–M10). What is done, what remains, +the current release blockers and the candidate features all live in +[docs/ROADMAP.md](docs/ROADMAP.md). + +## Licence + +Proprietary — all rights reserved. Change this if you intend to open-source it. diff --git a/app/build.gradle.kts b/app/build.gradle.kts new file mode 100644 index 0000000..325afaf --- /dev/null +++ b/app/build.gradle.kts @@ -0,0 +1,194 @@ +import java.util.Properties +import org.gradle.api.tasks.testing.Test + +plugins { + alias(libs.plugins.android.application) + alias(libs.plugins.kotlin.compose) +} + +// Optional release signing. +// +// Create a `keystore.properties` file in the project ROOT (it is git-ignored; see +// keystore.properties.example) with: +// +// storeFile=/absolute/path/to/upload-keystore.jks +// storePassword=******** +// keyAlias=upload +// keyPassword=******** +// +// Without that file the release build still runs R8/minification but produces an +// unsigned APK, which is fine for local checks but cannot be uploaded to Play. +val keystorePropertiesFile = rootProject.file("keystore.properties") +val keystoreProperties = Properties().apply { + if (keystorePropertiesFile.exists()) { + keystorePropertiesFile.inputStream().use { load(it) } + } +} +val hasReleaseSigning = keystoreProperties.getProperty("storeFile") != null + +if (hasReleaseSigning) { + // Fail early and readably instead of letting signReleaseBundle blow up later + // with an opaque NullPointerException. + val missing = listOf("storePassword", "keyAlias") + .filter { keystoreProperties.getProperty(it).isNullOrBlank() } + check(missing.isEmpty()) { + "keystore.properties is incomplete — missing: ${missing.joinToString()}" + } +} + +android { + namespace = "de.softwareapp_hb.fotodup.finder" + compileSdk = 36 + + defaultConfig { + applicationId = "de.softwareapp_hb.fotodup.finder" + minSdk = 26 + targetSdk = 36 + versionCode = 2 + versionName = "1.0.1" + + vectorDrawables { + useSupportLibrary = true + } + } + + signingConfigs { + if (hasReleaseSigning) { + create("release") { + storeFile = file(keystoreProperties.getProperty("storeFile")) + storePassword = keystoreProperties.getProperty("storePassword") + keyAlias = keystoreProperties.getProperty("keyAlias") + // Falls back to the store password. That matches reality: PKCS12 + // keystores (the default on modern JDKs) cannot have a key password + // that differs from the store password, and leaving this null makes + // AGP's signReleaseBundle crash with an NPE. + keyPassword = keystoreProperties.getProperty("keyPassword") + ?: keystoreProperties.getProperty("storePassword") + } + } + } + + buildTypes { + debug { + applicationIdSuffix = ".debug" + versionNameSuffix = "-debug" + } + release { + if (hasReleaseSigning) { + signingConfig = signingConfigs.getByName("release") + } + isMinifyEnabled = true + isShrinkResources = true + proguardFiles( + getDefaultProguardFile("proguard-android-optimize.txt"), + "proguard-rules.pro", + ) + } + } + + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } + + buildFeatures { + compose = true + } + + testOptions { + unitTests { + // Robolectric needs the merged resources and manifest to stand up the framework. + isIncludeAndroidResources = true + } + } + + packaging { + resources { + excludes += "/META-INF/{AL2.0,LGPL2.1}" + } + } + + lint { + // Dependency versions are pinned deliberately — Compose 1.12+/Coil 3.6+ need + // compileSdk 37, which AGP 9.1 does not support yet (see the README version + // note). These advisories are noise rather than signal here. + disable += setOf( + "GradleDependency", + "AndroidGradlePluginVersion", + "NewerVersionAvailable", + ) + } +} + +kotlin { + compilerOptions { + // With AGP built-in Kotlin this already defaults to + // android.compileOptions.targetCompatibility; set explicitly for clarity. + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + } +} + +// `java.awt` is used by the similarity benchmark (headless), and the benchmark itself is +// opt-in. Gradle's -D only reaches the Gradle JVM, so the flags are forwarded explicitly. +// Run it with: +// ./gradlew :app:testDebugUnitTest --tests '*SimilarityBenchmarkTest*' -DfotoBench=true +tasks.withType().configureEach { + systemProperty("java.awt.headless", "true") + // Robolectric reflects into JDK internals (notably java.io.FileDescriptor), which the module + // system blocks from JDK 17 onwards. Without these it fails with "Failed to interact with raw + // FileDescriptor internals". + jvmArgs( + "--add-opens=java.base/java.io=ALL-UNNAMED", + "--add-opens=java.base/java.lang=ALL-UNNAMED", + "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED", + "--add-opens=java.base/java.util=ALL-UNNAMED", + "--add-opens=java.base/java.text=ALL-UNNAMED", + "--add-opens=java.base/java.nio=ALL-UNNAMED", + "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", + // Robolectric reaches FileDescriptor internals through jdk.internal.access.SharedSecrets, and + // that needs an *export* rather than an *open*: without it the run dies with + // "IllegalAccessException ... module java.base does not export jdk.internal.access". + "--add-exports=java.base/jdk.internal.access=ALL-UNNAMED", + "--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED", + "--add-exports=java.base/jdk.internal.reflect=ALL-UNNAMED", + ) + systemProperty("fotoBench", providers.systemProperty("fotoBench").orNull ?: "false") + providers.systemProperty("fotoBench.dir").orNull?.let { systemProperty("fotoBench.dir", it) } + providers.systemProperty("fotoBench.count").orNull?.let { systemProperty("fotoBench.count", it) } + providers.systemProperty("fotoBench.scenes").orNull?.let { systemProperty("fotoBench.scenes", it) } + providers.systemProperty("fotoBench.density").orNull?.let { systemProperty("fotoBench.density", it) } + providers.systemProperty("fotoBench.max").orNull?.let { systemProperty("fotoBench.max", it) } + providers.systemProperty("fotoLocal.dir").orNull?.let { systemProperty("fotoLocal.dir", it) } + providers.systemProperty("fotoScale").orNull?.let { systemProperty("fotoScale", it) } + // Real-photo runs hold hundreds of decoded images in memory at once. + maxHeapSize = "2g" + testLogging { showStandardStreams = true } +} + +dependencies { + implementation(libs.androidx.core.ktx) + // EXIF orientation, so a photo hashes the same whichever decode path served it. + implementation(libs.androidx.exifinterface) + + implementation(libs.androidx.lifecycle.runtime.ktx) + implementation(libs.androidx.lifecycle.runtime.compose) + implementation(libs.androidx.lifecycle.viewmodel.compose) + + implementation(libs.androidx.activity.compose) + + implementation(platform(libs.androidx.compose.bom)) + implementation(libs.androidx.compose.ui) + implementation(libs.androidx.compose.ui.graphics) + implementation(libs.androidx.compose.ui.tooling.preview) + implementation(libs.androidx.compose.material3) + implementation(libs.androidx.compose.material.icons.core) + + implementation(libs.kotlinx.coroutines.android) + implementation(libs.coil.compose) + + debugImplementation(libs.androidx.compose.ui.tooling) + + testImplementation(libs.junit) + // Runs the real Android framework on the JVM, so the data layer is testable without a device. + testImplementation(libs.robolectric) +} diff --git a/app/proguard-rules.pro b/app/proguard-rules.pro new file mode 100644 index 0000000..45242f3 --- /dev/null +++ b/app/proguard-rules.pro @@ -0,0 +1,11 @@ +# ProGuard/R8 rules for the release build. +# +# The app relies on reflection only indirectly (Compose, Coil, AndroidX ship their +# own consumer rules), so we keep this list intentionally short. + +# Keep line numbers so crash reports stay readable without the original class names. +-keepattributes SourceFile,LineNumberTable +-renamesourcefileattribute SourceFile + +# Coroutines: keep the internal classes R8 sometimes struggles to strip correctly. +-dontwarn kotlinx.coroutines.** diff --git a/app/src/debug/AndroidManifest.xml b/app/src/debug/AndroidManifest.xml new file mode 100644 index 0000000..feebe44 --- /dev/null +++ b/app/src/debug/AndroidManifest.xml @@ -0,0 +1,44 @@ + + + + + + + + + + + + diff --git a/app/src/debug/java/de/softwareapp_hb/fotodup/finder/debug/DebugScanTrigger.kt b/app/src/debug/java/de/softwareapp_hb/fotodup/finder/debug/DebugScanTrigger.kt new file mode 100644 index 0000000..9034c44 --- /dev/null +++ b/app/src/debug/java/de/softwareapp_hb/fotodup/finder/debug/DebugScanTrigger.kt @@ -0,0 +1,57 @@ +package de.softwareapp_hb.fotodup.finder.debug + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.util.Log +import de.softwareapp_hb.fotodup.finder.FotoDupApplication +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.data.AppPermissions + +/** + * Debug-only hook that lets a device scan be driven over `adb` — see `src/debug/AndroidManifest.xml`. + * + * Not part of the app: it lives in the debug source set, so it is absent from the release build. + * It reports everything it does through [TAG] so `adb logcat -s FotoDupDebug` shows the outcome. + */ +class DebugScanTrigger : BroadcastReceiver() { + + override fun onReceive(context: Context, intent: Intent) { + val app = context.applicationContext as? FotoDupApplication ?: return + val controller = app.scanController + + when (intent.action) { + ACTION_START_SCAN -> { + val policy = when (intent.getStringExtra(EXTRA_POLICY)?.lowercase()) { + "exact" -> DuplicatePolicy.EXACT_ONLY + "aggressive" -> DuplicatePolicy.AGGRESSIVE + else -> DuplicatePolicy.BALANCED + } + Log.i(TAG, "START_SCAN policy=$policy access=${AppPermissions.accessState(context)}") + controller.start(policy) + report(controller) + } + + ACTION_REPORT_STATE -> report(controller) + } + } + + private fun report(controller: de.softwareapp_hb.fotodup.finder.domain.ScanController) { + val state = controller.state.value + val result = state.result + Log.i( + TAG, + "STATE scanning=${state.scanning} phase=${state.phase} " + + "progress=${state.done}/${state.total} error=${state.error} " + + "photos=${result?.photoCount} groups=${result?.groupCount} " + + "unreadable=${result?.unreadableCount} timings=${result?.timings}", + ) + } + + companion object { + const val ACTION_START_SCAN = "de.softwareapp_hb.fotodup.finder.debug.START_SCAN" + const val ACTION_REPORT_STATE = "de.softwareapp_hb.fotodup.finder.debug.REPORT_STATE" + const val EXTRA_POLICY = "policy" + const val TAG = "FotoDupDebug" + } +} diff --git a/app/src/main/AndroidManifest.xml b/app/src/main/AndroidManifest.xml new file mode 100644 index 0000000..0b0fa52 --- /dev/null +++ b/app/src/main/AndroidManifest.xml @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/FotoDupApplication.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/FotoDupApplication.kt new file mode 100644 index 0000000..c4d9cec --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/FotoDupApplication.kt @@ -0,0 +1,28 @@ +package de.softwareapp_hb.fotodup.finder + +import android.app.Application +import de.softwareapp_hb.fotodup.finder.domain.ScanController + +/** + * Application entry point. + * + * Deliberately lightweight: the app has no background singletons, no dependency + * injection graph and no network access. Image loading uses Coil's default loader, + * and the duplicate index lives in a small SQLite database opened lazily by the + * data layer. + */ +class FotoDupApplication : Application() { + + /** + * Owns the running scan. Living on the Application means the scan survives activity + * recreation and keeps going while the app is backgrounded (kept alive by + * [de.softwareapp_hb.fotodup.finder.service.ScanService]). + */ + lateinit var scanController: ScanController + private set + + override fun onCreate() { + super.onCreate() + scanController = ScanController(this) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/MainActivity.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/MainActivity.kt new file mode 100644 index 0000000..4c3900f --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/MainActivity.kt @@ -0,0 +1,29 @@ +package de.softwareapp_hb.fotodup.finder + +import android.os.Bundle +import androidx.activity.ComponentActivity +import androidx.activity.compose.setContent +import androidx.activity.enableEdgeToEdge +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.ui.Modifier +import de.softwareapp_hb.fotodup.finder.ui.scanner.ScannerScreen +import de.softwareapp_hb.fotodup.finder.ui.theme.FotoDupTheme + +class MainActivity : ComponentActivity() { + override fun onCreate(savedInstanceState: Bundle?) { + enableEdgeToEdge() + super.onCreate(savedInstanceState) + setContent { + FotoDupTheme { + Surface( + modifier = Modifier.fillMaxSize(), + color = MaterialTheme.colorScheme.background, + ) { + ScannerScreen() + } + } + } + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThreshold.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThreshold.kt new file mode 100644 index 0000000..75fdb59 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThreshold.kt @@ -0,0 +1,105 @@ +package de.softwareapp_hb.fotodup.finder.core + +import kotlin.math.abs +import kotlin.math.exp +import kotlin.math.sqrt + +/** + * Chooses a Hamming threshold from the observed distribution of pair distances instead of + * from a fixed constant. + * + * A constant cannot work at every library size: unrelated pairs grow with the square of the + * library, so the closest accidentally-similar pair keeps getting closer as a library grows. + * Measured on the benchmark, the widest pHash threshold with zero false pairs fell from + * 13 (120 images) to 5 (2 880 images). + * + * This estimates the unrelated-pair distribution from the upper part of the observed + * histogram — where duplicates are effectively absent — and returns the widest threshold + * whose **expected number of accidental matches** stays within [budget]. Budgeting the count + * rather than the rate is deliberate: what the user notices is how many *wrong* duplicate + * groups appear, not what fraction of all pairs they represent. + */ +object AdaptiveThreshold { + + /** Distances at or above this are assumed to contain no duplicates. */ + private const val CLEAN_FROM = 20 + + /** Below this many observed pairs the tail cannot be modelled, so the ceiling is used. */ + private const val MIN_SAMPLE = 200 + + private const val SQRT_TWO = 1.4142135623730951 + + /** + * @property threshold effective maximum Hamming distance. + * @property expectedFalsePairs estimated accidental matches at that threshold (NaN when + * the fallback was used). + * @property usedFallback true when there was too little data to model the tail. + */ + data class Choice( + val threshold: Int, + val expectedFalsePairs: Double, + val usedFallback: Boolean, + ) + + fun choose( + histogram: IntArray, + ceiling: Int, + budget: Double, + cleanFrom: Int = CLEAN_FROM, + minSample: Int = MIN_SAMPLE, + ): Choice { + require(ceiling >= 0) { "ceiling must not be negative" } + val population = histogram.sum().toDouble() + if (population < minSample) { + return Choice(ceiling, Double.NaN, usedFallback = true) + } + + var count = 0L + var sum = 0.0 + var sumSquares = 0.0 + for (distance in cleanFrom until histogram.size) { + val observed = histogram[distance] + count += observed + sum += observed.toDouble() * distance + sumSquares += observed.toDouble() * distance * distance + } + if (count < minSample) { + return Choice(ceiling, Double.NaN, usedFallback = true) + } + + val mean = sum / count + val variance = (sumSquares / count) - mean * mean + val sd = sqrt(variance.coerceAtLeast(0.25)) + + var threshold = -1 + var expected = 0.0 + for (candidate in 0..ceiling) { + // Continuity correction: a distance of `candidate` means "at most this many bits". + val z = (candidate + 0.5 - mean) / sd + val estimate = population * normalCdf(z) + if (estimate > budget) break + threshold = candidate + expected = estimate + } + + if (threshold < 0) { + // Even zero bits of tolerance blows the budget (a very large library): stay as strict + // as a Hamming test can be rather than reporting a threshold of -1. + val strict = population * normalCdf((0.5 - mean) / sd) + return Choice(0, strict, usedFallback = false) + } + return Choice(threshold, expected, usedFallback = false) + } + + private fun normalCdf(z: Double): Double = 0.5 * (1.0 + erf(z / SQRT_TWO)) + + /** Abramowitz & Stegun 7.1.26; |error| < 1.5e-7, ample for a threshold heuristic. */ + private fun erf(x: Double): Double { + val sign = if (x < 0) -1.0 else 1.0 + val magnitude = abs(x) + val t = 1.0 / (1.0 + 0.3275911 * magnitude) + val polynomial = ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + + 0.254829592) * t + return sign * (1.0 - polynomial * exp(-magnitude * magnitude)) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/ByteUtils.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/ByteUtils.kt new file mode 100644 index 0000000..7a40a13 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/ByteUtils.kt @@ -0,0 +1,46 @@ +package de.softwareapp_hb.fotodup.finder.core + +import java.io.InputStream + +private const val HEX_DIGITS = "0123456789abcdef" + +/** Lower-case hexadecimal representation of these bytes. */ +fun ByteArray.toHex(): String { + val sb = StringBuilder(size * 2) + for (b in this) { + val v = b.toInt() and 0xFF + sb.append(HEX_DIGITS[v ushr 4]) + sb.append(HEX_DIGITS[v and 0x0F]) + } + return sb.toString() +} + +/** 8-byte big-endian representation, used to stir the file size into a digest. */ +fun Long.toBigEndianBytes(): ByteArray { + val out = ByteArray(8) + for (i in 0 until 8) { + out[i] = ((this ushr (56 - i * 8)) and 0xFF).toByte() + } + return out +} + +/** + * Reads up to [max] bytes from this stream, returning fewer only when the stream + * ends. Unlike [InputStream.readNBytes] this is available on every API level the + * app supports. + */ +fun InputStream.readAtMost(max: Int): ByteArray { + require(max >= 0) { "max must not be negative" } + if (max == 0) return ByteArray(0) + val out = ByteArray(max) + var total = 0 + while (total < max) { + val read = read(out, total, max - total) + // A well-behaved stream never returns 0 for a non-empty request, but a non-blocking + // pipe-backed descriptor can report "no bytes, try again" *indefinitely*. Spinning on that + // hangs the caller, so treat it as the end of the data. + if (read <= 0) break + total += read + } + return if (total == max) out else out.copyOf(total) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DecodeMath.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DecodeMath.kt new file mode 100644 index 0000000..59a1027 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DecodeMath.kt @@ -0,0 +1,41 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** + * Pure arithmetic used when decoding images defensively. + * + * Kept in `core` with no Android imports so the awkward cases — zero sizes, and the absurd + * dimensions a crafted "decompression bomb" can declare — are unit-testable on the JVM. + */ +object DecodeMath { + + /** + * Dimensions beyond this are treated as suspect. A genuine ultra-high-resolution scan is + * nowhere near it, and clamping keeps the computed sample size within a range a decoder will + * actually accept. + */ + const val MAX_DIMENSION = 100_000 + + /** Upper bound on the returned sample, so `sample * 2` can never overflow. */ + private const val MAX_SAMPLE = 1 shl 16 + + /** + * Largest power-of-two subsample that keeps both edges within roughly twice [target]. + * + * Deliberately written with **division rather than multiplication**: a crafted image can + * declare 100 000 x 100 000, and `width * height` or `sample * 2` in the wrong form overflows + * `Int` and yields a nonsense (or negative) sample size. + */ + fun sampleSizeFor(width: Int, height: Int, target: Int): Int { + if (target <= 0) return 1 + val safeWidth = width.coerceIn(0, MAX_DIMENSION) + val safeHeight = height.coerceIn(0, MAX_DIMENSION) + var sample = 1 + while ( + sample < MAX_SAMPLE && + (safeWidth / (sample * 2) >= target || safeHeight / (sample * 2) >= target) + ) { + sample *= 2 + } + return sample + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DisjointSet.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DisjointSet.kt new file mode 100644 index 0000000..dd6fb60 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DisjointSet.kt @@ -0,0 +1,49 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** + * Minimal union-find (disjoint set) with path compression and union by rank. + * + * Used to merge photos into duplicate clusters: similarity is transitive within a + * scan, so a chain A~B, B~C puts A, B and C in one group. + */ +internal class DisjointSet(private val size: Int) { + + private val parent = IntArray(size) { it } + private val rank = IntArray(size) + + fun find(x: Int): Int { + var root = x + while (parent[root] != root) root = parent[root] + // Path compression. + var current = x + while (parent[current] != root) { + val next = parent[current] + parent[current] = root + current = next + } + return root + } + + fun union(a: Int, b: Int) { + val rootA = find(a) + val rootB = find(b) + if (rootA == rootB) return + when { + rank[rootA] < rank[rootB] -> parent[rootA] = rootB + rank[rootA] > rank[rootB] -> parent[rootB] = rootA + else -> { + parent[rootB] = rootA + rank[rootA]++ + } + } + } + + /** All clusters, each sorted ascending by index; includes singletons. */ + fun groups(): List> { + val byRoot = LinkedHashMap>() + for (i in 0 until size) { + byRoot.getOrPut(find(i)) { mutableListOf() }.add(i) + } + return byRoot.values.toList() + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroup.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroup.kt new file mode 100644 index 0000000..edf3e5d --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroup.kt @@ -0,0 +1,88 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** Why a set of photos was considered duplicates. */ +enum class GroupKind { + /** Byte-identical copies. */ + EXACT, + + /** Visually similar photos (resized / recompressed / lightly edited). */ + PERCEPTUAL, +} + +/** + * A set of photos the engine believes are duplicates of each other. + * + * @property id stable-ish identifier for UI list keys. + * @property members never smaller than two, ordered newest first. + */ +data class DuplicateGroup( + val id: String, + val kind: GroupKind, + val members: List, +) { + /** Combined size of every copy in the group. */ + val totalBytes: Long get() = members.sumOf { it.sizeBytes } + + /** Size of the largest copy — the one the app suggests keeping. */ + val largestBytes: Long get() = members.maxOfOrNull { it.sizeBytes } ?: 0L + + /** + * Bytes that could be reclaimed by keeping only the largest copy. This is an + * estimate: the user decides what to actually delete. + */ + val reclaimableBytes: Long get() = (totalBytes - largestBytes).coerceAtLeast(0L) + + /** + * The copy the app suggests keeping: the largest file, newest on a tie. + * + * This is the single source of truth — the "BEST" badge, the "select all but best" + * helper and the per-group delete action all use it, so they can never disagree. + */ + fun suggestedKeep(): PhotoFingerprint? = + members.maxWithOrNull( + compareBy { it.sizeBytes }.thenBy { it.dateModifiedSeconds }, + ) + + /** Every member except [suggestedKeep] — i.e. what the app would remove by default. */ + fun suggestedRemovals(): List { + val keepId = suggestedKeep()?.id ?: return emptyList() + return members.filterNot { it.id == keepId } + } + + /** + * The members the user has actually ticked in this group. + * + * This is what a row-wise "remove" acts on, and what the row's button counts, so the label + * can never disagree with the action. Right after a scan the selection holds + * [suggestedRemovals], so the default one-tap behaviour is unchanged. + */ + fun selectedMembers(selectedIds: Set): List = + members.filter { it.id in selectedIds } + + /** + * Members in the order they should be shown: [suggestedKeep] **first**, so it is + * always visible as the reference point, even when the group is collapsed and only + * the first few thumbnails are displayed. The remaining copies keep the engine's + * order (newest first). + */ + fun displayOrder(): List { + val keep = suggestedKeep() ?: return members + return buildList(members.size) { + add(keep) + members.forEach { member -> if (member.id != keep.id) add(member) } + } + } + + /** + * This group without [ids]. + * + * Used while photos are staged for removal, so the list shows what is *still* + * duplicated. Returns `null` when fewer than two copies would remain, because at that + * point the set is no longer a duplicate group and should not be listed at all. + */ + fun without(ids: Set): DuplicateGroup? { + if (ids.isEmpty()) return this + val remaining = members.filterNot { it.id in ids } + return if (remaining.size < 2) null else copy(members = remaining) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouper.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouper.kt new file mode 100644 index 0000000..50a0366 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouper.kt @@ -0,0 +1,214 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** + * Turns a list of fingerprinted photos into groups of duplicates. + * + * Two passes run, in order: + * 1. **Exact** — photos sharing a content hash are byte-identical. + * 2. **Perceptual** — photos not already in an exact group that are *visually* + * similar. Because near-duplicate comparison is the expensive part, candidates + * are indexed by 8-bit bands of the 64-bit hash (a standard locality-sensitive + * trick) so only likely pairs are ever compared. + * + * Pure Kotlin, deterministic and unit-testable. + */ +object DuplicateGrouper { + + /** Bands used to index perceptual hashes for fast candidate lookup. */ + private const val BANDS = 8 + private const val BAND_BITS = 64 / BANDS // 8 + private const val BAND_MASK = (1L shl BAND_BITS) - 1 + + /** + * Maximum aspect-ratio difference (as a factor) before two photos are assumed + * not to be the same shot. Guards against e.g. a square crop matching a wide + * photo purely by coincidence. + */ + private const val MAX_ASPECT_FACTOR = 1.15f + + /** Widest Hamming distance a 64-bit hash can have. */ + private const val MAX_DISTANCE = 64 + + /** + * How many accidental matches the adaptive threshold will tolerate per scan. Budgeting the + * count rather than the rate is deliberate: what a user notices is how many *wrong* + * duplicate groups appear, not what share of all pairs they represent. + */ + const val DEFAULT_FALSE_POSITIVE_BUDGET = 2.0 + + /** Outcome of a grouping pass, including the thresholds that were actually used. */ + data class GroupingResult( + val groups: List, + val thresholds: PerceptualThresholds, + /** True when the library was too small to model the distance tail. */ + val usedFallbackThreshold: Boolean, + /** Candidate pairs the banded index proposed (the population the threshold was fitted to). */ + val observedPairs: Int, + ) + + fun findGroups( + photos: List, + policy: DuplicatePolicy = DuplicatePolicy.DEFAULT, + ): List = findGroupsDetailed(photos, policy).groups + + fun findGroupsDetailed( + photos: List, + policy: DuplicatePolicy = DuplicatePolicy.DEFAULT, + falsePositiveBudget: Double = DEFAULT_FALSE_POSITIVE_BUDGET, + ): GroupingResult { + val groups = mutableListOf() + val alreadyGrouped = HashSet() + + // --- Pass 1: exact duplicates by content hash ------------------------- + val byContentHash = photos + .filter { it.contentHash != null } + .groupBy { it.contentHash!! } + + for ((hash, members) in byContentHash) { + if (members.size < 2) continue + groups += DuplicateGroup( + id = "exact:$hash", + kind = GroupKind.EXACT, + members = orderMembers(members), + ) + members.forEach { alreadyGrouped += it.id } + } + + // --- Pass 2: perceptual duplicates ----------------------------------- + var thresholds = PerceptualThresholds.forPolicy(policy) + var usedFallback = false + var observedPairs = 0 + + if (policy.includePerceptual) { + val candidates = photos.filter { + it.perceptualHash != null && it.rgbDHash != null && it.id !in alreadyGrouped + } + if (candidates.size >= 2) { + val hashes = LongArray(candidates.size) { candidates[it].perceptualHash!! } + val buckets = buildBuckets(hashes) + + // Pass A: fit the threshold to this library rather than trusting a constant. + val histogram = IntArray(MAX_DISTANCE + 1) + var pairs = 0 + forEachCandidatePair(hashes, buckets) { i, j -> + histogram[PerceptualHash.hammingDistance(hashes[i], hashes[j])]++ + pairs++ + } + observedPairs = pairs + + val choice = AdaptiveThreshold.choose( + histogram = histogram, + ceiling = policy.maxPhashDistance, + budget = falsePositiveBudget, + ) + usedFallback = choice.usedFallback + thresholds = PerceptualThresholds(choice.threshold, policy.maxRgbDistance) + + // Pass B: apply the conjunction. pHash alone is not enough once a library is + // large: requiring the colour structure to agree too is what holds precision. + val set = DisjointSet(candidates.size) + forEachCandidatePair(hashes, buckets) { i, j -> + val a = candidates[i] + val b = candidates[j] + if (aspectCompatible(a, b) && + PerceptualHash.hammingDistance(hashes[i], hashes[j]) <= thresholds.phashDistance && + a.rgbDHash!!.distanceTo(b.rgbDHash!!) <= thresholds.rgbDistance + ) { + set.union(i, j) + } + } + + set.groups() + .filter { it.size >= 2 } + .forEach { cluster -> + groups += DuplicateGroup( + id = "near:${candidates[cluster.first()].id}", + kind = GroupKind.PERCEPTUAL, + members = orderMembers(cluster.map { candidates[it] }), + ) + } + } + } + + return GroupingResult( + groups = groups.sortedByDescending { it.reclaimableBytes }, + thresholds = thresholds, + usedFallbackThreshold = usedFallback, + observedPairs = observedPairs, + ) + } + + /** Indexes hashes by each of their 8-bit bands: the locality-sensitive candidate filter. */ + private fun buildBuckets(hashes: LongArray): Array>> { + val buckets = Array(BANDS) { HashMap>() } + hashes.forEachIndexed { index, hash -> + for (band in 0 until BANDS) { + val value = (hash ushr (band * BAND_BITS)) and BAND_MASK + buckets[band].getOrPut(value) { mutableListOf() }.add(index) + } + } + return buckets + } + + /** + * Calls [action] once for every candidate pair `(i, j)` with `i < j`. + * + * Each photo gathers its band-neighbours into a reused scratch array — sorted, so the + * duplicates a pair shares across several bands collapse — and therefore **no global set of + * compared pairs is kept**. The previous implementation stored one boxed `Long` per + * candidate pair, which for a 20 000-photo library meant millions of entries and hundreds of + * megabytes of retained garbage. Here memory is bounded by the largest neighbour set. + * + * Candidate recall is exact for thresholds up to 7: with [BANDS] bands, two hashes differing + * in at most 7 bits must share an identical band. [DuplicatePolicy.AGGRESSIVE] reaches 18, so + * it can miss the occasional pair — the price of not scanning all n² pairs. + */ + private inline fun forEachCandidatePair( + hashes: LongArray, + buckets: Array>>, + action: (Int, Int) -> Unit, + ) { + var scratch = IntArray(64) + for (i in hashes.indices) { + val hash = hashes[i] + var size = 0 + for (band in 0 until BANDS) { + val bucket = buckets[band][(hash ushr (band * BAND_BITS)) and BAND_MASK] ?: continue + val required = size + bucket.size + if (required > scratch.size) { + var capacity = scratch.size + while (capacity < required) capacity *= 2 + scratch = scratch.copyOf(capacity) + } + for (index in bucket) { + if (index != i) scratch[size++] = index + } + } + if (size == 0) continue + + java.util.Arrays.sort(scratch, 0, size) + var previous = -1 + for (k in 0 until size) { + val j = scratch[k] + if (j == previous) continue + previous = j + if (j > i) action(i, j) + } + } + } + + private fun aspectCompatible(a: PhotoFingerprint, b: PhotoFingerprint): Boolean { + val ratioA = a.aspectRatio ?: return true + val ratioB = b.aspectRatio ?: return true + if (ratioA <= 0f || ratioB <= 0f) return true + val factor = if (ratioA > ratioB) ratioA / ratioB else ratioB / ratioA + return factor <= MAX_ASPECT_FACTOR + } + + /** Newest first, then largest — a sane default order for review. */ + private fun orderMembers(members: List): List = + members.sortedWith( + compareByDescending { it.dateModifiedSeconds } + .thenByDescending { it.sizeBytes }, + ) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicatePolicy.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicatePolicy.kt new file mode 100644 index 0000000..a1f603c --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/DuplicatePolicy.kt @@ -0,0 +1,49 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** The thresholds actually used for a perceptual pass, after any adaptive adjustment. */ +data class PerceptualThresholds( + val phashDistance: Int, + val rgbDistance: Int, +) { + companion object { + fun forPolicy(policy: DuplicatePolicy) = + PerceptualThresholds(policy.maxPhashDistance, policy.maxRgbDistance) + } +} + +/** + * How aggressively the engine considers two photos to be duplicates. + * + * These values are **policy ceilings, not tuned operating points**. The real operating point is + * fitted per scan by [AdaptiveThreshold]; the ceiling only bounds what that fit may decide, and + * is used directly when a library is too small to fit. Values come from measurement across + * three datasets (COCO, BSDS500, DIV2K), not from guesswork — but see the note on the harness. + * + * @property maxPhashDistance loosest permitted pHash Hamming distance (0..64). + * @property maxRgbDistance loosest permitted summed RGB-channel dHash distance (0..192). + */ +enum class DuplicatePolicy(val maxPhashDistance: Int, val maxRgbDistance: Int) { + /** Only byte-identical files. Fastest, zero false positives. */ + EXACT_ONLY(maxPhashDistance = -1, maxRgbDistance = -1), + + /** + * Byte-identical files plus visually very similar ones. Sensible default: the ceiling sits + * at the widest cut that still produced zero false pairs on 300 real photographs, so a small + * library gets the full benefit while a large one is tightened by the fit. + */ + BALANCED(maxPhashDistance = 15, maxRgbDistance = 60), + + /** + * Also catches heavier re-edits/recompressions, at the cost of more review. The ceiling is + * set above the measured F1 optimum (~18 on real photographs) so it stays a bound rather + * than becoming the operating point. + */ + AGGRESSIVE(maxPhashDistance = 22, maxRgbDistance = 84), + ; + + val includePerceptual: Boolean get() = this != EXACT_ONLY + + companion object { + val DEFAULT: DuplicatePolicy = BALANCED + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/Hashing.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/Hashing.kt new file mode 100644 index 0000000..43c3645 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/Hashing.kt @@ -0,0 +1,59 @@ +package de.softwareapp_hb.fotodup.finder.core + +import java.io.InputStream +import java.security.MessageDigest + +/** + * Content hashing used to find byte-identical photos. + * + * Deliberately pure Kotlin/JVM (only the standard library and `java.security`), so + * it is fully unit-testable without an Android device or Robolectric. + * + * The engine works in tiers, cheapest first: + * 1. group by file size (two files of different size can never be identical), + * 2. compare a cheap [quickSignature] built from size + head/tail bytes, + * 3. only then compute the full [sha256] content hash. + */ +object Hashing { + + /** Bytes sampled from the head (and, when available, the tail) of a file. */ + const val SIGNATURE_WINDOW = 32 * 1024 + + private const val BUFFER_SIZE = 64 * 1024 + private const val ALGORITHM = "SHA-256" + + /** Full SHA-256 of the given bytes, hex encoded. */ + fun sha256(bytes: ByteArray): String = + MessageDigest.getInstance(ALGORITHM).digest(bytes).toHex() + + /** Full SHA-256 of everything readable from [input]. The stream is not closed. */ + fun sha256(input: InputStream): String { + val digest = MessageDigest.getInstance(ALGORITHM) + val buffer = ByteArray(BUFFER_SIZE) + while (true) { + val read = input.read(buffer) + // `read == 0` means "no bytes, try again" — a non-blocking pipe-backed descriptor can + // report that indefinitely, and spinning on it hangs the whole scan. + if (read <= 0) break + digest.update(buffer, 0, read) + } + return digest.digest().toHex() + } + + /** + * Cheap "probably identical" signature: SHA-256 over the file size plus the + * first and last bytes of the file. + * + * Two files whose size differs, or whose head/tail bytes differ, are guaranteed + * to produce different signatures, which removes the vast majority of + * same-size candidates before a full hash is computed. Callers that cannot + * cheaply read the tail may pass an empty [tail]. + */ + fun quickSignature(size: Long, head: ByteArray, tail: ByteArray = ByteArray(0)): String { + val digest = MessageDigest.getInstance(ALGORITHM) + digest.update(size.toBigEndianBytes()) + digest.update(head) + digest.update(tail) + return digest.digest().toHex() + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHash.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHash.kt new file mode 100644 index 0000000..ebba2b0 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHash.kt @@ -0,0 +1,204 @@ +package de.softwareapp_hb.fotodup.finder.core + +import kotlin.math.PI +import kotlin.math.cos +import kotlin.math.sqrt + +/** + * Perceptual hashing used to find *near* duplicates: photos that were resized, + * recompressed or slightly edited. + * + * Two 64-bit descriptors are available, both compared with a Hamming distance: + * + * - [fromArgb] — **dHash**: the image is reduced to a 9x8 grayscale grid and each pixel + * is compared with its right-hand neighbour. Cheap (no arithmetic beyond subtraction). + * - [phashFromArgb] — **pHash**: a 2-D DCT is applied to a 32x32 grayscale grid and the + * low-frequency 8x8 block is thresholded at its median. Costs a small DCT but is more + * stable under brightness/gamma shifts and re-encoding. + * + * `core/bench` compares the two on transformed images so the default can be chosen from + * measurements rather than opinion. + * + * Pure Kotlin: the only input is an ARGB pixel array, so this is unit-testable on the JVM. + */ +object PerceptualHash { + + const val GRID_WIDTH = 9 + const val GRID_HEIGHT = 8 + const val BITS = GRID_WIDTH * GRID_HEIGHT // 64 bits total, (9-1)*8 comparisons + + /** + * Builds the 64-bit difference hash from a row-major grayscale grid of + * [GRID_WIDTH] x [GRID_HEIGHT] luminance values (0..255). + * + * A bit is set when a pixel is strictly brighter than the pixel on its right. + */ + fun fromLumaGrid(luma: IntArray): Long { + require(luma.size == GRID_WIDTH * GRID_HEIGHT) { + "expected ${GRID_WIDTH * GRID_HEIGHT} luminance values but got ${luma.size}" + } + var hash = 0L + var bit = 0 + for (y in 0 until GRID_HEIGHT) { + val row = y * GRID_WIDTH + for (x in 0 until GRID_WIDTH - 1) { + if (luma[row + x] > luma[row + x + 1]) { + hash = hash or (1L shl bit) + } + bit++ + } + } + return hash + } + + /** + * Downsamples a [width] x [height] ARGB pixel array into the + * [GRID_WIDTH] x [GRID_HEIGHT] grayscale grid used by [fromLumaGrid]. + */ + fun lumaGridFromArgb(pixels: IntArray, width: Int, height: Int): IntArray = + lumaGrid(pixels, width, height, GRID_WIDTH, GRID_HEIGHT) + + /** + * Area-averaged grayscale downsample of [pixels] to [outWidth] x [outHeight]. + * + * Each target cell is the average luminance of the source region mapped onto it + * (area averaging), which is more robust than sampling a single pixel. + */ + internal fun lumaGrid( + pixels: IntArray, + width: Int, + height: Int, + outWidth: Int, + outHeight: Int, + ): IntArray = grid(pixels, width, height, outWidth, outHeight) { luminance(it) } + + /** Area-averaged downsample with an arbitrary per-pixel sampling function. */ + private fun grid( + pixels: IntArray, + width: Int, + height: Int, + outWidth: Int, + outHeight: Int, + sample: (Int) -> Int, + ): IntArray { + require(width > 0 && height > 0) { "width and height must be positive" } + require(pixels.size >= width * height) { + "pixel array holds ${pixels.size} values but ${width * height} are required" + } + val out = IntArray(outWidth * outHeight) + for (gy in 0 until outHeight) { + val y0 = gy * height / outHeight + val y1 = maxOf(y0 + 1, (gy + 1) * height / outHeight) + for (gx in 0 until outWidth) { + val x0 = gx * width / outWidth + val x1 = maxOf(x0 + 1, (gx + 1) * width / outWidth) + var sum = 0L + var count = 0 + for (y in y0 until minOf(y1, height)) { + val row = y * width + for (x in x0 until minOf(x1, width)) { + sum += sample(pixels[row + x]) + count++ + } + } + out[gy * outWidth + gx] = if (count == 0) 0 else (sum / count).toInt() + } + } + return out + } + + /** Convenience: ARGB pixels straight to a 64-bit difference hash. */ + fun fromArgb(pixels: IntArray, width: Int, height: Int): Long = + fromLumaGrid(lumaGridFromArgb(pixels, width, height)) + + // --- RGB difference hash (per-channel structure) ------------------------- + + /** + * One 64-bit difference hash per RGB channel, computed from the same pixels as the pHash. + */ + fun rgbDHashFromArgb(pixels: IntArray, width: Int, height: Int): RgbDHash = RgbDHash( + red = fromLumaGrid(channelGrid(pixels, width, height, shift = 16)), + green = fromLumaGrid(channelGrid(pixels, width, height, shift = 8)), + blue = fromLumaGrid(channelGrid(pixels, width, height, shift = 0)), + ) + + private fun channelGrid(pixels: IntArray, width: Int, height: Int, shift: Int): IntArray = + grid(pixels, width, height, GRID_WIDTH, GRID_HEIGHT) { (it shr shift) and 0xFF } + + // --- pHash (DCT) -------------------------------------------------------- + + /** Edge length of the grayscale grid fed into the DCT. */ + private const val PHASH_SIZE = 32 + + /** Size of the low-frequency block that becomes the 64 bits. */ + private const val PHASH_BLOCK = 8 + + private val COS_TABLE: Array = Array(PHASH_SIZE) { x -> + DoubleArray(PHASH_SIZE) { u -> cos((2 * x + 1) * u * PI / (2 * PHASH_SIZE)) } + } + + private val NORM: DoubleArray = DoubleArray(PHASH_SIZE) { u -> + if (u == 0) sqrt(1.0 / PHASH_SIZE) else sqrt(2.0 / PHASH_SIZE) + } + + /** + * 64-bit DCT perceptual hash ("pHash") from ARGB pixels. + * + * The image is reduced to a 32x32 grayscale grid, a 2-D DCT-II is applied, and the + * top-left 8x8 low-frequency block becomes the hash: a bit is set where the + * coefficient is above the median of the block. + */ + fun phashFromArgb(pixels: IntArray, width: Int, height: Int): Long = + phashFromLumaGrid(lumaGrid(pixels, width, height, PHASH_SIZE, PHASH_SIZE)) + + /** [phashFromArgb] from an already downsampled [PHASH_SIZE]² grayscale grid. */ + internal fun phashFromLumaGrid(luma: IntArray): Long { + require(luma.size == PHASH_SIZE * PHASH_SIZE) { + "expected ${PHASH_SIZE * PHASH_SIZE} luminance values but got ${luma.size}" + } + + // Separable DCT-II. Only the coefficients inside the low-frequency block are + // needed, so each 1-D pass stops after PHASH_BLOCK terms instead of all 32 — + // about 10k multiply-adds per image instead of 1M for a naive 2-D DCT. + val rows = Array(PHASH_SIZE) { DoubleArray(PHASH_BLOCK) } + for (y in 0 until PHASH_SIZE) { + val base = y * PHASH_SIZE + for (v in 0 until PHASH_BLOCK) { + var sum = 0.0 + for (x in 0 until PHASH_SIZE) sum += luma[base + x] * COS_TABLE[x][v] + rows[y][v] = sum * NORM[v] + } + } + + val block = DoubleArray(PHASH_BLOCK * PHASH_BLOCK) + for (u in 0 until PHASH_BLOCK) { + for (v in 0 until PHASH_BLOCK) { + var sum = 0.0 + for (y in 0 until PHASH_SIZE) sum += rows[y][v] * COS_TABLE[y][u] + block[u * PHASH_BLOCK + v] = sum * NORM[u] + } + } + + val sorted = block.clone() + sorted.sort() + val mid = sorted.size / 2 + val median = (sorted[mid - 1] + sorted[mid]) / 2.0 + + var hash = 0L + for (i in block.indices) { + if (block[i] > median) hash = hash or (1L shl i) + } + return hash + } + + /** Number of differing bits between two hashes. */ + fun hammingDistance(a: Long, b: Long): Int = java.lang.Long.bitCount(a xor b) + + /** Integer luminance (0..255) of a packed ARGB pixel using Rec. 601 weights. */ + fun luminance(argb: Int): Int { + val r = (argb shr 16) and 0xFF + val g = (argb shr 8) and 0xFF + val b = argb and 0xFF + return (r * 299 + g * 587 + b * 114) / 1000 + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PhotoFingerprint.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PhotoFingerprint.kt new file mode 100644 index 0000000..6c70b29 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/PhotoFingerprint.kt @@ -0,0 +1,33 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** + * Everything the duplicate engine knows about one photo. + * + * The hashes are nullable because the engine fills them in progressively: cheap + * ones first, and a value stays `null` when the corresponding tier was skipped + * (e.g. [perceptualHash] on an [DuplicatePolicy.EXACT_ONLY] scan). + * + * Pure data: no Android types, so the engine and its tests run on the JVM. + */ +data class PhotoFingerprint( + val id: Long, + val sizeBytes: Long, + val dateModifiedSeconds: Long, + val width: Int, + val height: Int, + /** Cheap size + head/tail signature. */ + val quickSignature: String? = null, + /** Full content SHA-256, present only once the full hash was computed. */ + val contentHash: String? = null, + /** 64-bit DCT perceptual hash, present once the perceptual pass ran. */ + val perceptualHash: Long? = null, + /** + * Per-channel difference hash. Required, together with [perceptualHash], for two photos to + * count as near-duplicates: pHash sees luminance structure, this sees colour structure. + */ + val rgbDHash: RgbDHash? = null, +) { + /** width / height, or `null` when the dimensions are unknown. */ + val aspectRatio: Float? + get() = if (width > 0 && height > 0) width.toFloat() / height.toFloat() else null +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/RgbDHash.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/RgbDHash.kt new file mode 100644 index 0000000..eb674e8 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/core/RgbDHash.kt @@ -0,0 +1,33 @@ +package de.softwareapp_hb.fotodup.finder.core + +/** + * Both perceptual descriptors for one photo. + * + * They are always computed together from the same decoded thumbnail: the pHash sees luminance + * structure, the [RgbDHash] sees colour structure, and a pair must satisfy both to count as a + * near-duplicate. + */ +data class PerceptualFingerprint( + val phash: Long, + val rgb: RgbDHash, +) + +/** + * One 64-bit difference hash per RGB channel. + * + * Complements the grayscale pHash: pHash captures **luminance** structure, these capture + * **colour** structure. Requiring both to agree is what keeps precision intact as a library + * grows, because unrelated pairs accumulate with the square of the library size and a single + * threshold cannot hold out against that. + */ +data class RgbDHash( + val red: Long, + val green: Long, + val blue: Long, +) { + /** Sum of the three channel Hamming distances, 0..192. */ + fun distanceTo(other: RgbDHash): Int = + java.lang.Long.bitCount(red xor other.red) + + java.lang.Long.bitCount(green xor other.green) + + java.lang.Long.bitCount(blue xor other.blue) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/AppPermissions.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/AppPermissions.kt new file mode 100644 index 0000000..7f4ef29 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/AppPermissions.kt @@ -0,0 +1,83 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.Manifest +import android.content.Context +import android.content.pm.PackageManager +import android.os.Build +import androidx.core.content.ContextCompat + +/** How much of the photo library the user has granted access to. */ +enum class MediaAccess { + /** No access yet — the app must ask. */ + NONE, + + /** Android 14+ "Selected photos" mode: only a user-picked subset is visible. */ + PARTIAL, + + /** Full access to the photo library. */ + FULL, +} + +/** + * Central place for the runtime permissions the app asks for. + * + * Photo access differs per API level: + * + * - API 33+ (`TIRAMISU`): `READ_MEDIA_IMAGES` + * - API 34+ (`UPSIDE_DOWN_CAKE`): optionally `READ_MEDIA_VISUAL_USER_SELECTED` for + * partial access + * - older: `READ_EXTERNAL_STORAGE` + * + * `POST_NOTIFICATIONS` (API 33+) is requested in the same dialog so the user only + * sees one prompt. It is used solely for the scan-progress notification; if it is + * denied the scan still runs, the notification is just hidden. + */ +object AppPermissions { + + /** + * The permissions to request for the running API level. Requesting the legacy and + * the granular photo permissions together is the documented way to support both + * old and new devices with a single dialog. + */ + fun requestablePermissions(): Array = buildList { + when { + Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE -> { + add(Manifest.permission.READ_MEDIA_IMAGES) + add(Manifest.permission.READ_MEDIA_VISUAL_USER_SELECTED) + } + + Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU -> { + add(Manifest.permission.READ_MEDIA_IMAGES) + } + + else -> { + add(Manifest.permission.READ_EXTERNAL_STORAGE) + } + } + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { + add(Manifest.permission.POST_NOTIFICATIONS) + } + }.toTypedArray() + + fun hasFullAccess(context: Context): Boolean = when { + Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU -> + isGranted(context, Manifest.permission.READ_MEDIA_IMAGES) + + else -> isGranted(context, Manifest.permission.READ_EXTERNAL_STORAGE) + } + + /** True when only the user-selected subset of photos is visible. */ + fun hasPartialAccess(context: Context): Boolean = + Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE && + !hasFullAccess(context) && + isGranted(context, Manifest.permission.READ_MEDIA_VISUAL_USER_SELECTED) + + fun accessState(context: Context): MediaAccess = when { + hasFullAccess(context) -> MediaAccess.FULL + hasPartialAccess(context) -> MediaAccess.PARTIAL + else -> MediaAccess.NONE + } + + private fun isGranted(context: Context, permission: String): Boolean = + ContextCompat.checkSelfPermission(context, permission) == PackageManager.PERMISSION_GRANTED +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/BoundedCall.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/BoundedCall.kt new file mode 100644 index 0000000..d9c0a5b --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/BoundedCall.kt @@ -0,0 +1,118 @@ +package de.softwareapp_hb.fotodup.finder.data + +import kotlinx.coroutines.CancellationException +import java.util.concurrent.Callable +import java.util.concurrent.ExecutionException +import java.util.concurrent.ExecutorService +import java.util.concurrent.Executors +import java.util.concurrent.Future +import java.util.concurrent.ThreadFactory +import java.util.concurrent.TimeUnit +import java.util.concurrent.TimeoutException +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicLong + +/** + * Runs a blocking platform call on its own thread and abandons it if it overruns a deadline. + * + * `ContentResolver` calls can block *inside the provider* for a long time — or for ever. On + * the test device `ContentResolver.loadThumbnail` never returned for one perfectly valid HEIC, + * because the provider hung in its own decode. That froze the entire scan at a single photo, + * and it cannot be bounded with `withTimeoutOrNull`: coroutine cancellation is **cooperative**, + * so a thread parked in a binder transaction or a native read never reaches a cancellation + * point. The only way to put a deadline on such a call is to run it on a thread of its own and + * stop *waiting* for it. + * + * When a call is abandoned the thread carrying it stays parked in the provider. That leak is + * deliberate and bounded: the thread does no work of its own, the pool is daemon-only so it can + * never keep the process alive, dropping one worker is strictly better than losing the scan — and + * once several calls in a row have been abandoned, [orNull] stops submitting altogether until the + * provider answers again (see [BREAKER_TRIP]). + * + * Deliberately free of Android imports, so it is testable on the JVM like the rest of the engine. + */ +internal object BoundedCall { + + /** + * Consecutive abandoned calls after which the provider is treated as *stalled*. + * + * One slow call is a bad file. Several in a row with nothing succeeding in between is the + * provider itself — a `MediaProvider` that is restarting, or a blocked FUSE daemon. At that + * point waiting out the deadline for every remaining photo is pointless, and it is what made + * a device scan sit at one progress value looking frozen for tens of minutes. + */ + private const val BREAKER_TRIP = 3 + + private val executor: ExecutorService = Executors.newCachedThreadPool( + ThreadFactory { runnable -> Thread(runnable, "provider-call").apply { isDaemon = true } }, + ) + + /** Abandoned calls since the provider last answered — i.e. threads parked inside it. */ + private val parked = AtomicInteger(0) + + /** While `now < stalledUntil`, submissions fast-fail instead of parking more threads. */ + private val stalledUntil = AtomicLong(0L) + + /** + * Returns the result of [block], or `null` if it threw or did not finish within [timeoutMs] + * (in which case it is abandoned — including anything it still holds, e.g. an open + * `ParcelFileDescriptor`). + * + * While the provider looks stalled, calls return `null` immediately instead of submitting: + * that bounds the number of parked threads to roughly one per cooldown and lets the scan + * reach its end (reporting the unreadable files) rather than grinding. + */ + fun orNull(timeoutMs: Long, block: () -> T?): T? { + if (System.currentTimeMillis() < stalledUntil.get()) return null + + val future: Future = executor.submit(Callable { block() }) + return try { + val value = future.get(timeoutMs, TimeUnit.MILLISECONDS) + // The provider answered: whatever stalled is over. + providerAlive() + value + } catch (e: TimeoutException) { + // Give back what we can; a thread stuck in a binder call will not notice. + future.cancel(true) + onTimeout(timeoutMs) + null + } catch (e: ExecutionException) { + val cause = e.cause + // Cancelling a scan has to keep propagating through here. + if (cause is CancellationException) throw cause + // The provider answered, just with an error (missing file, revoked permission). + providerAlive() + null + } catch (e: InterruptedException) { + future.cancel(true) + Thread.currentThread().interrupt() + throw CancellationException("interrupted while waiting for a provider call") + } + } + + private fun onTimeout(timeoutMs: Long) { + if (parked.incrementAndGet() >= BREAKER_TRIP) { + // Cooldown scales with the call deadline: long enough not to hammer a stalled + // provider (at most a thread or two per window), short enough that a brief + // MediaProvider restart is picked up again. + stalledUntil.set(System.currentTimeMillis() + timeoutMs.coerceAtLeast(1_000L)) + } + } + + private fun providerAlive() { + parked.set(0) + stalledUntil.set(0L) + } + + /** Threads currently abandoned inside the provider (diagnostics and tests). */ + internal fun parkedCount(): Int = parked.get() + + /** True while submissions are being suppressed because the provider looks stalled. */ + internal fun isStalled(): Boolean = System.currentTimeMillis() < stalledUntil.get() + + /** Test hook: forget any stall carried over from an earlier test in the same JVM. */ + internal fun resetForTest() { + parked.set(0) + stalledUntil.set(0L) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaDeleter.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaDeleter.kt new file mode 100644 index 0000000..a5f1a69 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaDeleter.kt @@ -0,0 +1,56 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.app.PendingIntent +import android.content.ContentResolver +import android.content.Context +import android.net.Uri +import android.os.Build +import android.provider.MediaStore + +/** + * Deletes user-selected photos. + * + * On Android 11+ deletion *always* goes through a system consent dialog + * (`MediaStore.createDeleteRequest` / `createTrashRequest`): the app can never + * delete silently and therefore never needs the heavily restricted + * `MANAGE_EXTERNAL_STORAGE` permission, which Google Play scrutinises closely. + * + * On Android 10 the legacy `ContentResolver.delete` path is used, guarded by + * `WRITE_EXTERNAL_STORAGE` (declared with `maxSdkVersion="29"`). + */ +class MediaDeleter(context: Context) { + + private val resolver: ContentResolver = context.applicationContext.contentResolver + + /** True when the platform requires a user-consented system dialog to delete. */ + val requiresSystemConsent: Boolean get() = Build.VERSION.SDK_INT >= Build.VERSION_CODES.R + + /** True when photos can be moved to the system trash instead of deleted. */ + val supportsTrash: Boolean get() = Build.VERSION.SDK_INT >= Build.VERSION_CODES.R + + /** + * Builds the pending intent that shows the system "allow deleting" dialog. + * Returns `null` on API levels that do not support the request API. + */ + fun createDeleteRequest(uris: List, useTrash: Boolean): PendingIntent? { + if (Build.VERSION.SDK_INT < Build.VERSION_CODES.R || uris.isEmpty()) return null + return if (useTrash) { + MediaStore.createTrashRequest(resolver, uris, true) + } else { + MediaStore.createDeleteRequest(resolver, uris) + } + } + + /** Direct deletion used on API 26-29. Returns how many rows were removed. */ + fun deleteDirectly(uris: List): Int { + var deleted = 0 + uris.forEach { uri -> + try { + deleted += resolver.delete(uri, null, null) + } catch (e: SecurityException) { + // Not allowed to remove this item — skip it and keep going. + } + } + return deleted + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaStorePhotoSource.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaStorePhotoSource.kt new file mode 100644 index 0000000..66dc61d --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/MediaStorePhotoSource.kt @@ -0,0 +1,103 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.content.ContentUris +import android.content.Context +import android.database.Cursor +import android.net.Uri +import android.os.Build +import android.provider.MediaStore +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext + +/** + * Reads the device image library through MediaStore. + * + * Only metadata is read here; pixel data is touched later (and lazily) by + * [PhotoHasher], so listing a large library stays cheap. + */ +class MediaStorePhotoSource(context: Context) { + + private val resolver = context.applicationContext.contentResolver + + suspend fun loadPhotos(): List = withContext(Dispatchers.IO) { + val collection = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { + MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL) + } else { + @Suppress("DEPRECATION") + MediaStore.Images.Media.EXTERNAL_CONTENT_URI + } + + val projection = buildList { + add(MediaStore.Images.Media._ID) + add(MediaStore.Images.Media.DISPLAY_NAME) + add(MediaStore.Images.Media.SIZE) + add(MediaStore.Images.Media.DATE_MODIFIED) + add(MediaStore.Images.Media.WIDTH) + add(MediaStore.Images.Media.HEIGHT) + add(MediaStore.Images.Media.MIME_TYPE) + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { + add(MediaStore.Images.Media.RELATIVE_PATH) + } else { + @Suppress("DEPRECATION") + add(MediaStore.Images.Media.BUCKET_DISPLAY_NAME) + } + }.toTypedArray() + + val photos = ArrayList() + resolver.query( + collection, + projection, + null, + null, + "${MediaStore.Images.Media.DATE_MODIFIED} DESC", + )?.use { cursor -> readRows(cursor, photos) } + photos + } + + private fun readRows(cursor: Cursor, out: MutableList) { + val idColumn = cursor.getColumnIndexOrThrow(MediaStore.Images.Media._ID) + val sizeColumn = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.SIZE) + val dateColumn = cursor.getColumnIndex(MediaStore.Images.Media.DATE_MODIFIED) + val widthColumn = cursor.getColumnIndex(MediaStore.Images.Media.WIDTH) + val heightColumn = cursor.getColumnIndex(MediaStore.Images.Media.HEIGHT) + val nameColumn = cursor.getColumnIndex(MediaStore.Images.Media.DISPLAY_NAME) + val mimeColumn = cursor.getColumnIndex(MediaStore.Images.Media.MIME_TYPE) + val folderColumn = cursor.getColumnIndex( + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { + MediaStore.Images.Media.RELATIVE_PATH + } else { + @Suppress("DEPRECATION") + MediaStore.Images.Media.BUCKET_DISPLAY_NAME + }, + ) + + while (cursor.moveToNext()) { + val id = cursor.getLong(idColumn) + val size = cursor.getLong(sizeColumn) + // Ignore empty/zero-length entries; they cannot be real photos. + if (size <= 0L) continue + + out += PhotoRecord( + id = id, + uri = ContentUris.withAppendedId( + MediaStore.Images.Media.EXTERNAL_CONTENT_URI, + id, + ), + displayName = nameColumn.takeIf { it >= 0 }?.let { cursor.getStringOrNull(it) }, + sizeBytes = size, + dateModifiedSeconds = dateColumn.takeIf { it >= 0 }?.let { cursor.getLong(it) } ?: 0L, + width = widthColumn.takeIf { it >= 0 }?.let { cursor.getInt(it) } ?: 0, + height = heightColumn.takeIf { it >= 0 }?.let { cursor.getInt(it) } ?: 0, + mimeType = mimeColumn.takeIf { it >= 0 }?.let { cursor.getStringOrNull(it) }, + folder = folderColumn.takeIf { it >= 0 }?.let { cursor.getStringOrNull(it) }, + ) + } + } + + private fun Cursor.getStringOrNull(columnIndex: Int): String? = + if (isNull(columnIndex)) null else getString(columnIndex) + + /** Kept for symmetry with the delete flow which needs photo uris by id. */ + fun uriFor(id: Long): Uri = + ContentUris.withAppendedId(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, id) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoHasher.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoHasher.kt new file mode 100644 index 0000000..d00f6eb --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoHasher.kt @@ -0,0 +1,239 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.content.Context +import android.graphics.Bitmap +import android.graphics.BitmapFactory +import android.graphics.Matrix +import android.net.Uri +import android.os.Build +import android.provider.MediaStore +import android.util.Size +import androidx.exifinterface.media.ExifInterface +import de.softwareapp_hb.fotodup.finder.core.DecodeMath +import de.softwareapp_hb.fotodup.finder.core.Hashing +import de.softwareapp_hb.fotodup.finder.core.PerceptualFingerprint +import de.softwareapp_hb.fotodup.finder.core.PerceptualHash +import de.softwareapp_hb.fotodup.finder.core.readAtMost +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext +import java.io.FileInputStream +import java.io.IOException +import java.nio.ByteBuffer + +/** + * Computes the three hashes the engine needs for one photo. + * + * All work happens on [Dispatchers.IO]: the heavy parts are file I/O and decoding a + * deliberately tiny version of the bitmap. + */ +class PhotoHasher( + context: Context, + /** + * Deadline for a single provider call. + * + * Injectable so JVM tests can drive the "provider blocks for ever" path without waiting + * the production 10 s per file. Production always uses [PROVIDER_CALL_TIMEOUT_MS]. + */ + private val providerCallTimeoutMs: Long = PROVIDER_CALL_TIMEOUT_MS, +) { + + private val resolver = context.applicationContext.contentResolver + + /** + * Cheap signature over size + head/tail bytes. + * + * Opens the file **once** and reads both the head and the tail through the same + * channel, so this costs a single file descriptor per photo. Returns `null` when + * the file cannot be read (revoked permission, corrupt entry, ...). + */ + suspend fun quickSignature(photo: PhotoRecord): String? = withContext(Dispatchers.IO) { + BoundedCall.orNull(providerCallTimeoutMs) { + val window = Hashing.SIGNATURE_WINDOW + val descriptor = resolver.openFileDescriptor(photo.uri, "r") ?: return@orNull null + descriptor.use { pfd -> + val stream = FileInputStream(pfd.fileDescriptor) + val head = readAt(stream, offset = 0L, length = window) + if (head.isEmpty()) return@orNull null + val tail = if (photo.sizeBytes > head.size) { + readAt( + stream, + offset = (photo.sizeBytes - window).coerceAtLeast(0L), + length = window, + ) + } else { + ByteArray(0) + } + Hashing.quickSignature(photo.sizeBytes, head, tail) + } + } + } + + /** Full content hash; equal hashes mean byte-identical files. */ + suspend fun contentHash(photo: PhotoRecord): String? = withContext(Dispatchers.IO) { + BoundedCall.orNull(providerCallTimeoutMs) { + resolver.openInputStream(photo.uri)?.use { stream -> Hashing.sha256(stream) } + } + } + + /** + * Both perceptual descriptors, from a single decoded thumbnail. + * + * Prefers the media provider's **thumbnail cache** + * ([ContentResolver.loadThumbnail]); thumbnails are usually already generated by the + * gallery, so this avoids decoding the full-resolution original — the single most expensive + * step of a scan. Falls back to a subsampled decode when no thumbnail exists. + * + * Returns `null` for anything it cannot decode: a truncated file, a corrupt entry, an + * unrecognised format such as RAW/DNG, or a provider handing back something unusable. Those + * photos simply take no part in near-duplicate matching. + */ + suspend fun perceptualFingerprint(photo: PhotoRecord): PerceptualFingerprint? = + withContext(Dispatchers.IO) { + BoundedCall.orNull(providerCallTimeoutMs) { + val bitmap = systemThumbnail(photo) + ?: decodeSampled(photo.uri) + ?: return@orNull null + + try { + val width = bitmap.width + val height = bitmap.height + // Guard the allocation: a provider could hand back a full-resolution bitmap + // where a thumbnail was asked for, and a bogus size would otherwise blow up + // deep inside IntArray(). + val pixelCount = width.toLong() * height + if (pixelCount <= 0L || pixelCount > MAX_PIXELS) return@orNull null + + val pixels = IntArray(pixelCount.toInt()) + bitmap.getPixels(pixels, 0, width, 0, 0, width, height) + PerceptualFingerprint( + phash = PerceptualHash.phashFromArgb(pixels, width, height), + rgb = PerceptualHash.rgbDHashFromArgb(pixels, width, height), + ) + } finally { + bitmap.recycle() + } + } + } + + /** Small bitmap straight from the media provider's thumbnail cache, or `null`. */ + private fun systemThumbnail(photo: PhotoRecord): Bitmap? = if ( + Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q + ) { + resolver.loadThumbnail(photo.uri, Size(THUMBNAIL_EDGE, THUMBNAIL_EDGE), null) + } else { + @Suppress("DEPRECATION") + MediaStore.Images.Thumbnails.getThumbnail( + resolver, + photo.id, + MediaStore.Images.Thumbnails.MICRO_KIND, + null, + ) + } + + /** Fallback: decode the original at a heavily subsampled size, EXIF-oriented. */ + private fun decodeSampled(uri: Uri): Bitmap? { + val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true } + resolver.openInputStream(uri)?.use { BitmapFactory.decodeStream(it, null, bounds) } + val width = bounds.outWidth + val height = bounds.outHeight + if (width <= 0 || height <= 0) return null + + val options = BitmapFactory.Options().apply { + inSampleSize = DecodeMath.sampleSizeFor(width, height, MAX_DECODE_EDGE) + inPreferredConfig = Bitmap.Config.RGB_565 + } + val decoded = resolver.openInputStream(uri)?.use { + BitmapFactory.decodeStream(it, null, options) + } ?: return null + return applyExifOrientation(uri, decoded) + } + + /** + * Rotates/flips according to the EXIF orientation tag. + * + * `BitmapFactory` ignores EXIF while the media provider's thumbnail does not, so without + * this the *same photo* would hash differently depending on which path happened to serve it + * — and since the thumbnail cache can be evicted between scans, a cached hash could stop + * matching a recomputed one. Normalising here makes both paths agree. + */ + private fun applyExifOrientation(uri: Uri, bitmap: Bitmap): Bitmap { + val orientation = try { + resolver.openInputStream(uri)?.use { + ExifInterface(it).getAttributeInt( + ExifInterface.TAG_ORIENTATION, + ExifInterface.ORIENTATION_NORMAL, + ) + } ?: ExifInterface.ORIENTATION_NORMAL + } catch (e: IOException) { + ExifInterface.ORIENTATION_NORMAL + } + + val matrix = Matrix() + when (orientation) { + ExifInterface.ORIENTATION_ROTATE_90 -> matrix.postRotate(90f) + ExifInterface.ORIENTATION_ROTATE_180 -> matrix.postRotate(180f) + ExifInterface.ORIENTATION_ROTATE_270 -> matrix.postRotate(270f) + ExifInterface.ORIENTATION_FLIP_HORIZONTAL -> matrix.postScale(-1f, 1f) + ExifInterface.ORIENTATION_FLIP_VERTICAL -> matrix.postScale(1f, -1f) + ExifInterface.ORIENTATION_TRANSPOSE -> { + matrix.postRotate(90f) + matrix.postScale(-1f, 1f) + } + + ExifInterface.ORIENTATION_TRANSVERSE -> { + matrix.postRotate(270f) + matrix.postScale(-1f, 1f) + } + + else -> return bitmap + } + + val oriented = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true) + if (oriented !== bitmap) bitmap.recycle() + return oriented + } + + /** + * Reads [length] bytes at [offset] through [stream]'s channel. Providers that + * cannot seek fall back to a sequential read, which only makes sense at offset 0. + */ + private fun readAt(stream: FileInputStream, offset: Long, length: Int): ByteArray { + if (length <= 0) return ByteArray(0) + return try { + val channel = stream.channel + channel.position(offset) + val buffer = ByteBuffer.allocate(length) + var total = 0 + while (buffer.hasRemaining()) { + val read = channel.read(buffer) + // Same hazard as readAtMost: a pipe-backed descriptor can report 0 for ever, + // and spinning on it would hang the scan. + if (read <= 0) break + total += read + } + buffer.array().copyOf(total) + } catch (e: IOException) { + if (offset == 0L) stream.readAtMost(length) else ByteArray(0) + } + } + + private companion object { + /** + * Deadline for a single call into the media provider. + * + * A provider call can block indefinitely (see [BoundedCall]); this bounds how long one + * photo may hold up a scan before it is skipped. Deliberately shorter than the engine's + * per-photo budget so a slow provider still leaves room for the fallback decode. + */ + const val PROVIDER_CALL_TIMEOUT_MS = 10_000L + + /** Decode target for the fallback path — the 9x8 grid needs very little. */ + const val MAX_DECODE_EDGE = 64 + + /** Edge length requested from the media provider's thumbnail cache. */ + const val THUMBNAIL_EDGE = 96 + + /** Hard ceiling on the pixels we will allocate for one fingerprint. */ + const val MAX_PIXELS = 4_000_000L + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoIndexStore.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoIndexStore.kt new file mode 100644 index 0000000..2789382 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoIndexStore.kt @@ -0,0 +1,173 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.content.Context +import android.database.sqlite.SQLiteDatabase +import de.softwareapp_hb.fotodup.finder.core.RgbDHash +import android.database.sqlite.SQLiteOpenHelper +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext + +/** One cached row of the duplicate index. */ +data class IndexEntry( + val id: Long, + val sizeBytes: Long, + val dateModifiedSeconds: Long, + val quickSignature: String?, + val contentHash: String?, + val perceptualHash: Long?, + val rgbDHash: RgbDHash?, +) + +/** + * Persistent cache of computed hashes, keyed by MediaStore id. + * + * Re-scanning a big library is then fast: a row is only recomputed when the file's + * size or last-modified stamp changed. Implemented on top of the framework + * `SQLiteOpenHelper` (no Room/KSP) to keep the build lean. + */ +class PhotoIndexStore(context: Context) { + + private val helper = Helper(context.applicationContext) + + suspend fun loadAll(): Map = withContext(Dispatchers.IO) { + val result = HashMap() + helper.readableDatabase.rawQuery( + "SELECT id, size, date_modified, quick_sig, content_hash, perceptual_hash, " + + "rgb_red, rgb_green, rgb_blue FROM $TABLE", + null, + ).use { cursor -> + while (cursor.moveToNext()) { + val id = cursor.getLong(0) + val rgb = if (cursor.isNull(6) || cursor.isNull(7) || cursor.isNull(8)) { + null + } else { + RgbDHash(cursor.getLong(6), cursor.getLong(7), cursor.getLong(8)) + } + result[id] = IndexEntry( + id = id, + sizeBytes = cursor.getLong(1), + dateModifiedSeconds = cursor.getLong(2), + quickSignature = cursor.getString(3), + contentHash = cursor.getString(4), + perceptualHash = if (cursor.isNull(5)) null else cursor.getLong(5), + rgbDHash = rgb, + ) + } + } + result + } + + /** + * Writes entries in one transaction through a compiled statement. A large library + * means tens of thousands of rows, and rebuilding a `ContentValues` per row is + * needless overhead. + */ + suspend fun upsert(entries: Collection) = withContext(Dispatchers.IO) { + if (entries.isEmpty()) return@withContext + val db = helper.writableDatabase + val statement = db.compileStatement( + "INSERT OR REPLACE INTO $TABLE " + + "(id, size, date_modified, quick_sig, content_hash, perceptual_hash, " + + "rgb_red, rgb_green, rgb_blue) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + ) + db.beginTransaction() + try { + entries.forEach { entry -> + statement.clearBindings() + statement.bindLong(1, entry.id) + statement.bindLong(2, entry.sizeBytes) + statement.bindLong(3, entry.dateModifiedSeconds) + entry.quickSignature + ?.let { statement.bindString(4, it) } + ?: statement.bindNull(4) + entry.contentHash + ?.let { statement.bindString(5, it) } + ?: statement.bindNull(5) + entry.perceptualHash + ?.let { statement.bindLong(6, it) } + ?: statement.bindNull(6) + val rgb = entry.rgbDHash + if (rgb == null) { + statement.bindNull(7) + statement.bindNull(8) + statement.bindNull(9) + } else { + statement.bindLong(7, rgb.red) + statement.bindLong(8, rgb.green) + statement.bindLong(9, rgb.blue) + } + statement.executeInsert() + } + db.setTransactionSuccessful() + } finally { + db.endTransaction() + statement.close() + } + } + + /** Removes rows for photos that no longer exist in MediaStore. */ + suspend fun pruneTo(knownIds: Set) = withContext(Dispatchers.IO) { + val db = helper.writableDatabase + db.beginTransaction() + try { + // Deleting by iterating avoids building a huge "NOT IN (...)" clause. + val stale = ArrayList() + db.rawQuery("SELECT id FROM $TABLE", null).use { cursor -> + while (cursor.moveToNext()) { + val id = cursor.getLong(0) + if (id !in knownIds) stale += id + } + } + stale.forEach { db.delete(TABLE, "id = ?", arrayOf(it.toString())) } + db.setTransactionSuccessful() + } finally { + db.endTransaction() + } + } + + suspend fun clear() = withContext(Dispatchers.IO) { + helper.writableDatabase.delete(TABLE, null, null) + Unit + } + + private class Helper(context: Context) : SQLiteOpenHelper(context, DB_NAME, null, DB_VERSION) { + + override fun onCreate(db: SQLiteDatabase) { + db.execSQL( + """ + CREATE TABLE $TABLE ( + id INTEGER PRIMARY KEY, + size INTEGER NOT NULL, + date_modified INTEGER NOT NULL, + quick_sig TEXT, + content_hash TEXT, + perceptual_hash INTEGER, + rgb_red INTEGER, + rgb_green INTEGER, + rgb_blue INTEGER + ) + """.trimIndent(), + ) + } + + override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { + // The index is a pure cache: dropping and rebuilding is always safe. + db.execSQL("DROP TABLE IF EXISTS $TABLE") + onCreate(db) + } + } + + private companion object { + const val DB_NAME = "photo_index.db" + + /** + * Bump whenever the *meaning* of a stored hash changes. Version 2 switched the + * perceptual hash to the media provider's thumbnail; version 3 replaced dHash with pHash + * and added the RGB channel hashes. Stored hashes from an older version are not + * comparable, so the index is dropped and rebuilt. + */ + const val DB_VERSION = 3 + const val TABLE = "photo_index" + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoRecord.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoRecord.kt new file mode 100644 index 0000000..f80d43a --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/data/PhotoRecord.kt @@ -0,0 +1,42 @@ +package de.softwareapp_hb.fotodup.finder.data + +import android.net.Uri + +/** + * One photo as reported by MediaStore. + * + * This is the UI/hashing-facing view of a library item; the pure duplicate engine + * only ever sees [de.softwareapp_hb.fotodup.finder.core.PhotoFingerprint]. + */ +data class PhotoRecord( + val id: Long, + val uri: Uri, + val displayName: String?, + val sizeBytes: Long, + val dateModifiedSeconds: Long, + val width: Int, + val height: Int, + val mimeType: String?, + /** Folder the photo lives in, e.g. `DCIM/Camera` (API 29+) or a bucket name. */ + val folder: String?, +) { + /** Human readable size, e.g. "3.4 MB". */ + val readableSize: String get() = humanReadableBytes(sizeBytes) +} + +/** Formats a byte count using binary (KiB/MiB/GiB) units. */ +fun humanReadableBytes(bytes: Long): String { + if (bytes < 1024) return "$bytes B" + val units = arrayOf("KB", "MB", "GB", "TB") + var value = bytes.toDouble() / 1024.0 + var unitIndex = 0 + while (value >= 1024.0 && unitIndex < units.lastIndex) { + value /= 1024.0 + unitIndex++ + } + return if (value >= 100) { + "${value.toInt()} ${units[unitIndex]}" + } else { + String.format(java.util.Locale.US, "%.1f %s", value, units[unitIndex]) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/Parallel.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/Parallel.kt new file mode 100644 index 0000000..fa5297f --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/Parallel.kt @@ -0,0 +1,46 @@ +package de.softwareapp_hb.fotodup.finder.domain + +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.async +import kotlinx.coroutines.awaitAll +import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.sync.Semaphore +import kotlinx.coroutines.sync.withPermit + +/** Items handed to a worker at a time: far fewer coroutines, less scheduling churn. */ +private const val BATCH_SIZE = 32 + +/** + * Runs [block] for every item, with at most [concurrency] invocations in flight. + * + * Items are processed in small batches, so a library of tens of thousands of photos + * creates a few hundred coroutines instead of tens of thousands — the per-item + * coroutine bookkeeping used to be a measurable part of scan time. + */ +internal suspend fun List.forEachParallel( + concurrency: Int, + block: suspend (T) -> Unit, +) { + if (isEmpty()) return + val permits = Semaphore(concurrency.coerceAtLeast(1)) + coroutineScope { + chunked(BATCH_SIZE).map { batch -> + async(Dispatchers.IO) { + permits.withPermit { + for (item in batch) block(item) + } + } + }.awaitAll() + } +} + +/** + * Worker count for hashing. + * + * The work is a mix of file I/O and CPU-bound image decoding. Going much past the + * number of performance cores makes decoding contend instead of scale, and saturating + * every core starves the UI thread — which makes the app *feel* slow even when the + * scan finishes sooner. Hence the ceiling of 6 rather than all cores. + */ +internal fun defaultConcurrency(): Int = + Runtime.getRuntime().availableProcessors().coerceIn(2, 6) diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanController.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanController.kt new file mode 100644 index 0000000..31c5eaa --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanController.kt @@ -0,0 +1,150 @@ +package de.softwareapp_hb.fotodup.finder.domain + +import android.content.Context +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.data.MediaStorePhotoSource +import de.softwareapp_hb.fotodup.finder.data.PhotoHasher +import de.softwareapp_hb.fotodup.finder.data.PhotoIndexStore +import de.softwareapp_hb.fotodup.finder.service.ScanService +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.Job +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow +import kotlinx.coroutines.flow.update +import kotlinx.coroutines.launch + +/** + * Process-wide state of the (single) running scan. + * + * @property generation increments every time a scan completes, so observers can tell a + * *new* result from a re-emission of the previous one (which happens whenever the + * collector restarts, e.g. after a configuration change). Only a new result should + * reset the user's selection. + */ +data class ScanState( + val scanning: Boolean = false, + val phase: ScanPhase? = null, + val done: Int = 0, + val total: Int = 0, + val result: ScanResult? = null, + val error: String? = null, + val generation: Int = 0, +) { + /** 0f..1f, or `null` when the current phase has no measurable total. */ + val progressFraction: Float? + get() = if (total > 0 && done <= total) done.toFloat() / total else null +} + +/** + * Owns the running scan for the whole process. + * + * The scan deliberately does **not** live in a ViewModel: the first scan of a large + * library takes a while and the user will leave the screen. Running it here — with + * [ScanService] holding a foreground service for the duration — means the scan keeps + * going when the app is backgrounded, and its result survives activity recreation. + */ +class ScanController(context: Context) { + + private val appContext = context.applicationContext + + private val engine = ScanEngine( + source = MediaStorePhotoSource(appContext), + hasher = PhotoHasher(appContext), + index = PhotoIndexStore(appContext), + ) + private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default) + + private val _state = MutableStateFlow(ScanState()) + val state: StateFlow = _state.asStateFlow() + + private var job: Job? = null + + val isRunning: Boolean get() = job?.isActive == true + + /** Starts a scan; does nothing if one is already running. */ + fun start(policy: DuplicatePolicy) { + if (isRunning) return + + // Flip to "scanning" *before* starting the service: the service observes this + // state and would otherwise stop itself immediately. + _state.update { + it.copy( + scanning = true, + phase = ScanPhase.LOADING, + done = 0, + total = 0, + result = null, + error = null, + ) + } + ScanService.start(appContext) + + job = scope.launch { + try { + engine.scan(policy).collect(::onProgress) + } finally { + // Covers completion, failure and cancellation alike. + _state.update { current -> + if (current.scanning) current.copy(scanning = false, phase = null) else current + } + job = null + ScanService.stop(appContext) + } + } + } + + fun cancel() { + job?.cancel() + job = null + _state.update { it.copy(scanning = false, phase = null) } + ScanService.stop(appContext) + } + + private fun onProgress(progress: ScanProgress) { + when (progress) { + is ScanProgress.Working -> _state.update { + it.copy(phase = progress.phase, done = progress.done, total = progress.total) + } + + is ScanProgress.Finished -> _state.update { + it.copy( + scanning = false, + phase = null, + result = progress.result, + generation = it.generation + 1, + ) + } + + is ScanProgress.Failed -> _state.update { + it.copy(scanning = false, phase = null, error = progress.message) + } + } + } + + /** + * Keeps the shared result in sync when the UI deletes photos, so the state is still + * correct after an activity restart. + */ + fun removePhotos(deletedIds: Set) { + _state.update { current -> + val result = current.result ?: return@update current + val groups = result.groups + .map { group -> group.copy(members = group.members.filterNot { it.id in deletedIds }) } + .filter { it.members.size >= 2 } + val records = result.recordsById.filterKeys { it !in deletedIds } + current.copy( + result = result.copy( + photoCount = records.size, + groups = groups, + recordsById = records, + ), + ) + } + } + + /** Clears a finished error after the UI has shown it. */ + fun clearError() = _state.update { it.copy(error = null) } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngine.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngine.kt new file mode 100644 index 0000000..39306c1 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngine.kt @@ -0,0 +1,290 @@ +package de.softwareapp_hb.fotodup.finder.domain + +import de.softwareapp_hb.fotodup.finder.core.DuplicateGrouper +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.core.PerceptualFingerprint +import de.softwareapp_hb.fotodup.finder.core.PhotoFingerprint +import de.softwareapp_hb.fotodup.finder.data.IndexEntry +import de.softwareapp_hb.fotodup.finder.data.MediaStorePhotoSource +import de.softwareapp_hb.fotodup.finder.data.PhotoHasher +import de.softwareapp_hb.fotodup.finder.data.PhotoIndexStore +import de.softwareapp_hb.fotodup.finder.data.PhotoRecord +import kotlinx.coroutines.CancellationException +import kotlinx.coroutines.flow.Flow +import kotlinx.coroutines.flow.channelFlow +import kotlinx.coroutines.withTimeoutOrNull +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicInteger + +/** + * Orchestrates a full duplicate scan and streams progress. + * + * The pipeline is deliberately tiered so that only the photos that *can* be + * duplicates get the expensive treatment: + * + * 1. list the library from MediaStore, + * 2. compute a cheap size+head/tail signature for every photo, + * 3. full-hash only same-size files that share a signature, + * 4. perceptual-hash everything (needed to find *near* duplicates), + * 5. persist the hashes so the next scan skips unchanged files, + * 6. group the results. + * + * Steps 2-4 are parallelised across CPU cores. Unchanged photos reuse their + * cached hashes, which makes repeat scans cheap. + */ +/** + * Abandon a photo whose decode has not finished in this long. + * + * A single pathological file (a corrupt entry a provider blocks on, an unusually expensive + * HEIC/RAW decode) would otherwise stall the whole scan. Cancelling leaves the file out of the + * results instead, which is the same outcome as failing to decode it. + */ +private const val PER_PHOTO_TIMEOUT_MS = 15_000L + +class ScanEngine( + private val source: MediaStorePhotoSource, + private val hasher: PhotoHasher, + private val index: PhotoIndexStore, +) { + + fun scan(policy: DuplicatePolicy = DuplicatePolicy.DEFAULT): Flow = channelFlow { + val startedAt = System.currentTimeMillis() + + suspend fun report(phase: ScanPhase, done: Int, total: Int) { + send(ScanProgress.Working(phase, done, total)) + } + + try { + send(ScanProgress.Working(ScanPhase.LOADING)) + val photos = source.loadPhotos() + if (photos.isEmpty()) { + send(ScanProgress.Finished(ScanResult(0, emptyList(), emptyMap(), elapsed(startedAt)))) + return@channelFlow + } + + // Only cached rows whose size and timestamp still match are trustworthy. + val freshPrior = freshPriorEntries(photos, index.loadAll()) + val concurrency = defaultConcurrency() + val progressStep = (photos.size / 100).coerceAtLeast(1) + + // KEY OPTIMISATION: a file can only be a byte-for-byte duplicate if + // another file has *exactly* the same size. In a typical library the + // overwhelming majority of photos have a unique size, so for those we + // skip signature and content hashing entirely — no file is ever opened. + // Only same-size groups can produce exact duplicates. + val ambiguousSizes = photos + .groupingBy { it.sizeBytes } + .eachCount() + .filterValues { count -> count >= 2 } + .keys + val exactCandidates = photos.filter { it.sizeBytes in ambiguousSizes } + + val signaturesStartedAt = System.currentTimeMillis() + val quickSignatures = computeQuickSignatures( + exactCandidates, freshPrior, concurrency, progressStep, + ) { done, total -> report(ScanPhase.SIGNATURES, done, total) } + val signaturesMs = System.currentTimeMillis() - signaturesStartedAt + + val contentStartedAt = System.currentTimeMillis() + val contentHashes = computeContentHashes( + exactCandidates, quickSignatures, freshPrior, concurrency, progressStep, + ) { done, total -> report(ScanPhase.CONTENT, done, total) } + val contentMs = System.currentTimeMillis() - contentStartedAt + + val perceptualStartedAt = System.currentTimeMillis() + val perceptual = computePerceptualFingerprints( + photos, freshPrior, policy, concurrency, progressStep, + ) { done, total -> report(ScanPhase.PERCEPTUAL, done, total) } + val perceptualMs = System.currentTimeMillis() - perceptualStartedAt + + report(ScanPhase.GROUPING, 0, 0) + // The index is a cache: losing a write must not discard the scan that just ran, so a + // failure here costs the next scan some re-hashing and nothing else. + try { + persist(photos, freshPrior, quickSignatures, contentHashes, perceptual) + } catch (e: CancellationException) { + throw e + } catch (t: Throwable) { + // ignored on purpose + } + + val fingerprints = photos.map { photo -> + PhotoFingerprint( + id = photo.id, + sizeBytes = photo.sizeBytes, + dateModifiedSeconds = photo.dateModifiedSeconds, + width = photo.width, + height = photo.height, + quickSignature = quickSignatures[photo.id], + contentHash = contentHashes[photo.id], + perceptualHash = perceptual[photo.id]?.phash, + rgbDHash = perceptual[photo.id]?.rgb, + ) + } + val grouping = DuplicateGrouper.findGroupsDetailed(fingerprints, policy) + + // Photos we could not decode still matched exactly; report how many so they are not a + // silent gap. + val unreadableCount = if (policy.includePerceptual) { + photos.count { perceptual[it.id] == null } + } else { + 0 + } + + val totalMs = elapsed(startedAt) + send( + ScanProgress.Finished( + ScanResult( + photoCount = photos.size, + groups = grouping.groups, + recordsById = photos.associateBy { it.id }, + durationMs = totalMs, + timings = ScanTimings( + signaturesMs = signaturesMs, + contentMs = contentMs, + perceptualMs = perceptualMs, + totalMs = totalMs, + ), + perceptualThresholds = grouping.thresholds, + thresholdWasFallback = grouping.usedFallbackThreshold, + unreadableCount = unreadableCount, + ), + ), + ) + } catch (e: CancellationException) { + throw e + } catch (t: Throwable) { + send(ScanProgress.Failed(t.message ?: t::class.java.simpleName)) + } + } + + private fun freshPriorEntries(photos: List, cached: Map): Map { + if (cached.isEmpty()) return emptyMap() + val byId = photos.associateBy { it.id } + val fresh = HashMap() + cached.forEach { (id, entry) -> + val photo = byId[id] ?: return@forEach + if (photo.sizeBytes == entry.sizeBytes && photo.dateModifiedSeconds == entry.dateModifiedSeconds) { + fresh[id] = entry + } + } + return fresh + } + + /** + * Tier 1: a cheap signature for every photo. Reuses the cached value when the + * file is unchanged. + */ + private suspend fun computeQuickSignatures( + photos: List, + prior: Map, + concurrency: Int, + progressStep: Int, + onProgress: suspend (Int, Int) -> Unit, + ): Map { + val result = ConcurrentHashMap() + val counter = AtomicInteger() + val total = photos.size + photos.forEachParallel(concurrency) { photo -> + val signature = withTimeoutOrNull(PER_PHOTO_TIMEOUT_MS) { + prior[photo.id]?.quickSignature ?: hasher.quickSignature(photo) + } + if (signature != null) result[photo.id] = signature + val done = counter.incrementAndGet() + if (done % progressStep == 0 || done > total - progressStep) onProgress(done, total) + } + return result + } + + /** + * Tier 2: full content hashes, but only for photos that share a size *and* a + * quick signature with at least one other photo. + */ + private suspend fun computeContentHashes( + photos: List, + quickSignatures: Map, + prior: Map, + concurrency: Int, + progressStep: Int, + onProgress: suspend (Int, Int) -> Unit, + ): Map { + val result = ConcurrentHashMap() + val candidates = photos + .filter { quickSignatures[it.id] != null } + .groupBy { it.sizeBytes to quickSignatures.getValue(it.id) } + .values + .filter { it.size >= 2 } + .flatten() + if (candidates.isEmpty()) return result + + val counter = AtomicInteger() + val total = candidates.size + candidates.forEachParallel(concurrency) { photo -> + val hash = withTimeoutOrNull(PER_PHOTO_TIMEOUT_MS) { + prior[photo.id]?.contentHash ?: hasher.contentHash(photo) + } + if (hash != null) result[photo.id] = hash + val done = counter.incrementAndGet() + if (done % progressStep == 0 || done > total - progressStep) onProgress(done, total) + } + return result + } + + /** + * Tier 3: both perceptual descriptors for every photo, from a single decode each. + * Skipped entirely for [DuplicatePolicy.EXACT_ONLY]. + */ + private suspend fun computePerceptualFingerprints( + photos: List, + prior: Map, + policy: DuplicatePolicy, + concurrency: Int, + progressStep: Int, + onProgress: suspend (Int, Int) -> Unit, + ): Map { + val result = ConcurrentHashMap() + if (!policy.includePerceptual) return result + + val counter = AtomicInteger() + val total = photos.size + photos.forEachParallel(concurrency) { photo -> + val cached = prior[photo.id] + val fingerprint = withTimeoutOrNull(PER_PHOTO_TIMEOUT_MS) { + if (cached?.perceptualHash != null && cached.rgbDHash != null) { + PerceptualFingerprint(cached.perceptualHash, cached.rgbDHash) + } else { + hasher.perceptualFingerprint(photo) + } + } + if (fingerprint != null) result[photo.id] = fingerprint + val done = counter.incrementAndGet() + if (done % progressStep == 0 || done > total - progressStep) onProgress(done, total) + } + return result + } + + private suspend fun persist( + photos: List, + prior: Map, + quickSignatures: Map, + contentHashes: Map, + perceptual: Map, + ) { + val entries = photos.map { photo -> + val previous = prior[photo.id] + IndexEntry( + id = photo.id, + sizeBytes = photo.sizeBytes, + dateModifiedSeconds = photo.dateModifiedSeconds, + quickSignature = quickSignatures[photo.id] ?: previous?.quickSignature, + contentHash = contentHashes[photo.id] ?: previous?.contentHash, + perceptualHash = perceptual[photo.id]?.phash ?: previous?.perceptualHash, + rgbDHash = perceptual[photo.id]?.rgb ?: previous?.rgbDHash, + ) + } + index.upsert(entries) + index.pruneTo(photos.mapTo(HashSet(photos.size)) { it.id }) + } + + private fun elapsed(startedAt: Long): Long = System.currentTimeMillis() - startedAt +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanModels.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanModels.kt new file mode 100644 index 0000000..749669f --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/domain/ScanModels.kt @@ -0,0 +1,67 @@ +package de.softwareapp_hb.fotodup.finder.domain + +import de.softwareapp_hb.fotodup.finder.core.DuplicateGroup +import de.softwareapp_hb.fotodup.finder.core.PerceptualThresholds +import de.softwareapp_hb.fotodup.finder.data.PhotoRecord + +/** + * The stage of a scan, used to label progress in the UI and the notification. + * + * Deliberately carries no text: the wording lives in resources and is mapped by + * `text.labelRes()`, so the screen and the notification cannot drift apart. + */ +enum class ScanPhase { + LOADING, + SIGNATURES, + CONTENT, + PERCEPTUAL, + GROUPING, +} + +/** Streamed while a scan runs. */ +sealed interface ScanProgress { + + /** [total] is 0 while the phase is indeterminate (e.g. loading the library). */ + data class Working( + val phase: ScanPhase, + val done: Int = 0, + val total: Int = 0, + ) : ScanProgress + + data class Finished(val result: ScanResult) : ScanProgress + + data class Failed(val message: String) : ScanProgress +} + +/** + * Wall-clock cost of each scan stage. Surfaced in the UI so it is obvious which part + * of a scan is slow on a given device (the perceptual pass is normally dominant). + */ +data class ScanTimings( + val signaturesMs: Long, + val contentMs: Long, + val perceptualMs: Long, + val totalMs: Long, +) + +/** Everything the UI needs after a scan. */ +data class ScanResult( + val photoCount: Int, + val groups: List, + val recordsById: Map, + val durationMs: Long, + val timings: ScanTimings? = null, + /** Thresholds the perceptual pass actually used (the pHash one is adapted per scan). */ + val perceptualThresholds: PerceptualThresholds? = null, + /** True when the library was too small to fit the distance distribution. */ + val thresholdWasFallback: Boolean = false, + /** + * Photos whose pixels could not be read — corrupt, truncated, RAW, or an unsupported format. + * They still take part in exact-duplicate matching but not in the perceptual pass. + */ + val unreadableCount: Int = 0, +) { + val groupCount: Int get() = groups.size + val duplicatePhotoCount: Int get() = groups.sumOf { it.members.size } + val reclaimableBytes: Long get() = groups.sumOf { it.reclaimableBytes } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/service/ScanService.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/service/ScanService.kt new file mode 100644 index 0000000..4eb651d --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/service/ScanService.kt @@ -0,0 +1,182 @@ +package de.softwareapp_hb.fotodup.finder.service + +import android.app.Notification +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.app.Service +import android.content.Context +import android.content.Intent +import android.content.pm.ServiceInfo +import android.os.IBinder +import androidx.core.app.NotificationCompat +import androidx.core.app.ServiceCompat +import de.softwareapp_hb.fotodup.finder.FotoDupApplication +import de.softwareapp_hb.fotodup.finder.MainActivity +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.domain.ScanState +import de.softwareapp_hb.fotodup.finder.text.labelRes +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.Job +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.cancel +import kotlinx.coroutines.launch + +/** + * Foreground service that keeps a running scan alive while the user is elsewhere. + * + * It contains no scan logic — [de.softwareapp_hb.fotodup.finder.domain.ScanController] + * does the work and publishes progress, and this service mirrors that progress into an + * ongoing notification. The notification is also how the user cancels a long scan, and + * its presence is what stops Android from freezing the app's process when the app is not + * visible. + */ +class ScanService : Service() { + + private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate) + private var observer: Job? = null + + override fun onBind(intent: Intent?): IBinder? = null + + override fun onCreate() { + super.onCreate() + createChannel() + } + + // The service-type constant is inlined at compile time, so referencing it on older + // devices is harmless — ServiceCompat ignores the type below API 29. + @Suppress("InlinedApi") + override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { + val controller = (application as FotoDupApplication).scanController + + if (intent?.action == ACTION_CANCEL) { + controller.cancel() + stopEverything() + return START_NOT_STICKY + } + + // This must happen within a few seconds of startForegroundService(), so it is + // the very first thing we do. + ServiceCompat.startForeground( + this, + NOTIFICATION_ID, + buildNotification(controller.state.value), + ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC, + ) + + if (observer == null) { + observer = scope.launch { + controller.state.collect { state -> + if (state.scanning) { + notifyProgress(state) + } else { + stopEverything() + } + } + } + } + return START_NOT_STICKY + } + + /** + * Android 15+ caps how long a `dataSync` foreground service may run (6 hours in any + * 24). If that ever fires, stop cleanly rather than let the system kill the app. + */ + override fun onTimeout(startId: Int, fgsType: Int) { + (application as FotoDupApplication).scanController.cancel() + stopEverything() + } + + override fun onDestroy() { + observer = null + scope.cancel() + super.onDestroy() + } + + private fun stopEverything() { + ServiceCompat.stopForeground(this, ServiceCompat.STOP_FOREGROUND_REMOVE) + stopSelf() + } + + private fun notifyProgress(state: ScanState) { + getSystemService(NotificationManager::class.java) + ?.notify(NOTIFICATION_ID, buildNotification(state)) + } + + private fun buildNotification(state: ScanState): Notification { + val openApp = PendingIntent.getActivity( + this, + 0, + Intent(this, MainActivity::class.java).apply { + flags = Intent.FLAG_ACTIVITY_SINGLE_TOP or Intent.FLAG_ACTIVITY_CLEAR_TOP + }, + PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, + ) + val cancel = PendingIntent.getService( + this, + 1, + Intent(this, ScanService::class.java).setAction(ACTION_CANCEL), + PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, + ) + + val builder = NotificationCompat.Builder(this, CHANNEL_ID) + .setSmallIcon(R.drawable.ic_notification) + .setContentTitle(getString(R.string.notification_title)) + .setContentText(getString(state.phase.labelRes())) + .setContentIntent(openApp) + .addAction(0, getString(R.string.action_cancel), cancel) + .setOngoing(true) + .setOnlyAlertOnce(true) + .setSilent(true) + .setPriority(NotificationCompat.PRIORITY_LOW) + .setForegroundServiceBehavior(NotificationCompat.FOREGROUND_SERVICE_IMMEDIATE) + + val fraction = state.progressFraction + if (fraction == null) { + builder.setProgress(0, 0, true) + } else { + builder.setProgress(PROGRESS_MAX, (fraction * PROGRESS_MAX).toInt(), false) + } + return builder.build() + } + + private fun createChannel() { + val manager = getSystemService(NotificationManager::class.java) ?: return + if (manager.getNotificationChannel(CHANNEL_ID) != null) return + manager.createNotificationChannel( + NotificationChannel( + CHANNEL_ID, + getString(R.string.notification_channel_name), + NotificationManager.IMPORTANCE_LOW, + ).apply { + description = getString(R.string.notification_channel_description) + setShowBadge(false) + }, + ) + } + + companion object { + private const val CHANNEL_ID = "scan_progress" + private const val NOTIFICATION_ID = 1 + private const val PROGRESS_MAX = 100 + private const val ACTION_CANCEL = "de.softwareapp_hb.fotodup.finder.action.CANCEL_SCAN" + + /** + * Asks the system to run the service. Safe to call from the background: if the + * start is not permitted the exception is swallowed, because the scan itself does + * not depend on the service — it only loses the foreground guarantee. + */ + fun start(context: Context) { + try { + context.startForegroundService(Intent(context, ScanService::class.java)) + } catch (e: IllegalStateException) { + // e.g. ForegroundServiceStartNotAllowedException on Android 12+. + } + } + + fun stop(context: Context) { + context.stopService(Intent(context, ScanService::class.java)) + } + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/text/TextRes.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/text/TextRes.kt new file mode 100644 index 0000000..fe1015c --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/text/TextRes.kt @@ -0,0 +1,28 @@ +package de.softwareapp_hb.fotodup.finder.text + +import androidx.annotation.StringRes +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.domain.ScanPhase + +/** + * Maps domain enums to their resource ids. + * + * Lives outside both `ui` and `service` because the screen and the progress notification must + * show the *same* wording — duplicating the mapping is how they drift apart. + */ +@StringRes +fun ScanPhase?.labelRes(): Int = when (this) { + ScanPhase.SIGNATURES -> R.string.phase_signatures + ScanPhase.CONTENT -> R.string.phase_content + ScanPhase.PERCEPTUAL -> R.string.phase_perceptual + ScanPhase.GROUPING -> R.string.phase_grouping + null, ScanPhase.LOADING -> R.string.phase_loading +} + +@StringRes +fun DuplicatePolicy.labelRes(): Int = when (this) { + DuplicatePolicy.EXACT_ONLY -> R.string.policy_exact_only + DuplicatePolicy.BALANCED -> R.string.policy_balanced + DuplicatePolicy.AGGRESSIVE -> R.string.policy_aggressive +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/components/PhotoThumbnail.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/components/PhotoThumbnail.kt new file mode 100644 index 0000000..90db85f --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/components/PhotoThumbnail.kt @@ -0,0 +1,117 @@ +package de.softwareapp_hb.fotodup.finder.ui.components + +import androidx.compose.foundation.background +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.layout.ContentScale +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import coil3.compose.AsyncImage +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.data.PhotoRecord + +/** + * A single selectable photo tile. + * + * Selected tiles get a tint plus a check badge; the copy the app suggests keeping + * is marked "BEST" so the choice stays understandable. + */ +@Composable +fun PhotoThumbnail( + photo: PhotoRecord, + selected: Boolean, + isBest: Boolean, + onClick: () -> Unit, + modifier: Modifier = Modifier, +) { + Box( + modifier = modifier + .size(108.dp) + .clip(RoundedCornerShape(12.dp)) + .background(MaterialTheme.colorScheme.surfaceVariant) + .clickable(onClick = onClick), + ) { + AsyncImage( + model = photo.uri, + contentDescription = photo.displayName, + contentScale = ContentScale.Crop, + modifier = Modifier.fillMaxSize(), + ) + + if (selected) { + Box( + modifier = Modifier + .fillMaxSize() + .background(MaterialTheme.colorScheme.error.copy(alpha = 0.35f)), + ) + Box( + modifier = Modifier + .align(Alignment.TopEnd) + .padding(6.dp) + .size(22.dp) + .clip(RoundedCornerShape(11.dp)) + .background(MaterialTheme.colorScheme.error), + contentAlignment = Alignment.Center, + ) { + Text( + text = "✓", + color = MaterialTheme.colorScheme.onError, + style = MaterialTheme.typography.labelMedium, + fontWeight = FontWeight.Bold, + ) + } + } else if (isBest) { + Badge( + text = stringResource(R.string.badge_best), + container = MaterialTheme.colorScheme.primary, + content = MaterialTheme.colorScheme.onPrimary, + modifier = Modifier + .align(Alignment.TopStart) + .padding(6.dp), + ) + } + + Badge( + text = photo.readableSize, + container = Color.Black.copy(alpha = 0.55f), + content = Color.White, + modifier = Modifier + .align(Alignment.BottomStart) + .padding(6.dp), + ) + } +} + +@Composable +private fun Badge( + text: String, + container: Color, + content: Color, + modifier: Modifier = Modifier, +) { + Box( + modifier = modifier + .clip(RoundedCornerShape(6.dp)) + .background(container) + .padding(horizontal = 6.dp, vertical = 2.dp), + ) { + Text( + text = text, + color = content, + style = MaterialTheme.typography.labelSmall, + fontWeight = FontWeight.SemiBold, + ) + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/GroupSection.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/GroupSection.kt new file mode 100644 index 0000000..d8d6144 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/GroupSection.kt @@ -0,0 +1,320 @@ +package de.softwareapp_hb.fotodup.finder.ui.scanner + +import androidx.annotation.StringRes +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.PaddingValues +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.lazy.LazyColumn +import androidx.compose.foundation.lazy.LazyRow +import androidx.compose.foundation.lazy.items +import androidx.compose.material3.FilledTonalButton +import androidx.compose.material3.ElevatedCard +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.remember +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.res.pluralStringResource +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextAlign +import androidx.compose.ui.unit.dp +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.core.DuplicateGroup +import de.softwareapp_hb.fotodup.finder.core.GroupKind +import de.softwareapp_hb.fotodup.finder.data.PhotoRecord +import de.softwareapp_hb.fotodup.finder.data.humanReadableBytes +import de.softwareapp_hb.fotodup.finder.domain.ScanResult +import de.softwareapp_hb.fotodup.finder.ui.components.PhotoThumbnail +import java.util.Locale + +/** How many thumbnails a collapsed group shows before offering "Show all". */ +private const val COLLAPSED_LIMIT = 4 + +@Composable +internal fun ResultSection(state: ScannerUiState, viewModel: ScannerViewModel) { + val result = state.result ?: return + + if (result.groups.isEmpty()) { + NoDuplicates(result) + return + } + + // Staged photos are taken out of the list, so a row disappears the moment the user + // removes it — and a group that is no longer duplicated disappears entirely. + val groups = state.visibleGroups() + + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(16.dp), + verticalArrangement = Arrangement.spacedBy(12.dp), + ) { + item { SummaryCard(groups, result) } + item { + Column { + Row(horizontalArrangement = Arrangement.spacedBy(4.dp)) { + TextButton(onClick = viewModel::selectAllButBest) { + Text(stringResource(R.string.action_select_all_but_best)) + } + TextButton(onClick = viewModel::clearSelection) { + Text(stringResource(R.string.action_clear_selection)) + } + } + Text( + text = stringResource(R.string.hint_select_then_remove), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(horizontal = 12.dp), + ) + } + } + if (groups.isEmpty()) { + item { EverythingStagedCard(state.markedForRemovalIds.size) } + } + items(groups, key = { it.id }) { group -> + GroupCard( + group = group, + records = result.recordsById, + selectedIds = state.selectedIds, + expanded = group.id in state.expandedGroupIds, + onToggleExpand = { viewModel.toggleGroupExpanded(group.id) }, + onTogglePhoto = viewModel::toggleSelected, + onSelectGroup = { select -> viewModel.setGroupSelection(group.id, select) }, + onRemoveDuplicates = { viewModel.markGroupForRemoval(group.id) }, + ) + } + } +} + +/** Shown when every duplicate is staged, so the empty list is not mistaken for a bug. */ +@Composable +private fun EverythingStagedCard(stagedCount: Int) { + ElevatedCard(Modifier.fillMaxWidth()) { + Column(Modifier.padding(16.dp)) { + Text( + text = stringResource(R.string.staged_title), + style = MaterialTheme.typography.titleMedium, + fontWeight = FontWeight.SemiBold, + ) + Spacer(Modifier.height(4.dp)) + Text( + text = pluralStringResource(R.plurals.staged_body, stagedCount, stagedCount), + style = MaterialTheme.typography.bodyMedium, + ) + } + } +} + +@Composable +private fun SummaryCard(groups: List, result: ScanResult) { + val reclaimableBytes = groups.sumOf { it.reclaimableBytes } + val involvedPhotos = groups.sumOf { it.members.size } + + ElevatedCard(Modifier.fillMaxWidth()) { + Column(Modifier.padding(16.dp)) { + Text( + text = pluralStringResource(R.plurals.summary_groups, groups.size, groups.size), + style = MaterialTheme.typography.titleLarge, + fontWeight = FontWeight.SemiBold, + ) + Spacer(Modifier.height(4.dp)) + Text( + text = stringResource(R.string.summary_involved, involvedPhotos, result.photoCount), + style = MaterialTheme.typography.bodyMedium, + ) + Spacer(Modifier.height(8.dp)) + Text( + text = stringResource(R.string.summary_reclaimable, humanReadableBytes(reclaimableBytes)), + style = MaterialTheme.typography.titleMedium, + color = MaterialTheme.colorScheme.primary, + fontWeight = FontWeight.Bold, + ) + Text( + text = stringResource(R.string.summary_scanned_in, formatDuration(result.durationMs)), + style = MaterialTheme.typography.bodySmall, + ) + result.timings?.let { timings -> + Text( + text = stringResource( + R.string.summary_timings, + formatDuration(timings.signaturesMs), + formatDuration(timings.contentMs), + formatDuration(timings.perceptualMs), + ), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + result.perceptualThresholds?.let { thresholds -> + Text( + text = pluralStringResource( + if (result.thresholdWasFallback) { + R.plurals.summary_cut_default + } else { + R.plurals.summary_cut_fitted + }, + thresholds.phashDistance, + thresholds.phashDistance, + ), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + if (result.unreadableCount > 0) { + Text( + text = pluralStringResource( + R.plurals.summary_unreadable, + result.unreadableCount, + result.unreadableCount, + ), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + } + } +} + +@Composable +private fun NoDuplicates(result: ScanResult) { + Column( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.Center, + ) { + Text(text = "\u2728", style = MaterialTheme.typography.displayMedium) + Spacer(Modifier.height(12.dp)) + Text( + text = stringResource(R.string.no_duplicates_title), + style = MaterialTheme.typography.titleLarge, + ) + Spacer(Modifier.height(8.dp)) + Text( + text = pluralStringResource( + R.plurals.no_duplicates_body, + result.photoCount, + result.photoCount, + formatDuration(result.durationMs), + ), + style = MaterialTheme.typography.bodyMedium, + textAlign = TextAlign.Center, + ) + } +} + +@Composable +private fun GroupCard( + group: DuplicateGroup, + records: Map, + selectedIds: Set, + expanded: Boolean, + onToggleExpand: () -> Unit, + onTogglePhoto: (Long) -> Unit, + onSelectGroup: (Boolean) -> Unit, + onRemoveDuplicates: () -> Unit, +) { + // Same rule the delete button and "select all but best" use, so the badge shown + // here always matches what would actually be kept. + val bestId = remember(group) { group.suggestedKeep()?.id } + // The suggested keep is shown first, so it stays visible as a reference even when + // the group is collapsed. The engine works with pure fingerprints; the UI needs the + // full records so it can show the photo and its size. + val ordered = remember(group) { group.displayOrder() } + val visible = (if (expanded) ordered else ordered.take(COLLAPSED_LIMIT)) + .mapNotNull { records[it.id] } + // Counted from the current tick state, not from the suggestion, so deselecting a copy in + // this row immediately updates the label. + val removableCount = group.selectedMembers(selectedIds).size + + ElevatedCard(Modifier.fillMaxWidth()) { + Column(Modifier.padding(12.dp)) { + Row(verticalAlignment = Alignment.CenterVertically) { + Column(Modifier.weight(1f)) { + Text( + text = stringResource(group.titleRes()), + style = MaterialTheme.typography.titleSmall, + fontWeight = FontWeight.SemiBold, + ) + Text( + text = stringResource( + R.string.group_copies_reclaimable, + group.members.size, + humanReadableBytes(group.reclaimableBytes), + ), + style = MaterialTheme.typography.bodySmall, + ) + } + TextButton(onClick = { onSelectGroup(true) }) { + Text(stringResource(R.string.group_action_all)) + } + TextButton(onClick = { onSelectGroup(false) }) { + Text(stringResource(R.string.group_action_none)) + } + } + Spacer(Modifier.height(8.dp)) + LazyRow(horizontalArrangement = Arrangement.spacedBy(8.dp)) { + items(visible, key = { it.id }) { photo -> + PhotoThumbnail( + photo = photo, + selected = photo.id in selectedIds, + isBest = bestId == photo.id, + onClick = { onTogglePhoto(photo.id) }, + ) + } + } + if (group.members.size > COLLAPSED_LIMIT) { + TextButton(onClick = onToggleExpand, modifier = Modifier.align(Alignment.End)) { + Text( + text = if (expanded) { + stringResource(R.string.group_show_less) + } else { + stringResource(R.string.group_show_all, group.members.size) + }, + ) + } + } + Spacer(Modifier.height(4.dp)) + // Row-wise staging: hides this row's ticked copies right away and marks them for + // removal. Nothing is deleted and no system dialog appears until the user taps Apply. + FilledTonalButton( + onClick = onRemoveDuplicates, + enabled = removableCount > 0, + modifier = Modifier.fillMaxWidth(), + ) { + Text( + text = if (removableCount == 0) { + stringResource(R.string.group_nothing_selected) + } else { + pluralStringResource( + R.plurals.group_remove_duplicates, + removableCount, + removableCount, + ) + }, + ) + } + } + } +} + +@StringRes +private fun DuplicateGroup.titleRes(): Int = when (kind) { + GroupKind.EXACT -> R.string.group_exact_copies + GroupKind.PERCEPTUAL -> R.string.group_visually_similar +} + +internal fun formatDuration(millis: Long): String = if (millis < 1000) { + "$millis ms" +} else { + String.format(Locale.US, "%.1f s", millis / 1000.0) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerScreen.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerScreen.kt new file mode 100644 index 0000000..5d49306 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerScreen.kt @@ -0,0 +1,239 @@ +package de.softwareapp_hb.fotodup.finder.ui.scanner + +import android.app.Activity +import androidx.activity.compose.rememberLauncherForActivityResult +import androidx.activity.result.IntentSenderRequest +import androidx.activity.result.contract.ActivityResultContracts +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.navigationBarsPadding +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.width +import androidx.compose.material3.Button +import androidx.compose.material3.DropdownMenu +import androidx.compose.material3.DropdownMenuItem +import androidx.compose.material3.ExperimentalMaterial3Api +import androidx.compose.material3.HorizontalDivider +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Scaffold +import androidx.compose.material3.SnackbarHost +import androidx.compose.material3.SnackbarHostState +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.material3.TopAppBar +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import androidx.lifecycle.compose.collectAsStateWithLifecycle +import androidx.lifecycle.viewmodel.compose.viewModel +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.data.MediaAccess +import de.softwareapp_hb.fotodup.finder.data.AppPermissions +import de.softwareapp_hb.fotodup.finder.data.humanReadableBytes +import de.softwareapp_hb.fotodup.finder.text.labelRes + +@Composable +fun ScannerScreen(viewModel: ScannerViewModel = viewModel()) { + val state by viewModel.state.collectAsStateWithLifecycle() + + val permissionLauncher = rememberLauncherForActivityResult( + ActivityResultContracts.RequestMultiplePermissions(), + ) { viewModel.refreshAccess() } + + val deleteLauncher = rememberLauncherForActivityResult( + ActivityResultContracts.StartIntentSenderForResult(), + ) { result -> viewModel.onSystemDeleteResult(result.resultCode == Activity.RESULT_OK) } + + // System delete/trash consent dialogs are one-shot; collect and launch each one. + LaunchedEffect(viewModel) { + viewModel.deleteRequests.collect { sender -> + deleteLauncher.launch(IntentSenderRequest.Builder(sender).build()) + } + } + + val snackbarHostState = remember { SnackbarHostState() } + LaunchedEffect(state.message) { + state.message?.let { message -> + snackbarHostState.showSnackbar(message) + viewModel.dismissMessage() + } + } + + Scaffold( + topBar = { ScannerTopBar(state, viewModel) }, + snackbarHost = { SnackbarHost(snackbarHostState) }, + bottomBar = { BottomActionBar(state, viewModel) }, + ) { innerPadding -> + Column( + modifier = Modifier + .fillMaxSize() + .padding(innerPadding), + ) { + AccessBanner(state.access) { + permissionLauncher.launch(AppPermissions.requestablePermissions()) + } + + when { + state.scanning -> ProgressSection(state, viewModel::cancelScan) + state.result != null -> ResultSection(state, viewModel) + else -> EmptySection( + hasAccess = state.access != MediaAccess.NONE, + onScan = viewModel::startScan, + ) + } + } + } +} + +@OptIn(ExperimentalMaterial3Api::class) +@Composable +private fun ScannerTopBar(state: ScannerUiState, viewModel: ScannerViewModel) { + var menuOpen by remember { mutableStateOf(false) } + + TopAppBar( + title = { Text(stringResource(R.string.app_name)) }, + actions = { + if (state.scanning) { + TextButton(onClick = viewModel::cancelScan) { + Text(stringResource(R.string.action_cancel)) + } + } else { + TextButton( + onClick = viewModel::startScan, + enabled = state.access != MediaAccess.NONE, + ) { Text(stringResource(R.string.action_scan)) } + } + TextButton(onClick = { menuOpen = true }) { + Text(stringResource(R.string.action_options)) + } + DropdownMenu(expanded = menuOpen, onDismissRequest = { menuOpen = false }) { + DropdownMenuItem( + text = { Text(stringResource(R.string.sensitivity_title)) }, + onClick = {}, + enabled = false, + ) + DuplicatePolicy.entries.forEach { policy -> + DropdownMenuItem( + text = { Text(stringResource(policy.labelRes())) }, + onClick = { + viewModel.setPolicy(policy) + menuOpen = false + }, + leadingIcon = { Text(if (state.policy == policy) "✓" else " ") }, + ) + } + HorizontalDivider() + DropdownMenuItem( + text = { + Text( + stringResource( + if (state.useTrash) { + R.string.menu_use_trash + } else { + R.string.menu_delete_permanently + }, + ), + ) + }, + onClick = { + viewModel.setUseTrash(!state.useTrash) + menuOpen = false + }, + ) + } + }, + ) +} + +/** + * The bottom action bar. + * + * Two-step by design: **"Remove all selected"** only *stages* photos (rows vanish from the + * list immediately, nothing is deleted), and **"Apply"** performs the real removal — which + * on Android 11+ shows a **single** system confirmation for the whole batch instead of one + * dialog per group. + */ +@Composable +private fun BottomActionBar(state: ScannerUiState, viewModel: ScannerViewModel) { + val hasMarked = state.markedForRemovalIds.isNotEmpty() + val hasSelection = state.selectedIds.isNotEmpty() + if (!hasMarked && !hasSelection) return + + Surface(tonalElevation = 3.dp) { + Column( + modifier = Modifier + .fillMaxWidth() + // Reserve room for the system navigation bar: Scaffold does not inset a + // custom bottom bar itself. + .navigationBarsPadding() + .padding(horizontal = 16.dp, vertical = 10.dp), + ) { + if (hasMarked) { + Text( + text = stringResource( + R.string.bar_marked_for_removal, + state.markedForRemovalIds.size, + humanReadableBytes(state.markedBytes), + ), + style = MaterialTheme.typography.titleSmall, + fontWeight = FontWeight.SemiBold, + ) + Text( + text = stringResource( + if (state.useTrash) R.string.bar_will_trash else R.string.bar_will_delete, + ), + style = MaterialTheme.typography.bodySmall, + ) + } + if (hasSelection) { + Text( + text = stringResource( + R.string.bar_selected, + state.selectedIds.size, + humanReadableBytes(state.selectedBytes), + ), + style = MaterialTheme.typography.bodySmall, + ) + } + + Spacer(Modifier.height(6.dp)) + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.End, + verticalAlignment = Alignment.CenterVertically, + ) { + if (hasMarked) { + TextButton(onClick = viewModel::discardMarkedForRemoval) { + Text(stringResource(R.string.action_discard)) + } + } + if (hasSelection) { + TextButton(onClick = viewModel::markSelectedForRemoval) { + Text(stringResource(R.string.action_remove_selected)) + } + } + if (hasMarked) { + Spacer(Modifier.width(8.dp)) + Button(onClick = viewModel::applyMarkedForRemoval) { + Text(stringResource(R.string.action_apply)) + } + } + } + } + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerSections.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerSections.kt new file mode 100644 index 0000000..a083fd9 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerSections.kt @@ -0,0 +1,126 @@ +package de.softwareapp_hb.fotodup.finder.ui.scanner + +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.material3.Button +import androidx.compose.material3.LinearProgressIndicator +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.res.stringResource +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextAlign +import androidx.compose.ui.unit.dp +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.data.MediaAccess +import de.softwareapp_hb.fotodup.finder.text.labelRes + +@Composable +internal fun AccessBanner(access: MediaAccess, onGrant: () -> Unit) { + if (access == MediaAccess.FULL) return + + Surface( + color = MaterialTheme.colorScheme.secondaryContainer, + modifier = Modifier.fillMaxWidth(), + ) { + Column(Modifier.padding(16.dp)) { + Text( + text = stringResource( + if (access == MediaAccess.PARTIAL) { + R.string.access_limited_title + } else { + R.string.access_needed_title + }, + ), + style = MaterialTheme.typography.titleSmall, + fontWeight = FontWeight.SemiBold, + ) + Spacer(Modifier.height(4.dp)) + Text( + text = stringResource( + if (access == MediaAccess.PARTIAL) { + R.string.access_limited_body + } else { + R.string.access_needed_body + }, + ), + style = MaterialTheme.typography.bodySmall, + ) + Spacer(Modifier.height(10.dp)) + Button(onClick = onGrant) { Text(stringResource(R.string.action_grant_access)) } + } + } +} + +@Composable +internal fun ProgressSection(state: ScannerUiState, onCancel: () -> Unit) { + Column( + modifier = Modifier + .fillMaxWidth() + .padding(24.dp), + horizontalAlignment = Alignment.CenterHorizontally, + ) { + Text( + text = stringResource(state.phase.labelRes()), + style = MaterialTheme.typography.titleMedium, + textAlign = TextAlign.Center, + ) + Spacer(Modifier.height(16.dp)) + val fraction = state.progressFraction + if (fraction == null) { + LinearProgressIndicator(modifier = Modifier.fillMaxWidth()) + } else { + LinearProgressIndicator(progress = { fraction }, modifier = Modifier.fillMaxWidth()) + } + if (state.progressTotal > 0) { + Spacer(Modifier.height(8.dp)) + Text( + text = stringResource( + R.string.progress_counter, + state.progressDone, + state.progressTotal, + ), + style = MaterialTheme.typography.bodySmall, + ) + } + Spacer(Modifier.height(24.dp)) + Button(onClick = onCancel) { Text(stringResource(R.string.action_cancel)) } + } +} + +@Composable +internal fun EmptySection(hasAccess: Boolean, onScan: () -> Unit) { + Column( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.Center, + ) { + Text(text = "\uD83E\uDDF9", style = MaterialTheme.typography.displayMedium) + Spacer(Modifier.height(12.dp)) + Text( + text = stringResource(R.string.intro_title), + style = MaterialTheme.typography.titleLarge, + textAlign = TextAlign.Center, + ) + Spacer(Modifier.height(8.dp)) + Text( + text = stringResource(R.string.intro_body), + style = MaterialTheme.typography.bodyMedium, + textAlign = TextAlign.Center, + ) + Spacer(Modifier.height(24.dp)) + Button(onClick = onScan, enabled = hasAccess) { + Text(stringResource(R.string.action_start_scan)) + } + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerUiState.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerUiState.kt new file mode 100644 index 0000000..de4c0b1 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerUiState.kt @@ -0,0 +1,57 @@ +package de.softwareapp_hb.fotodup.finder.ui.scanner + +import de.softwareapp_hb.fotodup.finder.core.DuplicateGroup +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.data.MediaAccess +import de.softwareapp_hb.fotodup.finder.domain.ScanPhase +import de.softwareapp_hb.fotodup.finder.domain.ScanResult + +/** Everything the scanner screen renders. */ +data class ScannerUiState( + val access: MediaAccess = MediaAccess.NONE, + val policy: DuplicatePolicy = DuplicatePolicy.DEFAULT, + val scanning: Boolean = false, + val phase: ScanPhase? = null, + val progressDone: Int = 0, + val progressTotal: Int = 0, + val result: ScanResult? = null, + /** Ids the user currently has ticked (the orange/red selection on the tiles). */ + val selectedIds: Set = emptySet(), + /** + * Ids the user has staged for removal. These disappear from the list immediately but + * are **not** deleted until they tap Apply, which triggers the one system dialog. + */ + val markedForRemovalIds: Set = emptySet(), + val expandedGroupIds: Set = emptySet(), + /** Move to trash instead of permanent delete (Android 11+). */ + val useTrash: Boolean = true, + val message: String? = null, +) { + val selectedBytes: Long + get() = bytesOf(selectedIds) + + val markedBytes: Long + get() = bytesOf(markedForRemovalIds) + + private fun bytesOf(ids: Set): Long = result?.recordsById.orEmpty() + .filterKeys { it in ids } + .values + .sumOf { it.sizeBytes } + + /** 0f..1f, or null when the current phase has no measurable total. */ + val progressFraction: Float? + get() = if (progressTotal > 0) progressDone.toFloat() / progressTotal else null + + /** + * The groups that are still duplicated, i.e. with everything staged for removal taken + * out. A group whose staged copies leave fewer than two members is no longer a + * duplicate set and is dropped — that is what makes a row "disappear" when the user + * removes it. + */ + fun visibleGroups(): List = + result?.groups.orEmpty().mapNotNull { it.without(markedForRemovalIds) } + + /** A single group with staged photos removed, or `null` if it is no longer listed. */ + fun visibleGroup(groupId: String): DuplicateGroup? = + result?.groups?.firstOrNull { it.id == groupId }?.without(markedForRemovalIds) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerViewModel.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerViewModel.kt new file mode 100644 index 0000000..ff685fb --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/scanner/ScannerViewModel.kt @@ -0,0 +1,261 @@ +package de.softwareapp_hb.fotodup.finder.ui.scanner + +import android.app.Application +import android.content.IntentSender +import androidx.annotation.PluralsRes +import androidx.annotation.StringRes +import androidx.lifecycle.AndroidViewModel +import androidx.lifecycle.viewModelScope +import de.softwareapp_hb.fotodup.finder.FotoDupApplication +import de.softwareapp_hb.fotodup.finder.R +import de.softwareapp_hb.fotodup.finder.core.DuplicateGroup +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.data.AppPermissions +import de.softwareapp_hb.fotodup.finder.data.MediaDeleter +import de.softwareapp_hb.fotodup.finder.domain.ScanState +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.channels.Channel +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow +import kotlinx.coroutines.flow.receiveAsFlow +import kotlinx.coroutines.flow.update +import kotlinx.coroutines.launch +import kotlinx.coroutines.withContext + +/** + * Drives the duplicate scanner screen: permissions, selection and deletion. + * + * Scanning itself is *not* owned here — [de.softwareapp_hb.fotodup.finder.domain.ScanController] + * owns it, so a long scan can outlive this ViewModel and keep running while the app is + * backgrounded. This class mirrors the controller's progress into UI state and keeps the + * selection local, which is why a selection survives configuration changes. + */ +class ScannerViewModel(application: Application) : AndroidViewModel(application) { + + private val scanController = (application as FotoDupApplication).scanController + private val deleter = MediaDeleter(application) + + private val _state = MutableStateFlow(ScannerUiState()) + val state: StateFlow = _state.asStateFlow() + + /** One-shot system consent intents the UI has to launch. */ + private val deleteRequestChannel = Channel(Channel.BUFFERED) + val deleteRequests = deleteRequestChannel.receiveAsFlow() + + /** + * Which completed scan the current selection belongs to. Without this, the automatic + * "select all but best" would wipe a manual selection every time the controller + * re-emits its unchanged result. + */ + private var appliedGeneration = -1 + + /** Which photos the in-flight delete request covers. */ + private var pendingDeleteIds: Set = emptySet() + + /** + * True while a system confirmation dialog is on screen. Only one can be in flight: a + * second request would simply queue *another* approval for the user to dismiss. + */ + private var deleteDialogPending = false + + init { + refreshAccess() + viewModelScope.launch { + scanController.state.collect(::onScanState) + } + } + + fun refreshAccess() { + _state.update { it.copy(access = AppPermissions.accessState(getApplication())) } + } + + fun setPolicy(policy: DuplicatePolicy) = _state.update { it.copy(policy = policy) } + + fun setUseTrash(useTrash: Boolean) = _state.update { it.copy(useTrash = useTrash) } + + fun startScan() = scanController.start(_state.value.policy) + + fun cancelScan() = scanController.cancel() + + private fun onScanState(scan: ScanState) { + val freshResult = scan.result?.takeIf { scan.generation != appliedGeneration } + if (freshResult != null) appliedGeneration = scan.generation + + _state.update { current -> + current.copy( + scanning = scan.scanning, + phase = scan.phase, + progressDone = scan.done, + progressTotal = scan.total, + result = scan.result, + selectedIds = when { + freshResult != null -> autoSelection(freshResult.groups) + scan.scanning -> emptySet() + else -> current.selectedIds + }, + markedForRemovalIds = when { + // A fresh scan rebuilds the list, so stale staging would point at + // photos that may no longer be there. + freshResult != null -> emptySet() + scan.scanning -> emptySet() + else -> current.markedForRemovalIds + }, + expandedGroupIds = when { + freshResult != null -> + freshResult.groups.take(DEFAULT_EXPANDED_GROUPS).map(DuplicateGroup::id).toSet() + + scan.scanning -> emptySet() + else -> current.expandedGroupIds + }, + message = scan.error ?: current.message, + ) + } + if (scan.error != null) scanController.clearError() + } + + // --- Selection (ticking tiles) and staging ------------------------------ + + fun toggleSelected(id: Long) = _state.update { current -> + val next = current.selectedIds.toMutableSet() + if (!next.add(id)) next.remove(id) + current.copy(selectedIds = next) + } + + fun setGroupSelection(groupId: String, selectAll: Boolean) = _state.update { current -> + val group = current.visibleGroup(groupId) ?: return@update current + val next = current.selectedIds.toMutableSet() + group.members.forEach { member -> + if (selectAll) next += member.id else next -= member.id + } + current.copy(selectedIds = next) + } + + fun toggleGroupExpanded(groupId: String) = _state.update { current -> + val next = current.expandedGroupIds.toMutableSet() + if (!next.add(groupId)) next.remove(groupId) + current.copy(expandedGroupIds = next) + } + + /** Ticks every copy that could go, across all groups that are still duplicated. */ + fun selectAllButBest() = _state.update { current -> + val ids = current.visibleGroups() + .flatMap { group -> group.suggestedRemovals().map { it.id } } + .toSet() + current.copy(selectedIds = ids) + } + + fun clearSelection() = _state.update { it.copy(selectedIds = emptySet()) } + + fun dismissMessage() = _state.update { it.copy(message = null) } + + /** + * Suggests removing every copy except the best one per group. The keep/remove rule + * lives on [DuplicateGroup] so it matches the "BEST" badge exactly. + */ + private fun autoSelection(groups: List): Set = + groups.flatMapTo(HashSet()) { group -> group.suggestedRemovals().map { it.id } } + + // --- Staging: mark now, delete once ------------------------------------- + + /** + * Stages the members of one group the user has ticked, and the row disappears from the list + * straight away. + * + * Nothing is deleted and **no dialog appears** — the actual removal happens when the user + * taps Apply, at which point one system confirmation covers everything staged. + */ + fun markGroupForRemoval(groupId: String) = _state.update { current -> + val group = current.visibleGroup(groupId) ?: return@update current + val chosen = group.selectedMembers(current.selectedIds).map { it.id }.toSet() + if (chosen.isEmpty()) return@update current + current.copy( + markedForRemovalIds = current.markedForRemovalIds + chosen, + selectedIds = current.selectedIds - chosen, + ) + } + + /** Stages everything the user has ticked. */ + fun markSelectedForRemoval() = _state.update { current -> + current.copy( + markedForRemovalIds = current.markedForRemovalIds + current.selectedIds, + selectedIds = emptySet(), + ) + } + + /** Undoes all staging — the rows come back and nothing was deleted. */ + fun discardMarkedForRemoval() = _state.update { it.copy(markedForRemovalIds = emptySet()) } + + /** + * The single "apply" action: really deletes everything staged. On Android 11+ this + * shows **one** system confirmation for the whole batch. + */ + fun applyMarkedForRemoval() = requestDelete(_state.value.markedForRemovalIds) + + private fun requestDelete(ids: Set) { + if (ids.isEmpty()) return + // Only one system dialog at a time — otherwise rapid taps queue up several + // confirmations the user has to dismiss. + if (deleteDialogPending) return + + val result = _state.value.result ?: return + val uris = ids.mapNotNull { result.recordsById[it]?.uri } + if (uris.isEmpty()) return + + pendingDeleteIds = ids + if (deleter.requiresSystemConsent) { + val intent = deleter.createDeleteRequest(uris, _state.value.useTrash) ?: return + deleteDialogPending = true + viewModelScope.launch { deleteRequestChannel.send(intent.intentSender) } + } else { + viewModelScope.launch { + val removed = withContext(Dispatchers.IO) { deleter.deleteDirectly(uris) } + applyDeletion(ids, removed) + } + } + } + + /** Result of the system delete dialog that the UI launched for us. */ + fun onSystemDeleteResult(confirmed: Boolean) { + deleteDialogPending = false + if (!confirmed) { + // Keep the staging so the user can simply tap Apply again. + pendingDeleteIds = emptySet() + _state.update { it.copy(message = text(R.string.message_not_confirmed)) } + return + } + val ids = pendingDeleteIds + applyDeletion(ids, ids.size) + } + + private fun applyDeletion(deletedIds: Set, removedCount: Int) { + pendingDeleteIds = emptySet() + deleteDialogPending = false + // The controller owns the result, so keep it in sync instead of mutating a copy — + // otherwise the deleted photos would reappear after an activity restart. + scanController.removePhotos(deletedIds) + _state.update { current -> + current.copy( + // Only drop what actually went away, so one delete does not throw away + // unrelated selection or staging. + selectedIds = current.selectedIds - deletedIds, + markedForRemovalIds = current.markedForRemovalIds - deletedIds, + message = if (removedCount > 0) { + quantity(R.plurals.message_deleted, removedCount) + } else { + text(R.string.message_nothing_deleted) + }, + ) + } + } + + /** Resource helpers: the ViewModel has no Compose context, only the Application. */ + private fun text(@StringRes id: Int): String = getApplication().getString(id) + + private fun quantity(@PluralsRes id: Int, count: Int): String = + getApplication().resources.getQuantityString(id, count, count) + + private companion object { + const val DEFAULT_EXPANDED_GROUPS = 3 + } +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Color.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Color.kt new file mode 100644 index 0000000..1795b9f --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Color.kt @@ -0,0 +1,27 @@ +package de.softwareapp_hb.fotodup.finder.ui.theme + +import androidx.compose.ui.graphics.Color + +// Brand palette: a calm green ("keep it clean") accent with a teal secondary. +// Only the roles the app actually uses are overridden; the rest come from the +// Material 3 baseline, which keeps the scheme consistent. + +val GreenPrimaryLight = Color(0xFF2E7D32) +val GreenOnPrimaryLight = Color(0xFFFFFFFF) +val GreenContainerLight = Color(0xFFB7F2B2) +val GreenOnContainerLight = Color(0xFF00210A) +val TealSecondaryLight = Color(0xFF00696E) +val SurfaceLight = Color(0xFFFFFBFE) +val SurfaceVariantLight = Color(0xFFDEE4DD) +val OnSurfaceLight = Color(0xFF171D18) +val ErrorLight = Color(0xFFBA1A1A) + +val GreenPrimaryDark = Color(0xFF9BD79A) +val GreenOnPrimaryDark = Color(0xFF00390F) +val GreenContainerDark = Color(0xFF15521C) +val GreenOnContainerDark = Color(0xFFB7F2B2) +val TealSecondaryDark = Color(0xFF4CDADE) +val SurfaceDark = Color(0xFF101418) +val SurfaceVariantDark = Color(0xFF3F4943) +val OnSurfaceDark = Color(0xFFDEE4DD) +val ErrorDark = Color(0xFFFFB4AB) diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Theme.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Theme.kt new file mode 100644 index 0000000..730012a --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Theme.kt @@ -0,0 +1,64 @@ +package de.softwareapp_hb.fotodup.finder.ui.theme + +import android.os.Build +import androidx.compose.foundation.isSystemInDarkTheme +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.darkColorScheme +import androidx.compose.material3.dynamicDarkColorScheme +import androidx.compose.material3.dynamicLightColorScheme +import androidx.compose.material3.lightColorScheme +import androidx.compose.runtime.Composable +import androidx.compose.ui.platform.LocalContext + +private val LightColors = lightColorScheme( + primary = GreenPrimaryLight, + onPrimary = GreenOnPrimaryLight, + primaryContainer = GreenContainerLight, + onPrimaryContainer = GreenOnContainerLight, + secondary = TealSecondaryLight, + background = SurfaceLight, + surface = SurfaceLight, + surfaceVariant = SurfaceVariantLight, + onSurface = OnSurfaceLight, + error = ErrorLight, +) + +private val DarkColors = darkColorScheme( + primary = GreenPrimaryDark, + onPrimary = GreenOnPrimaryDark, + primaryContainer = GreenContainerDark, + onPrimaryContainer = GreenOnContainerDark, + secondary = TealSecondaryDark, + background = SurfaceDark, + surface = SurfaceDark, + surfaceVariant = SurfaceVariantDark, + onSurface = OnSurfaceDark, + error = ErrorDark, +) + +/** + * App theme. Uses Material You dynamic colour on Android 12+ when available and + * falls back to the branded green scheme otherwise. + */ +@Composable +fun FotoDupTheme( + darkTheme: Boolean = isSystemInDarkTheme(), + dynamicColor: Boolean = true, + content: @Composable () -> Unit, +) { + val colorScheme = when { + dynamicColor && Build.VERSION.SDK_INT >= Build.VERSION_CODES.S -> { + val context = LocalContext.current + if (darkTheme) dynamicDarkColorScheme(context) else dynamicLightColorScheme(context) + } + + darkTheme -> DarkColors + else -> LightColors + } + + MaterialTheme( + colorScheme = colorScheme, + typography = FotoDupTypography, + content = content, + ) +} diff --git a/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Type.kt b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Type.kt new file mode 100644 index 0000000..498d824 --- /dev/null +++ b/app/src/main/java/de/softwareapp_hb/fotodup/finder/ui/theme/Type.kt @@ -0,0 +1,9 @@ +package de.softwareapp_hb.fotodup.finder.ui.theme + +import androidx.compose.material3.Typography + +/** + * The app sticks with the Material 3 default type scale; a dedicated object keeps + * a single place to tune typography later (e.g. larger "wasted space" figures). + */ +val FotoDupTypography = Typography() diff --git a/app/src/main/res/drawable/ic_launcher_background.xml b/app/src/main/res/drawable/ic_launcher_background.xml new file mode 100644 index 0000000..a0c1e46 --- /dev/null +++ b/app/src/main/res/drawable/ic_launcher_background.xml @@ -0,0 +1,10 @@ + + + + diff --git a/app/src/main/res/drawable/ic_launcher_foreground.xml b/app/src/main/res/drawable/ic_launcher_foreground.xml new file mode 100644 index 0000000..1421fa9 --- /dev/null +++ b/app/src/main/res/drawable/ic_launcher_foreground.xml @@ -0,0 +1,44 @@ + + + + + + + + + + + + + + + + + + diff --git a/app/src/main/res/drawable/ic_notification.xml b/app/src/main/res/drawable/ic_notification.xml new file mode 100644 index 0000000..1aba17a --- /dev/null +++ b/app/src/main/res/drawable/ic_notification.xml @@ -0,0 +1,22 @@ + + + + + + + + + + diff --git a/app/src/main/res/mipmap-anydpi/ic_launcher.xml b/app/src/main/res/mipmap-anydpi/ic_launcher.xml new file mode 100644 index 0000000..b3e26b4 --- /dev/null +++ b/app/src/main/res/mipmap-anydpi/ic_launcher.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/app/src/main/res/mipmap-anydpi/ic_launcher_round.xml b/app/src/main/res/mipmap-anydpi/ic_launcher_round.xml new file mode 100644 index 0000000..b3e26b4 --- /dev/null +++ b/app/src/main/res/mipmap-anydpi/ic_launcher_round.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/app/src/main/res/values-night/colors.xml b/app/src/main/res/values-night/colors.xml new file mode 100644 index 0000000..32429ef --- /dev/null +++ b/app/src/main/res/values-night/colors.xml @@ -0,0 +1,5 @@ + + + #101418 + #1B5E20 + diff --git a/app/src/main/res/values-night/themes.xml b/app/src/main/res/values-night/themes.xml new file mode 100644 index 0000000..63f87c5 --- /dev/null +++ b/app/src/main/res/values-night/themes.xml @@ -0,0 +1,9 @@ + + + + diff --git a/app/src/main/res/values/colors.xml b/app/src/main/res/values/colors.xml new file mode 100644 index 0000000..f1cab3a --- /dev/null +++ b/app/src/main/res/values/colors.xml @@ -0,0 +1,5 @@ + + + #FFFBFE + #1B5E20 + diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml new file mode 100644 index 0000000..30794d3 --- /dev/null +++ b/app/src/main/res/values/strings.xml @@ -0,0 +1,123 @@ + + + Foto Dup Finder + + + Scan + Cancel + Options + Apply + Discard + Remove all selected + Select all but best + Clear selection + Grant access + Start scan + Sensitivity + Move to trash + Delete permanently + + + Exact copies only + Balanced (recommended) + Aggressive — finds more + + + Photo access needed + The app needs permission to read your photos so it can find duplicates. Nothing ever leaves your device. + Limited photo access + You granted access to selected photos only. Grant full access to scan your whole library. + + + Find and remove duplicate photos + Fast, ad-free and completely offline. Your photos are never uploaded. + + + Reading your photo library… + Comparing file signatures… + Verifying identical candidates… + Looking for visually similar photos… + Grouping duplicates… + %1$d / %2$d + + + + %1$d duplicate group + %1$d duplicate groups + + + %1$d of %2$d photos are involved + %1$s reclaimable + Scanned in %1$s + signatures %1$s · exact %2$s · similar %3$s + + + similarity cut: pHash ≤ %1$d bit (default) + similarity cut: pHash ≤ %1$d bits (default) + + + similarity cut: pHash ≤ %1$d bit (fitted to this library) + similarity cut: pHash ≤ %1$d bits (fitted to this library) + + + %1$d photo could not be read (RAW, corrupt or unsupported) and was skipped for similar-photo matching + %1$d photos could not be read (RAW, corrupt or unsupported) and were skipped for similar-photo matching + + Tip: select what you want gone, then use \"Remove all selected\". Android asks for a single confirmation for the whole batch. + + + Exact copies + Visually similar + + %1$d copies · %2$s reclaimable + All + None + Show less + Show all %1$d + + Remove %1$d duplicate + Remove %1$d duplicates + + Nothing selected in this row + BEST + + + All duplicates are ready to go + + %1$d photo is marked for removal. Tap Apply to finish, or Discard to bring the groups back. + %1$d photos are marked for removal. Tap Apply to finish, or Discard to bring the groups back. + + + + No duplicates found + + + %1$d photo scanned in %2$s. + %1$d photos scanned in %2$s. + + + + + %1$d marked for removal · %2$s + Will be moved to the trash when you apply + Will be deleted permanently when you apply + %1$d selected · %2$s + + + + Deleted %1$d photo + Deleted %1$d photos + + Nothing was deleted + Deletion was not confirmed + + + Scan progress + Shows progress while duplicate photos are being searched for + Finding duplicate photos + diff --git a/app/src/main/res/values/themes.xml b/app/src/main/res/values/themes.xml new file mode 100644 index 0000000..8803c84 --- /dev/null +++ b/app/src/main/res/values/themes.xml @@ -0,0 +1,14 @@ + + + + + diff --git a/app/src/main/res/xml/backup_rules.xml b/app/src/main/res/xml/backup_rules.xml new file mode 100644 index 0000000..de943d0 --- /dev/null +++ b/app/src/main/res/xml/backup_rules.xml @@ -0,0 +1,8 @@ + + + + + diff --git a/app/src/main/res/xml/data_extraction_rules.xml b/app/src/main/res/xml/data_extraction_rules.xml new file mode 100644 index 0000000..5cb4e6f --- /dev/null +++ b/app/src/main/res/xml/data_extraction_rules.xml @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/CascadeBench.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/CascadeBench.kt new file mode 100644 index 0000000..580355a --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/CascadeBench.kt @@ -0,0 +1,187 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import de.softwareapp_hb.fotodup.finder.core.PerceptualHash +import java.util.Locale + +/** + * Models the two-stage cascade: a cheap binary hash proposes candidate pairs, and an + * expensive verifier is only consulted for pairs that survive stage 1. + * + * The experiment answers two questions with numbers: + * 1. can stage 1 run at a **loose** threshold (high recall) and have the verifier restore + * precision, and + * 2. how much of the library would the verifier actually have to touch? + */ +internal object CascadeBench { + + private class PairRef( + val a: Int, + val b: Int, + val stage1: Int, + val positive: Boolean, + ) + + private class Policy( + val name: String, + val tp: Int, + val fp: Int, + val fn: Int, + /** Pairs stage 1 lets through — what the verifier would have to look at. */ + val stage1Candidates: Int, + /** Of those, how many are not true duplicates (the verifier's extra load). */ + val falseCandidates: Int, + ) { + val precision = if (tp + fp == 0) 1.0 else tp.toDouble() / (tp + fp) + val recall = if (tp + fn == 0) 1.0 else tp.toDouble() / (tp + fn) + val f1 = if (precision + recall == 0.0) 0.0 else 2 * precision * recall / (precision + recall) + } + + fun run(bases: List): String { + val transforms = ImageLab.transforms + val baseCount = bases.size + val transformCount = transforms.size + val imageCount = baseCount * (1 + transformCount) + val report = StringBuilder() + + // --- signatures: transforms are streamed, only signatures are retained -------- + val stage1 = LongArray(imageCount) + val verifiers = listOf(ColourHistogramVerifier, ColourDHashVerifier) + val signatures = Array(verifiers.size) { Array(imageCount) { LongArray(0) } } + + fun sign(imageIndex: Int, img: Img) { + stage1[imageIndex] = PerceptualHash.phashFromArgb(img.pixels, img.width, img.height) + verifiers.forEachIndexed { v, verifier -> + signatures[v][imageIndex] = verifier.signature(img) + } + } + + bases.forEachIndexed { i, img -> sign(i, img) } + transforms.forEachIndexed { t, transform -> + bases.forEachIndexed { i, img -> + sign(baseCount + i * transformCount + t, transform.apply(img)) + } + } + + // --- pairs (identical for every verifier) ------------------------------------ + val pairs = ArrayList() + for (i in 0 until baseCount) { + for (t in 0 until transformCount) { + val copy = baseCount + i * transformCount + t + pairs += PairRef(i, copy, PerceptualHash.hammingDistance(stage1[i], stage1[copy]), true) + } + } + for (i in 0 until baseCount) { + for (j in i + 1 until baseCount) { + pairs += PairRef(i, j, PerceptualHash.hammingDistance(stage1[i], stage1[j]), false) + } + } + val positivePairs = pairs.count { it.positive } + + // The cheapest stage-1 setting that still yields no false positives, plus a looser + // one — the loose setting is what a cascade would use to buy recall back. + val negatives = pairs.filterNot { it.positive }.map { it.stage1 } + // A dataset that already contains near-identical pairs can leave no threshold with zero + // false pairs at all. That is a property of the *data*, not a failure, so fall back to the + // strictest setting and say so rather than throwing. + val tight = (0..64).lastOrNull { threshold -> negatives.none { it <= threshold } } ?: 0 + val loose = tight + 6 + val hasCleanThreshold = negatives.none { it <= tight } + + report.appendLine( + "images=$imageCount positive pairs=$positivePairs " + + "negative pairs=${pairs.size - positivePairs}", + ) + report.appendLine("stage 1 = pHash; tight threshold (no false positives) = $tight; loose = $loose") + if (!hasCleanThreshold) { + report.appendLine( + "NOTE: no threshold has zero false pairs — this set contains genuine duplicates " + + "that the harness treats as negatives, so precision numbers below are pessimistic.", + ) + } + report.appendLine() + + for ((v, verifier) in verifiers.withIndex()) { + val verified = DoubleArray(pairs.size) { p -> + val ref = pairs[p] + verifier.distance(signatures[v][ref.a], signatures[v][ref.b]) + } + + fun evaluate( + name: String, + costFilter: (Int) -> Boolean, + accept: (Int, Double) -> Boolean, + ): Policy { + var tp = 0 + var fp = 0 + var fn = 0 + var candidates = 0 + var falseCandidates = 0 + pairs.forEachIndexed { p, ref -> + if (costFilter(ref.stage1)) { + candidates++ + if (!ref.positive) falseCandidates++ + } + if (accept(ref.stage1, verified[p])) { + if (ref.positive) tp++ else fp++ + } else if (ref.positive) { + fn++ + } + } + return Policy(name, tp, fp, fn, candidates, falseCandidates) + } + + val tightOnly = evaluate("A) stage 1 only, thr $tight", { it <= tight }) { s, _ -> s <= tight } + val looseOnly = evaluate("B) stage 1 only, thr $loose", { it <= loose }) { s, _ -> s <= loose } + + val cascades = verifier.cutoffs().map { cutoff -> + evaluate( + "C) thr $loose + verifier <= %.2f".format(Locale.US, cutoff), + { it <= loose }, + ) { s, d -> s <= loose && d <= cutoff } + } + val bestCascade = cascades.maxBy { it.f1 } + + val verifierOnly = verifier.cutoffs().map { cutoff -> + evaluate( + "D) verifier alone <= %.2f".format(Locale.US, cutoff), + { true }, + ) { _, d -> d <= cutoff } + } + val bestVerifierOnly = verifierOnly.maxBy { it.f1 } + + report.appendLine("=== verifier: ${verifier.label} ===") + report.appendLine( + " policy prec recall F1 | st1 cands false cands", + ) + listOf(tightOnly, looseOnly, bestCascade, bestVerifierOnly).forEach { policy -> + report.appendLine( + " %-38s %6.3f %6.3f %6.3f | %9d %11d".format( + Locale.US, policy.name, policy.precision, policy.recall, policy.f1, + policy.stage1Candidates, policy.falseCandidates, + ), + ) + } + report.appendLine() + report.appendLine( + " cascade sweep at stage-1 thr $loose (cutoff -> precision, recall, F1, false cands):", + ) + verifier.cutoffs().forEachIndexed { index, cutoff -> + val policy = cascades[index] + report.appendLine( + " %6.2f -> %6.3f %6.3f %6.3f %6d".format( + Locale.US, cutoff, policy.precision, policy.recall, policy.f1, + policy.falseCandidates, + ), + ) + } + report.appendLine() + report.appendLine( + " verdict: cascade(C) %.3f vs loose-only(B) %.3f vs tight-only(A) %.3f".format( + Locale.US, bestCascade.f1, looseOnly.f1, tightOnly.f1, + ), + ) + report.appendLine() + } + return report.toString() + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/GroupingScaleTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/GroupingScaleTest.kt new file mode 100644 index 0000000..66deb81 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/GroupingScaleTest.kt @@ -0,0 +1,70 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import de.softwareapp_hb.fotodup.finder.core.DuplicateGrouper +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.core.PhotoFingerprint +import de.softwareapp_hb.fotodup.finder.core.RgbDHash +import org.junit.Assume.assumeTrue +import org.junit.Test +import java.util.Locale +import kotlin.random.Random + +/** + * Times the grouping pass at realistic library sizes, including a large cluster of mutually + * similar photos — the case that makes the banded candidate index behave quadratically. + * + * Opt-in: + * ``` + * ./gradlew :app:testDebugUnitTest --tests '*GroupingScaleTest*' -DfotoScale=true --rerun-tasks + * ``` + */ +class GroupingScaleTest { + + @Test + fun `time the grouper at library scale`() { + assumeTrue("set -DfotoScale=true to run", System.getProperty("fotoScale") == "true") + + val librarySize = 5_363 + val report = StringBuilder() + report.appendLine("library=$librarySize, BALANCED".format(Locale.US)) + + for (clusterSize in listOf(0, 10, 100, 500, 1_000, 2_500)) { + val random = Random(7) + val photos = ArrayList(librarySize) + repeat(librarySize - clusterSize) { index -> + photos += PhotoFingerprint( + id = index.toLong(), + sizeBytes = 1_000, + dateModifiedSeconds = 0, + width = 100, + height = 100, + perceptualHash = random.nextLong(), + rgbDHash = RgbDHash(random.nextLong(), random.nextLong(), random.nextLong()), + ) + } + // A cluster of mutually near-identical photos: all within a couple of bits, which is + // exactly what a folder of dark screenshots or a repeated export looks like. + repeat(clusterSize) { offset -> + photos += PhotoFingerprint( + id = (librarySize + offset).toLong(), + sizeBytes = 1_000, + dateModifiedSeconds = 0, + width = 100, + height = 100, + perceptualHash = offset.toLong() and 0x3L, + rgbDHash = RgbDHash(0L, 0L, 0L), + ) + } + + val started = System.nanoTime() + val result = DuplicateGrouper.findGroupsDetailed(photos, DuplicatePolicy.BALANCED) + val millis = (System.nanoTime() - started) / 1_000_000 + report.appendLine( + " identical cluster of %5d -> %6d ms, candidate pairs=%d, groups=%d".format( + Locale.US, clusterSize, millis, result.observedPairs, result.groups.size, + ), + ) + } + println(report) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageLab.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageLab.kt new file mode 100644 index 0000000..ada81ee --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageLab.kt @@ -0,0 +1,148 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import java.io.File +import java.security.MessageDigest +import javax.imageio.ImageIO +import kotlin.math.PI +import kotlin.math.cos +import kotlin.math.sin +import kotlin.random.Random + +/** A decoded image as packed ARGB pixels. */ +internal class Img(val pixels: IntArray, val width: Int, val height: Int) + +/** A transformation applied to a base image to produce a "near duplicate". */ +internal class Transform(val label: String, val apply: (Img) -> Img) + +/** + * Image sources and the transformations a real photo library inflicts on a picture. + * + * Runs entirely on the JVM (headless AWT), so the benchmark needs no device and no sample + * photos: it synthesises structured images, or loads a real folder when one is provided. + */ +internal object ImageLab { + + /** Transformations a genuine duplicate should survive — plus one it will not. */ + val transforms: List = listOf( + Transform("resize 50%") { ImageOps.resample(it, it.width / 2, it.height / 2) }, + Transform("resize 200%") { ImageOps.resample(it, it.width * 2, it.height * 2) }, + Transform("resize 33%") { ImageOps.resample(it, it.width / 3, it.height / 3) }, + Transform("jpeg q85") { ImageOps.jpeg(it, 0.85f) }, + Transform("jpeg q45") { ImageOps.jpeg(it, 0.45f) }, + Transform("brightness +12%") { ImageOps.levels(it, gain = 1.12, gamma = 1.0) }, + Transform("brightness -12%") { ImageOps.levels(it, gain = 0.88, gamma = 1.0) }, + Transform("gamma 1.25") { ImageOps.levels(it, gain = 1.0, gamma = 1.25) }, + Transform("crop 90%") { ImageOps.crop(it, 0.9) }, + // Geometry the descriptors are not designed for — measured on purpose so the + // limits are visible rather than assumed. + Transform("shift x 5%") { ImageOps.shift(it, percentX = 5, percentY = 0) }, + Transform("shift y 5%") { ImageOps.shift(it, percentX = 0, percentY = 5) }, + Transform("rotate 2°") { ImageOps.rotate(it, 2.0) }, + Transform("rotate 5°") { ImageOps.rotate(it, 5.0) }, + Transform("rotate 10°") { ImageOps.rotate(it, 10.0) }, + Transform("rotate 90") { ImageOps.rotate90(it) }, + Transform("rotate 180") { ImageOps.rotate180(it) }, + Transform("flip horizontal") { ImageOps.flipHorizontal(it) }, + ) + + /** + * Distinct, structured synthetic images. + * + * Deliberately *not* random noise: each has a gradient background plus solid blocks, so + * different seeds genuinely look different. Pure noise would make the "negative" pairs + * trivially separable and the numbers meaningless. + */ + fun syntheticBases(count: Int, width: Int = 320, height: Int = 240, seed: Int = 42): List = + (0 until count).map { syntheticImage(seed + it, width, height) } + + /** + * Loads real images from [dir] (recursively). + * + * Only [maxImages] are decoded — chosen by a seeded shuffle so the subset is not biased by + * filename order — because the harness holds every base image in memory. COCO photographs + * are 640 px on the long edge, so ~1.2 MB each decoded. + */ + fun loadFrom(dir: File, maxImages: Int = DEFAULT_MAX_IMAGES, seed: Int = 42): List { + require(dir.isDirectory) { "$dir is not a directory" } + val files = dir.walkTopDown() + .filter { it.isFile && it.extension.lowercase() in SUPPORTED } + .sortedBy { it.absolutePath } + .toList() + val selected = if (files.size <= maxImages) { + files + } else { + files.shuffled(Random(seed)).take(maxImages) + } + + // Two files with identical bytes are the same photo, not an unrelated pair. Datasets do + // ship such duplicates (BSDS500 includes its bench folder twice), and a real folder the + // user points at may too — but for *measuring* false positives they must go, otherwise + // the harness counts genuine duplicates as negatives. + val seen = HashSet() + val distinct = selected.filter { file -> + val digest = runCatching { + MessageDigest.getInstance("SHA-256").digest(file.readBytes()) + .joinToString("") { "%02x".format(it) } + }.getOrNull() + digest == null || seen.add(digest) + } + + return distinct + .mapNotNull { file -> runCatching { ImageIO.read(file) }.getOrNull() } + .map { ImageOps.fromBufferedImage(it) } + .filter { it.width >= 64 && it.height >= 64 } + } + + internal fun syntheticImage(seed: Int, width: Int, height: Int): Img { + val rnd = Random(seed) + val angle = rnd.nextDouble() * PI + val cosA = cos(angle) + val sinA = sin(angle) + val from = randomColor(rnd) + val to = randomColor(rnd) + val diagonal = (width + height).toDouble() + + val pixels = IntArray(width * height) + for (y in 0 until height) { + for (x in 0 until width) { + val t = (((x * cosA + y * sinA) / diagonal) + 0.5).coerceIn(0.0, 1.0) + pixels[y * width + x] = lerp(from, to, t) + } + } + + repeat(3 + rnd.nextInt(6)) { + val colour = randomColor(rnd) + val w = 20 + rnd.nextInt(maxOf(1, width / 2)) + val h = 20 + rnd.nextInt(maxOf(1, height / 2)) + val x0 = rnd.nextInt(maxOf(1, width - w)) + val y0 = rnd.nextInt(maxOf(1, height - h)) + for (yy in y0 until minOf(y0 + h, height)) { + for (xx in x0 until minOf(x0 + w, width)) { + pixels[yy * width + xx] = colour + } + } + } + return Img(pixels, width, height) + } + + private fun randomColor(rnd: Random): Int { + val r = 25 + rnd.nextInt(210) + val g = 25 + rnd.nextInt(210) + val b = 25 + rnd.nextInt(210) + return (0xFF shl 24) or (r shl 16) or (g shl 8) or b + } + + private fun lerp(a: Int, b: Int, t: Double): Int { + fun channel(shift: Int): Int { + val ca = (a shr shift) and 0xFF + val cb = (b shr shift) and 0xFF + return (ca + (cb - ca) * t).toInt().coerceIn(0, 255) + } + return (0xFF shl 24) or (channel(16) shl 16) or (channel(8) shl 8) or channel(0) + } + + private val SUPPORTED = setOf("jpg", "jpeg", "png", "bmp") + + /** Default cap on how many real photographs are decoded at once. */ + const val DEFAULT_MAX_IMAGES = 300 +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageOps.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageOps.kt new file mode 100644 index 0000000..6214be2 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/ImageOps.kt @@ -0,0 +1,159 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import java.awt.RenderingHints +import java.awt.geom.AffineTransform +import java.awt.image.BufferedImage +import java.io.ByteArrayInputStream +import java.io.ByteArrayOutputStream +import javax.imageio.IIOImage +import javax.imageio.ImageIO +import javax.imageio.ImageWriteParam +import kotlin.math.pow + +/** + * Image operations backed by `java.awt` in headless mode. + * + * Used by the similarity benchmark to simulate what a real photo library does to a + * picture: rescaling, JPEG re-encoding and tone changes. + */ +internal object ImageOps { + + fun resample(img: Img, width: Int, height: Int): Img { + val targetW = maxOf(1, width) + val targetH = maxOf(1, height) + val dst = BufferedImage(targetW, targetH, BufferedImage.TYPE_INT_RGB) + val g = dst.createGraphics() + g.setRenderingHint( + RenderingHints.KEY_INTERPOLATION, + RenderingHints.VALUE_INTERPOLATION_BILINEAR, + ) + g.drawImage(toBufferedImage(img), 0, 0, targetW, targetH, null) + g.dispose() + return fromBufferedImage(dst) + } + + /** Re-encode through a real JPEG codec at the given quality. */ + fun jpeg(img: Img, quality: Float): Img { + val bytes = ByteArrayOutputStream() + val writer = ImageIO.getImageWritersByFormatName("jpeg").next() + val param = writer.defaultWriteParam.apply { + compressionMode = ImageWriteParam.MODE_EXPLICIT + compressionQuality = quality + } + ImageIO.createImageOutputStream(bytes).use { stream -> + writer.output = stream + writer.write(null, IIOImage(toBufferedImage(img), null, null), param) + } + writer.dispose() + val decoded = ImageIO.read(ByteArrayInputStream(bytes.toByteArray())) + return fromBufferedImage(decoded) + } + + /** Brightness/gamma change through a 256-entry lookup table. */ + fun levels(img: Img, gain: Double, gamma: Double): Img { + val lut = IntArray(256) { v -> + ((v / 255.0).pow(gamma) * gain * 255.0).toInt().coerceIn(0, 255) + } + val out = IntArray(img.pixels.size) + for (i in img.pixels.indices) { + val p = img.pixels[i] + out[i] = (0xFF shl 24) or + (lut[(p shr 16) and 0xFF] shl 16) or + (lut[(p shr 8) and 0xFF] shl 8) or + lut[p and 0xFF] + } + return Img(out, img.width, img.height) + } + + /** Centre crop to [ratio] of the original size. */ + fun crop(img: Img, ratio: Double): Img { + val w = maxOf(1, (img.width * ratio).toInt()) + val h = maxOf(1, (img.height * ratio).toInt()) + val x0 = (img.width - w) / 2 + val y0 = (img.height - h) / 2 + val out = IntArray(w * h) + for (y in 0 until h) { + System.arraycopy(img.pixels, (y0 + y) * img.width + x0, out, y * w, w) + } + return Img(out, w, h) + } + + /** + * Rotates by [degrees] about the centre, keeping the original canvas — i.e. the + * corners are clipped, which is what happens when a print is photographed slightly + * crooked. + */ + fun rotate(img: Img, degrees: Double): Img { + val dst = BufferedImage(img.width, img.height, BufferedImage.TYPE_INT_RGB) + val g = dst.createGraphics() + g.setRenderingHint( + RenderingHints.KEY_INTERPOLATION, + RenderingHints.VALUE_INTERPOLATION_BILINEAR, + ) + g.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_QUALITY) + val transform = AffineTransform() + transform.rotate(Math.toRadians(degrees), img.width / 2.0, img.height / 2.0) + g.drawImage(toBufferedImage(img), transform, null) + g.dispose() + return fromBufferedImage(dst) + } + + fun rotate90(img: Img): Img { + val w = img.height + val out = IntArray(w * img.width) + for (y in 0 until img.height) { + for (x in 0 until img.width) { + out[x * w + (img.height - 1 - y)] = img.pixels[y * img.width + x] + } + } + return Img(out, w, img.width) + } + + fun rotate180(img: Img): Img { + val out = IntArray(img.pixels.size) + val last = img.pixels.size - 1 + for (i in img.pixels.indices) out[i] = img.pixels[last - i] + return Img(out, img.width, img.height) + } + + fun flipHorizontal(img: Img): Img { + val out = IntArray(img.pixels.size) + for (y in 0 until img.height) { + val row = y * img.width + for (x in 0 until img.width) out[row + x] = img.pixels[row + (img.width - 1 - x)] + } + return Img(out, img.width, img.height) + } + + /** + * Translates the image by a percentage of its own size, replicating the edge pixels. + * Models a photo that is off-centre within a resaved/re-rendered frame. + */ + fun shift(img: Img, percentX: Int, percentY: Int): Img { + val dx = img.width * percentX / 100 + val dy = img.height * percentY / 100 + val out = IntArray(img.pixels.size) + for (y in 0 until img.height) { + val sy = (y - dy).coerceIn(0, img.height - 1) + for (x in 0 until img.width) { + val sx = (x - dx).coerceIn(0, img.width - 1) + out[y * img.width + x] = img.pixels[sy * img.width + sx] + } + } + return Img(out, img.width, img.height) + } + + fun toBufferedImage(img: Img): BufferedImage { + val bi = BufferedImage(img.width, img.height, BufferedImage.TYPE_INT_RGB) + bi.setRGB(0, 0, img.width, img.height, img.pixels, 0, img.width) + return bi + } + + fun fromBufferedImage(bi: BufferedImage): Img { + val w = bi.width + val h = bi.height + val pixels = IntArray(w * h) + bi.getRGB(0, 0, w, h, pixels, 0, w) + return Img(pixels, w, h) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryBench.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryBench.kt new file mode 100644 index 0000000..9d4aee6 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryBench.kt @@ -0,0 +1,179 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import java.util.Locale + +/** + * Answers the cost question the all-duplicates benchmark could not: + * + * **how much of a realistic library would an expensive second stage have to touch?** + * + * The floor is the share of photos that genuinely have a near-duplicate — a verifier cannot + * avoid those. Everything above that floor is what stage 1's false positives cost. + */ +internal object LibraryBench { + + /** Slack added to the tight stage-1 threshold to produce the "loose" one. */ + private const val LOOSE_SLACK = 6 + + private class Candidate(val a: Int, val b: Int, val stage1: Int, val positive: Boolean) + + private class Outcome( + val name: String, + val tp: Int, + val fp: Int, + val fn: Int, + val candidatePairs: Int, + val candidatePhotos: Int, + ) { + val precision = if (tp + fp == 0) 1.0 else tp.toDouble() / (tp + fp) + val recall = if (tp + fn == 0) 1.0 else tp.toDouble() / (tp + fn) + val f1 = if (precision + recall == 0.0) 0.0 else 2 * precision * recall / (precision + recall) + } + + fun run( + sceneCount: Int, + density: Double, + verifiers: List, + seed: Int = 42, + ): String { + val library = LibraryModel.build(sceneCount, density, seed, verifiers) + val n = library.imageCount + val text = StringBuilder() + + // `LibraryModel.build` appends a duplicate copy immediately after its original, so a + // duplicate pair is always (k, k+1). Skipping those is then an array lookup instead of + // a hash-set probe on millions of pairs. + val isDuplicateCopy = BooleanArray(n) + library.duplicatePairs.forEach { pair -> + require(pair[1] == pair[0] + 1) { "duplicate copies must be adjacent" } + isDuplicateCopy[pair[1]] = true + } + fun isDuplicatePair(a: Int, b: Int) = b == a + 1 && isDuplicateCopy[b] + + // Pass 1: exact negative-distance histogram — millions of pairs, no per-pair storage. + val negativeHistogram = IntArray(65) + for (a in 0 until n) { + for (b in a + 1 until n) { + if (isDuplicatePair(a, b)) continue + negativeHistogram[library.stage1Distance(a, b)]++ + } + } + + fun falsePositivesUpTo(threshold: Int) = (0..threshold).sumOf { negativeHistogram[it] } + val tight = (0..64).last { falsePositivesUpTo(it) == 0 } + val loose = tight + LOOSE_SLACK + val totalDuplicates = library.duplicatePairs.size + val duplicatePhotos = totalDuplicates * 2 + + // Pass 2: only the pairs stage 1 proposes — a tiny fraction of the negative pairs. + val candidates = ArrayList() + for (a in 0 until n) { + for (b in a + 1 until n) { + if (isDuplicatePair(a, b)) continue + val distance = library.stage1Distance(a, b) + if (distance <= loose) { + candidates += Candidate(a, b, distance, positive = false) + } + } + } + library.duplicatePairs.forEach { pair -> + val distance = library.stage1Distance(pair[0], pair[1]) + if (distance <= loose) { + candidates += Candidate(pair[0], pair[1], distance, positive = true) + } + } + + fun photosInPairs(threshold: Int): Int { + val seen = HashSet() + candidates.forEach { candidate -> + if (candidate.stage1 <= threshold) { + seen += candidate.a + seen += candidate.b + } + } + return seen.size + } + + text.appendLine( + "scenes=$sceneCount duplicate rate=%.0f%% library images=%d duplicate pairs=%d".format( + Locale.US, density * 100, n, totalDuplicates, + ), + ) + text.appendLine( + "photos genuinely in a duplicate pair: $duplicatePhotos " + + "(%.1f%% of the library — the floor a verifier cannot go below)".format( + Locale.US, 100.0 * duplicatePhotos / n, + ), + ) + text.appendLine("stage 1 = pHash; tight=$tight (zero false positives); loose=$loose") + text.appendLine() + text.appendLine( + "policy prec recall F1 | cand pairs stage-2 photos", + ) + + fun line(outcome: Outcome) { + text.appendLine( + " %-38s %6.3f %6.3f %6.3f | %10d %6d (%.1f%% of library)".format( + Locale.US, outcome.name, outcome.precision, outcome.recall, outcome.f1, + outcome.candidatePairs, outcome.candidatePhotos, + 100.0 * outcome.candidatePhotos / n, + ), + ) + } + + fun stage1Only(name: String, threshold: Int): Outcome { + val missed = library.duplicatePairs.count { + library.stage1Distance(it[0], it[1]) > threshold + } + val tp = totalDuplicates - missed + val fp = falsePositivesUpTo(threshold) + return Outcome(name, tp, fp, missed, tp + fp, photosInPairs(threshold)) + } + + line(stage1Only("A) stage 1 only, thr $tight", tight)) + line(stage1Only("B) stage 1 only, thr $loose", loose)) + + // The verifier's cost is fixed by stage 1: it needs a signature for every photo that + // appears in a pair stage 1 proposed. The cutoff only changes what it decides. + val stage2Photos = photosInPairs(loose) + + verifiers.forEachIndexed { v, verifier -> + val best = verifier.cutoffs().map { cutoff -> + var tp = 0 + var fp = 0 + candidates.forEach { candidate -> + if (candidate.stage1 > loose) return@forEach + val distance = verifier.distance( + library.verifierSignatures[v][candidate.a], + library.verifierSignatures[v][candidate.b], + ) + if (distance <= cutoff) { + if (candidate.positive) tp++ else fp++ + } + } + Outcome( + name = "C) thr $loose + verifier <= %.2f".format(Locale.US, cutoff), + tp = tp, + fp = fp, + fn = totalDuplicates - tp, + candidatePairs = tp + fp, + candidatePhotos = stage2Photos, + ) + }.maxBy { it.f1 } + line(best) + text.appendLine(" ^ best cutoff for ${verifier.label}") + } + + text.appendLine() + text.appendLine( + "reading: a stage-2 verifier earns its place only if row C beats row B on recall", + ) + text.appendLine( + "without losing precision, and with a stage-2 photo count close to the floor above.", + ) + text.appendLine( + "false-positive candidate pairs at the loose threshold: ${candidates.count { !it.positive }}", + ) + return text.toString() + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryModel.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryModel.kt new file mode 100644 index 0000000..6148eee --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LibraryModel.kt @@ -0,0 +1,84 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import de.softwareapp_hb.fotodup.finder.core.PerceptualHash +import kotlin.random.Random + +/** + * A synthetic photo library — mostly unique photos plus a minority of near-duplicate copies + * — described **only by signatures**. + * + * Images are generated, hashed and discarded immediately, so a library of a few thousand + * photos costs a couple of megabytes instead of gigabytes of pixels. + */ +internal object LibraryModel { + + const val WIDTH = 320 + const val HEIGHT = 240 + + /** + * Transformations a real re-save or edit produces. + * + * Extreme geometry (90°/180°/mirror) is excluded: those are unusual duplicates and *no* + * binary descriptor handles them, so including them would depress every number by the + * same amount without telling us anything. + */ + val duplicateTransforms = ImageLab.transforms.filterNot { + it.label in setOf("rotate 90", "rotate 180", "flip horizontal") + } + + class Library( + val imageCount: Int, + val stage1: LongArray, + /** Ground-truth near-duplicate pairs, as image indices. */ + val duplicatePairs: List, + /** The transform that produced each duplicate pair, aligned with [duplicatePairs]. */ + val duplicateLabels: List, + val verifierSignatures: List>, + ) { + fun stage1Distance(a: Int, b: Int): Int = + PerceptualHash.hammingDistance(stage1[a], stage1[b]) + + /** Share of photo pairs that are actually duplicates, as a percentage. */ + val densityPercent: Double + get() = 100.0 * duplicatePairs.size / imageCount + } + + fun build( + sceneCount: Int, + density: Double, + seed: Int, + verifiers: List, + ): Library { + val random = Random(seed) + val stage1 = ArrayList() + val signatures = List(verifiers.size) { ArrayList() } + val pairs = ArrayList() + val labels = ArrayList() + + fun add(img: Img): Int { + val index = stage1.size + stage1 += PerceptualHash.phashFromArgb(img.pixels, img.width, img.height) + verifiers.forEachIndexed { v, verifier -> signatures[v] += verifier.signature(img) } + return index + } + + for (scene in 0 until sceneCount) { + val original = ImageLab.syntheticImage(seed + scene, WIDTH, HEIGHT) + val originalIndex = add(original) + if (random.nextDouble() < density) { + val transform = duplicateTransforms[random.nextInt(duplicateTransforms.size)] + val copyIndex = add(transform.apply(original)) + pairs += intArrayOf(originalIndex, copyIndex) + labels += transform.label + } + } + + return Library( + imageCount = stage1.size, + stage1 = stage1.toLongArray(), + duplicatePairs = pairs, + duplicateLabels = labels, + verifierSignatures = signatures.map { it.toTypedArray() }, + ) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LocalGroupingTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LocalGroupingTest.kt new file mode 100644 index 0000000..4ac48e9 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/LocalGroupingTest.kt @@ -0,0 +1,121 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import de.softwareapp_hb.fotodup.finder.core.DuplicateGrouper +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.core.PerceptualHash +import de.softwareapp_hb.fotodup.finder.core.PhotoFingerprint +import de.softwareapp_hb.fotodup.finder.data.humanReadableBytes +import org.junit.Assume.assumeTrue +import org.junit.Test +import java.io.File +import javax.imageio.ImageIO + +/** + * Runs the app's **real** grouping logic against a folder of images, on the desktop. + * + * Only MediaStore, the SQLite index and the progress notification need Android — the duplicate + * engine is pure Kotlin, so the algorithm can be exercised without a phone. Prepare a folder with: + * + * ``` + * python3 tools/normalise-for-local-grouping.py ~/fixtures/foto-dup-edge /tmp/fotonorm + * ./gradlew :app:testDebugUnitTest --tests '*LocalGroupingTest*' -DfotoLocal.dir=/tmp/fotonorm + * ``` + * + * Faithful: hashing, the adaptive threshold, banded candidate generation, the pHash∧RGB + * conjunction, union-find clustering and the resulting groups. + * Not faithful: MediaStore's view of what is indexed, the provider's thumbnail decode, and the + * SQLite cache. + */ +class LocalGroupingTest { + + private class Row( + val name: String, + val sizeBytes: Long, + val modifiedSeconds: Long, + val sha256: String, + val normalised: String?, + val width: Int, + val height: Int, + val error: String?, + ) + + private class Pixels(val values: IntArray, val width: Int, val height: Int) + + @Test + fun `group a prepared folder`() { + val dir = System.getProperty("fotoLocal.dir").orEmpty() + assumeTrue("set -DfotoLocal.dir=", dir.isNotBlank()) + val root = File(dir) + val manifest = File(root, "manifest.tsv") + assumeTrue("missing ${manifest.path}", manifest.isFile) + + val rows = manifest.readLines().filter { it.isNotBlank() }.map { line -> + val parts = line.split('\t') + Row( + name = parts[0], + sizeBytes = parts[1].toLong(), + modifiedSeconds = parts[2].toLong(), + sha256 = parts[3], + normalised = parts[4].ifBlank { null }, + width = parts[5].toInt(), + height = parts[6].toInt(), + error = parts.getOrNull(7)?.ifBlank { null }, + ) + } + + val fingerprints = rows.mapIndexed { index, row -> + val pixels = row.normalised?.let { readPixels(File(root, it)) } + PhotoFingerprint( + id = index.toLong(), + sizeBytes = row.sizeBytes, + dateModifiedSeconds = row.modifiedSeconds, + width = row.width, + height = row.height, + contentHash = row.sha256, + perceptualHash = pixels?.let { + PerceptualHash.phashFromArgb(it.values, it.width, it.height) + }, + rgbDHash = pixels?.let { + PerceptualHash.rgbDHashFromArgb(it.values, it.width, it.height) + }, + ) + } + + val result = DuplicateGrouper.findGroupsDetailed(fingerprints, DuplicatePolicy.BALANCED) + val nameOf = fingerprints.associate { it.id to rows[it.id.toInt()].name } + val unreadable = rows.filter { it.error != null } + + val report = StringBuilder() + report.appendLine("images=${fingerprints.size} groups=${result.groups.size}") + report.appendLine( + "effective cut: pHash ≤ ${result.thresholds.phashDistance}, " + + "RGB ≤ ${result.thresholds.rgbDistance}" + + if (result.usedFallbackThreshold) " (fallback to the ceiling)" else " (fitted)", + ) + report.appendLine("candidate pairs proposed by the band index: ${result.observedPairs}") + report.appendLine("unreadable (${unreadable.size}): ${unreadable.joinToString { it.name }}") + report.appendLine() + if (result.groups.isEmpty()) report.appendLine(" (no duplicate groups)") + result.groups.forEach { group -> + report.appendLine( + "${group.kind} — ${group.members.size} copies · " + + "${humanReadableBytes(group.reclaimableBytes)} reclaimable", + ) + group.members.forEach { member -> + report.appendLine( + " ${nameOf[member.id]} (${humanReadableBytes(member.sizeBytes)})", + ) + } + } + println(report) + } + + private fun readPixels(file: File): Pixels? { + val image = runCatching { ImageIO.read(file) }.getOrNull() ?: return null + val width = image.width + val height = image.height + val values = IntArray(width * height) + image.getRGB(0, 0, width, height, values, 0, width) + return Pixels(values, width, height) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBench.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBench.kt new file mode 100644 index 0000000..9e48e01 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBench.kt @@ -0,0 +1,170 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import de.softwareapp_hb.fotodup.finder.core.PerceptualHash +import java.util.Locale + +/** A 64-bit descriptor under test. */ +internal enum class Descriptor(val label: String) { + DHASH("dHash (current)") { + override fun hash(img: Img): Long = + PerceptualHash.fromArgb(img.pixels, img.width, img.height) + }, + PHASH("pHash (DCT)") { + override fun hash(img: Img): Long = + PerceptualHash.phashFromArgb(img.pixels, img.width, img.height) + }, + ; + + abstract fun hash(img: Img): Long +} + +/** + * Measures how well each descriptor separates real near-duplicates (a base image vs a + * transformed copy of itself) from unrelated images. + * + * Reports, per descriptor: hashing cost, the Hamming-distance distributions, a + * precision/recall/F1 sweep over thresholds, and per-transform recall. That makes the + * dHash-vs-pHash choice a measurement rather than an opinion. + */ +internal object SimilarityBench { + + /** One operating point of the threshold sweep. */ + private class Point(val threshold: Int, val tp: Int, val fp: Int, val fn: Int) { + val precision = if (tp + fp == 0) 1.0 else tp.toDouble() / (tp + fp) + val recall = if (tp + fn == 0) 1.0 else tp.toDouble() / (tp + fn) + val f1 = if (precision + recall == 0.0) 0.0 else 2 * precision * recall / (precision + recall) + } + + fun run(bases: List, thresholds: IntRange = 0..24): String { + val report = StringBuilder() + val transforms = ImageLab.transforms + val positivePairs = bases.size * transforms.size + val negativePairs = bases.size * (bases.size - 1) / 2 + + report.appendLine( + "images=${bases.size} transforms=${transforms.size} " + + "positive pairs=$positivePairs negative pairs=$negativePairs", + ) + report.appendLine( + "note: rotation and flip transforms are included on purpose — neither descriptor " + + "is rotation invariant, so expect those rows to fail; see the per-transform table.", + ) + report.appendLine() + + val verdicts = mutableListOf() + + for (descriptor in Descriptor.entries) { + // --- accuracy ----------------------------------------------------- + // Each transformed image is produced, hashed and discarded immediately, so + // memory stays flat no matter how large the image set is. + val baseHashes = bases.map { descriptor.hash(it) } + val perTransform = ArrayList(transforms.size) + transforms.forEach { transform -> + val distances = IntArray(bases.size) + bases.forEachIndexed { index, base -> + val transformed = transform.apply(base) + distances[index] = PerceptualHash.hammingDistance( + baseHashes[index], + descriptor.hash(transformed), + ) + } + perTransform += distances + } + + val positives = ArrayList(positivePairs) + perTransform.forEach { distances -> distances.forEach { positives += it } } + + val negatives = ArrayList(negativePairs) + for (i in bases.indices) { + for (j in i + 1 until bases.size) { + negatives += PerceptualHash.hammingDistance(baseHashes[i], baseHashes[j]) + } + } + + // --- hash cost, measured on its own ------------------------------- + // Only the descriptor is inside the timed region: image transforms cost far + // more than hashing and would otherwise drown the number we care about. + repeat(3) { bases.forEach { descriptor.hash(it) } } // JIT warm-up + val repeats = 5 + val timingStarted = System.nanoTime() + repeat(repeats) { bases.forEach { descriptor.hash(it) } } + val elapsedMs = (System.nanoTime() - timingStarted) / 1_000_000.0 + val timedImages = bases.size * repeats + + report.appendLine("== ${descriptor.label} ==") + report.appendLine( + " hash cost %.3f ms/image (%.0f images/s; %d hashes in %.0f ms)".format( + Locale.US, + elapsedMs / timedImages, + timedImages / (elapsedMs / 1000.0), + timedImages, + elapsedMs, + ), + ) + report.appendLine(" duplicate distances: ${describe(positives)}") + report.appendLine(" unrelated distances: ${describe(negatives)}") + report.appendLine(" thr | TP FP FN | precision recall F1") + + val sweep = thresholds.map { t -> + Point( + threshold = t, + tp = positives.count { it <= t }, + fp = negatives.count { it <= t }, + fn = positives.count { it > t }, + ) + } + sweep.forEach { point -> + report.appendLine( + " %3d | %4d %5d %5d | %9.3f %6.3f %6.3f".format( + Locale.US, point.threshold, point.tp, point.fp, point.fn, + point.precision, point.recall, point.f1, + ), + ) + } + + val bestF1 = sweep.maxBy { it.f1 } + report.appendLine( + " best F1 %.3f at threshold %d%s".format( + Locale.US, + bestF1.f1, + bestF1.threshold, + if (bestF1.threshold == thresholds.last) " (raise the upper bound!)" else "", + ), + ) + + // The operating point that matters in production: the widest threshold that + // still produces no false positives at all. + val safe = sweep.lastOrNull { it.fp == 0 } + val operatingThreshold = safe?.threshold ?: sweep.first().threshold + report.appendLine( + " zero false positives up to threshold $operatingThreshold " + + "-> recall %.3f, F1 %.3f".format(Locale.US, safe?.recall ?: 0.0, safe?.f1 ?: 0.0), + ) + verdicts += " %-16s thr %2d recall %.3f F1 %.3f (precision %.3f)".format( + Locale.US, descriptor.label, operatingThreshold, + safe?.recall ?: 0.0, safe?.f1 ?: 0.0, safe?.precision ?: 0.0, + ) + + report.appendLine(" recall per transform at threshold $operatingThreshold:") + transforms.forEachIndexed { index, transform -> + val hits = perTransform[index].count { it <= operatingThreshold } + report.appendLine( + " %-16s %3d%%".format(Locale.US, transform.label, hits * 100 / bases.size), + ) + } + report.appendLine() + } + + report.appendLine("== verdict: best threshold with ZERO false positives ==") + verdicts.forEach { report.appendLine(it) } + return report.toString() + } + + private fun describe(values: List): String { + if (values.isEmpty()) return "n/a" + val sorted = values.sorted() + fun percentile(q: Double) = sorted[((sorted.size - 1) * q).toInt()] + return "min=${sorted.first()} p5=${percentile(0.05)} median=${percentile(0.5)} " + + "p95=${percentile(0.95)} max=${sorted.last()}" + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBenchmarkTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBenchmarkTest.kt new file mode 100644 index 0000000..dc55960 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/SimilarityBenchmarkTest.kt @@ -0,0 +1,63 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import org.junit.Assume.assumeTrue +import org.junit.Test +import java.io.File + +/** + * Opt-in benchmark comparing the perceptual descriptors. + * + * It is skipped in a normal test run. To execute it: + * + * ``` + * # synthetic images + * ./gradlew :app:testDebugUnitTest --tests '*SimilarityBenchmarkTest*' -DfotoBench=true + * + * # your own photos + * ./gradlew :app:testDebugUnitTest --tests '*SimilarityBenchmarkTest*' \ + * -DfotoBench=true -DfotoBench.dir=/path/to/photos + * ``` + * + * The report is written to `app/build/reports/similarity-benchmark.txt`. + */ +class SimilarityBenchmarkTest { + + @Test + fun `compare dHash and pHash on transformed images`() { + assumeTrue("set -DfotoBench=true to run", System.getProperty("fotoBench") == "true") + + val dir = System.getProperty("fotoBench.dir").orEmpty() + val bases = if (dir.isNotBlank()) { + val max = System.getProperty("fotoBench.max")?.toIntOrNull() ?: ImageLab.DEFAULT_MAX_IMAGES + val loaded = ImageLab.loadFrom(File(dir), maxImages = max) + assumeTrue("fewer than 5 usable images in $dir", loaded.size >= 5) + println("loaded ${loaded.size} real photographs from $dir") + loaded + } else { + ImageLab.syntheticBases(count = System.getProperty("fotoBench.count")?.toIntOrNull() ?: 60) + } + + val report = SimilarityBench.run(bases) + write("similarity-benchmark.txt", report) + + val cascadeReport = CascadeBench.run(bases) + write("cascade-benchmark.txt", cascadeReport) + + val density = System.getProperty("fotoBench.density")?.toDoubleOrNull() ?: 0.15 + val scenes = System.getProperty("fotoBench.scenes")?.toIntOrNull() ?: 2500 + val libraryReport = LibraryBench.run( + sceneCount = scenes, + density = density, + verifiers = listOf(ColourHistogramVerifier, ColourDHashVerifier), + ) + write("library-benchmark.txt", libraryReport) + } + + private fun write(name: String, report: String) { + val out = File("build/reports/$name") + out.parentFile?.mkdirs() + out.writeText(report) + println(report) + println("report written to ${out.absolutePath}") + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/Verifiers.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/Verifiers.kt new file mode 100644 index 0000000..a588ba7 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/bench/Verifiers.kt @@ -0,0 +1,122 @@ +package de.softwareapp_hb.fotodup.finder.bench + +import kotlin.math.abs + +/** + * A second-stage verifier: comparatively expensive per photo, but only ever run on photos + * that already passed the cheap stage-1 filter. + * + * Signatures are `LongArray` so both packed binary hashes and quantised histograms fit the + * same shape. + */ +internal interface Verifier { + val label: String + + fun signature(img: Img): LongArray + + /** Distance between two signatures; the scale is verifier-specific. */ + fun distance(a: LongArray, b: LongArray): Double + + /** Cutoffs to sweep, ordered from strict to loose. */ + fun cutoffs(): List +} + +/** + * Normalised 4x4x4 RGB histogram compared with L1 distance. + * + * This captures **colour only, with no spatial information** — which is exactly why it is + * orthogonal to a grayscale structural hash and could catch hash collisions that share + * structure but differ in colour. + */ +internal object ColourHistogramVerifier : Verifier { + + override val label = "colour histogram 4x4x4 (L1)" + + private const val BINS = 64 + private const val SCALE = 10_000L + + override fun signature(img: Img): LongArray { + val counts = IntArray(BINS) + val pixels = img.pixels + for (i in pixels.indices) { + val p = pixels[i] + val r = ((p shr 16) and 0xFF) shr 6 + val g = ((p shr 8) and 0xFF) shr 6 + val b = (p and 0xFF) shr 6 + counts[(r shl 4) or (g shl 2) or b]++ + } + val total = pixels.size.coerceAtLeast(1) + return LongArray(BINS) { (counts[it].toLong() * SCALE) / total } + } + + override fun distance(a: LongArray, b: LongArray): Double { + var sum = 0L + for (i in a.indices) sum += abs(a[i] - b[i]) + return sum.toDouble() / SCALE + } + + override fun cutoffs(): List = + generateSequence(0.04) { it + 0.04 }.takeWhile { it <= 0.80 }.toList() +} + +/** + * One 64-bit difference hash per RGB channel, compared by the sum of the three Hamming + * distances. Structural, unlike the colour histogram, so a good cross-check on it. + */ +internal object ColourDHashVerifier : Verifier { + + override val label = "RGB dHash 3x64 bit (Hamming sum)" + + private const val GRID_WIDTH = 9 + private const val GRID_HEIGHT = 8 + + override fun signature(img: Img): LongArray = LongArray(3) { channel -> + differenceHash(img, shift = when (channel) { + 0 -> 16 + 1 -> 8 + else -> 0 + }) + } + + override fun distance(a: LongArray, b: LongArray): Double { + var sum = 0 + for (i in a.indices) sum += java.lang.Long.bitCount(a[i] xor b[i]) + return sum.toDouble() + } + + override fun cutoffs(): List = + (0..60 step 3).map { it.toDouble() } + + private fun differenceHash(img: Img, shift: Int): Long { + val grid = IntArray(GRID_WIDTH * GRID_HEIGHT) + for (y in 0 until GRID_HEIGHT) { + val y0 = y * img.height / GRID_HEIGHT + val y1 = maxOf(y0 + 1, (y + 1) * img.height / GRID_HEIGHT) + for (x in 0 until GRID_WIDTH) { + val x0 = x * img.width / GRID_WIDTH + val x1 = maxOf(x0 + 1, (x + 1) * img.width / GRID_WIDTH) + var sum = 0L + var count = 0 + for (yy in y0 until minOf(y1, img.height)) { + val row = yy * img.width + for (xx in x0 until minOf(x1, img.width)) { + sum += (img.pixels[row + xx] shr shift) and 0xFF + count++ + } + } + grid[y * GRID_WIDTH + x] = if (count == 0) 0 else (sum / count).toInt() + } + } + + var hash = 0L + var bit = 0 + for (y in 0 until GRID_HEIGHT) { + val row = y * GRID_WIDTH + for (x in 0 until GRID_WIDTH - 1) { + if (grid[row + x] > grid[row + x + 1]) hash = hash or (1L shl bit) + bit++ + } + } + return hash + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThresholdTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThresholdTest.kt new file mode 100644 index 0000000..9eaf00e --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/AdaptiveThresholdTest.kt @@ -0,0 +1,85 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import kotlin.math.exp + +class AdaptiveThresholdTest { + + private fun gaussianHistogram( + population: Int, + mean: Double, + sd: Double, + spike: IntArray = IntArray(0), + ): IntArray { + val histogram = IntArray(65) + for (d in 0..64) { + val z = (d - mean) / sd + val density = exp(-0.5 * z * z) / (sd * SQRT_TWO_PI) + histogram[d] = (population * density).toInt() + } + spike.forEachIndexed { index, count -> + if (index < histogram.size) histogram[index] += count + } + return histogram + } + + /** Rough shape of the unrelated-pair distances seen in the benchmark runs. */ + private fun unrelated(population: Int, spike: IntArray = IntArray(0)) = + gaussianHistogram(population, mean = 32.0, sd = 8.0, spike = spike) + + @Test + fun `too little data falls back to the configured ceiling`() { + val choice = AdaptiveThreshold.choose(unrelated(10), ceiling = 12, budget = 2.0) + assertTrue(choice.usedFallback) + assertEquals(12, choice.threshold) + } + + @Test + fun `a realistic library narrows the threshold below the ceiling`() { + val choice = AdaptiveThreshold.choose(unrelated(2_000), ceiling = 12, budget = 2.0) + assertTrue("expected no fallback but got one", !choice.usedFallback) + assertTrue("expected a threshold below the ceiling, got ${choice.threshold}", choice.threshold < 12) + assertTrue("expected a positive threshold, got ${choice.threshold}", choice.threshold > 0) + assertTrue("expected the budget to hold, got ${choice.expectedFalsePairs}", choice.expectedFalsePairs <= 2.0) + } + + @Test + fun `a bigger library gets a stricter threshold`() { + val small = AdaptiveThreshold.choose(unrelated(2_000), ceiling = 18, budget = 2.0).threshold + val large = AdaptiveThreshold.choose(unrelated(200_000), ceiling = 18, budget = 2.0).threshold + assertTrue("expected $large < $small", large < small) + } + + @Test + fun `a huge library is as strict as possible`() { + val choice = AdaptiveThreshold.choose(unrelated(50_000_000), ceiling = 18, budget = 2.0) + assertEquals(0, choice.threshold) + } + + @Test + fun `duplicate pairs at low distances barely shift the threshold`() { + val spike = IntArray(6) { 200 } // a few hundred genuine duplicates at distances 0..5 + val without = AdaptiveThreshold.choose(unrelated(2_000), ceiling = 12, budget = 2.0) + val with = AdaptiveThreshold.choose(unrelated(2_000, spike), ceiling = 12, budget = 2.0) + // The fit itself only uses distances >= 20, so the spike does not change the estimated + // distribution. It does inflate the pair population, which moves the threshold at most + // one bit in the safe direction. + assertTrue( + "expected at most a one-bit shift, got ${without.threshold} -> ${with.threshold}", + with.threshold <= without.threshold && without.threshold - with.threshold <= 1, + ) + } + + @Test + fun `a smaller budget never allows a wider threshold`() { + val generous = AdaptiveThreshold.choose(unrelated(50_000), ceiling = 18, budget = 5.0) + val strict = AdaptiveThreshold.choose(unrelated(50_000), ceiling = 18, budget = 0.5) + assertTrue(strict.threshold <= generous.threshold) + } + + private companion object { + const val SQRT_TWO_PI = 2.5066282746310002 + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/ByteUtilsTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/ByteUtilsTest.kt new file mode 100644 index 0000000..1b95701 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/ByteUtilsTest.kt @@ -0,0 +1,27 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Test + +class ByteUtilsTest { + + @Test + fun `toHex encodes with leading zeroes`() { + assertEquals("00", byteArrayOf(0).toHex()) + assertEquals("0f", byteArrayOf(0x0F).toHex()) + assertEquals("ff", byteArrayOf(0xFF.toByte()).toHex()) + assertEquals("0102ff", byteArrayOf(1, 2, 0xFF.toByte()).toHex()) + } + + @Test + fun `toBigEndianBytes is eight bytes long and ordered`() { + val bytes = 0x0102030405060708L.toBigEndianBytes() + assertEquals(8, bytes.size) + assertEquals(listOf(1, 2, 3, 4, 5, 6, 7, 8), bytes.toList()) + } + + @Test + fun `toBigEndianBytes round trips the low values`() { + assertEquals(listOf(0, 0, 0, 0, 0, 0, 0, 42), 42L.toBigEndianBytes().toList()) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DecodeMathTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DecodeMathTest.kt new file mode 100644 index 0000000..dc8a477 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DecodeMathTest.kt @@ -0,0 +1,56 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +class DecodeMathTest { + + private fun isPowerOfTwo(value: Int) = value > 0 && Integer.bitCount(value) == 1 + + @Test + fun `a typical camera photo is subsampled to about 100 px`() { + // 4000x3000 -> 32 gives 125x93, the first power of two inside the ~2x target window. + assertEquals(32, DecodeMath.sampleSizeFor(4000, 3000, target = 64)) + } + + @Test + fun `small images are barely subsampled`() { + // 100 px on the long edge is already inside the ~2x target window, so no subsampling. + assertEquals(1, DecodeMath.sampleSizeFor(100, 80, target = 64)) + // 128 is exactly 2x the target, so a single halving is what the rule asks for. + assertEquals(2, DecodeMath.sampleSizeFor(128, 128, target = 64)) + } + + @Test + fun `zero and negative dimensions yield a usable sample`() { + assertEquals(1, DecodeMath.sampleSizeFor(0, 0, target = 64)) + assertEquals(1, DecodeMath.sampleSizeFor(-5, -5, target = 64)) + assertEquals(1, DecodeMath.sampleSizeFor(4000, 3000, target = 0)) + } + + /** + * The decompression-bomb case: a crafted file declaring enormous dimensions must not overflow, + * and must still produce a sample that decodes to something tiny. + */ + @Test + fun `absurd dimensions do not overflow`() { + val bomb = DecodeMath.sampleSizeFor(Int.MAX_VALUE, Int.MAX_VALUE, target = 64) + assertTrue("expected a positive power of two but got $bomb", isPowerOfTwo(bomb)) + // 2^31 / 2^19 is still ~4000 px, well inside what a decoder will accept. + assertTrue("sample should stay bounded but was $bomb", bomb <= 1 shl 20) + } + + @Test + fun `sample always decodes the declared dimensions to roughly the target`() { + val cases = listOf(200 to 150, 640 to 480, 4000 to 3000, 12000 to 9000, 100_000 to 100_000) + for ((width, height) in cases) { + val sample = DecodeMath.sampleSizeFor(width, height, target = 64) + assertTrue("not a power of two for $width x $height: $sample", isPowerOfTwo(sample)) + assertTrue( + "decoded edge stayed too large for $width x $height: ${width / sample}", + width / sample <= 256 || width > 100_000, + ) + } + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroupTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroupTest.kt new file mode 100644 index 0000000..1ba76ca --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGroupTest.kt @@ -0,0 +1,136 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Test + +class DuplicateGroupTest { + + private fun fp(id: Long, size: Long = 1_000, date: Long = 0) = PhotoFingerprint( + id = id, + sizeBytes = size, + dateModifiedSeconds = date, + width = 100, + height = 100, + contentHash = "content", + ) + + private fun group(vararg members: PhotoFingerprint) = DuplicateGroup( + id = "g", + kind = GroupKind.EXACT, + members = members.toList(), + ) + + @Test + fun `suggested keep is the largest file`() { + val g = group(fp(1, size = 100), fp(2, size = 300), fp(3, size = 200)) + assertEquals(2L, g.suggestedKeep()?.id) + } + + @Test + fun `suggested keep breaks ties by newest`() { + val g = group(fp(1, size = 300, date = 10), fp(2, size = 300, date = 20)) + assertEquals(2L, g.suggestedKeep()?.id) + } + + @Test + fun `suggested removals exclude the keep`() { + val g = group(fp(1, size = 100), fp(2, size = 300), fp(3, size = 200)) + assertEquals(setOf(1L, 3L), g.suggestedRemovals().map { it.id }.toSet()) + } + + @Test + fun `display order puts the keep first and leaves the rest untouched`() { + val g = group( + fp(1, size = 100, date = 30), + fp(2, size = 300, date = 10), + fp(3, size = 200, date = 20), + ) + // id 2 is the largest, so it moves to the front; 1 and 3 keep their order. + assertEquals(listOf(2L, 1L, 3L), g.displayOrder().map { it.id }) + } + + @Test + fun `display order needs no reshuffle when the keep is already first`() { + val g = group(fp(1, size = 300, date = 5), fp(2, size = 100, date = 9)) + assertEquals(listOf(1L, 2L), g.displayOrder().map { it.id }) + } + + @Test + fun `empty group has no keep and an empty display order`() { + val g = group() + assertNull(g.suggestedKeep()) + assertEquals(emptyList(), g.displayOrder().map { it.id }) + } + + @Test + fun `reclaimable bytes keep the largest copy`() { + val g = group(fp(1, size = 100), fp(2, size = 300)) + assertEquals(100L, g.reclaimableBytes) + } + + @Test + fun `without keeps the group while at least two copies remain`() { + val g = group(fp(1), fp(2), fp(3)) + val remaining = g.without(setOf(1L)) + assertEquals(setOf(2L, 3L), remaining?.members?.map { it.id }?.toSet()) + } + + @Test + fun `without drops the group once fewer than two copies remain`() { + val g = group(fp(1), fp(2), fp(3)) + assertNull(g.without(setOf(1L, 2L))) + assertNull(g.without(setOf(1L, 2L, 3L))) + } + + @Test + fun `without with no ids returns the same group`() { + val g = group(fp(1), fp(2)) + assertEquals(g, g.without(emptySet())) + } + + @Test + fun `without keeps the group id so list keys stay stable`() { + val g = group(fp(1), fp(2), fp(3)) + assertEquals("g", g.without(setOf(1L))?.id) + } + + @Test + fun `display order after staging picks the best of what is left`() { + val g = group(fp(1, size = 100), fp(2, size = 300), fp(3, size = 200)) + // Stage the largest copy: the next-largest of the survivors becomes BEST. + val remaining = g.without(setOf(2L)) + assertEquals(listOf(3L, 1L), remaining?.displayOrder()?.map { it.id }) + assertEquals(3L, remaining?.suggestedKeep()?.id) + } + + @Test + fun `selected members are only the ticked ones`() { + val g = group(fp(1), fp(2), fp(3)) + assertEquals(setOf(1L, 2L), g.selectedMembers(setOf(1L, 2L, 99L)).map { it.id }.toSet()) + assertEquals(emptyList(), g.selectedMembers(emptySet()).map { it.id }) + } + + /** + * The reported bug: a group of BEST + DUP1 + DUP2 showed "Remove 2 duplicates" even after + * DUP2 was unticked, because the count came from the *suggestion* rather than the selection. + */ + @Test + fun `unticking a copy reduces the row removal count`() { + val g = group( + fp(1, size = 300), // BEST + fp(2, size = 100), // DUP1 + fp(3, size = 100), // DUP2 + ) + + // Right after a scan everything but BEST is ticked: the row offers to remove 2. + val ticked = g.suggestedRemovals().map { it.id }.toSet() + assertEquals(setOf(2L, 3L), ticked) + assertEquals(2, g.selectedMembers(ticked).size) + + // Unticking DUP2 must immediately read as 1. + val afterUntick = ticked - 3L + assertEquals(1, g.selectedMembers(afterUntick).size) + assertEquals(listOf(2L), g.selectedMembers(afterUntick).map { it.id }) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouperTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouperTest.kt new file mode 100644 index 0000000..72004f2 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/DuplicateGrouperTest.kt @@ -0,0 +1,238 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +class DuplicateGrouperTest { + + private fun fp( + id: Long, + size: Long = 1_000, + contentHash: String? = null, + perceptualHash: Long? = null, + // Derived from the pHash by default so existing tests keep their meaning; tests for the + // colour gate pass an explicit value. + rgbDHash: RgbDHash? = perceptualHash?.let { RgbDHash(it, it, it) }, + width: Int = 0, + height: Int = 0, + date: Long = 0, + ) = PhotoFingerprint( + id = id, + sizeBytes = size, + dateModifiedSeconds = date, + width = width, + height = height, + contentHash = contentHash, + perceptualHash = perceptualHash, + rgbDHash = rgbDHash, + ) + + /** + * The `solid-*.png` fixtures are flat colour, so every one of them hashes to exactly the same + * perceptual fingerprint. They must therefore come back as a *single* group, not split. + */ + @Test + fun `many photos with identical perceptual hashes form one group`() { + val photos = (1L..8L).map { + fp(it, perceptualHash = 1L, rgbDHash = RgbDHash(0L, 0L, 0L)) + } + val groups = DuplicateGrouper.findGroups(photos, DuplicatePolicy.BALANCED) + assertEquals(1, groups.size) + assertEquals(8, groups.single().members.size) + } + + /** Same, but large enough that the adaptive threshold actually runs instead of falling back. */ + @Test + fun `identical hashes stay together even when the adaptive fit runs`() { + // 400 flat photos + 400 unrelated ones: the 400 identical pairs dominate the low end of + // the candidate histogram, which must not split them apart. + val clones = (1L..400L).map { fp(it, perceptualHash = 1L, rgbDHash = RgbDHash(0L, 0L, 0L)) } + val others = (1L..400L).map { + fp(1_000L + it, perceptualHash = 0x0F0F0F0F0F0F0F0FL, rgbDHash = RgbDHash(it, it, it)) + } + val groups = DuplicateGrouper.findGroups(clones + others, DuplicatePolicy.BALANCED) + val biggest = groups.maxBy { it.members.size } + assertEquals(400, biggest.members.size) + } + + @Test + fun `exact duplicates are grouped`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, contentHash = "h1"), + fp(2, contentHash = "h1"), + fp(3, contentHash = "h1"), + fp(4, contentHash = "h2"), + ), + DuplicatePolicy.EXACT_ONLY, + ) + + assertEquals(1, groups.size) + val group = groups.single() + assertEquals(GroupKind.EXACT, group.kind) + assertEquals(setOf(1L, 2L, 3L), group.members.map { it.id }.toSet()) + } + + @Test + fun `unique photos produce no groups`() { + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, contentHash = "a"), fp(2, contentHash = "b")), + DuplicatePolicy.EXACT_ONLY, + ) + assertTrue(groups.isEmpty()) + } + + @Test + fun `perceptually close photos are grouped`() { + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0b11L)), + DuplicatePolicy.BALANCED, + ) + assertEquals(1, groups.size) + assertEquals(GroupKind.PERCEPTUAL, groups.single().kind) + assertEquals(setOf(1L, 2L), groups.single().members.map { it.id }.toSet()) + } + + @Test + fun `a pHash beyond the balanced ceiling is not grouped`() { + // Bits 0..19 differ => distance 20, above BALANCED's pHash ceiling of 12. + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0b11111111111111111111L)), + DuplicatePolicy.BALANCED, + ) + assertTrue(groups.isEmpty()) + } + + @Test + fun `a pHash within the balanced ceiling is grouped`() { + // Distance 10 is inside BALANCED's ceiling of 12. + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0b1111111111L)), + DuplicatePolicy.BALANCED, + ) + assertEquals(1, groups.size) + } + + @Test + fun `differing colour structure blocks a near duplicate`() { + // Identical luminance structure, but the colour hashes are 64 bits apart on every + // channel — beyond the RGB ceiling, so this must not be reported as a duplicate. + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, perceptualHash = 0L, rgbDHash = RgbDHash(0L, 0L, 0L)), + fp(2, perceptualHash = 0L, rgbDHash = RgbDHash(-1L, -1L, -1L)), + ), + DuplicatePolicy.BALANCED, + ) + assertTrue(groups.isEmpty()) + } + + @Test + fun `aggressive policy groups a distance ten pair`() { + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0b1111111111L)), + DuplicatePolicy.AGGRESSIVE, + ) + assertEquals(1, groups.size) + } + + @Test + fun `exact only policy ignores perceptual hashes`() { + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0L)), + DuplicatePolicy.EXACT_ONLY, + ) + assertTrue(groups.isEmpty()) + } + + @Test + fun `transitive similarity forms one cluster`() { + // 0 ~ 7 (distance 3), 7 ~ 63 (distance 3), 0 ~ 63 (distance 6). + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 7L), fp(3, perceptualHash = 63L)), + DuplicatePolicy.BALANCED, + ) + assertEquals(1, groups.size) + assertEquals(setOf(1L, 2L, 3L), groups.single().members.map { it.id }.toSet()) + } + + @Test + fun `exact matches are not repeated in the perceptual pass`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, contentHash = "same", perceptualHash = 0L), + fp(2, contentHash = "same", perceptualHash = 0L), + fp(3, contentHash = "other", perceptualHash = 0L), + ), + DuplicatePolicy.BALANCED, + ) + // One exact group of {1,2}; photo 3 differs in content and has no peer left. + assertEquals(1, groups.size) + assertEquals(GroupKind.EXACT, groups.single().kind) + assertEquals(setOf(1L, 2L), groups.single().members.map { it.id }.toSet()) + } + + @Test + fun `different aspect ratios are not grouped`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, perceptualHash = 0L, width = 100, height = 100), // 1.0 + fp(2, perceptualHash = 0L, width = 300, height = 100), // 3.0 + ), + DuplicatePolicy.BALANCED, + ) + assertTrue(groups.isEmpty()) + } + + @Test + fun `unknown dimensions do not block grouping`() { + val groups = DuplicateGrouper.findGroups( + listOf(fp(1, perceptualHash = 0L), fp(2, perceptualHash = 0L)), + DuplicatePolicy.BALANCED, + ) + assertEquals(1, groups.size) + } + + @Test + fun `reclaimable bytes keep the largest copy`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, size = 100, contentHash = "h"), + fp(2, size = 300, contentHash = "h"), + ), + DuplicatePolicy.EXACT_ONLY, + ) + val group = groups.single() + assertEquals(400, group.totalBytes) + assertEquals(300, group.largestBytes) + assertEquals(100, group.reclaimableBytes) + } + + @Test + fun `members are ordered newest first`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, contentHash = "h", date = 10), + fp(2, contentHash = "h", date = 20), + ), + DuplicatePolicy.EXACT_ONLY, + ) + assertEquals(listOf(2L, 1L), groups.single().members.map { it.id }) + } + + @Test + fun `groups are sorted by reclaimable bytes descending`() { + val groups = DuplicateGrouper.findGroups( + listOf( + fp(1, size = 100, contentHash = "small"), + fp(2, size = 100, contentHash = "small"), + fp(3, size = 5_000, contentHash = "big"), + fp(4, size = 5_000, contentHash = "big"), + ), + DuplicatePolicy.EXACT_ONLY, + ) + assertEquals(2, groups.size) + assertEquals("big", groups.first().id.removePrefix("exact:")) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/HashingTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/HashingTest.kt new file mode 100644 index 0000000..75a6e23 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/HashingTest.kt @@ -0,0 +1,103 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import java.io.ByteArrayInputStream +import java.io.InputStream + +class HashingTest { + + @Test + fun `sha256 matches the well known empty vector`() { + assertEquals( + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + Hashing.sha256(ByteArray(0)), + ) + } + + @Test + fun `sha256 matches the well known abc vector`() { + assertEquals( + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", + Hashing.sha256("abc".toByteArray(Charsets.UTF_8)), + ) + } + + @Test + fun `streaming sha256 equals in-memory sha256`() { + val data = ByteArray(200_000) { (it % 251).toByte() } + assertEquals(Hashing.sha256(data), Hashing.sha256(ByteArrayInputStream(data))) + } + + @Test + fun `quick signature is stable for identical input`() { + val a = Hashing.quickSignature(size = 3, head = "abc".toByteArray(), tail = "xyz".toByteArray()) + val b = Hashing.quickSignature(size = 3, head = "abc".toByteArray(), tail = "xyz".toByteArray()) + assertEquals(a, b) + } + + @Test + fun `quick signature changes with size, head and tail`() { + val base = Hashing.quickSignature(100, "AAAA".toByteArray(), "ZZZZ".toByteArray()) + assertNotEquals(base, Hashing.quickSignature(101, "AAAA".toByteArray(), "ZZZZ".toByteArray())) + assertNotEquals(base, Hashing.quickSignature(100, "BBBB".toByteArray(), "ZZZZ".toByteArray())) + assertNotEquals(base, Hashing.quickSignature(100, "AAAA".toByteArray(), "YYYY".toByteArray())) + } + + @Test + fun `quick signature works without a tail`() { + val a = Hashing.quickSignature(100, "AAAA".toByteArray()) + assertEquals("quick signature should be deterministic", a, Hashing.quickSignature(100, "AAAA".toByteArray())) + assertNotEquals(a, Hashing.quickSignature(100, "BBBB".toByteArray())) + } + + @Test + fun `readAtMost returns fewer bytes at end of stream`() { + assertArrayEquals(byteArrayOf(1, 2, 3), ByteArrayInputStream(byteArrayOf(1, 2, 3)).readAtMost(8)) + assertEquals(0, ByteArrayInputStream(byteArrayOf(1)).readAtMost(0).size) + } + + @Test + fun `readAtMost stops at the requested length`() { + val stream = ByteArrayInputStream(byteArrayOf(1, 2, 3, 4, 5)) + assertArrayEquals(byteArrayOf(1, 2), stream.readAtMost(2)) + } + + @Test + fun `readAtMost stops when the stream keeps reporting no data`() { + // A non-blocking pipe-backed descriptor can return 0 for ever. The loop must treat that as + // the end rather than spinning; the watchdog makes a regression fail instead of hanging. + val stream = object : InputStream() { + override fun read(): Int = 0 + override fun read(b: ByteArray, off: Int, len: Int): Int = 0 + } + var result: ByteArray? = null + val worker = Thread { result = stream.readAtMost(16) }.apply { + isDaemon = true + start() + } + worker.join(2_000) + assertTrue("readAtMost did not terminate", !worker.isAlive) + assertEquals(0, result!!.size) + } + + @Test + fun `sha256 stops when the stream keeps reporting no data`() { + val stream = object : InputStream() { + override fun read(): Int = 0 + override fun read(b: ByteArray, off: Int, len: Int): Int = 0 + } + var digest: String? = null + val worker = Thread { digest = Hashing.sha256(stream) }.apply { + isDaemon = true + start() + } + worker.join(2_000) + assertTrue("sha256 did not terminate", !worker.isAlive) + // Nothing was readable, so it hashes the empty input. + assertEquals(Hashing.sha256(ByteArray(0)), digest) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHashTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHashTest.kt new file mode 100644 index 0000000..7d0281e --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/core/PerceptualHashTest.kt @@ -0,0 +1,164 @@ +package de.softwareapp_hb.fotodup.finder.core + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +class PerceptualHashTest { + + private fun gray(v: Int): Int { + val c = v.coerceIn(0, 255) + return (0xFF shl 24) or (c shl 16) or (c shl 8) or c + } + + private fun image(width: Int, height: Int, luma: (x: Int, y: Int) -> Int): IntArray { + val pixels = IntArray(width * height) + for (y in 0 until height) { + for (x in 0 until width) { + pixels[y * width + x] = gray(luma(x, y)) + } + } + return pixels + } + + @Test + fun `hamming distance basics`() { + assertEquals(0, PerceptualHash.hammingDistance(0L, 0L)) + assertNotEquals(0, PerceptualHash.hammingDistance(0L, 1L)) + // 0L xor -1L has all 64 bits set. + assertEquals(64, PerceptualHash.hammingDistance(0L, -1L)) + } + + @Test + fun `uniform image has an empty hash`() { + val pixels = image(PerceptualHash.GRID_WIDTH, PerceptualHash.GRID_HEIGHT) { _, _ -> 128 } + assertEquals(0L, PerceptualHash.fromArgb(pixels, PerceptualHash.GRID_WIDTH, PerceptualHash.GRID_HEIGHT)) + } + + @Test + fun `strictly decreasing luminance sets every bit`() { + // 9x8 image where each pixel is brighter than the one on its right. + val pixels = image(PerceptualHash.GRID_WIDTH, PerceptualHash.GRID_HEIGHT) { x, _ -> 250 - x * 20 } + val hash = PerceptualHash.fromArgb(pixels, PerceptualHash.GRID_WIDTH, PerceptualHash.GRID_HEIGHT) + assertEquals(-1L, hash) + } + + @Test + fun `increasing luminance clears every bit and is far from the inverse`() { + val width = PerceptualHash.GRID_WIDTH + val height = PerceptualHash.GRID_HEIGHT + val decreasing = PerceptualHash.fromArgb(image(width, height) { x, _ -> 250 - x * 20 }, width, height) + val increasing = PerceptualHash.fromArgb(image(width, height) { x, _ -> 10 + x * 20 }, width, height) + assertEquals(0L, increasing) + assertEquals(64, PerceptualHash.hammingDistance(decreasing, increasing)) + } + + @Test + fun `fromLumaGrid sets the expected bits`() { + // Row-major 9x8 grid, all rows identical: bright for x<=4, dark afterwards. + val luma = IntArray(PerceptualHash.GRID_WIDTH * PerceptualHash.GRID_HEIGHT) { index -> + val x = index % PerceptualHash.GRID_WIDTH + if (x <= 4) 200 else 0 + } + val expected = (0 until 8).fold(0L) { acc, row -> acc or (1L shl (row * 8 + 4)) } + assertEquals(expected, PerceptualHash.fromLumaGrid(luma)) + } + + @Test + fun `fromLumaGrid rejects the wrong grid size`() { + try { + PerceptualHash.fromLumaGrid(IntArray(10)) + throw AssertionError("expected IllegalArgumentException") + } catch (expected: IllegalArgumentException) { + // expected + } + } + + @Test + fun `similar images stay close`() { + val width = 64 + val height = 64 + val base = image(width, height) { x, _ -> x * 4 } + // Same image with a subtle brightness bump on the left half. + val tweaked = image(width, height) { x, _ -> (x * 4 + if (x < 32) 6 else 0) } + val distance = PerceptualHash.hammingDistance( + PerceptualHash.fromArgb(base, width, height), + PerceptualHash.fromArgb(tweaked, width, height), + ) + assertTrue("expected a small distance but was $distance", distance <= 6) + } + + @Test + fun `luminance uses Rec 601 weights`() { + assertEquals(255, PerceptualHash.luminance(gray(255))) + assertEquals(0, PerceptualHash.luminance(gray(0))) + // Pure green is much brighter than pure blue for the human eye. + val green = (0xFF shl 24) or (0x00 shl 16) or (0xFF shl 8) or 0x00 + val blue = (0xFF shl 24) or (0x00 shl 16) or (0x00 shl 8) or 0xFF + assertTrue(PerceptualHash.luminance(green) > PerceptualHash.luminance(blue)) + } + + // --- pHash (DCT) -------------------------------------------------------- + + /** Structured 32x32 test image; r=g=b so the luminance is exact, not rounded. */ + private fun pattern(offset: Int) = image(32, 32) { x, y -> offset + (x * 3 + y * 5) % 60 } + + @Test + fun `phash is deterministic`() { + val pixels = pattern(30) + assertEquals( + PerceptualHash.phashFromArgb(pixels, 32, 32), + PerceptualHash.phashFromArgb(pixels, 32, 32), + ) + } + + /** + * A uniform brightness offset changes only the DC coefficient, so the AC coefficients + * — and therefore the median and every bit — are unchanged. This is the robustness + * that motivated the descriptor. + */ + @Test + fun `phash ignores a uniform brightness offset`() { + assertEquals( + PerceptualHash.phashFromArgb(pattern(offset = 40), 32, 32), + PerceptualHash.phashFromArgb(pattern(offset = 80), 32, 32), + ) + } + + /** Likewise a gain scales every coefficient equally, preserving the comparison. */ + @Test + fun `phash ignores a uniform brightness gain`() { + val base = pattern(offset = 40).copyOf() + val doubled = IntArray(base.size) { i -> + val p = base[i] + val v = ((p and 0xFF) * 2).coerceAtMost(255) + (0xFF shl 24) or (v shl 16) or (v shl 8) or v + } + assertEquals( + PerceptualHash.phashFromArgb(base, 32, 32), + PerceptualHash.phashFromArgb(doubled, 32, 32), + ) + } + + @Test + fun `phash separates structurally opposite images`() { + val increasing = image(32, 32) { x, _ -> x * 6 } + val decreasing = image(32, 32) { x, _ -> 250 - x * 6 } + val distance = PerceptualHash.hammingDistance( + PerceptualHash.phashFromArgb(increasing, 32, 32), + PerceptualHash.phashFromArgb(decreasing, 32, 32), + ) + assertTrue("expected a large distance but was $distance", distance >= 30) + } + + @Test + fun `phash rejects the wrong grid size`() { + try { + PerceptualHash.phashFromLumaGrid(IntArray(10)) + throw AssertionError("expected IllegalArgumentException") + } catch (expected: IllegalArgumentException) { + // expected + } + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/data/BoundedCallTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/data/BoundedCallTest.kt new file mode 100644 index 0000000..f504bf8 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/data/BoundedCallTest.kt @@ -0,0 +1,104 @@ +package de.softwareapp_hb.fotodup.finder.data + +import kotlinx.coroutines.CancellationException +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Before +import org.junit.Test +import java.util.concurrent.CountDownLatch +import kotlin.system.measureTimeMillis + +/** + * The regression behind the device hang: a `ContentResolver` call that never returns used to + * freeze the whole scan, because a coroutine timeout cannot interrupt a thread parked in a + * binder transaction. These tests pin the behaviour that fixes it — the caller gives up, the + * scan keeps moving. + */ +class BoundedCallTest { + + @Before + fun resetStallState() { + // The singleton is shared across tests in this JVM; a stall triggered by one test must + // not leak into the next. + BoundedCall.resetForTest() + } + + @Test + fun `returns the result of a call that finishes in time`() { + assertEquals("42", BoundedCall.orNull(5_000) { "42" }) + } + + @Test + fun `returns null when the call throws`() { + val result: String? = BoundedCall.orNull(5_000) { throw IllegalStateException("boom") } + assertNull(result) + } + + @Test + fun `abandons a call stuck for ever instead of waiting for it`() { + // Stands in for a provider call parked in a binder transaction: it would never finish. + val parkedForever = CountDownLatch(1) + var result: String? = "sentinel" + val elapsed = measureTimeMillis { + result = BoundedCall.orNull(200) { + parkedForever.await() + "too late" + } + } + assertNull("a call that overran must count as no result", result) + assertTrue("gave up only after ${elapsed}ms", elapsed < 5_000) + } + + @Test + fun `never waits longer than the deadline plus a small margin`() { + val parkedForever = CountDownLatch(1) + val elapsed = measureTimeMillis { BoundedCall.orNull(200) { parkedForever.await(); null } } + assertTrue("returned after ${elapsed}ms", elapsed in 150..3_000) + } + + @Test + fun `propagates cancellation so cancelling a scan still works`() { + assertThrows(CancellationException::class.java) { + BoundedCall.orNull(5_000) { throw CancellationException("cancelled") } + } + } + + @Test + fun `caps the threads parked in a stalled provider instead of one per file`() { + val parkedForever = CountDownLatch(1) + // 50 photos that would each park a thread for the full deadline: 50 x 50 ms = 2.5 s. + val elapsed = measureTimeMillis { + repeat(50) { + assertNull(BoundedCall.orNull(50) { parkedForever.await(); null }) + } + } + assertTrue("50 blocking calls took ${elapsed}ms", elapsed < 1_500) + assertTrue("parked ${BoundedCall.parkedCount()} threads", BoundedCall.parkedCount() <= 6) + assertTrue("the breaker should be open", BoundedCall.isStalled()) + } + + @Test + fun `recovers once the provider answers again`() { + val parkedForever = CountDownLatch(1) + repeat(3) { assertNull(BoundedCall.orNull(100) { parkedForever.await(); null }) } + assertTrue("the breaker should have tripped", BoundedCall.isStalled()) + + val refused = measureTimeMillis { + assertNull(BoundedCall.orNull(100) { parkedForever.await(); null }) + } + assertTrue("a stalled provider must be refused immediately, took ${refused}ms", refused < 50) + + // The provider comes back; the cooldown must elapse before a probe is admitted. + parkedForever.countDown() + var recovered: String? = null + val deadline = System.currentTimeMillis() + 8_000 + while (recovered == null && System.currentTimeMillis() < deadline) { + Thread.sleep(100) + recovered = BoundedCall.orNull(2_000) { "ok" } + } + assertEquals("the call must succeed once the provider answers again", "ok", recovered) + assertEquals("no thread should be counted as parked", 0, BoundedCall.parkedCount()) + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngineEndToEndTest.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngineEndToEndTest.kt new file mode 100644 index 0000000..e14b21f --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/domain/ScanEngineEndToEndTest.kt @@ -0,0 +1,205 @@ +package de.softwareapp_hb.fotodup.finder.domain + +import android.app.Application +import de.softwareapp_hb.fotodup.finder.core.DuplicatePolicy +import de.softwareapp_hb.fotodup.finder.core.GroupKind +import de.softwareapp_hb.fotodup.finder.data.BoundedCall +import de.softwareapp_hb.fotodup.finder.data.MediaStorePhotoSource +import de.softwareapp_hb.fotodup.finder.data.PhotoHasher +import de.softwareapp_hb.fotodup.finder.data.PhotoIndexStore +import de.softwareapp_hb.fotodup.finder.testutil.FakeMediaStoreProvider +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Before +import org.junit.Test +import org.junit.runner.RunWith +import org.robolectric.RobolectricTestRunner +import org.robolectric.RuntimeEnvironment +import org.robolectric.annotation.Config +import java.awt.image.BufferedImage +import java.io.File +import javax.imageio.IIOImage +import javax.imageio.ImageIO +import javax.imageio.ImageWriteParam + +/** + * Runs the **real** scan pipeline on the JVM: the MediaStore query, head/tail signatures, SHA-256, + * the decoded thumbnail, the SQLite index and the grouper — against [FakeMediaStoreProvider]. + * + * What it can prove: the query and projection handling, the head/tail read, the exact-duplicate + * path over genuine bytes, the index, the unreadable-file reporting, and that the pipeline runs + * to completion instead of stalling (which is what froze on a device). + * + * What it cannot: perceptual hashing. Robolectric's `ShadowBitmapFactory` does not decode real + * pixel data, so any assertion about near-duplicates has to come from the image-based harness in + * `bench/` or from a device. + */ +@RunWith(RobolectricTestRunner::class) +@Config(sdk = [36], application = Application::class) +class ScanEngineEndToEndTest { + + private lateinit var library: File + + @Before + fun setUp() { + val context = RuntimeEnvironment.getApplication() + library = File(context.cacheDir, "library").apply { + deleteRecursively() + mkdirs() + } + context.getDatabasePath("photo_index.db").delete() + // The provider-call breaker is a process-wide singleton; a stall from another test must + // not decide whether this scan is allowed to talk to the (fake) provider. + BoundedCall.resetForTest() + FakeMediaStoreProvider.install() + } + + @Test + fun `finds exact and near duplicates and survives unreadable files`() { + val scene = saveJpeg("scene.jpg", quality = 0.90f, tint = 0) + // A byte-identical copy: only the exact-duplicate path can catch this. + val copy = File(library, "scene-copy.jpg").apply { writeBytes(scene.readBytes()) } + // The same scene re-encoded twice: different bytes, so only the perceptual path can match. + val low = saveJpeg("scene-low.jpg", quality = 0.55f, tint = 0) + val low2 = saveJpeg("scene-low2.jpg", quality = 0.50f, tint = 0) + // An unrelated photo. + val other = saveJpeg("other.jpg", quality = 0.90f, tint = 96) + // Non-empty but undecodable, so MediaStore reports it and the hasher must give up. + val garbage = File(library, "garbage.jpg").apply { + writeBytes(ByteArray(4_096) { (it % 251).toByte() }) + } + + addPhoto(1, scene) + addPhoto(2, copy) + addPhoto(3, low) + addPhoto(4, low2) + addPhoto(5, other) + addPhoto(6, garbage) + + val result = runScan() + + assertEquals("every library entry should be seen", 6, result.photoCount) + + // The exact-duplicate path is fully exercised here: real bytes off disk, read through + // ContentResolver, head/tail signature, then a full SHA-256. + val exact = result.groups.filter { it.kind == GroupKind.EXACT } + assertEquals("the byte-identical copy is one exact group", 1, exact.size) + assertEquals(setOf(1L, 2L), exact.single().members.map { it.id }.toSet()) + + // Robolectric's ShadowBitmapFactory does not decode real pixel data, so every perceptual + // fingerprint comes back null and all six files are reported as unreadable. That is a + // limitation of the shadow, not of the app. What this test proves is the I/O half of the + // scan — the MediaStore query, the head/tail read, the exact path, the index and the + // reporting of unreadable files — and above all that it runs to completion rather than + // stalling, which is what froze on a device. + assertEquals("undecodable files must be reported, not dropped", 6, result.unreadableCount) + + // A second run must agree; it takes the cached path through the SQLite index. + val again = runScan() + assertEquals(result.groups.size, again.groups.size) + assertEquals(result.reclaimableBytes, again.reclaimableBytes) + assertEquals(result.unreadableCount, again.unreadableCount) + } + + /** + * The release blocker: on a real device the scan appeared frozen with the phase label still + * showing, while every worker was parked in a provider read (MediaProvider restarting, FUSE + * blocked). A provider call that never returns must be *abandoned*, never waited on. + */ + @Test + fun `a provider read that blocks for ever cannot freeze the scan`() { + val scene = saveJpeg("scene.jpg", quality = 0.90f, tint = 0) + // Byte-identical copies, so every one of them is read for the signature phase. + val copy = File(library, "scene-copy.jpg").apply { writeBytes(scene.readBytes()) } + val stuck = File(library, "stuck.jpg").apply { writeBytes(scene.readBytes()) } + + addPhoto(1, scene) + addPhoto(2, copy) + addPhoto(3, stuck) + + // Media id 3 never answers; the scan must still run to completion. + FakeMediaStoreProvider.blockReads(3) + val result = try { + runScan(providerTimeoutMs = 250L) + } finally { + FakeMediaStoreProvider.releaseBlockedReads() + } + + assertEquals("every library entry should be seen", 3, result.photoCount) + val exact = result.groups.filter { it.kind == GroupKind.EXACT } + assertEquals("the readable identical copies still group", 1, exact.size) + assertEquals( + "the blocked copy is reported as unreadable, not silently dropped", + 3, + result.unreadableCount, + ) + } + + private fun runScan(providerTimeoutMs: Long = 10_000L): ScanResult = runBlocking { + val context = RuntimeEnvironment.getApplication() + val engine = ScanEngine( + source = MediaStorePhotoSource(context), + hasher = PhotoHasher(context, providerTimeoutMs), + index = PhotoIndexStore(context), + ) + var finished: ScanResult? = null + engine.scan(DuplicatePolicy.BALANCED).collect { progress -> + when (progress) { + is ScanProgress.Finished -> finished = progress.result + is ScanProgress.Failed -> throw AssertionError("scan failed: ${progress.message}") + is ScanProgress.Working -> Unit + } + } + requireNotNull(finished) { "the scan produced no result" } + } + + private fun addPhoto(id: Long, file: File, width: Int = 160, height: Int = 120) { + FakeMediaStoreProvider.addFile( + id = id, + file = file, + width = width, + height = height, + modifiedSeconds = 1_700_000_000L + id, + ) + } + + /** + * A deterministic textured JPEG: solid colour blocks with hard edges. + * + * Deliberately not a smooth gradient — on a smooth ramp the per-channel difference hash is + * decided by compression noise rather than by structure, so two qualities can diverge. Real + * photographs have edges, which is what the descriptor is meant to key on. + */ + private fun saveJpeg(name: String, quality: Float, tint: Int): File { + val width = 160 + val height = 120 + val image = BufferedImage(width, height, BufferedImage.TYPE_INT_RGB) + val random = kotlin.random.Random(42 + tint) + for (blockY in 0 until 6) { + for (blockX in 0 until 8) { + val colour = java.awt.Color( + random.nextInt(60, 250), + random.nextInt(60, 250), + random.nextInt(60, 250), + ).rgb + for (y in blockY * height / 6 until (blockY + 1) * height / 6) { + for (x in blockX * width / 8 until (blockX + 1) * width / 8) { + image.setRGB(x, y, colour) + } + } + } + } + val file = File(library, name) + val writer = ImageIO.getImageWritersByFormatName("jpeg").next() + val param = writer.defaultWriteParam.apply { + compressionMode = ImageWriteParam.MODE_EXPLICIT + compressionQuality = quality + } + ImageIO.createImageOutputStream(file).use { output -> + writer.output = output + writer.write(null, IIOImage(image, null, null), param) + } + writer.dispose() + return file + } +} diff --git a/app/src/test/java/de/softwareapp_hb/fotodup/finder/testutil/FakeMediaStoreProvider.kt b/app/src/test/java/de/softwareapp_hb/fotodup/finder/testutil/FakeMediaStoreProvider.kt new file mode 100644 index 0000000..a58c4a8 --- /dev/null +++ b/app/src/test/java/de/softwareapp_hb/fotodup/finder/testutil/FakeMediaStoreProvider.kt @@ -0,0 +1,119 @@ +package de.softwareapp_hb.fotodup.finder.testutil + +import android.content.ContentProvider +import android.content.ContentValues +import android.database.Cursor +import android.database.MatrixCursor +import android.net.Uri +import android.os.ParcelFileDescriptor +import android.provider.MediaStore +import org.robolectric.Robolectric +import java.io.File +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.CountDownLatch + +/** + * A stand-in for the system media provider, so the *real* scan pipeline can run on the JVM. + * + * Registering this under the MediaStore authority means `MediaStorePhotoSource` queries it, and + * `ContentResolver.openFileDescriptor` reaches [openFile] — so `PhotoHasher` reads genuine files + * through the same code path a phone would exercise. + * + * The query answers with exactly the columns the caller asked for, so the source under test can + * request whatever projection it likes without this fake having to be kept in step. + */ +class FakeMediaStoreProvider : ContentProvider() { + + class Entry( + val id: Long, + val file: File, + val width: Int, + val height: Int, + val modifiedSeconds: Long, + val relativePath: String = "DCIM/", + ) + + override fun onCreate(): Boolean = true + + override fun query( + uri: Uri, + projection: Array?, + selection: String?, + selectionArgs: Array?, + sortOrder: String?, + ): Cursor { + val columns = projection?.toList() + ?: listOf(MediaStore.Images.Media._ID, MediaStore.Images.Media.SIZE) + val cursor = MatrixCursor(columns.toTypedArray()) + media.forEach { entry -> cursor.addRow(columns.map { valueFor(it, entry) }) } + return cursor + } + + override fun getType(uri: Uri): String = "image/jpeg" + + override fun openFile(uri: Uri, mode: String): ParcelFileDescriptor { + val id = uri.lastPathSegment?.toLongOrNull() ?: error("no media id in $uri") + // Stands in for a wedged provider/FUSE daemon: the call never returns until the test + // releases it. Used to prove the scan survives a provider call that blocks for ever. + blockedReads[id]?.await() + val entry = media.firstOrNull { it.id == id } ?: error("unknown media id $id") + return ParcelFileDescriptor.open(entry.file, ParcelFileDescriptor.MODE_READ_ONLY) + } + + override fun insert(uri: Uri, values: ContentValues?): Uri? = + throw UnsupportedOperationException("the fake library is read-only") + + override fun delete(uri: Uri, selection: String?, selectionArgs: Array?): Int = + throw UnsupportedOperationException("the fake library is read-only") + + override fun update( + uri: Uri, + values: ContentValues?, + selection: String?, + selectionArgs: Array?, + ): Int = throw UnsupportedOperationException("the fake library is read-only") + + private fun valueFor(column: String, entry: Entry): Any? = when (column) { + MediaStore.Images.Media._ID -> entry.id + MediaStore.Images.Media.DISPLAY_NAME -> entry.file.name + MediaStore.Images.Media.SIZE -> entry.file.length() + MediaStore.Images.Media.DATE_MODIFIED -> entry.modifiedSeconds + MediaStore.Images.Media.WIDTH -> entry.width + MediaStore.Images.Media.HEIGHT -> entry.height + MediaStore.Images.Media.MIME_TYPE -> "image/jpeg" + MediaStore.Images.Media.RELATIVE_PATH -> entry.relativePath + else -> null + } + + companion object { + val media: MutableList = mutableListOf() + + /** Media ids whose [openFile] blocks until [releaseBlockedReads] is called. */ + private val blockedReads: MutableMap = ConcurrentHashMap() + + /** Registers the provider under the MediaStore authority and clears the library. */ + fun install(): FakeMediaStoreProvider { + media.clear() + blockedReads.clear() + return Robolectric.buildContentProvider(FakeMediaStoreProvider::class.java) + .create(MediaStore.AUTHORITY) + .get() + } + + /** Makes every read of media [id] block, simulating a provider that never answers. */ + fun blockReads(id: Long) { + blockedReads[id] = CountDownLatch(1) + } + + /** Lets every blocked read finish, so a test can tear down cleanly. */ + fun releaseBlockedReads() { + blockedReads.values.forEach(CountDownLatch::countDown) + blockedReads.clear() + } + + /** Adds [file] to the fake library, deriving the reported size from the file itself. */ + fun addFile(id: Long, file: File, width: Int, height: Int, modifiedSeconds: Long = 0L) { + media += Entry(id, file, width, height, modifiedSeconds) + } + } +} diff --git a/build.gradle.kts b/build.gradle.kts new file mode 100644 index 0000000..0b90cd8 --- /dev/null +++ b/build.gradle.kts @@ -0,0 +1,10 @@ +// Top-level build file. Plugins are declared here (without applying them) so the +// versions are resolved once for the whole build, and applied per-module. +// +// NOTE: since AGP 9.0 Kotlin support is built into AGP ("built-in Kotlin"), so the +// `org.jetbrains.kotlin.android` plugin must NOT be applied anymore. Only the +// Compose compiler plugin is still applied explicitly. +plugins { + alias(libs.plugins.android.application) apply false + alias(libs.plugins.kotlin.compose) apply false +} diff --git a/docs/EDGE_CASE_TESTING.md b/docs/EDGE_CASE_TESTING.md new file mode 100644 index 0000000..d770d6e --- /dev/null +++ b/docs/EDGE_CASE_TESTING.md @@ -0,0 +1,212 @@ +# Edge-case testing + +Fixtures are generated, not committed: + +```bash +# text/PNG/JPEG fixtures +python3 tools/make-edge-case-fixtures.py + +# ...plus HEIC (needs an encoder; ImageMagick's HEIC support is read-only) +python3 -m venv /tmp/heicvenv && /tmp/heicvenv/bin/pip install pillow-heif +/tmp/heicvenv/bin/python tools/make-edge-case-fixtures.py + +# push to a phone +adb push fixtures/foto-dup-edge /sdcard/DCIM/foto-dup-edge +``` + +Everything lands in `fixtures/foto-dup-edge/` **inside the repo**. The directory is git-ignored +— the fixtures are regenerated on demand, so no binary test data is committed. + +## What each fixture is for + +| Fixture | Expected behaviour | +|---|---| +| `valid.jpg`, `tiny-1px.jpg`, `grayscale.jpg`, `with-alpha.png` | read normally | +| `zero-bytes.jpg`, `truncated.jpg`, `garbage-body.jpg` | skipped, counted in "could not be read", **no crash** | +| `renamed-png.jpg` (PNG named `.jpg`) | depends on the MIME reported by MediaStore — must not crash | +| `no-extension` | skipped by MediaStore; must not crash if it appears | +| `decompression-bomb.png` (declares 60 000²) | skipped, never allocates; exercises `DecodeMath` | +| `dup-a.jpg` / `dup-b.jpg` | **one exact-duplicate group** | +| `solid-0..5.png` | Flat colour: every one hashes *identically*, so they all belong in **one** group | +| `exif-orientation-1..8.jpg` | Identical **pixels** with different orientation **tags**, so they *display differently*. Only the upright one groups with the untagged PNG copies; the other seven are correctly distinct. A small group here therefore **proves EXIF normalisation is applied** — if it were ignored, all eight share identical pixels and would collapse into one group with the PNGs | +| `heic-basic.heic` | decoded and hashed like any other photo | +| `heic-dup.heic` | byte-identical to `heic-basic` → one exact-duplicate group | +| `heic-large.heic` (2400×1800) | exercises the subsampled fallback decode | +| `heic-solid-0/1.heic` | ditto the PNG solids, but they should join them: flat images of *any* colour are indistinguishable to the current descriptors | +| `scene.jpg` + `scene.heic` (one scene, two codecs) | informative: is a codec conversion still reported? | + +## Licence + +All fixtures are **generated locally from synthetic images**, so they carry no third-party +licence and can be pushed, kept or discarded freely. + +The obvious sources for ready-made HEIC samples are **not** clean: + +- `nokiatech/heif_conformance` — the canonical HEIC test set, but the repository has **no + LICENSE file** and the README states no terms, so it is all-rights-reserved by default. +- `strukturag/libheif` `examples/example.heic` — the repo is LGPL (library) / MIT (example + apps), but the **image file itself has no stated licence**. +- Wikimedia Commons — **hosts no HEIC files** (not an accepted upload format). + +That is why the HEIC fixtures are encoded here with `pillow-heif` instead. + +## Known gap this fixture set exposed + +**Flat-colour images are indistinguishable to the current descriptors.** A uniform image has no +structure, so its pHash is always `1` and its RGB dHash is always `0` — regardless of the colour. +A black screenshot, a white scan and a grey solid therefore all hash *identically* and get reported +as mutual duplicates. + +This is a genuine false-positive risk on real libraries (dark UI screenshots, blank scans). The fix +is to add the colour dimension the descriptors deliberately discard — a small colour histogram is +orthogonal to a grayscale structural hash and would separate them (see the benchmark harness, which +already has a colour-histogram implementation). + +## Running the grouping logic on a desktop + +The duplicate engine is pure Kotlin, so the algorithm can be exercised without a phone. Only +MediaStore, the SQLite index and the notification need Android. + +```bash +# normalise EXIF orientation with Pillow, then fingerprint and group in Kotlin +python3 tools/normalise-for-local-grouping.py fixtures/foto-dup-edge build/fotonorm +./gradlew :app:testDebugUnitTest --tests '*LocalGroupingTest*' \ + -DfotoLocal.dir=/tmp/fotonorm --rerun-tasks +``` + +(`--rerun-tasks` because Gradle does not always notice a changed `-D` value on the test task.) + +Faithful: hashing, the adaptive threshold, banded candidate generation, the pHash∧RGB conjunction, +union-find clustering and the resulting groups. +Not faithful: MediaStore's view of what is indexed, the provider's thumbnail decode, and the +SQLite hash cache. + +The last two matter — see below. + +### Why the device and a desktop run can disagree on flat images + +Local run of the fixture set: **all eight solids in one group**, as the algorithm demands. +On a real device the same eight came back as 3 + 2 + 2. + +Reproduced locally by feeding the harness *thumbnail-sized JPEGs* (96×72, q75) instead of the +lossless solids: the result becomes **3 + 2 + 2**, matching the device. + +Cause: JPEG quantises the DC coefficient, so a one-unit colour difference between neighbouring +solids falls below the quantisation step and they become effectively identical. On a device the +perceptual hash is taken from the provider's **lossy thumbnail**, so on a flat image it is +hash-of-quantisation-noise — there is no signal left to group by. + +Two consequences: + +1. Flat images (blank scans, dark UI screenshots) are grouped arbitrarily. **A colour signature is + the fix** — it is the one dimension the current descriptors discard. +2. Because the hash comes from the provider's thumbnail, it depends on the thumbnail's + re-encoding, not purely on the file. Real near-duplicates survive this (they are close + anyway), but the margin is thinner than a self-decoded hash would give. + +## Running the scan pipeline on the JVM (Robolectric) + +`ScanEngineEndToEndTest` drives the **real** pipeline — MediaStore query, head/tail signature, +SHA-256, SQLite index and grouper — against `FakeMediaStoreProvider`, a stand-in ContentProvider +registered under the MediaStore authority and backed by real files on disk. + +```bash +./gradlew :app:testDebugUnitTest --tests '*ScanEngineEndToEndTest*' +``` + +**What it covers:** query and projection handling, the head/tail read, the exact-duplicate path over +genuine bytes, the index across two runs, unreadable-file reporting, and that the scan **runs to +completion instead of stalling** — the failure that froze on a device. + +**What it cannot cover:** perceptual hashing. Robolectric's `ShadowBitmapFactory` does not decode +real pixel data, so every fingerprint is `null` and all files are reported as unreadable. The test +asserts that reality explicitly rather than pretending otherwise; near-duplicate behaviour has to +come from `bench/` or from a device. + +### JDK 17+ needs two different module flags + +Robolectric reflects into JDK internals, and on a modern JDK that fails unless the test JVM is +opened up. Two *distinct* flags are needed, and the second is easy to miss: + +``` +--add-opens=java.base/java.io=ALL-UNNAMED # FileDescriptor fields +--add-exports=java.base/jdk.internal.access=ALL-UNNAMED # SharedSecrets accessor +``` + +The second must be an **export**, not an open: Robolectric calls +`jdk.internal.access.SharedSecrets#getJavaIOFileDescriptorAccess` reflectively, and the failure +otherwise surfaces as an opaque `IllegalAccessException` at `Reflection.java:400`, wrapped in +*"Failed to interact with raw FileDescriptor internals; perhaps JRE has changed?"*. + +## What cannot be generated locally + +- **HEIC with an EXIF orientation tag.** `pillow-heif` accepts `exif=` on save but the + orientation tag reads back as `1` regardless, so a generated fixture would misrepresent what + it tests. This case needs a **real camera/iPhone HEIC** with a non-upright orientation. +- **HEIC from a real encoder** in general — worth spot-checking one real iPhone file. +- **Burst sequences / same-scene shots.** The one behaviour the benchmark cannot model, and the + most likely source of false groups. Needs real photos. +- **Permission revoked mid-scan**, and the Android 14 "Selected photos" flow: device only. + +## Driving a scan on a device without touch input + +The test device (Xiaomi M2101K7AG, Android 12 / MIUI, serial `2a2ae895`) denies the usual +automation, so a scan has to be driven from the app side instead: + +| Usual approach | Result on this device | +|---|---| +| `adb shell input tap/keyevent` | `SecurityException: INJECT_EVENTS` — MIUI blocks it, and the toggle that would allow it has no adb-reachable switch | +| `adb shell pm grant READ_EXTERNAL_STORAGE` | `SecurityException: Neither user 2000 nor current process has GRANT_RUNTIME_PERMISSIONS` | +| `adb install -g` | same refusal | + +Two **debug-only** hooks exist for this, under `app/src/debug/`. Both are absent from the +release build — re-check with the audit command in `PLAY_RELEASE_CHECKLIST.md` § 5. + +### Starting a scan + +```bash +adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.START_SCAN \ + -n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger +``` + +Add `--es policy exact|balanced|aggressive` to override the default (`balanced`). The receiver +logs what it did and the resulting `ScanState` under the `FotoDupDebug` tag, so progress **and** +the final timings are readable without the screen: + +```bash +adb logcat -s FotoDupDebug +# poll the current state at any time: +adb shell am broadcast -a de.softwareapp_hb.fotodup.finder.debug.REPORT_STATE \ + -n de.softwareapp_hb.fotodup.finder.debug/de.softwareapp_hb.fotodup.finder.debug.DebugScanTrigger +``` + +### Granting photo access + +`MANAGE_EXTERNAL_STORAGE` is the one storage permission that *can* be granted from adb, which is +why the **debug** manifest declares it: + +```bash +adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE allow +# ...run the scan, then put it back: +adb shell appops set de.softwareapp_hb.fotodup.finder.debug MANAGE_EXTERNAL_STORAGE default +``` + +The app's own banner still reads *“Photo access needed”*, because `checkSelfPermission` is +unaffected by an appops grant. That is expected: it only affects the on-screen prompt, not the scan. + +### Attaching a debugger for thread stacks + +`jdb` attaches over JDWP to the debug build: + +```bash +adb forward tcp:8700 jdwp:$(adb shell pidof de.softwareapp_hb.fotodup.finder.debug | tr -d '\r') +printf 'suspend\nwhere all\nquit\n' | jdb -attach localhost:8700 +``` + +Two things learned the hard way: + +- `print` needs a **selected frame**. With no current frame even `java.lang.Class` reports + "Name unknown", so expressions cannot be evaluated until a breakpoint has been reached. +- Leaving the main thread suspended for more than a few seconds **during an activity launch** + gets the process killed by the launch timeout. Evaluate and `cont` quickly. + diff --git a/docs/PLAY_RELEASE_CHECKLIST.md b/docs/PLAY_RELEASE_CHECKLIST.md new file mode 100644 index 0000000..266ea8c --- /dev/null +++ b/docs/PLAY_RELEASE_CHECKLIST.md @@ -0,0 +1,227 @@ +# Google Play release checklist + +Work top to bottom. Items marked **[code]** are already handled in this repository; +the rest are things you must do in the Play Console / with your own accounts. + +## 1. Identity and signing + +- [x] Final **application id** chosen: `de.softwareapp_hb.fotodup.finder` + (`applicationId` in `app/build.gradle.kts`). It can **never be changed after the + first release**, so treat it as locked. `namespace` and the Kotlin package of + the same name are aligned with it; debug builds append `.debug`. +- [ ] Generate an **upload keystore** (keep it safe; if you lose it you cannot + publish updates): + ```bash + keytool -genkeypair -v -keystore upload-keystore.jks \ + -alias upload -keyalg RSA -keysize 4096 -validity 10000 + ``` +- [ ] Copy `keystore.properties.example` to `keystore.properties` and fill it in. + **[code]** The Gradle build picks it up automatically and signs release + builds; without it the release APK stays unsigned. +- [ ] Enable **Play App Signing** in the Play Console and upload the certificate of + your *upload* key. +- [ ] Store the keystore + passwords in a password manager / secure vault. Back it up. + +## 2. Versioning + +- [ ] Set `versionName` (user-visible, e.g. `1.0.0`) **[code]** in + `app/build.gradle.kts`. +- [ ] Increment `versionCode` for **every** upload. + +## 3. Build the release artefacts + +```bash +./gradlew :app:bundleRelease # -> app/build/outputs/bundle/release/app-release.aab +./gradlew :app:assembleRelease # -> APK for local testing +``` + +- [ ] Upload the **`.aab`** (not the APK) to the Play Console. +- [ ] Keep `app/build/outputs/mapping/release/mapping.txt` for every release so you + can de-obfuscate crash reports (upload it under *Release > App bundle explorer*). + +## 4. Store listing + +Ready-to-paste copy and the generated graphics live in +[docs/STORE_LISTING.md](STORE_LISTING.md) (milestone M10). Everything below is a Console +action except the items marked **[code]**. + +- [ ] App name (≤ 30 chars): "Foto Dup Finder". +- [ ] Short description (≤ 80 chars) — two options are drafted and length-checked in + STORE_LISTING.md. +- [ ] Full description (≤ 4000 chars) — drafted and length-checked in STORE_LISTING.md. +- [x] **App icon** 512×512 PNG **[code]** — `tools/make-store-assets.sh` renders it from the + same vectors as the adaptive icon in `res/drawable/ic_launcher_*`. +- [x] **Feature graphic** 1024×500 PNG **[code]** — same script. +- [ ] At least 2 phone screenshots (up to 8). Three are captured (empty state, scanning, + results), but **re-shoot them after granting photo access on a fresh install**: the + existing ones still show the "Photo access needed" banner — see the caveat in + STORE_LISTING.md. + ⚠️ Play rejects a screenshot whose longer side is more than 2× its shorter side, so a + 1080×2400 phone capture has to be cropped to 1080×2160; + `tools/make-store-screenshots.sh` does exactly that (status and navigation bars only). +- [ ] Category: *Tools* (or *Photography*). Tags: photo cleanup, duplicates. + +## 5. Policy and compliance + +- [ ] Host [PRIVACY.md](../PRIVACY.md) at a public URL and paste that URL into the + Play Console. **This is mandatory** because the app accesses photos. + ⚠️ **Before hosting:** replace the `support@example.com` placeholder in the + *Contact* section with a real address — a policy whose contact is a placeholder is a + rejection risk. + + > **Note on Play's automatic translation:** it translates the *store listing*, not + > the hosted policy. If you want the policy in another language, host a translated + > copy yourself and link it from the listing. +- [ ] Complete the **Data safety** form: + - *Does your app collect or share any required user data types?* → **No**. + - Data types accessed: **Photos** — processed on-device only, not collected. + - *Is all user data encrypted in transit?* → N/A (no transmission). + - *Do you provide a way for users to request data deletion?* → Uninstalling + removes all app data. +- [ ] Confirm the **permission set** the Console will show. Read it from the *merged* + release manifest, not the source one, so nothing a library quietly adds slips past: + + | Declared permission | Why | From | + |---|---|---| + | `READ_MEDIA_IMAGES` | read the photo library | app | + | `READ_MEDIA_VISUAL_USER_SELECTED` | Android 14 partial access | app | + | `READ_EXTERNAL_STORAGE` (`maxSdkVersion=32`) | same, on Android ≤ 12 | app | + | `WRITE_EXTERNAL_STORAGE` (`maxSdkVersion=29`) | legacy delete on Android 10 only | app | + | `FOREGROUND_SERVICE`, `FOREGROUND_SERVICE_DATA_SYNC` | keep a scan alive | app | + | `POST_NOTIFICATIONS` | scan-progress notification | app | + | `…DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION` | framework-generated, not user-facing | AndroidX | + + Deliberately **absent**: `INTERNET` and `MANAGE_EXTERNAL_STORAGE`. Both are + load-bearing for the "nothing leaves the device" claim, so re-check after any new + dependency: + + ```bash + ./gradlew :app:processReleaseMainManifest + grep -A1 ' Foto Dup Finder is a photo-cleanup tool. Reading the photo library **is** its core + > functionality: it compares photos on the device to find exact and near duplicates + > and lets the user remove the copies they select. Nothing is uploaded — the app + > declares no network permission. Deletion always goes through the system consent + > dialog (`MediaStore.createDeleteRequest`); the app never requests + > `MANAGE_EXTERNAL_STORAGE`. + +- [ ] Declare the **foreground service type**. This is the release-blocking item for the + Android 14+ review. **[code]** The manifest already matches what you will declare: + + | Where | Value | + |---|---| + | `AndroidManifest.xml` | `` | + | `AndroidManifest.xml` | `` | + | `ScanService.kt` | `ServiceCompat.startForeground(…, ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC)` | + | `ScanService.kt` | `onTimeout()` stops the scan at Android 15's 6-hour `dataSync` cap | + + In the Console go to **App content › Foreground service permissions**, select + **Data sync**, and paste: + + > The app finds duplicate photos on the device. The user starts a scan and may leave + > the app while it runs; the foreground service keeps that user-initiated scan alive + > and shows its progress. The scan reads the photo library and writes a small local + > index of file hashes. It performs no network access and transfers no data. + + The declaration is **per type**: if a new foreground-service type is ever added to + the manifest, it has to be declared here too. +- [ ] Complete the **content rating** questionnaire (no objectionable content → + typically rated Everyone / 3+). +- [ ] Select the target audience (not designed for children). +- [ ] Accept the *Developer Program Policies* and *US export laws* declarations. + +## 6. Testing before production + +- [ ] Create an **internal testing** track and add yourself as a tester; install via + the Play link on a real device. +- [ ] Test on the oldest supported Android version you can (minSdk 26) and on the + newest you have (Android 16 / API 36). +- [ ] Test the permission flows: + - [ ] Deny access — app must not crash and must explain what is needed. + - [ ] Android 14+ **partial access** ("Select photos") — banner appears and the + app scans the subset. + - [ ] Grant full access — scan works. +- [ ] Test deletion: + - [ ] Move to trash (Android 11+). + - [ ] Permanent delete (Android 11+). + - [ ] Refuse the system dialog — nothing is deleted, the app says so. +- [ ] Test with a library of several thousand photos to sanity-check scan time. +- [ ] Verify no crashes in *Play Console > Quality > Android vitals* after rollout. + +## 7. Release notes ("What's new") and rollout + +Release notes are **not stored in this repo** — they are typed into the Play Console +for each release. + +Where: + +- Production: **Test and release > Production**, then **Create new release** +- Testing: **Test and release > Testing > Internal / Closed / Open testing** + +Rules (per Google's *Prepare and roll out a release* guide): + +- Up to **500 Unicode characters per language**. +- Language tags go on their own line, wrapping the text — a block starts with a line + containing `` and ends with a line containing ``. Several + `` blocks can be entered together. +- Notes belong to a **release**, so every track has its own; "Copy from a previous + release" avoids retyping. +- Google explicitly forbids using release notes for **promotion or to solicit user + actions** — keep them factual about what changed. +- For app updates list only user-visible changes; the full developer history belongs + in [CHANGELOG.md](../CHANGELOG.md). + +Ready-to-paste block for the first release: + + + Initial release. + + • Finds exact and near-duplicate photos + • Suggests which copy to keep + • Moves photos to the trash or deletes them via the system dialog + • No ads, no account, works fully offline + + +Ready-to-paste block for **1.0.1** (the scan-robustness patch): + + + • Fixed a scan that could appear to stop on very large photo libraries + • The app now recovers gracefully when the photo library is temporarily unresponsive + + +- [ ] Upload the `.aab`, paste the notes above, and roll out to **internal** → + **closed** → **production** (staged rollout recommended). +- [ ] Monitor Android vitals and reviews; respond to feedback. + +## 8. Post-launch maintenance + +- [ ] Play requires new apps/updates to target recent API levels (API 36 from + August 2026 **[code]**). Re-check this requirement each year and bump + `targetSdk`. +- [ ] Watch for newer AGP; when it supports it, update `compileSdk` to 37 and bump + the Compose BOM and Coil together (see the version note in the README). +- [ ] Optionally add crash reporting later — but note this would change the Data + safety answers, so weigh it against the "no data leaves the device" promise. + +## 9. Optional monetisation + +The app currently has no ads and no billing, which is a deliberate differentiator. +If you later add a paid tier, use **Play Billing** (e.g. a one-time "Pro" +purchase) and update: +- the Data safety form (purchase history), +- the privacy policy, +- the store description. + +Avoid ad SDKs entirely — they would contradict the app's main selling point. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..37ed4d9 --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,58 @@ +# Roadmap + +Foto Dup Finder is built against a fixed milestone plan (**M0–M10**). This file is the +single source of truth for what is done and what remains. Before this file existed the +plan lived only in developer notes, and the [CHANGELOG](../CHANGELOG.md) refers to +milestone labels (`M7`, `M8`). + +Legend: ✅ done · 🚧 in progress · ⏳ not started · 💡 candidate (proposed, not scheduled) + +## Milestones + +| # | Milestone | Status | Evidence / notes | +|---|---|---|---| +| M0 | Toolchain & scaffold | ✅ | User-local SDK, Gradle 9.3.1 + AGP 9.1.0, wrapper and version catalog, empty debug APK builds | +| M1 | Pure-Kotlin duplicate engine + JVM tests | ✅ | SHA-256 quick/full hashing, pHash + RGB dHash, union-find grouping, Exact/Balanced/Aggressive policies; tests under `app/src/test` | +| M2 | Data layer | ✅ | MediaStore query, coroutine-parallel hashing, incremental SQLite hash index, permission + partial-access handling | +| M3 | Compose UI | ✅ | Scan progress, grouped results with thumbnails, two-step staging, selection helpers | +| M4 | Safe deletion | ✅ | `MediaStore.createDeleteRequest` / `createTrashRequest`; legacy direct delete on Android 10 | +| M5 | Performance pass | ✅ | Size short-circuit, provider-thumbnail hashing, batched hashing (folded into M2/M3) | +| M6 | Release config & Play readiness | ✅ | R8/minify, optional signing, `.aab`, [PLAY_RELEASE_CHECKLIST](PLAY_RELEASE_CHECKLIST.md), [PRIVACY.md](../PRIVACY.md), README | +| M7 | On-device validation & hardening | ✅ | Foreground `dataSync` service, one failure boundary per file, `core/DecodeMath`, hostile fixtures ([EDGE_CASE_TESTING](EDGE_CASE_TESTING.md)), `versionName` → 1.0.0 | +| M8 | Localisation groundwork | ✅ | Every user-visible string in `res/values/strings.xml` with real ``, shared `text/TextRes.kt`; **English is the only locale** | +| M9 | Monetisation | ⏳ | No billing code. Blocked on a product decision (free forever vs. one-time unlock vs. limited free scans); changes the Data Safety answers | +| M10 | Store assets | ✅ | `tools/make-store-assets.sh` renders the 512×512 icon and the 1024×500 feature graphic from the app's own vectors; `tools/make-store-screenshots.sh` normalises device captures to Play's 2:1 limit (1080×2160); all listing copy is drafted and length-checked in [STORE_LISTING](STORE_LISTING.md). The captured screenshots still show the permission banner — see blockers | + +## Release blockers (work that is not a numbered milestone) + +| Item | Detail | +|---|---| +| ~~Large-library scan stall~~ ✅ **fixed — verified on device** | The 5,363-photo library used to freeze at *“74 / 100 · Comparing file signatures…”* (a **blocked syscall**: zero CPU, workers sleeping). `BoundedCall` now abandons a stuck provider call **and** trips a stall breaker after three consecutive timeouts. **Verified on the real test device (Android 12 / MIUI): the scan completes — 20.2 s cold, 11.0 s warm, 15 groups, 6 unreadable — where it used to hang.** Covered by `BoundedCallTest` and `ScanEngineEndToEndTest` | +| ~~Unreleased `BoundedCall` fix~~ ✅ | The provider-call deadline shipped in **1.0.1** (`versionCode` 2) | +| **Play Console: foreground-service declaration** | Declaring the `dataSync` type (App content › Foreground service permissions) is release-blocking. **[code]** the manifest already declares it; the ready-to-paste justification is in [PLAY_RELEASE_CHECKLIST](PLAY_RELEASE_CHECKLIST.md) step 5 — this is a Console action, not a code change | +| **Remaining Play paperwork** | Keystore / Play App Signing, Data Safety form, hosted privacy policy and store listing — see [PLAY_RELEASE_CHECKLIST](PLAY_RELEASE_CHECKLIST.md) | +| **Store screenshots still show the permission banner** | The three captures were taken with the debug-only `MANAGE_EXTERNAL_STORAGE` app-op, so the app's own permission check (correctly) still reported *no access* and the banner is in frame. Re-shoot on a clean install after tapping **Grant access** — the same run that closes the permission-flow verification gap. Files, specs and caveats: [STORE_LISTING](STORE_LISTING.md) | +| **Device-only verifications** | RAW/DNG files, a *real* iPhone HEIC with a non-upright EXIF orientation, permission revoked mid-scan, and the Android 14 “Selected photos” flow. ✅ The **index-reuse** check is done: after a scan the index holds **5,363 rows, 5,357 with a perceptual hash**, and a second scan runs in **11.0 s vs 20.2 s** cold | + +## Candidate milestones (proposed, not scheduled) + +| Candidate | Rationale | +|---|---| +| **Colour signature** | Flat-colour images (blank scans, dark UI screenshots) hash identically regardless of colour, so they are reported as mutual duplicates — see the “Known gap” in [EDGE_CASE_TESTING](EDGE_CASE_TESTING.md). A small colour histogram is orthogonal to the grayscale structure hash, and the benchmark harness already has an implementation to measure against | +| **Video support** | Only images are scanned today (README “Known limitations”) | +| **“Similar shots” section** | Bursts / same-subject shots as a separate, never-pre-selected list. Needs a different algorithm and must not feed the deletion flow | +| **Warm-scan speed** | A second scan is faster (11.0 s) but its “similar” stage still costs ~10 s even though all 5,357 perceptual hashes are cached — the per-photo `withTimeoutOrNull` wrapper still runs for all 5,363 photos. Profiling this could make an unchanged library re-scan in well under a second | + +## Suggested order + +1. ✅ **Done** — the scan-stall hardening and the `BoundedCall` fix shipped as **1.0.1** + (`versionCode` 2). Only the on-device re-check of the 5,363-photo scan remains. +2. Finish the Play Console paperwork and roll out through **internal → closed → production**. +3. ✅ **Done** — **M10 store assets**: the icon and feature graphic are generated from the + app's own vectors, the screenshot normaliser enforces Play's 2:1 limit, and the listing + copy is drafted and length-checked. The only remainder is the screenshot re-shoot after a + real permission grant (see blockers). +4. **Colour signature** — the flat-colour false-positive fix. +5. **M9 — monetisation**, once the product decision is made. + +Deliberately deferred: **video support** and the **“similar shots”** section. diff --git a/docs/STORE_LISTING.md b/docs/STORE_LISTING.md new file mode 100644 index 0000000..34593d1 --- /dev/null +++ b/docs/STORE_LISTING.md @@ -0,0 +1,137 @@ +# Play Store listing (M10) + +Everything the *Main store listing* page in the Play Console asks for, ready to copy. The +graphics are generated from the app's own vectors, so they can be regenerated after any +branding change instead of being edited by hand. + +See [PLAY_RELEASE_CHECKLIST](PLAY_RELEASE_CHECKLIST.md) § 4 for where these fit in the +release. + +## Graphics + +Generated into `store-assets/` **inside the repository**, never into `$HOME`. These PNGs are +the files that get uploaded to the Play Console, so they are committed and versioned with the +release they belong to. Only the uncropped captures the screenshots are cut from +(`store-assets/raw-screenshots/`) are git-ignored. + +```bash +./tools/make-store-assets.sh # icon + feature graphic -> store-assets/ +./tools/make-store-screenshots.sh # crop the adb captures -> store-assets/screenshots/ +``` + +| Asset | Play's spec | Result | Source | +|---|---|---|---| +| App icon | 512×512, 32-bit PNG with alpha, ≤ 1 MB | `store-assets/icon-512.png` — 512×512, RGBA, 6 KB | `tools/store-assets/icon.svg`, mirrored from `app/src/main/res/drawable/ic_launcher_background.xml` + `ic_launcher_foreground.xml` (paths and group transforms are identical, so the store icon *is* the launcher icon) | +| Feature graphic | 1024×500, JPEG or 24-bit PNG without alpha | `store-assets/feature-graphic-1024x500.png` — 1024×500, RGB | `tools/store-assets/feature-graphic.svg` | +| Phone screenshots | JPEG or 24-bit PNG without alpha, shorter side ≥ 320 px, longer side ≤ 3840 px, **longer side ≤ 2× shorter side** | `store-assets/screenshots/0{1,2,3}-*.png` — 1080×2160, RGB | `tools/make-store-screenshots.sh`, from raw `adb exec-out screencap` captures in `store-assets/raw-screenshots/` | + +Two details worth keeping in mind: + +- The icon is **full bleed**: the adaptive-icon background covers all 108 dp, so every pixel + of the PNG ends up opaque and Play's mask can crop to any shape without showing a gap. +- A phone screenshot at the test device's native 1080×2400 is **2.22:1 and therefore + rejected** — Play's limit is 2:1. The normaliser crops the status bar (93 px) and the + navigation bar (130 px) and clamps to 1080×2160, which removes system chrome only, never + app content. Re-measure both insets if you shoot on another device and pass + `STATUS_BAR_PX` / `NAV_BAR_PX`. + +## Screenshots + +Captured on the test device (1080×2400, Android 12 / MIUI), cropped to 1080×2160: + +| File | Screen | +|---|---| +| `01-empty.png` | Intro / empty state before the first scan | +| `02-scanning.png` | Scanning progress — *“Looking for visually similar photos…”* 5341 / 5363 | +| `03-results.png` | Results — 15 duplicate groups, 39.8 MB reclaimable, `BEST` badge, selection bar | + +⚠️ **Two caveats before these go live.** + +1. All three still show the *“Photo access needed”* banner. The test device (Xiaomi M2101K7AG + / MIUI) refuses `pm grant`, `install -g` and synthetic taps, so the runtime permission was + never actually granted there — the scan was driven with a debug-only + `MANAGE_EXTERNAL_STORAGE` app-op, which the app's permission check does not honour (see + [EDGE_CASE_TESTING](EDGE_CASE_TESTING.md)). **Re-shoot on a clean install after tapping + “Grant access”**: that also closes the one remaining device-only verification gap. +2. The results screenshot shows real photos from the test library. They are the developer's + own, so they can be published, but check that nothing personal is recognisable first. + +Screens still worth capturing once a device grants access through the real flow — each needs +one tap, which is exactly what the MIUI test device cannot be driven to do: + +- the **Options** sheet with the three sensitivity presets, +- the staged state (*“All duplicates are ready to go”*) after selecting duplicates, +- Android's **confirmation dialog** for moving photos to the trash. + +## Store listing copy + +### App name (max 30 characters) + + Foto Dup Finder + +### Short description (max 80 characters) + + Find and remove duplicate photos. Free, ad-free, works offline. + +Alternative, if the no-upload promise should lead: + + Find and delete duplicate photos. No ads, no uploads, no account. + +### Full description (max 4000 characters) + + Find and remove duplicate photos, then get the space back. Foto Dup Finder does one + job and does it without ads, without an account and without ever uploading anything. + + WHY IT IS DIFFERENT + • No ads — no banners, no interstitials, no upsells. + • No internet permission at all, so the app has no way to upload your photos. + • No account, no sign-in, no tracking, no analytics. + + WHAT IT DOES + • Finds exact copies: files that are byte-for-byte identical, even when their names + differ. + • Finds near-duplicates: the same picture saved twice, or two shots of the same + scene. A perceptual hash compares how the images actually look rather than only + their bytes, so a re-compressed or slightly re-cropped copy is still caught. + • Suggests what to keep. The best copy of each group is marked BEST, and + "Select all but best" marks the rest in a single tap. + • Lets you choose how far to go: Exact copies only, Balanced (recommended) or + Aggressive. + • Deletes safely. Every removal goes through Android's own confirmation dialog, and + you choose between moving photos to the trash (recoverable) and deleting them + permanently. Nothing is removed without your confirmation. + • Copes with a messy library. RAW files and unreadable or corrupt images are skipped + and reported instead of stopping the scan. + • Much faster the second time. File hashes are remembered on the device, so + re-scanning an unchanged library takes a fraction of the first run. + + HOW IT WORKS + 1. Grant photo access — everything runs on your device. + 2. Tap Start scan. A library of several thousand photos is done in seconds. + 3. Review the groups, keep the copies you want and select the rest. + 4. Tap "Remove all selected" and confirm once for the whole batch. + + YOUR PHOTOS STAY YOURS + Foto Dup Finder declares no internet permission, ships no advertising or analytics + SDKs, and stores nothing besides a local index of file hashes so the next scan can be + quick. No photo ever leaves your phone, and uninstalling the app removes the index + with it. + + GOOD TO KNOW + • Requires Android 8.0 or newer. + • Photos moved to the trash can be restored from your gallery app until the trash is + emptied (Android 11 and newer). + • Works fully offline, in flight mode, always. + +Every claim above is checkable against this repository: no `INTERNET` permission in the +merged manifest, no ad or analytics dependency in `gradle/libs.versions.toml`, `minSdk 26` +in `app/build.gradle.kts`, and the second-run speed-up is the persistent hash index in +`data/PhotoIndexStore.kt`. + +### Category and tags + +- **Category:** Tools (alternative: Photography — *Tools* matches the one-job utility framing + better, and Photography is more crowded). +- **Tags:** duplicate photos, photo cleanup, storage. +- **Contact details:** the same e-mail as in [PRIVACY.md](../PRIVACY.md) — replace the + `support@example.com` placeholder there first. diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..620f1c7 --- /dev/null +++ b/gradle.properties @@ -0,0 +1,20 @@ +# Project-wide Gradle settings. + +# JVM memory for the Gradle daemon (large Android builds need headroom). +org.gradle.jvmargs=-Xmx4g -XX:MaxMetaspaceSize=1g -Dfile.encoding=UTF-8 + +# Build performance. +org.gradle.parallel=true +org.gradle.caching=true + +# AndroidX is required for Jetpack libraries. +android.useAndroidX=true + +# Only generate R class entries for the module's own resources (faster builds). +android.nonTransitiveRClass=true + +# Let the Android Gradle Plugin auto-download any missing SDK components +# (platform/build-tools) using the licenses that were accepted during setup. +android.builder.sdkDownload=true + +kotlin.code.style=official diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml new file mode 100644 index 0000000..7ffbb19 --- /dev/null +++ b/gradle/libs.versions.toml @@ -0,0 +1,54 @@ +[versions] +agp = "9.1.0" +kotlin = "2.4.10" + +# AndroidX / Compose +coreKtx = "1.18.0" +lifecycle = "2.10.0" +activityCompose = "1.13.0" +exifinterface = "1.4.1" +# Compose BOM 2026.06.01 -> Compose UI/foundation 1.11.4, material3 1.4.0. +# Newer BOMs (2026.08.00+) ship Compose 1.12.x, which requires compileSdk 37 — +# that is newer than what AGP 9.1.0 supports (36) and newer than the API level +# Google Play requires for targetSdk, so we deliberately stay on 1.11.4. +composeBom = "2026.06.01" +# Coil 3.6.x requires compileSdk 37; 3.5.0 requires 36. +coil = "3.5.0" + +# Kotlin libraries +coroutines = "1.9.0" + +# Testing +junit = "4.13.2" +robolectric = "4.17" + +[libraries] +androidx-core-ktx = { group = "androidx.core", name = "core-ktx", version.ref = "coreKtx" } +androidx-exifinterface = { group = "androidx.exifinterface", name = "exifinterface", version.ref = "exifinterface" } + +androidx-lifecycle-runtime-ktx = { group = "androidx.lifecycle", name = "lifecycle-runtime-ktx", version.ref = "lifecycle" } +androidx-lifecycle-runtime-compose = { group = "androidx.lifecycle", name = "lifecycle-runtime-compose", version.ref = "lifecycle" } +androidx-lifecycle-viewmodel-compose = { group = "androidx.lifecycle", name = "lifecycle-viewmodel-compose", version.ref = "lifecycle" } + +androidx-activity-compose = { group = "androidx.activity", name = "activity-compose", version.ref = "activityCompose" } + +androidx-compose-bom = { group = "androidx.compose", name = "compose-bom", version.ref = "composeBom" } +androidx-compose-ui = { group = "androidx.compose.ui", name = "ui" } +androidx-compose-ui-graphics = { group = "androidx.compose.ui", name = "ui-graphics" } +androidx-compose-ui-tooling = { group = "androidx.compose.ui", name = "ui-tooling" } +androidx-compose-ui-tooling-preview = { group = "androidx.compose.ui", name = "ui-tooling-preview" } +androidx-compose-material3 = { group = "androidx.compose.material3", name = "material3" } +androidx-compose-material-icons-core = { group = "androidx.compose.material", name = "material-icons-core" } + +kotlinx-coroutines-android = { group = "org.jetbrains.kotlinx", name = "kotlinx-coroutines-android", version.ref = "coroutines" } + +coil-compose = { group = "io.coil-kt.coil3", name = "coil-compose", version.ref = "coil" } + +junit = { group = "junit", name = "junit", version.ref = "junit" } +robolectric = { group = "org.robolectric", name = "robolectric", version.ref = "robolectric" } + +[plugins] +android-application = { id = "com.android.application", version.ref = "agp" } +# NOTE: `org.jetbrains.kotlin.android` is intentionally absent — AGP 9.x provides +# built-in Kotlin support and applying that plugin is now an error. +kotlin-compose = { id = "org.jetbrains.kotlin.plugin.compose", version.ref = "kotlin" } diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..61285a6 Binary files /dev/null and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..5f38436 --- /dev/null +++ b/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,7 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-9.3.1-all.zip +networkTimeout=10000 +validateDistributionUrl=true +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists diff --git a/gradlew b/gradlew new file mode 100755 index 0000000..adff685 --- /dev/null +++ b/gradlew @@ -0,0 +1,248 @@ +#!/bin/sh + +# +# Copyright © 2015 the original authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# + +############################################################################## +# +# Gradle start up script for POSIX generated by Gradle. +# +# Important for running: +# +# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is +# noncompliant, but you have some other compliant shell such as ksh or +# bash, then to run this script, type that shell name before the whole +# command line, like: +# +# ksh Gradle +# +# Busybox and similar reduced shells will NOT work, because this script +# requires all of these POSIX shell features: +# * functions; +# * expansions «$var», «${var}», «${var:-default}», «${var+SET}», +# «${var#prefix}», «${var%suffix}», and «$( cmd )»; +# * compound commands having a testable exit status, especially «case»; +# * various built-in commands including «command», «set», and «ulimit». +# +# Important for patching: +# +# (2) This script targets any POSIX shell, so it avoids extensions provided +# by Bash, Ksh, etc; in particular arrays are avoided. +# +# The "traditional" practice of packing multiple parameters into a +# space-separated string is a well documented source of bugs and security +# problems, so this is (mostly) avoided, by progressively accumulating +# options in "$@", and eventually passing that to Java. +# +# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS, +# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly; +# see the in-line comments for details. +# +# There are tweaks for specific operating systems such as AIX, CygWin, +# Darwin, MinGW, and NonStop. +# +# (3) This script is generated from the Groovy template +# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt +# within the Gradle project. +# +# You can find Gradle at https://github.com/gradle/gradle/. +# +############################################################################## + +# Attempt to set APP_HOME + +# Resolve links: $0 may be a link +app_path=$0 + +# Need this for daisy-chained symlinks. +while + APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path + [ -h "$app_path" ] +do + ls=$( ls -ld "$app_path" ) + link=${ls#*' -> '} + case $link in #( + /*) app_path=$link ;; #( + *) app_path=$APP_HOME$link ;; + esac +done + +# This is normally unused +# shellcheck disable=SC2034 +APP_BASE_NAME=${0##*/} +# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036) +APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD=maximum + +warn () { + echo "$*" +} >&2 + +die () { + echo + echo "$*" + echo + exit 1 +} >&2 + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "$( uname )" in #( + CYGWIN* ) cygwin=true ;; #( + Darwin* ) darwin=true ;; #( + MSYS* | MINGW* ) msys=true ;; #( + NONSTOP* ) nonstop=true ;; +esac + + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD=$JAVA_HOME/jre/sh/java + else + JAVACMD=$JAVA_HOME/bin/java + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD=java + if ! command -v java >/dev/null 2>&1 + then + die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +fi + +# Increase the maximum file descriptors if we can. +if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then + case $MAX_FD in #( + max*) + # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + MAX_FD=$( ulimit -H -n ) || + warn "Could not query maximum file descriptor limit" + esac + case $MAX_FD in #( + '' | soft) :;; #( + *) + # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + ulimit -n "$MAX_FD" || + warn "Could not set maximum file descriptor limit to $MAX_FD" + esac +fi + +# Collect all arguments for the java command, stacking in reverse order: +# * args from the command line +# * the main class name +# * -classpath +# * -D...appname settings +# * --module-path (only if needed) +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables. + +# For Cygwin or MSYS, switch paths to Windows format before running java +if "$cygwin" || "$msys" ; then + APP_HOME=$( cygpath --path --mixed "$APP_HOME" ) + + JAVACMD=$( cygpath --unix "$JAVACMD" ) + + # Now convert the arguments - kludge to limit ourselves to /bin/sh + for arg do + if + case $arg in #( + -*) false ;; # don't mess with options #( + /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath + [ -e "$t" ] ;; #( + *) false ;; + esac + then + arg=$( cygpath --path --ignore --mixed "$arg" ) + fi + # Roll the args list around exactly as many times as the number of + # args, so each arg winds up back in the position where it started, but + # possibly modified. + # + # NB: a `for` loop captures its iteration list before it begins, so + # changing the positional parameters here affects neither the number of + # iterations, nor the values presented in `arg`. + shift # remove old arg + set -- "$@" "$arg" # push replacement arg + done +fi + + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Collect all arguments for the java command: +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments, +# and any embedded shellness will be escaped. +# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be +# treated as '${Hostname}' itself on the command line. + +set -- \ + "-Dorg.gradle.appname=$APP_BASE_NAME" \ + -jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \ + "$@" + +# Stop when "xargs" is not available. +if ! command -v xargs >/dev/null 2>&1 +then + die "xargs is not available" +fi + +# Use "xargs" to parse quoted args. +# +# With -n1 it outputs one arg per line, with the quotes and backslashes removed. +# +# In Bash we could simply go: +# +# readarray ARGS < <( xargs -n1 <<<"$var" ) && +# set -- "${ARGS[@]}" "$@" +# +# but POSIX shell has neither arrays nor command substitution, so instead we +# post-process each arg (as a line of input to sed) to backslash-escape any +# character that might be a shell metacharacter, then use eval to reverse +# that process (while maintaining the separation between arguments), and wrap +# the whole thing up as a single "set" statement. +# +# This will of course break if any of these variables contains a newline or +# an unmatched quote. +# + +eval "set -- $( + printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" | + xargs -n1 | + sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' | + tr '\n' ' ' + )" '"$@"' + +exec "$JAVACMD" "$@" diff --git a/gradlew.bat b/gradlew.bat new file mode 100644 index 0000000..e509b2d --- /dev/null +++ b/gradlew.bat @@ -0,0 +1,93 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem +@rem SPDX-License-Identifier: Apache-2.0 +@rem + +@if "%DEBUG%"=="" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%"=="" set DIRNAME=. +@rem This is normally unused +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if %ERRORLEVEL% equ 0 goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +goto fail + +:execute +@rem Setup the command line + + + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %* + +:end +@rem End local scope for the variables with windows NT shell +if %ERRORLEVEL% equ 0 goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +set EXIT_CODE=%ERRORLEVEL% +if %EXIT_CODE% equ 0 set EXIT_CODE=1 +if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE% +exit /b %EXIT_CODE% + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/keystore.properties.example b/keystore.properties.example new file mode 100644 index 0000000..623c674 --- /dev/null +++ b/keystore.properties.example @@ -0,0 +1,24 @@ +# Template for release signing. Copy to `keystore.properties` and fill in real values. +# +# `keystore.properties` is git-ignored and must NEVER be committed. +# +# Generate an upload keystore (keep the file and passwords safe — losing the +# upload key means you cannot ship updates to Google Play): +# +# keytool -genkeypair -v \ +# -keystore upload-keystore.jks \ +# -alias upload \ +# -keyalg RSA -keysize 4096 -validity 10000 +# +# Then enable Play App Signing in the Play Console and register this key as the +# *upload* key (Google keeps the app signing key). + +# Path to your keystore (absolute is safest). +storeFile=/absolute/path/to/fotodup-upload.jks +storePassword=change-me +keyAlias=upload + +# Optional — defaults to storePassword when omitted or left blank. +# PKCS12 keystores (the default on modern JDKs) cannot have a key password that +# differs from the store password, so this is normally the same value. +keyPassword=change-me diff --git a/settings.gradle.kts b/settings.gradle.kts new file mode 100644 index 0000000..86d7460 --- /dev/null +++ b/settings.gradle.kts @@ -0,0 +1,24 @@ +pluginManagement { + repositories { + google { + content { + includeGroupByRegex("com\\.android.*") + includeGroupByRegex("com\\.google.*") + includeGroupByRegex("androidx.*") + } + } + mavenCentral() + gradlePluginPortal() + } +} + +dependencyResolutionManagement { + repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) + repositories { + google() + mavenCentral() + } +} + +rootProject.name = "FotoDuplicateFinder" +include(":app") diff --git a/store-assets/feature-graphic-1024x500.png b/store-assets/feature-graphic-1024x500.png new file mode 100644 index 0000000..5453c32 Binary files /dev/null and b/store-assets/feature-graphic-1024x500.png differ diff --git a/store-assets/icon-512.png b/store-assets/icon-512.png new file mode 100644 index 0000000..01bf31e Binary files /dev/null and b/store-assets/icon-512.png differ diff --git a/store-assets/screenshots/01-empty.png b/store-assets/screenshots/01-empty.png new file mode 100644 index 0000000..825ee10 Binary files /dev/null and b/store-assets/screenshots/01-empty.png differ diff --git a/store-assets/screenshots/02-scanning.png b/store-assets/screenshots/02-scanning.png new file mode 100644 index 0000000..403c9b8 Binary files /dev/null and b/store-assets/screenshots/02-scanning.png differ diff --git a/store-assets/screenshots/03-results.png b/store-assets/screenshots/03-results.png new file mode 100644 index 0000000..f8231c7 Binary files /dev/null and b/store-assets/screenshots/03-results.png differ diff --git a/tools/make-edge-case-fixtures.py b/tools/make-edge-case-fixtures.py new file mode 100644 index 0000000..e1f5b8a --- /dev/null +++ b/tools/make-edge-case-fixtures.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Generate hostile / edge-case image fixtures for Foto Dup Finder. + +Every file here is something a real photo library can contain: an interrupted download, a +half-written JPEG, a crafted "decompression bomb", a photo whose pixels are fine but whose +EXIF says it is rotated, two byte-identical copies, and so on. + +They are written to fixtures/foto-dup-edge/ inside the repository. That directory is +git-ignored — the fixtures are regenerated on demand, so no binary test data is committed: + + python3 tools/make-edge-case-fixtures.py + adb push fixtures/foto-dup-edge /sdcard/DCIM/foto-dup-edge + +Then scan in the app and check: nothing crashes, unreadable files are reported rather than +silently dropped, the EXIF variants of the same picture are *not* reported as duplicates of +each other (they normalise to the same pixels), and dup-a/dup-b *are*. +""" +import random +import struct +import zlib +from pathlib import Path + +from PIL import Image + +REPO = Path(__file__).resolve().parent.parent +OUT = REPO / "fixtures" / "foto-dup-edge" +random.seed(7) + + +def write(name: str, data: bytes) -> Path: + path = OUT / name + path.write_bytes(data) + return path + + +def gradient(width: int = 320, height: int = 240) -> Image.Image: + """A structured base image — distinct enough from the solids to matter.""" + img = Image.new("RGB", (width, height)) + px = img.load() + for y in range(height): + for x in range(width): + px[x, y] = ((60 + x) % 256, (90 + y) % 256, 200) + return img + + +def png_chunk(kind: bytes, data: bytes) -> bytes: + return ( + struct.pack(">I", len(data)) + + kind + + data + + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF) + ) + + +def heic_section(base: Image.Image) -> None: + """HEIC samples, encoded locally so they carry no third-party licence. + + Android can decode HEIC from API 28, so these exercise that path — and because we encode + them ourselves from our own images, they can be shipped/pushed freely. + """ + try: + import pillow_heif + + pillow_heif.register_heif_opener() + except ImportError: + print("\nHEIC fixtures skipped — needs an encoder. Generate them with:") + print(" python3 -m venv /tmp/heicvenv") + print(" /tmp/heicvenv/bin/pip install pillow-heif") + print(" /tmp/heicvenv/bin/python tools/make-edge-case-fixtures.py") + return + + # The iPhone default format: control, plus a byte-identical copy for the exact-duplicate path. + basic = OUT / "heic-basic.heic" + base.save(basic, "HEIF", quality=90) + write("heic-dup.heic", basic.read_bytes()) + + # Bigger than a thumbnail, so the subsampled fallback decode gets exercised. + gradient(2400, 1800).save(OUT / "heic-large.heic", "HEIF", quality=85) + + # NOTE: HEIC EXIF *orientation* cannot be set with this encoder. Both `exif=` and + # `exif=` are written, but the orientation tag comes back as 1 regardless, which + # would make a fixture that lies about what it tests. That one case therefore has to be + # checked with a real camera/iPhone HEIC — see docs/EDGE_CASE_TESTING.md. + + # One scene encoded both ways: is a codec conversion still worth reporting? + scene = gradient(800, 600) + scene.save(OUT / "scene.jpg", "JPEG", quality=92) + scene.save(OUT / "scene.heic", "HEIF", quality=90) + + # Near-identical solids in HEIC too — the band-index stress case. + for index in (0, 1): + Image.new("RGB", (400, 300), (20 + index, 40, 60)).save( + OUT / f"heic-solid-{index}.heic", + "HEIF", + quality=90, + ) + + +def main() -> None: + OUT.mkdir(parents=True, exist_ok=True) + base = gradient() + + # A distinct control scene, so the duplicate pair below is unambiguous. (Previously this was + # byte-identical to dup-a/dup-b, which produced a confusing group of three.) + control = gradient(240, 180) + control = control.transpose(Image.FLIP_LEFT_RIGHT) + write("valid.jpg", _jpeg_bytes(control)) + + good = _jpeg_bytes(base) + + # 2. Interrupted download / half-written file: valid header, no end-of-image marker. + write("truncated.jpg", good[: len(good) * 4 // 10]) + + # 3. Right header, garbage body. + write("garbage-body.jpg", good[:120] + bytes(random.getrandbits(8) for _ in range(4096))) + + # 4. Zero bytes. MediaStore normally filters these, but a sideloaded file can slip through. + write("zero-bytes.jpg", b"") + + # 5. A PNG wearing a .jpg extension — the MIME type lies. + png_bytes = _png_bytes(base) + write("renamed-png.jpg", png_bytes) + + # 6. A valid JPEG with no extension at all. + write("no-extension", good) + + # 7. Decompression bomb: the header declares 60000 x 60000 (14 GB decoded) but carries + # almost no pixel data. A naive decoder allocates until it dies. + ihdr = struct.pack(">IIBBBBB", 60000, 60000, 8, 2, 0, 0, 0) + write( + "decompression-bomb.png", + b"\x89PNG\r\n\x1a\n" + + png_chunk(b"IHDR", ihdr) + + png_chunk(b"IDAT", zlib.compress(b"\x00" * 64)) + + png_chunk(b"IEND", b""), + ) + + # 8. The same picture written with every EXIF orientation value. All eight are the same + # photo and must normalise to the same fingerprint. + for orientation in range(1, 9): + exif = Image.Exif() + exif[0x0112] = orientation + path = OUT / f"exif-orientation-{orientation}.jpg" + base.save(path, "JPEG", quality=90, exif=exif) + + # 9. Byte-identical copies — the exact-duplicate path must find these. + write("dup-a.jpg", good) + write("dup-b.jpg", good) + + # 10. Odd but legal encodings that must not break the hasher. + base.convert("L").save(OUT / "grayscale.jpg", "JPEG", quality=90) + base.convert("RGBA").save(OUT / "with-alpha.png", "PNG") + Image.new("RGB", (1, 1), (255, 0, 0)).save(OUT / "tiny-1px.jpg", "JPEG") + + # 11. Near-identical solids: colour-only similarity bait, and a stress case for the banded + # index (all of these may share hash bands). + for index in range(6): + Image.new("RGB", (400, 300), (20 + index, 40, 60)).save( + OUT / f"solid-{index}.png", "PNG" + ) + + total = sorted(OUT.iterdir()) + heic_section(base) + total = sorted(OUT.iterdir()) + print(f"wrote {len(total)} fixtures to {OUT}") + for path in total: + print(f" {path.stat().st_size:>9,} {path.name}") + + +def _jpeg_bytes(img: Image.Image) -> bytes: + import io + + buffer = io.BytesIO() + img.save(buffer, "JPEG", quality=92) + return buffer.getvalue() + + +def _png_bytes(img: Image.Image) -> bytes: + import io + + buffer = io.BytesIO() + img.save(buffer, "PNG") + return buffer.getvalue() + + +if __name__ == "__main__": + main() diff --git a/tools/make-store-assets.sh b/tools/make-store-assets.sh new file mode 100755 index 0000000..11186e6 --- /dev/null +++ b/tools/make-store-assets.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# +# Generates the Google Play store graphics from the app's own icon vectors. +# +# ./tools/make-store-assets.sh [output-dir] # default: ./store-assets +# +# Produces, matching Play's asset specs (support.google.com/.../answer/9866151): +# icon-512.png 512 x 512 32-bit PNG *with* alpha (ours is full-bleed, +# so every pixel ends up opaque anyway) +# feature-graphic-1024x500.png 1024 x 500 24-bit PNG, no alpha +# +# Output lands in ./store-assets/ *inside* the repo, never in $HOME: the editable sources are +# the SVGs in tools/store-assets/ and these PNGs are the files that get uploaded to the Play +# Console, so they belong with the release they were made for. Pass a directory to override. +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +repo="$(dirname "$here")" +out="${1:-$repo/store-assets}" +mkdir -p "$out" + +render() { # + local svg="$1" width="$2" height="$3" name="$4" mode="$5" + inkscape "$svg" \ + --export-type=png --export-width="$width" --export-height="$height" \ + --export-filename="$out/$name" >/dev/null 2>&1 + if [ "$mode" = keep-alpha ]; then + # Play asks for a 32-bit PNG here. The SVG's background rect covers the whole + # canvas, so the alpha channel is fully opaque — no accidental transparency. + magick "$out/$name" -alpha on PNG32:"$out/$name" + else + magick "$out/$name" -background '#1B5E20' -alpha remove -alpha off PNG24:"$out/$name" + fi + printf ' %s ' "$out/$name" + identify -format '%wx%h %[channels] %b\n' "$out/$name" +} + +echo "Writing store assets to $out" +render "$here/store-assets/icon.svg" 512 512 icon-512.png keep-alpha +render "$here/store-assets/feature-graphic.svg" 1024 500 feature-graphic-1024x500.png flatten +echo "Done. Screenshots: ./tools/make-store-screenshots.sh # reads store-assets/raw-screenshots/" diff --git a/tools/make-store-screenshots.sh b/tools/make-store-screenshots.sh new file mode 100755 index 0000000..b5cf663 --- /dev/null +++ b/tools/make-store-screenshots.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# +# Turns raw device screenshots into Play-legal store screenshots. +# +# ./tools/make-store-screenshots.sh [raw-dir] [output-dir] +# +# Both paths default to inside the repo, never to $HOME: +# raw store-assets/raw-screenshots/ git-ignored staging for the `adb` captures +# output store-assets/screenshots/ the cropped files that get uploaded to Play +# +# Why this exists: Play rejects a screenshot whose longer side is more than twice its +# shorter side ("your screenshot's maximum dimension can't be more than twice its minimum +# dimension"). A 1080 x 2400 phone capture is 2.22:1, so it has to become 1080 x 2160. +# +# The crop removes system chrome only — the status bar at the top, the navigation bar at the +# bottom — never app content. Both heights were measured on the test device: +# +# STATUS_BAR_PX ITYPE_STATUS_BAR frame height 93 px (1080 x 2400 @ 440 dpi) +# NAV_BAR_PX ITYPE_NAVIGATION_BAR bottom inset 130 px (same device) +# +# adb shell dumpsys window displays | grep -oE 'ITYPE_STATUS_BAR frame=\[[^]]*\]\[[^]]*\]' +# adb shell dumpsys window displays | grep -oE 'control type=ITYPE_NAVIGATION_BAR.*mInsetsHint=\{[^}]*\}' +# +# Output is 24-bit PNG without alpha, which is what Play asks for. Captures are not shot by +# this script — take them yourself (the shot list is in docs/STORE_LISTING.md) and drop them +# into store-assets/raw-screenshots/, or pass the directory as the first argument. +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +repo="$(dirname "$here")" +raw="${1:-$repo/store-assets/raw-screenshots}" +out="${2:-$repo/store-assets/screenshots}" +status_bar="${STATUS_BAR_PX:-93}" +nav_bar="${NAV_BAR_PX:-130}" + +if [ ! -d "$raw" ]; then + echo "usage: $0 [raw-screenshot-dir] [output-dir]" >&2 + echo "error: no such directory: $raw" >&2 + echo " capture first, e.g. adb exec-out screencap -p > $raw/01-empty.png" >&2 + exit 2 +fi + +mkdir -p "$out" + +shopt -s nullglob +for f in "$raw"/*.png; do + name="$(basename "$f")" + width="$(identify -format '%w' "$f")" + height="$(identify -format '%h' "$f")" + + # Drop the two system bars, then clamp to a 2:1 portrait frame. + keep=$(( height - status_bar - nav_bar )) + cap=$(( width * 2 )) + if [ "$keep" -gt "$cap" ]; then + keep="$cap" + fi + + magick "$f" -crop "${width}x${keep}+0+${status_bar}" +repage \ + -background '#FFFBFE' -alpha remove -alpha off PNG24:"$out/$name" + printf ' %s %s -> ' "$name" "$(identify -format '%wx%h' "$f")" + identify -format '%wx%h %[channels] %b\n' "$out/$name" +done diff --git a/tools/normalise-for-local-grouping.py b/tools/normalise-for-local-grouping.py new file mode 100644 index 0000000..ce8f79e --- /dev/null +++ b/tools/normalise-for-local-grouping.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""Prepare an image folder for a *local* run of the app's grouping logic. + +The duplicate engine (`core/`) is pure Kotlin, so it can be exercised on a desktop instead of a +phone. The only Android-specific parts are MediaStore, the image decoder and the hash index — and +the decoder's one subtlety is EXIF orientation. + +So this script normalises orientation with Pillow (`ImageOps.exif_transpose`, the reference +behaviour) and writes lossless PNGs plus a manifest. `LocalGroupingTest` then fingerprints those +pixels with the app's own `PerceptualHash`, rebuilds the exact-duplicate path from the recorded +SHA-256, and runs the real `DuplicateGrouper`. + + python3 tools/normalise-for-local-grouping.py fixtures/foto-dup-edge build/fotonorm + ./gradlew :app:testDebugUnitTest --tests '*LocalGroupingTest*' -DfotoLocal.dir=/tmp/fotonorm + +What this reproduces faithfully: the algorithm — hashing, the adaptive threshold, banded candidate +generation, the conjunction, union-find clustering, and the resulting groups. +What it does NOT reproduce: MediaStore's idea of what is indexed, the provider's thumbnail +decode, or the SQLite cache. +""" +import hashlib +import json +import os +import sys +from pathlib import Path + +from PIL import Image, ImageOps + +SUPPORTED = {".jpg", ".jpeg", ".png", ".bmp", ".heic", ".heif", ".webp"} + + +def main() -> None: + if len(sys.argv) < 3: + print(__doc__) + raise SystemExit(2) + source = Path(os.path.expanduser(sys.argv[1])) + target = Path(os.path.expanduser(sys.argv[2])) + + try: + import pillow_heif + + pillow_heif.register_heif_opener() + except ImportError: + print("note: pillow-heif missing, HEIC files will be treated as unreadable") + + target.mkdir(parents=True, exist_ok=True) + for stale in target.glob("*.png"): + stale.unlink() + + entries = [] + for index, path in enumerate(sorted(source.iterdir())): + if not path.is_file() or path.suffix.lower() not in SUPPORTED: + continue + raw = path.read_bytes() + record = { + "name": path.name, + "sizeBytes": len(raw), + "modifiedSeconds": int(path.stat().st_mtime), + "sha256": hashlib.sha256(raw).hexdigest(), + } + try: + with Image.open(path) as img: + normalised = ImageOps.exif_transpose(img).convert("RGB") + out = target / f"{index:03d}.png" + normalised.save(out, "PNG") + record["normalised"] = out.name + record["width"], record["height"] = normalised.size + except Exception as error: # noqa: BLE001 - mirrors the app's "could not be read" + record["error"] = f"{type(error).__name__}: {error}" + entries.append(record) + + (target / "manifest.json").write_text(json.dumps(entries, indent=1)) + + # Also a line-oriented manifest so the Kotlin side needs no JSON parser. One record per line: + # name, sizeBytes, modifiedSeconds, sha256, normalisedFile, width, height, error + tsv = target / "manifest.tsv" + with tsv.open("w") as handle: + for entry in entries: + handle.write( + "\t".join( + [ + entry["name"], + str(entry["sizeBytes"]), + str(entry["modifiedSeconds"]), + entry["sha256"], + entry.get("normalised", ""), + str(entry.get("width", 0)), + str(entry.get("height", 0)), + entry.get("error", "").replace("\t", " ").replace("\n", " "), + ] + ) + + "\n" + ) + + readable = sum(1 for e in entries if "error" not in e) + print(f"prepared {readable} readable of {len(entries)} files in {target}") + for entry in entries: + if "error" in entry: + print(f" unreadable: {entry['name']} ({entry['error'][:60]})") + + +if __name__ == "__main__": + main() diff --git a/tools/store-assets/feature-graphic.svg b/tools/store-assets/feature-graphic.svg new file mode 100644 index 0000000..3547e74 --- /dev/null +++ b/tools/store-assets/feature-graphic.svg @@ -0,0 +1,40 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + Foto Dup Finder + Find and remove duplicate photos + Fast, ad-free and completely offline + diff --git a/tools/store-assets/icon.svg b/tools/store-assets/icon.svg new file mode 100644 index 0000000..245bb5f --- /dev/null +++ b/tools/store-assets/icon.svg @@ -0,0 +1,24 @@ + + + + + + + + + + + + + + + +