readest

akai/readest

Fork 1

Commit Graph

Author	SHA1	Message	Date
loveheaven	ff605e000d	feat(library): in-place import from registered external folders (#4315 ) * feat(library): in-place import with cloud sync and symmetric local delete Adds an `inPlace` option to importBook so a source file inside a registered external library folder is referenced directly via `book.filePath` instead of being copied into Books/<hash>/. Sidecars (cover, config, nav) still live under Books/<hash>/. ingestService routes through shouldImportInPlace, which marks an import in-place when the absolute source path lives under any of `settings.externalLibraryFolders` and is NOT inside a per-root `Books/` subtree. The Readest data dir (`customRootDir`) is intentionally excluded — that directory is Readest's home and should freely hold hash copies; in-place is for user-registered roots (Duokan, Calibre, Moon+ Reader, an iCloud mirror, …). Cloud sync treats in-place books as first-class: - uploadBook reads bytes from (book.filePath, 'None') when set. The cloud key is unchanged, so a peer downloading the book lands it under Books/<hash>/ as a normal hash copy. - useBooksSync strips `book.filePath` before pushing — it is a device-local path that is meaningless on any other device. - ingestService no longer skips upload for in-place books; autoUpload / forceUpload behave like any other book. Only transient imports opt out. - deleteBook 'local'/'both' now physically removes the source file at book.filePath (base 'None'). Local-delete semantics are symmetric with hash-copy books: the local copy is gone, the cloud backup remains, a future pull restores under Books/<hash>/. removeFile errors are swallowed. New `SystemSettings.externalLibraryFolders?: string[]` (no UI yet; registration entrypoint lands in a follow-up). Added to BACKUP_SETTINGS_BLACKLIST alongside `localBooksDir` / `customRootDir` so device-local paths don't ride cloud backups. Tests: cloud-service, ingest-service, and backup-settings suites cover in-place delete, multi-root matching, per-root `Books/` guard, and the backup-strip. * feat(library): one-tap "read in place" toggle in folder import Surface the in-place / copy choice as a single "Read books in place (don't copy)" checkbox in the Import-from-Folder dialog. When the user opts in, the chosen directory is registered in `settings.externalLibraryFolders` and ingestService's `shouldImportInPlace` will route the books straight to importBook with `inPlace: true` — no copy into Books/<hash>/, sync still works, local delete still removes the source file (the symmetry was set up in the previous in-place commit). User experience: - First-time users hit the toggle once per library folder. The choice is also persisted to localStorage so subsequent dialog opens default to whatever they picked last. - Repeat imports from a folder that's already registered as an external library folder force the toggle ON and disable it, with a help line explaining that imports from this folder are always in-place. The check is exact-string (after path normalization) so registering /Users/me/Duokan only locks the toggle for that exact path — picking /Users/me/Downloads after Duokan still shows the toggle in its normal state. - URL-ingress / drag-drop replays go through `runFolderImport` without the dialog and default `readInPlace: false`. They still benefit from in-place automatically when the dropped path lives under an already-registered root, because that decision is made by `shouldImportInPlace` based on settings, not by the dialog flag. Mechanics: - ImportFromFolderResult gains `readInPlace: boolean`. ImportFromFolderDialog gains an `initialReadInPlace` prop (seeded from the new `readest:lastImportFolderReadInPlace` localStorage key) and an `isRegisteredExternalRoot` predicate it uses to render the locked / unlocked toggle. - runFolderImport calls a new `registerExternalLibraryFolder` helper that appends the chosen directory to `settings.externalLibraryFolders` and persists settings, but only when `result.readInPlace` is true. `isRegisteredExternalRoot` does the inverse lookup the dialog needs. Both helpers normalize paths the same way `shouldImportInPlace` does so the predicate matches the ingest layer. - The new feature has no effect for users who never flip the toggle: `externalLibraryFolders` stays empty, the path-prefix check in `shouldImportInPlace` returns false for every import, and books continue to be copied into Books/<hash>/ exactly as before. Self-healing for externally-removed in-place books: Once the dialog lets users opt their library into in-place mode, the source file becomes a piece of state Readest doesn't control — another app may rewrite it (e.g. Duokan persisting reading progress into the epub), the user may move it in Finder, or an external drive may unmount between sessions. Previously, clicking such a book would navigate into the reader, fail inside loadBookContent's `fs.openFile(book.filePath, 'None')` with a low-level IO error, flash an "Unable to open book" toast, and auto-bounce back to the library — leaving the stale library record in place so the next tap reproduces the same dance. BookshelfItem.handleBookClick now probes availability before navigating, but only for purely-local in-place books (`book.filePath && !book.uploadedAt && !book.deletedAt`). If `appService.isBookAvailable` returns false — which for in-place books means the recorded `book.filePath` no longer exists at the OS level — we dispatch `delete-books` for that hash and show an info toast explaining the removal, instead of opening the reader. Scope is intentionally narrow: - Cloud-synced books still flow through `makeBookAvailable`'s on-demand download path; missing local copies trigger a re-download, not a deletion. - Hash-copy books (no `filePath` set) are not probed: a missing Books/<hash>/ file under normal use signals a bug or filesystem corruption, not user intent, and silently dropping the record would hide the real problem. - The dispatched delete-books event reuses the existing Bookshelf deletion path, so sidecar metadata and selection state are cleaned up the same way as a user-initiated delete. For in-place books that path doesn't touch any file outside Books/<hash>/, so the now-missing source location (or whatever the user did with it externally) is left alone — symmetric with 165f15a6. * fix(library): centralize book content resolution --------- Co-authored-by: Huang Xin <chrox.huang@gmail.com>	2026-05-27 12:08:36 +02:00
loveheaven	5ac8564e41	feat(library): add Import from Folder dialog with format/size filters (#4229 ) * feat(library): add Import from Folder dialog with format/size filters Replaces the silent "import every supported file recursively" behaviour of the directory import menu item with an explicit dialog that lets users pick which formats to include, set a minimum file size, and choose between mirroring subfolders as nested groups (legacy behaviour) or flattening every match into the current library view. The folder, the chosen Folder Structure radio, the ticked File Formats and the File Size threshold are all persisted in localStorage so re-opening the dialog seeds every field with the user's last choice. Cancelling the dialog does not write to storage so an aborted pick won't pollute the next session. Also hides the native number-input spinner via a small .no-spinner utility in globals.css; on macOS WebKit the spin buttons were drawing over the rounded input border and looked broken. The KB suffix now lives inside the input's bordered shell instead of beside it. Two correctness fixes the dialog flow exposed: * The library importer + ingestService now treat groupId as a tri-state — undefined means "don't touch the existing group", '' means "explicitly the library root", any other string means a specific group. Previously a falsy check in both layers conflated '' with undefined, so re-importing a deduped book under flatten mode silently kept its stale groupId/groupName from the prior keep-as-groups run, making the book reappear in the old subfolder group instead of moving into the library root. New regression tests in ingest-service.test.ts cover both the empty-string case and the omitted case. * Imports of arbitrary user paths (e.g. ~/Downloads) now go through a new allow_paths_in_scopes Tauri command that extends both fs_scope and asset_protocol_scope. The dialog plugin only auto-grants fs_scope, so reads through the asset protocol (RemoteFile / convertFileSrc) used to fail with "asset protocol not configured to allow the path". The shim is invoked after every selectFiles / selectDirectory call and once more at the start of runFolderImport so localStorage-restored paths are also covered. Granted scopes persist across restarts via tauri_plugin_persisted_scope. * fixup(library): harden Import-from-Folder scope grant + RTL/dialog polish Three review fixes on top of the Import-from-Folder feature: * lib.rs: refuse to extend asset_protocol_scope for paths not already in fs_scope. Without this gate, any frontend code (XSS via book content, OPDS HTML, dictionary lookups, or a compromised dependency) could call allow_paths_in_scopes with '/' or '~/.ssh' and gain persistent read access to arbitrary user files via the asset protocol — the grant survives restarts thanks to tauri_plugin_persisted_scope. Mirrors the defensive check in dir_scanner.rs. * ImportFromFolderDialog.tsx: migrate from a custom ModalPortal chassis to the project's shared <Dialog> primitive so eink mode auto-removes shadows, mobile gets the bottom-sheet treatment, RTL direction is applied, and focus management is correct. * ImportFromFolderDialog.tsx: swap directional Tailwind utilities for the logical equivalents (text-start, ps-/pe-, rounded-s-, text-end) per DESIGN.md §2.8 — Arabic/Hebrew users were getting a mirrored number-input row with the KB suffix on the wrong side. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * i18n(library): translate Import-from-Folder dialog strings across 33 locales Translates the 13 new strings introduced with the Import-from-Folder dialog (folder picker label, format-filter section, size-threshold input, folder-structure radios, OK button, empty-result toast). All 33 supported locales — including RTL fa/he/ar — are now complete; no __STRING_NOT_TRANSLATED__ placeholders remain in the catalog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Huang Xin <chrox.huang@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 14:51:53 +02:00
Huang Xin	0b18de0581	feat(send): Send to Readest — multi-channel capture into your library (#4230 ) * feat(send): Send to Readest — multi-channel capture into your library A Send-to-Kindle equivalent: email, web-upload, share, or one-click capture books and articles into the cloud library; they sync to every device. Architecture (client-side processing): out-of-app channels drop a raw payload into a per-user send_inbox; Readest clients drain it through one shared ingestService.ingestFile(). The server never parses or converts. - ingestService.ingestFile() — channel-agnostic import orchestration extracted from library/page.tsx (DI-based, forceUpload support). - send_addresses / send_allowed_senders / send_inbox tables + RLS + 4 SECURITY DEFINER claim/lease RPCs (migration 012_send_to_readest.sql). - Conversion subsystem (DOCX/RTF/HTML/article/TXT -> EPUB) in a Web Worker. - send-email Cloudflare Email Worker; inbox-drainer controller + useInboxDrainer hook; /api/send/* routes. - Send to Readest settings panel: inbound address, approved-sender allowlist, recent activity, per-device drain toggle. - /send web page (file drop + article URL) + SSRF-guarded fetch-url proxy. - OS-shared files routed through ingestFile; Manifest V3 browser extension. Security: inbox state changes only via SECURITY DEFINER RPCs (clients get SELECT-only on send_inbox); approved-sender allowlist gates email; SSRF guard on the one server-side URL fetch; inbox payload signed URLs authorize against send_inbox.user_id. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: run format:check in the pre-push hook Biome format checking is fast (~0.4s), so gate pushes on it too — catches mis-formatted files that bypassed the staged-only pre-commit hook before they reach CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(send): address CodeQL security findings - ReDoS (senders.ts): the email regex had ambiguous quantifiers around the literal dot. Rewrote it linear-time (domain labels exclude '.') and cap the input at 254 chars. - XSS (convertToEpub.ts): run untrusted HTML through DOMPurify (sanitizeForParsing — keeps document structure) before DOMParser, so title extraction and Readability never parse executable markup. - SSRF (fetch-url.ts): harden the host guard — block bare single-label hostnames, IPv4-mapped IPv6, CGNAT/benchmark/multicast ranges, and the unspecified address. DNS rebinding stays a documented residual risk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 20:06:52 +02:00

Author

SHA1

Message

Date

loveheaven

ff605e000d

feat(library): in-place import from registered external folders (#4315 )

* feat(library): in-place import with cloud sync and symmetric local delete

Adds an `inPlace` option to importBook so a source file inside a
registered external library folder is referenced directly via
`book.filePath` instead of being copied into Books/<hash>/. Sidecars
(cover, config, nav) still live under Books/<hash>/.

ingestService routes through shouldImportInPlace, which marks an
import in-place when the absolute source path lives under any of
`settings.externalLibraryFolders` and is NOT inside a per-root
`Books/` subtree. The Readest data dir (`customRootDir`) is
intentionally excluded — that directory is Readest's home and
should freely hold hash copies; in-place is for user-registered
roots (Duokan, Calibre, Moon+ Reader, an iCloud mirror, …).

Cloud sync treats in-place books as first-class:

- uploadBook reads bytes from (book.filePath, 'None') when set.
The cloud key is unchanged, so a peer downloading the book
lands it under Books/<hash>/ as a normal hash copy.

- useBooksSync strips `book.filePath` before pushing — it is a
device-local path that is meaningless on any other device.

- ingestService no longer skips upload for in-place books;
autoUpload / forceUpload behave like any other book. Only
transient imports opt out.

- deleteBook 'local'/'both' now physically removes the source
file at book.filePath (base 'None'). Local-delete semantics
are symmetric with hash-copy books: the local copy is gone,
the cloud backup remains, a future pull restores under
Books/<hash>/. removeFile errors are swallowed.

New `SystemSettings.externalLibraryFolders?: string[]` (no UI yet;
registration entrypoint lands in a follow-up). Added to
BACKUP_SETTINGS_BLACKLIST alongside `localBooksDir` /
`customRootDir` so device-local paths don't ride cloud backups.

Tests: cloud-service, ingest-service, and backup-settings suites
cover in-place delete, multi-root matching, per-root `Books/`
guard, and the backup-strip.

* feat(library): one-tap "read in place" toggle in folder import

Surface the in-place / copy choice as a single "Read books in place (don't copy)" checkbox in the Import-from-Folder dialog. When the user opts in, the chosen directory is registered in `settings.externalLibraryFolders` and ingestService's `shouldImportInPlace` will route the books straight to importBook with `inPlace: true` — no copy into Books/<hash>/, sync still works, local delete still removes the source file (the symmetry was set up in the previous in-place commit).

User experience:

- First-time users hit the toggle once per library folder. The choice is also persisted to localStorage so subsequent dialog opens default to whatever they picked last.

- Repeat imports from a folder that's already registered as an external library folder force the toggle ON and disable it, with a help line explaining that imports from this folder are always in-place. The check is exact-string (after path normalization) so registering /Users/me/Duokan only locks the toggle for that exact path — picking /Users/me/Downloads after Duokan still shows the toggle in its normal state.

- URL-ingress / drag-drop replays go through `runFolderImport` without the dialog and default `readInPlace: false`. They still benefit from in-place automatically when the dropped path lives under an already-registered root, because that decision is made by `shouldImportInPlace` based on settings, not by the dialog flag.

Mechanics:

- ImportFromFolderResult gains `readInPlace: boolean`. ImportFromFolderDialog gains an `initialReadInPlace` prop (seeded from the new `readest:lastImportFolderReadInPlace` localStorage key) and an `isRegisteredExternalRoot` predicate it uses to render the locked / unlocked toggle.

- runFolderImport calls a new `registerExternalLibraryFolder` helper that appends the chosen directory to `settings.externalLibraryFolders` and persists settings, but only when `result.readInPlace` is true. `isRegisteredExternalRoot` does the inverse lookup the dialog needs. Both helpers normalize paths the same way `shouldImportInPlace` does so the predicate matches the ingest layer.

- The new feature has no effect for users who never flip the toggle: `externalLibraryFolders` stays empty, the path-prefix check in `shouldImportInPlace` returns false for every import, and books continue to be copied into Books/<hash>/ exactly as before.

Self-healing for externally-removed in-place books:

Once the dialog lets users opt their library into in-place mode, the source file becomes a piece of state Readest doesn't control — another app may rewrite it (e.g. Duokan persisting reading progress into the epub), the user may move it in Finder, or an external drive may unmount between sessions. Previously, clicking such a book would navigate into the reader, fail inside loadBookContent's `fs.openFile(book.filePath, 'None')` with a low-level IO error, flash an "Unable to open book" toast, and auto-bounce back to the library — leaving the stale library record in place so the next tap reproduces the same dance.

BookshelfItem.handleBookClick now probes availability before navigating, but only for purely-local in-place books (`book.filePath && !book.uploadedAt && !book.deletedAt`). If `appService.isBookAvailable` returns false — which for in-place books means the recorded `book.filePath` no longer exists at the OS level — we dispatch `delete-books` for that hash and show an info toast explaining the removal, instead of opening the reader.

Scope is intentionally narrow:
- Cloud-synced books still flow through `makeBookAvailable`'s on-demand download path; missing local copies trigger a re-download, not a deletion.
- Hash-copy books (no `filePath` set) are not probed: a missing Books/<hash>/ file under normal use signals a bug or filesystem corruption, not user intent, and silently dropping the record would hide the real problem.
- The dispatched delete-books event reuses the existing Bookshelf deletion path, so sidecar metadata and selection state are cleaned up the same way as a user-initiated delete. For in-place books that path doesn't touch any file outside Books/<hash>/, so the now-missing source location (or whatever the user did with it externally) is left alone — symmetric with 165f15a6.

* fix(library): centralize book content resolution

---------

Co-authored-by: Huang Xin <chrox.huang@gmail.com>

2026-05-27 12:08:36 +02:00

loveheaven

5ac8564e41

feat(library): add Import from Folder dialog with format/size filters (#4229 )

* feat(library): add Import from Folder dialog with format/size filters

Replaces the silent "import every supported file recursively" behaviour of the directory import menu item with an explicit dialog that lets users pick which formats to include, set a minimum file size, and choose between mirroring subfolders as nested groups (legacy behaviour) or flattening every match into the current library view.

The folder, the chosen Folder Structure radio, the ticked File Formats and the File Size threshold are all persisted in localStorage so re-opening the dialog seeds every field with the user's last choice. Cancelling the dialog does not write to storage so an aborted pick won't pollute the next session.

Also hides the native number-input spinner via a small .no-spinner utility in globals.css; on macOS WebKit the spin buttons were drawing over the rounded input border and looked broken. The KB suffix now lives inside the input's bordered shell instead of beside it.

Two correctness fixes the dialog flow exposed:

* The library importer + ingestService now treat groupId as a tri-state — undefined means "don't touch the existing group", '' means "explicitly the library root", any other string means a specific group. Previously a falsy check in both layers conflated '' with undefined, so re-importing a deduped book under flatten mode silently kept its stale groupId/groupName from the prior keep-as-groups run, making the book reappear in the old subfolder group instead of moving into the library root. New regression tests in ingest-service.test.ts cover both the empty-string case and the omitted case.

* Imports of arbitrary user paths (e.g. ~/Downloads) now go through a new allow_paths_in_scopes Tauri command that extends both fs_scope and asset_protocol_scope. The dialog plugin only auto-grants fs_scope, so reads through the asset protocol (RemoteFile / convertFileSrc) used to fail with "asset protocol not configured to allow the path". The shim is invoked after every selectFiles / selectDirectory call and once more at the start of runFolderImport so localStorage-restored paths are also covered. Granted scopes persist across restarts via tauri_plugin_persisted_scope.

* fixup(library): harden Import-from-Folder scope grant + RTL/dialog polish

Three review fixes on top of the Import-from-Folder feature:

* lib.rs: refuse to extend asset_protocol_scope for paths not already
  in fs_scope. Without this gate, any frontend code (XSS via book
  content, OPDS HTML, dictionary lookups, or a compromised dependency)
  could call allow_paths_in_scopes with '/' or '~/.ssh' and gain
  persistent read access to arbitrary user files via the asset
  protocol — the grant survives restarts thanks to
  tauri_plugin_persisted_scope. Mirrors the defensive check in
  dir_scanner.rs.

* ImportFromFolderDialog.tsx: migrate from a custom ModalPortal chassis
  to the project's shared <Dialog> primitive so eink mode auto-removes
  shadows, mobile gets the bottom-sheet treatment, RTL direction is
  applied, and focus management is correct.

* ImportFromFolderDialog.tsx: swap directional Tailwind utilities for
  the logical equivalents (text-start, ps-/pe-, rounded-s-, text-end)
  per DESIGN.md §2.8 — Arabic/Hebrew users were getting a mirrored
  number-input row with the KB suffix on the wrong side.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* i18n(library): translate Import-from-Folder dialog strings across 33 locales

Translates the 13 new strings introduced with the Import-from-Folder
dialog (folder picker label, format-filter section, size-threshold
input, folder-structure radios, OK button, empty-result toast). All
33 supported locales — including RTL fa/he/ar — are now complete; no
__STRING_NOT_TRANSLATED__ placeholders remain in the catalog.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Huang Xin <chrox.huang@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-20 14:51:53 +02:00

Huang Xin

0b18de0581

feat(send): Send to Readest — multi-channel capture into your library (#4230 )

* feat(send): Send to Readest — multi-channel capture into your library

A Send-to-Kindle equivalent: email, web-upload, share, or one-click
capture books and articles into the cloud library; they sync to every
device.

Architecture (client-side processing): out-of-app channels drop a raw
payload into a per-user send_inbox; Readest clients drain it through one
shared ingestService.ingestFile(). The server never parses or converts.

- ingestService.ingestFile() — channel-agnostic import orchestration
  extracted from library/page.tsx (DI-based, forceUpload support).
- send_addresses / send_allowed_senders / send_inbox tables + RLS + 4
  SECURITY DEFINER claim/lease RPCs (migration 012_send_to_readest.sql).
- Conversion subsystem (DOCX/RTF/HTML/article/TXT -> EPUB) in a Web Worker.
- send-email Cloudflare Email Worker; inbox-drainer controller +
  useInboxDrainer hook; /api/send/* routes.
- Send to Readest settings panel: inbound address, approved-sender
  allowlist, recent activity, per-device drain toggle.
- /send web page (file drop + article URL) + SSRF-guarded fetch-url proxy.
- OS-shared files routed through ingestFile; Manifest V3 browser extension.

Security: inbox state changes only via SECURITY DEFINER RPCs (clients get
SELECT-only on send_inbox); approved-sender allowlist gates email;
SSRF guard on the one server-side URL fetch; inbox payload signed URLs
authorize against send_inbox.user_id.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: run format:check in the pre-push hook

Biome format checking is fast (~0.4s), so gate pushes on it too — catches
mis-formatted files that bypassed the staged-only pre-commit hook before
they reach CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(send): address CodeQL security findings

- ReDoS (senders.ts): the email regex had ambiguous quantifiers around
  the literal dot. Rewrote it linear-time (domain labels exclude '.')
  and cap the input at 254 chars.
- XSS (convertToEpub.ts): run untrusted HTML through DOMPurify
  (sanitizeForParsing — keeps document structure) before DOMParser, so
  title extraction and Readability never parse executable markup.
- SSRF (fetch-url.ts): harden the host guard — block bare single-label
  hostnames, IPv4-mapped IPv6, CGNAT/benchmark/multicast ranges, and the
  unspecified address. DNS rebinding stays a documented residual risk.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-19 20:06:52 +02:00

3 Commits