* docs: design for syncing updated book data (cover + file) (#4544)
Cover-change sync via a content hash (coverHash = partial MD5 of cover.png)
plus a cover_updated_at field-level merge timestamp; file updates ride the
existing re-import / metaHash dedupe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sync): sync updated book covers across devices (#4544)
Editing a book's cover wrote cover.png locally but changed no hash (the
cover is keyed by the file hash), so peers had no signal to re-download it
and the change never propagated.
Give the cover its own content-addressed version:
- coverHash = partial MD5 of cover.png; a peer re-downloads the cover iff
the synced hash differs from the local one (idempotent, no churn on
identical/re-extracted covers — compatible with the metaHash dedupe).
- coverUpdatedAt = field-level LWW timestamp so a page-turn that wins
whole-row LWW on updated_at can't clobber a cover edit (mirrors the
reading_status_updated_at fix for #4634).
Editing a cover recomputes the hash, bumps coverUpdatedAt, and re-uploads
only the cover; the server merges cover fields independently; peers
re-download on a hash diff. File updates continue to ride the existing
re-import / metaHash dedupe (changed file -> changed hash -> re-key).
Migration 016 adds cover_hash / cover_updated_at to books.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): decouple the incremental-pull cursor from updated_at (#4678)
`books.updated_at` was overloaded as both the incremental-pull cursor
(`GET /api/sync?since=…` filters `updated_at > since`, devices keep one
global `max(updated_at)` watermark) and the library "date read" sort key.
A server-resolved reading-status merge had to be written with a timestamp
greater than every peer's global cursor to propagate, which forced
`updated_at = now()` and reordered the date-read library by sync-processing
time (the #4677 symptom).
Introduce a server-assigned `synced_at` column on `books`, stamped by a
`BEFORE INSERT OR UPDATE` trigger on every write, used only as the pull
cursor. `updated_at` stays pure client event time used only for sorting.
- Migration 016 + baseline schema: add `synced_at` (NOT NULL DEFAULT now()),
index `(user_id, synced_at)`, trigger `set_books_synced_at`. Backfill
`synced_at = updated_at` before creating the trigger so existing devices'
cursors hand over without a re-sync storm.
- GET: books filters/orders on `synced_at > since` (a delete bumps synced_at,
so the deleted_at clause is dropped); configs/notes stay on updated_at.
- POST: extract `buildStatusPropagationRow` and drop the `updated_at = now()`
bump — the trigger advances synced_at so peers re-pull the status change
while updated_at (the sort key) stays put.
- Client `computeMaxTimestamp` keys on synced_at, falling back to
updated_at/deleted_at for pre-migration servers and configs/notes.
Backward-compatible: `synced_at >= updated_at` always, so `synced_at > since`
is a strict superset of `updated_at > since` — old web clients and the
koplugin keep working with no data loss (at worst a redundant idempotent
re-pull of rare server-merged rows). The koplugin's shared pull/push cursor
is left untouched; a proper split is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sync): make the books synced_at backfill safe for large live tables (#4678)
The single `UPDATE … WHERE synced_at IS NULL` deadlocked on a 3.8M-row
production `books` table: it rewrites every row in one transaction while the
live /api/sync push path upserts books rows, and the two lock rows in opposite
orders. `ALTER COLUMN … SET NOT NULL` (full-table ACCESS EXCLUSIVE scan) and a
plain CREATE INDEX (write-blocking SHARE lock) compounded it.
Rework migration 016 as an online migration (run via psql, not in a wrapping
transaction):
- backfill in small autocommitted batches via a procedure, using
FOR UPDATE SKIP LOCKED so it never waits on an app-locked row;
- CREATE INDEX CONCURRENTLY instead of a blocking build;
- install the trigger last (so it can't clobber the updated_at backfill);
- drop the hard SET NOT NULL (the default + trigger + backfill keep the column
populated and the client falls back to updated_at); a NOT VALID CHECK +
VALIDATE alternative is included, commented, for operators who want it.
The baseline schema.sql (fresh, empty installs) keeps the simple inline
NOT NULL DEFAULT now() + trigger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync POST handler rewrote books.updated_at = now() whenever a pushed
book's resolved reading_status differed from the server row's. A book that
was imported locally and never given a status sends reading_status:
undefined, while the server stores null, so `undefined !== null` reported a
spurious status change. The 1-day re-sync window re-pushes every recently
touched book on each sync, so the server stamped those books with a fresh,
batch-identical timestamp every cycle, floating them to the top of the
date-sorted library (and above a book the user had just read).
Normalize nullish reading_status values before comparing so a statusless
book never registers as a status change. Verified on-device via CDP:
PUSH_SENT carried the old timestamp while PUSH_RETURNED came back with a
fresh now() for exactly the statusless books.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): design spec for syncing reading status (#4634)
Field-level LWW for reading_status (dedicated reading_status_updated_at),
a new 'abandoned' status in the Readest UI, and a koplugin bridge to
KOReader's native summary.status (whole-library apply + capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): implementation plan for syncing reading status (#4634)
Bite-sized TDD tasks across 3 parts: A) cloud field-level LWW
(reading_status_updated_at on server upsert + client pull-merge),
B) 'abandoned'/On-hold status in the Readest UI, C) koplugin bridge
to KOReader summary.status (mapping + reconcile + whole-library
apply/capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): add reading_status_updated_at for field-level status LWW (#4634)
* feat(sync): stamp readingStatusUpdatedAt on status change in updateBookProgress
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): stamp status timestamp on explicit library status edits
* feat(sync): resolve reading status by its own timestamp in client pull-merge
* feat(sync): resolve reading status by its own timestamp in server upsert (#4634)
* fix(sync): tighten reading-status merge typing + strengthen test (A5 review)
Replace as-unknown-as double-casts at read sites with typed locals
(clientBook/serverBook); retain a single as-unknown-as only at the
server-wins construction site where the static type is too narrow.
Strengthen test 3 to assert both fields with toEqual.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(library): render the 'On hold' (abandoned) status badge
* feat(library): add 'Mark as On hold' actions + i18n for abandoned status
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* i18n: translate 'On hold' and 'Mark as On hold' for the abandoned status (#4634)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): add reading-status mapping + reconcile between Readest and KOReader
* feat(koplugin): persist + sync reading_status_updated_at in LibraryStore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): bridge reading status to KOReader summary.status on library sync (#4634)
* test(sync): cover koplugin v1->v2 migration + tighten status-sync tests (final review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): redesign KOReader first-sync (decisive-only + bootstrap) (#4634)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(koplugin): safe first-sync of reading status (decisive-only + bootstrap) (#4634)
KOReader auto-sets summary.status='reading' on first open, and legacy Readest
statuses have reading_status_updated_at=0, so pure timestamp LWW let opening a
finished book downgrade it. Restrict sync to deliberate statuses (finished/
complete, abandoned/on-hold, unread->clear); never capture KO 'reading'/'New'.
On the unsynced baseline (Readest ts=0) conflicts resolve Readest-authoritative,
then stamp now_ms to exit bootstrap into steady-state LWW. reconcile now returns
write_ko/write_store flags; statussync captures now_ms once and equalizes both
sides (convergent, idempotent, resumable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(koplugin): cover remaining first-sync graph cells + document sort effect (review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Server-side hardening for three reported web advisories:
- OPDS proxy (/api/opds/proxy): add http(s) scheme allowlist, internal/loopback/
link-local host blocklist, and manual per-hop redirect re-validation so a
public URL can't redirect into an internal address. Move isBlockedHost into
the shared src/utils/network.ts as the canonical blocklist and reimplement
isLanAddress to delegate to it (also tightens the /api/kosync LAN check);
fetch-url.ts re-exports it. (GHSA-c7mm-g2j2-98cx, GHSA-5g3f-mq2c-j65v)
- Storage upload (/api/storage/upload): validate the client-supplied fileName
with a new isSafeObjectKeyName helper before building the object key, so a
name can't escape the caller's own prefix. (GHSA-mfmj-2frf-vhgw)
- Stripe (/api/stripe/check): bind the entitlement to the session owner —
reject a Checkout Session whose metadata.userId differs from the authenticated
caller. (GHSA-pv88-3727-j7v8)
Unit tests added for each path; full suite + lint green. The Tauri-native
advisory (GHSA-55vr-pvq5-6fmg) is handled in a separate change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes #3156. Adds a reading-statistics system whose canonical data model
is KOReader's own (book + page_stat_data), so stats round-trip losslessly
between Readest and KOReader.
- Storage: a cross-platform Turso statistics.db in KOReader's exact schema
(web/Workers, desktop, iOS, Android) — replacing #3156's Node-only
better-sqlite3 + statistics.json.
- Tracking: per-page reading events (time-on-page, idle-capped) flushed on
page-change/idle/hide/close — the KOReader model — not session aggregates.
- Sync: legacy /api/sync extended with a stats type backed by self-contained
Supabase tables (stat_books, stat_pages); union/longer-duration-wins merge
keyed on book_hash. apps/readest.koplugin syncs through the same endpoint.
- Scale & robustness: per-tab singleton connection (avoids OPFS lock
conflicts) + explicit WAL checkpoint; transactional bulk apply; chunked
resumable push; client-driven paged pull with trailing-ms completion;
paginated/scoped server merge.
Verified: 5668 unit tests, 155 koplugin busted tests, biome+tsgo + luacheck
all green; web OPFS DB verified live.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#4222.
Three changes that together stabilize Readest sync across devices:
**Stop the artificial updatedAt bump in useProgressAutoSave**
saveConfig unconditionally bumps config.updatedAt = Date.now(), and
useProgressAutoSave used to fire saveConfig on the very first relocate
after book open — even though that relocate just reflects the position
loaded from disk, not user action. The stale local config then looked
"newer" than a fresher server-side push, so the next auto-push
overwrote the other device's real progress via last-writer-wins. The
hook now snapshots the loaded location and skips saveConfig when the
in-memory location still matches it.
**Retry the first config pull with backoff + release the gate**
useProgressSync gates pushes behind a successful pull (so a brand-new
import can't clobber the server's real progress). But handleAutoSync
only re-arms on progress.location changes, so a single failed pull
(Android cold-start contention, Wi-Fi/LTE handoff, captive portal)
used to block every push for the whole reader session. The new
pullWithRetry retries on backoff (1500/4000/10000 ms) and releases
the gate after exhaustion — server-side last-writer-wins still
protects the cross-device case (a stale local push with an older
updated_at loses to a fresher server record). sync-book-progress
events reset the chain so manual pull-to-refresh recovers cleanly.
Fetch timeout bumped from 8s to 15s to better tolerate slow networks
in that cold-start window.
**Server piggybacks books.progress off configs push**
/api/sync POST now updates books.progress + books.updated_at for each
upserted config, gated by .lt('updated_at') so a concurrent newer
books push is never downgraded and a missing row is a silent no-op
(useBooksSync still seeds new rows from the library page). The
in-reader syncBooks round-trip is dropped — the reader now sends one
POST per auto-save instead of two, and the books row stays consistent
with config pushes even while a reader stays open (#4198).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Tauri build uses `output: 'export'`, so the dynamic `/runtime-config.js`
route handler is never emitted. Requesting it returns the SPA fallback HTML
and crashes with `Unexpected token '<'`. Gate the script tag on
`NEXT_PUBLIC_APP_PLATFORM === 'web'`; Tauri consumers already fall back to
the `NEXT_PUBLIC_*` envs baked in at build time.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Email-in (`<user>@readest.com`) is now a paid feature. The other
Send channels — in-app /send page, mobile share-sheet, browser
extension — stay open to free users.
Three enforcement layers:
- `pages/api/send/address.ts` and `pages/api/send/senders.ts` return
403 with `{ code: 'plan_required', plan, requiredPlans }` for free
users. No `send_addresses` row is allocated on the blocked path.
`pages/api/send/inbox.ts` and `pages/api/send/inbox/file.ts` are
deliberately left open — they're shared with the file-upload and
extension channels.
- `workers/send-email` looks up `plans.plan` after resolving the
recipient and bounces (not silently drops) inbound mail for free
users with a one-sentence message pointing to upgrade plus the free
clip channels. Bounce rather than drop so a downgraded user
understands why their mail stops landing.
- `components/settings/integrations/SendToReadestForm.tsx` reads the
user's plan from the JWT before any API call. Free users see one
friendly card — headline, value prop, "View plans" CTA → /user, and
a softer line about the free alternatives — instead of address /
senders / activity sections of disabled controls. The
IntegrationsPanel NavigationRow stays visible so users can discover
the feature.
Single source of truth for the entitled tier set: `EMAIL_IN_PLANS` +
`isEmailInPlan(plan)` in `src/utils/access.ts`. Mirror copies live in
the Worker (no shared import surface) — keep them in sync.
Edge cases:
- Downgraded user: existing `send_addresses` row stays. All three
layers block; re-upgrading silently restores the same address.
- Loading flicker: `userPlan` starts as `null` so the loading skeleton
stays up rather than briefly flashing the upgrade card for a paid
user on a slow client.
12 new unit tests cover the gate on `/api/send/address` and
`/api/send/senders` (GET + POST blocked for free users, no Supabase
access on the blocked path, allowed for plus / pro / purchase).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the URL-only placeholder extension with a full MV3 page-clipper
that builds a self-contained EPUB on the user's machine and uploads it
to the inbox.
- Captures the rendered DOM in a content script, then runs Readability,
asset bundling, and EPUB build through the shared
`convertToEpub({kind: 'page'})` pipeline inside a Chrome offscreen
document (the SW lacks DOMParser).
- Uploads the resulting EPUB directly from the offscreen page to the
new `POST /api/send/inbox/file` endpoint — keeps the bytes in one
realm because `runtime.sendMessage` JSON-serialises ArrayBuffer to
`{}` between extension contexts.
- Adds a long-lived Port + ping handshake between SW, offscreen, and
the on-demand capture content script so neither idle-eviction nor
load-order races can hang the popup.
- Localised popup, badge feedback, key-as-content i18n (`_('English source')`)
with an extract script that seeds locale stubs from i18n-langs.json
and writes a static-imports map for the runtime. All 33 locales
fully translated.
- Server: `pages/api/send/inbox/file.ts` accepts a raw EPUB body
(Content-Type: application/epub+zip), enforces the inbox pending cap,
stores to the existing send-inbox R2 bucket as `kind='file'`.
`assetBundler` now sets `credentials: 'include'` in the non-Tauri
branch so the extension SW carries paywalled-CDN cookies.
- 47 vitest cases for the extension shell (upload, badge, auth, lazy,
popup state machine, auth-bridge token sync) + 8 cases for the new
server endpoint. CI's `test_web_app` invokes both via the extended
`test:pr:web` plus a `build-browser-ext` step that catches webpack
alias / Tauri-stub regressions.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Builds the URL-clipping path of the "Send to Readest" feature: paste a
link, the renderer ingests the rendered page, and a self-contained EPUB
lands in the library. No server proxy, no external CDN refs left in the
EPUB once it's saved.
Architecture
- New Rust `clip_url` command spawns a hidden Tauri WebviewWindow at the
target URL with a real Chrome UA + WebKit fingerprint mask, so TLS-
fingerprint and JS-challenge walls (Cloudflare, Medium, X, WeChat MP)
resolve naturally instead of bouncing the server proxy.
- Capture transport is URL-payload navigation to a one-shot
127.0.0.1:RANDOM_PORT/clip/{token}?d={url-safe-base64} listener.
Top-level navigation isn't governed by CSP connect-src / form-action /
WebKit Private Network Access — the four earlier transports
(fetch, <form>, custom URI scheme, window.name) were each blocked by
one of those.
- Page-to-EPUB bundler (`assetBundler`) walks <img>/<picture> with
src → data-src → data-original → data-srcset → srcset fallback so lazy-
loading sites don't ship a 60px LQIP; fetches assets in parallel with a
per-asset timeout + per-asset/total caps; failed images degrade to alt-
text placeholders. A per-site rules table (seeded with WeChat MP) + a
selector fallback catches articles Readability misextracts. Builder
prepends the article <h1> + byline so the EPUB has a proper opening.
- Nested EPUB TOC built from h1–h6.
UI surfaces
- "From Web URL" entry in the library Import menu, gated to Tauri; web
build hides the URL field and points at the browser extension.
- `ImportFromUrlDialog` with auto-height (overrides Dialog's `sm:h-[65%]`
default) and a dim placeholder for the URL field.
- Clip webview window styled to match Readest's main window — macOS
decorations + overlay title bar; other desktops decorationless with a
drop shadow; native background + in-page loading overlay pick up the
caller's `themeCode.bg`/`fg` so light/dark/eink/custom themes all
render correctly. Title localised, all five overlay/title strings
translated across 33 locales.
Notes
- Gates the macOS traffic-light positioner to main/reader-* windows so
the decorationless clip window no longer null-derefs in
`position_traffic_lights`.
- Stricter validation across the path: schemes restricted to http/https,
hex-color parsing rejects malformed values, server endpoint returns
400 on missing/invalid base64.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Email Worker rejects mail from senders that are not in the user's
approved-sender allowlist. With an empty allowlist the very first send
("email it to yourself") bounces with an approval-pending notice, which
is the wrong first impression for the feature.
Seed the caller's verified account email as `approved` the moment we
lazily create the user's send_addresses row. Best-effort: address
creation still succeeds if the seed insert fails (idempotent via the
existing UNIQUE (user_id, email) constraint).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(send): Send to Readest — multi-channel capture into your library
A Send-to-Kindle equivalent: email, web-upload, share, or one-click
capture books and articles into the cloud library; they sync to every
device.
Architecture (client-side processing): out-of-app channels drop a raw
payload into a per-user send_inbox; Readest clients drain it through one
shared ingestService.ingestFile(). The server never parses or converts.
- ingestService.ingestFile() — channel-agnostic import orchestration
extracted from library/page.tsx (DI-based, forceUpload support).
- send_addresses / send_allowed_senders / send_inbox tables + RLS + 4
SECURITY DEFINER claim/lease RPCs (migration 012_send_to_readest.sql).
- Conversion subsystem (DOCX/RTF/HTML/article/TXT -> EPUB) in a Web Worker.
- send-email Cloudflare Email Worker; inbox-drainer controller +
useInboxDrainer hook; /api/send/* routes.
- Send to Readest settings panel: inbound address, approved-sender
allowlist, recent activity, per-device drain toggle.
- /send web page (file drop + article URL) + SSRF-guarded fetch-url proxy.
- OS-shared files routed through ingestFile; Manifest V3 browser extension.
Security: inbox state changes only via SECURITY DEFINER RPCs (clients get
SELECT-only on send_inbox); approved-sender allowlist gates email;
SSRF guard on the one server-side URL fetch; inbox payload signed URLs
authorize against send_inbox.user_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: run format:check in the pre-push hook
Biome format checking is fast (~0.4s), so gate pushes on it too — catches
mis-formatted files that bypassed the staged-only pre-commit hook before
they reach CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(send): address CodeQL security findings
- ReDoS (senders.ts): the email regex had ambiguous quantifiers around
the literal dot. Rewrote it linear-time (domain labels exclude '.')
and cap the input at 254 chars.
- XSS (convertToEpub.ts): run untrusted HTML through DOMPurify
(sanitizeForParsing — keeps document structure) before DOMParser, so
title extraction and Readability never parse executable markup.
- SSRF (fetch-url.ts): harden the host guard — block bare single-label
hostnames, IPv4-mapped IPv6, CGNAT/benchmark/multicast ranges, and the
unspecified address. DNS rebinding stays a documented residual risk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auto-sync triggers (boot non-settings, focus, visibilitychange, online,
periodic) used to fan out N parallel `GET /api/sync/replicas?kind=…`
requests, one per replica kind. With 5 kinds today and the focus path
firing on every foreground transition, that's 5x the Cloudflare Worker
invocations of what the work actually requires.
Server: extend POST /api/sync/replicas to accept a batched-pull body
(`{ cursors: [{kind, since}, …] }`) alongside the existing push
(`{ rows: […] }`). Per-kind queries fan out via Promise.all — Supabase
calls inside the Worker aren't billed as Cloudflare requests, so DB
load is unchanged while Worker invocations collapse from N to 1.
Client/manager: add `client.pullBatch` and `manager.pullMany` that
share the existing cursor/HLC machinery. The boot path's `since=null`
override carries over via `pullMany(kinds, { since: null })`.
Orchestrator: `triggerIncrementalPullAll` now does ONE pullMany call
then fans out per-kind apply via Promise.allSettled. Boot does the
same for non-settings kinds (settings stays a single call to preserve
its apply-first ordering invariant).
Foreground triggers: listen to BOTH `focus` and `visibilitychange`,
sharing one throttle. focus is fastest on iOS Tauri WKWebView (~T=0,
~400ms ahead of visibilitychange). visibilitychange is the only
signal that fires on browser tab switching — focus does not. Drops
the Supabase user-ref-change listener (was the slowest of the three
foreground signals; redundant with the DOM events).
Bonus: `useBooksSync` now serializes `handleAutoSync` against
`pullLibrary` via the shared `isPullingRef` gate. The two paths used
to fire two concurrent `/api/sync?type=books` requests on the same
`since` value at startup; now whichever runs first claims the gate
and the other skips (throttle's `emitLast` retries afterwards).
Per session: boot 5→2 Worker calls. Per foreground trigger: 5→1.
`interactive-widget=resizes-content` was set in the SSR viewport
metadata so Android Chrome would shrink the layout viewport when
the on-screen keyboard opens (matching iOS default behavior).
Other browsers — Safari on macOS / iOS, desktop Chrome, Firefox —
log a console warning every page load because they don't recognize
the key.
Move the attachment client-side, gated on a UA sniff for Android,
so the meta tag stays clean for everyone else. The Android-specific
behavior (modals centered above the keyboard) is preserved on the
platform that actually needed it.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `interactive-widget=resizes-content` to the viewport meta. iOS
Safari already shrinks the layout viewport when the keyboard opens,
so existing `fixed inset-0` flex-centered modals (PassphrasePrompt,
GroupingModal, etc.) auto-center in the visible space. Android
Chrome defaults to `resizes-visual` — only the visual viewport
shrinks, layout stays full-height — leaving those modals rendered
under the keyboard. Switching to `resizes-content` makes Android
match iOS without any per-modal JS.
Updates both the App Router viewport export (src/app/layout.tsx)
and the Pages Router fallback meta (src/pages/_app.tsx).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): encrypt opds_catalog credentials end-to-end (TS path)
Wires encrypted-credential sync for opds_catalog via the CryptoSession
shipped in PR 4a (#4084) plus a new publish/pull crypto middleware.
TS-only — native still uses ephemeral storage (re-enter passphrase per
launch); PR 4d wires the OS keychain.
- ReplicaAdapter gains optional `encryptedFields: readonly string[]`.
Adapters stay sync; the middleware handles the crypto round trip.
- replicaCryptoMiddleware.ts: encryptPackedFields drops the named
fields from the push when the session is locked (no plaintext
leak); decryptRowFields drops them on pull failure (local
plaintext preserved by the store merge).
- replicaPublish / replicaPullAndApply invoke the middleware.
- OPDS adapter declares encryptedFields = [username, password] and
now pack/unpack them as plaintext.
- passphraseGate.ts: ensurePassphraseUnlocked coalesces concurrent
calls, prompts via the registered prompter with kind=setup|unlock,
throws NO_PASSPHRASE on cancel.
- PassphrasePromptModal mounted at the Providers root; registers
itself as the gate prompter.
- CryptoSession.forget() wipes server-side envelopes + salts.
- Migration 010 + replica_keys_forget RPC; DELETE
/api/sync/replica-keys + client wrapper.
- SyncPassphraseSection on the user page: status / Set / Unlock /
Lock / Forgot.
- CatalogManager pre-save: ensurePassphraseUnlocked when credentials
are present; user cancel saves locally without sync.
Plan updated: PR 4 split documented as 4a/4b/4c/4d.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): persist sync passphrase via OS keychain (Tauri)
Replaces the EphemeralPassphraseStore stub on native with real
OS-keychain storage so users don't re-enter their sync passphrase
every launch. Web stays on the in-memory ephemeral store by design.
Native bridge plugin gains 4 commands wired across all platforms:
- Rust desktop (`keyring` crate): macOS Keychain on apple-native,
Windows Credential Manager on windows-native, Linux libsecret/
Secret Service on sync-secret-service. Per-target features so each
platform compiles only the backend it needs.
- iOS Swift: Security framework Keychain (kSecClassGenericPassword,
SecItemAdd / Copy / Delete).
- Android Kotlin: androidx.security EncryptedSharedPreferences
(AndroidKeystore-derived AES-GCM master key,
AES256_SIV / AES256_GCM key/value encryption).
TS layer:
- TauriPassphraseStore wraps the bridge calls. set is fail-loud
(surfaces keychain rejection); get is fail-soft (returns null on
any error so the gate prompts).
- createPassphraseStore returns ephemeral synchronously;
upgradeToKeychainIfAvailable swaps the singleton to
TauriPassphraseStore on Tauri after probing the bridge. CryptoSession
resolves the store via createPassphraseStore() each touch so the
swap is transparent.
- CryptoSession.tryRestoreFromStore: silent unlock at boot. Stale-
entry recovery clears the store when the account has no salt
server-side. unlock/setup persist; forget also clears the store.
- Providers boot effect: upgrade keychain → silent restore.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): make encrypted-credential pull actually decrypt + UX polish
PR 4c shipped the encrypt path but the pull side silently dropped
ciphers when locked, the modal was busy with double-rings, and web
re-prompted on every page refresh. This rolls up the post-test
fixes + UX polish:
Pull-side decrypt:
- decryptRowFields takes an `onLocked` callback the orchestrator
wires to the passphrase gate; encountering a cipher field with a
locked session now triggers the lazy-prompt path instead of
dropping the field.
- replicaPullAndApply re-applies the unpacked row for metadata-only
kinds even when a local copy exists, so the now-decrypted creds
reach the store (the binary-kind skip-if-local optimization
doesn't apply).
- Cipher fingerprint comparison: capture the row's `cipher.c` for
each encrypted field, compare against the local record's
lastSeenCipher. Same → skip prompt + decrypt entirely. Different
(rotation / value change on another device) → prompt to
re-decrypt. Fingerprint persists via OPDSCatalog.lastSeenCipher.
Web persistence:
- SessionStoragePassphraseStore: passphrase survives page refresh
within the same tab, dies on tab close. Replaces
EphemeralPassphraseStore as the default on web. Avoids
localStorage / IndexedDB to keep the tab-scoped trust boundary.
UI:
- Renamed PassphrasePromptModal → PassphrasePrompt; modernized: filled
input style with single subtle focus border, btn-primary +
btn-ghost replaced with leaner custom buttons. eink-bordered +
btn-primary classes give the dialog correct e-paper rendering.
- globals.css: suppress redundant outline/box-shadow on focused text
inputs / textareas (the element's own border is the focus
indicator).
- AGENTS.md: documents the e-ink convention (`eink-bordered`,
`btn-primary` for inverted CTAs, etc.) so future widgets ship with
e-paper support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the per-account PBKDF2 salt endpoint and an in-memory CryptoSession
that derives keys lazily per saltId. Lets future kinds (OPDS catalogs in
PR 4b) encrypt/decrypt fields without re-deriving on every operation.
- Migration 008: replica_keys_{create,list} RPCs round-trip the bytea
salt as base64; SECURITY INVOKER, RLS-gated by the existing replica_keys
policies.
- /api/sync/replica-keys GET/POST endpoint matches the dual app/pages
shape used by /api/sync/replicas.
- ReplicaSyncClient.{listReplicaKeys,createReplicaKey} wraps the endpoint.
- CryptoSession.{unlock,setup,encryptField,decryptField,lock} caches
derived keys per saltId; foreign envelopes trigger a lazy re-list +
derive. Iterations injectable so tests run with PBKDF2 ITER=1000.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(sync): kind-agnostic replica primitives
Extract dict-only sync into shared primitives (registry, pull/apply
orchestrator, persist env, schema allowlist) so other kinds can plug
in. Companion changes: per-replica Storage Manager grouping,
useReplicaPull boot-race recovery, manifest=null reconciliation on
every boot pull, copyFile takes explicit srcBase + dstBase, settled-
event helpers, lenient webDownload Content-Length (R2/S3 signed URLs
commonly omit it), and generic "File" transfer toast copy any replica
kind can share.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): cross-device custom font sync
Plug the font replica adapter into the kind-agnostic primitives:
font store gains replica wiring, custom font import publishes the
replica row + queues a binary upload, and bootstrap registers the
font adapter and download-complete handler.
Includes legacy flat-path migration so pre-existing fonts sync
without re-import, full @font-face activation on auto-download
(load + mount the rule, mirroring manual import), and a fix to
createCustomFont so contentId / bundleDir / byteSize survive the
trip through addFont — otherwise import-time publish silently
no-oped on missing contentId.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): foundation for CRDT-based cross-device replica sync (Phase 1+2)
Adds the primitives and orchestration layer for syncing user-imported
assets (dictionaries, fonts, textures, OPDS catalogs, dict settings)
across devices via a polymorphic `replicas` table with field-level LWW
under HLC ordering. Phase 1 ships the foundation (CRDT, crypto, server
schemas, SQL migrations, push/pull endpoint); Phase 2 adds the adapter
registry, HTTP client, and sync manager. No modifications to existing
book sync — additive only.
Phase 1:
- src/libs/crdt.ts — HlcGenerator (monotonic + remote-absorption +
clock-regression-safe), per-field LWW with deviceId tiebreak,
remove-wins tombstones, reincarnation token revival.
- src/libs/crypto/{derive,encrypt,envelope,passphrase}.ts —
PBKDF2-600k key derivation (OWASP 2024), AES-GCM round-trip,
envelope {c,i,s,alg,h} with SHA-256 sidecar integrity check,
passphrase storage abstraction (web ephemeral; Tauri keychain stub).
- src/libs/replica-schemas.ts — Zod-backed allowlist (dictionary only
in PR 1), 64KiB row cap, 64-field cap, schemaVersion bounds,
filename validator.
- src/libs/replica-sync-server.ts — push batch validation
(auth + allowlist + schema + HLC ±60s skew clamp).
- src/pages/api/sync/replicas.ts — POST/GET endpoint wrapping the
Postgres crdt_merge_replica function via RPC.
- docker/volumes/db/migrations/003_add_replicas.sql — replicas table
+ replica_keys table + RLS.
- docker/volumes/db/migrations/004_crdt_merge_replica_fn.sql — atomic
per-field LWW merge function (forwards-compat preserves unknown
fields).
Phase 2:
- src/services/sync/replicaRegistry.ts — adapter contract
(core + optional BinaryCapability + LifecycleHooks per eng review).
- src/libs/replica-sync-client.ts — HTTP wrapper mapping status codes
to typed SyncError codes.
- src/services/sync/replicaSyncManager.ts — 5s debounced push,
immediate flush on visibilitychange/online, per-kind pull cursor,
remote HLC absorption.
Tests: 125 new (crdt 26, crypto 32, schemas 21, server 16, client 12,
registry 6, manager 12). Full suite 3656 passing, lint clean. Existing
book/config/note sync paths untouched.
Plan: ~/.claude/plans/vivid-orbiting-thimble.md
CEO plan: ~/.gstack/projects/readest-readest/ceo-plans/2026-05-06-replica-sync-cathedral.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): add kind="replica" path through TransferManager (Phase 3)
Adds the replica branch to the existing book-shaped transfer
infrastructure so dictionary (and future kinds) bundles can flow through
the same queue, retry, and progress UI as book uploads.
Existing book transfer paths remain unchanged. The book-side regression
suite (37 tests in transfer-store.test.ts, 37 in transfer-manager.test.ts)
all stay green.
Store (src/store/transferStore.ts):
- TransferItem gains kind: 'book' | 'replica' (default 'book' on legacy
persisted rows), replicaKind, replicaId, replicaFiles, replicaBase.
- New addReplicaTransfer(replicaKind, replicaId, displayTitle, type, opts)
with files + base in opts; auto-computes totalBytes from file sizes.
- New getReplicaTransfer(replicaKind, replicaId, type) lookup.
- getTransferByBookHash filters to kind === 'book' (defensive against
bookHash="" collisions on replica items).
- restoreTransfers fills kind: 'book' for legacy persisted rows.
Manager (src/services/transferManager.ts):
- queueReplicaUpload / queueReplicaDownload / queueReplicaDelete.
- executeTransfer dispatches by kind to the new executeReplicaTransfer
(iterates files, calls appService.uploadReplicaFile per file with
per-file progress aggregation) or the existing executeBookTransfer
(refactored out, byte-identical behavior).
- Dispatches replica-transfer-complete event on success so stores can
react (e.g., commit manifest_jsonb to the replica row).
Storage / cloud (src/libs/storage.ts, src/services/cloudService.ts):
- uploadReplicaFile bypasses the book-only File.name smuggling and
takes an explicit cfp (cloud file path).
- uploadReplicaFileToCloud / downloadReplicaFileFromCloud /
deleteReplicaBundleFromCloud orchestrate per-file operations under
${userId}/Readest/replicas/<kind>/<replicaId>/<filename>.
- replicaCloudKey() centralizes the path-construction rule.
- New CLOUD_REPLICAS_SUBDIR constant.
App service (src/services/appService.ts, src/types/system.ts):
- AppService gains uploadReplicaFile, downloadReplicaFile,
deleteReplicaBundle (file-level operations; orchestration lives in
TransferManager).
Tests: 18 new (12 in transfer-store.test.ts, 5 in transfer-manager.test.ts,
1 fixture). Full suite 3674 passing, lint clean. Existing book regression
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): dictionary replica adapter + bootstrap (Phase 4, partial)
Lands the safe-to-ship foundation of Phase 4 — adapter logic, registry
bootstrap, and SystemSettings hooks for replica sync. The on-disk
migration of legacy customDictionaries (bundleDir → content-hash id),
the live store wiring, and the Settings → Sync UI are deferred to a
follow-up PR so they can land with real-device QA.
Adapter (src/services/sync/adapters/dictionary.ts):
- dictionaryAdapter: kind='dictionary', schemaVersion=1.
- pack/unpack — only synced subset (name, kind, lang, addedAt,
unsupported{,Reason}). bundleDir / files / unavailable / deletedAt
stay per-device or are handled by the tombstone mechanism.
- BinaryCapability.enumerateFiles dispatches by bundle kind:
- mdict: mdx + mdd[] + css[]
- stardict: ifo + idx + dict + syn (skips .idx.offsets / .syn.offsets
sidecars — those are device-local indices)
- dict: dict + index
- slob: single .slob file
- primaryDictionaryFile() picks the anchor file per kind for
partialMD5 hashing.
- computeDictionaryReplicaId(partialMd5, byteSize, sortedFilenames)
produces a deterministic 32-hex content-hash id used at import time.
- 23 tests cover pack/unpack identity, kind dispatch, file enumeration,
id determinism, and per-device-field exclusion.
Bootstrap (src/services/sync/replicaBootstrap.ts):
- bootstrapReplicaAdapters() registers all known adapters once at app
start. Idempotent (safe to call multiple times). Wired into
EnvContext.tsx so the registry populates on app mount.
- 3 tests cover registration, idempotency, and the PR-1 allowlist.
SystemSettings (src/types/settings.ts, src/services/constants.ts):
- +SyncCategory = 'book' | 'progress' | 'note' | 'dictionary' — typed
union for the user-facing sync toggles. 'progress' gates the
existing book-config sync (reading progress); 'note' gates
annotations; 'book' gates book binaries + metadata; 'dictionary'
gates the new replica sync. Future replica kinds extend the union.
- +SYNC_CATEGORIES readonly array for UI iteration.
- +syncCategories: Partial<Record<SyncCategory, boolean>> — per-
category opt-in toggles in DEFAULT_SYSTEM_SETTINGS (default ON for
all four). UI panel ships in the follow-up.
- +lastSyncedAtReplicas: Record<string, string> — per-kind HLC pull
cursors (matches replicaSyncManager's CursorStore contract).
Registry type cleanup (src/services/sync/replicaRegistry.ts):
- BinaryCapability.enumerateFiles return shape: localRelPath → lfp
to match the existing TransferStore.ReplicaTransferFile convention.
Tests: 28 new (23 dict + 3 bootstrap + 2 syncCategories defaults).
Full suite 3702 passing, lint clean. Existing book/config/note sync
paths untouched.
Deferred to PR 1 follow-up (with real-device QA):
- customDictionaryStore migration: rehash legacy uniqueId() bundleDir
to content-hash id; preserve providerOrder mapping; staged
.legacy/<old-id>/ backup.
- Wire customDictionaryStore mutations through replicaSyncManager
(markDirty on add/rename/delete; pull on init).
- Settings → Sync panel: per-category toggles + last-sync timestamps.
- Sync passphrase modal: set / change / forgot flow (lazy first prompt
on encrypted-field push/pull).
- <CloudReplicaRow> in CustomDictionaries.tsx for "Download from cloud
(X MB)" affordance.
- Tauri keychain backend for sync passphrase storage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): replicaSync singleton + content-hash id at dict import (Phase 4b)
Two foundation pieces that everything UI-side will sit on top of, both
purely additive — no live store wiring, no behavior change for existing
dictionary imports.
replicaSync singleton (src/services/sync/replicaSync.ts):
- initReplicaSync({deviceId, cursorStore, hlcStore?, client?}) builds
one ReplicaSyncManager backed by an HlcGenerator with persistence
wrapped around .next()/.observe(). Idempotent (second init returns
the existing instance).
- LocalStorageHlcStore (src/libs/hlc-store.ts) snapshots the HLC
counter under 'readest_replica_hlc' so it survives restart. Falls
back silently when localStorage is unavailable (private mode, SSR);
client re-derives via the existing remote max(updated_at_ts) repair
path. InMemoryHlcStore is the test backend.
- 16 tests (9 hlc-store, 7 replicaSync) covering snapshot persistence,
restore-on-init, and idempotency.
- Wiring into EnvContext for production deferred to the follow-up that
also adds the cursor store backed by useSettingsStore.
Content-hash id at dictionary import
(src/services/dictionaries/contentId.ts):
- computeDictionaryContentId(primaryFile, filenames) wraps
computeDictionaryReplicaId(partialMd5(primary), byteSize,
sortedFilenames) — the cross-device id used as the replica_id when
the dict actually pushes/pulls.
- Wired into all four import paths in dictionaryService.ts:
- stardict primary = .ifo (small text, partialMD5 ≈ full hash)
- mdict primary = .mdx (body)
- dict primary = .dict.dz (gzipped body)
- slob primary = .slob (single-file bundle)
ImportedDictionary gains contentId?: string. Optional for backwards
compat; legacy bundles without contentId are flagged as
"needs rehash before sync" by the upcoming store-wiring follow-up.
- 6 tests cover identity determinism, byteSize sensitivity, filename-
set sensitivity, and order-independence.
Tests: 22 new (9 + 7 + 6). Full suite 3724 passing, lint clean.
Existing dictionary import flow unchanged for users — contentId is an
additional field, not a replacement for the bundleDir-based id.
Deferred to follow-up (with on-device QA):
- Production cursor store backed by useSettingsStore +
appService.saveSettings.
- EnvContext call to initReplicaSync after appService boot.
- customDictionaryStore mutation hooks → replicaSyncManager.markDirty.
- Legacy bundleDir → contentId migration with .legacy/ backup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(sync): land replica-sync design plan in repo
Moves the plan document that drove this PR's foundation work
(`~/.claude/plans/vivid-orbiting-thimble.md`) into the project tree at
`apps/readest-app/.claude/plans/` so reviewers and future contributors
can read it alongside the code without leaving the repo.
The plan went through three review passes — Codex (19 findings, all
absorbed), CEO/scope review (mode SCOPE EXPANSION; encrypted secrets
pulled forward to v1, "private-only forever" posture lock), and eng
review (FULL_REVIEW mode, 16 findings absorbed). The full review trail
lives in the file's `## GSTACK REVIEW REPORT` section at the bottom.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(crdt): point README at in-repo plan path
Now that vivid-orbiting-thimble.md lives at
apps/readest-app/.claude/plans/, the README link should point there
rather than at the home-dir copy that no longer exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(sync): rename src/libs files to camelCase per project convention
Test files renamed in lockstep. Imports + comment references updated
across cloudService, storage, transferManager, replicaSync,
replicaSyncManager, /api/sync/replicas, and all four test files. No
behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validate serverUrl in the kosync API proxy to block requests to
private/internal addresses and non-http(s) schemes. Also fix isLanAddress
to detect 0.0.0.0 and bracket-wrapped IPv6 private addresses.
Closes code-scanning alert #14.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Paginate Supabase select queries in the storage stats API to avoid
the 1000 row default limit. Align StorageManager's formatFileSize
with useQuotaStats calculation so quota values display consistently.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(sync): implement KOReader progress synchronization
This commit introduces a comprehensive feature to synchronize reading progress with a KOReader sync server. It includes a compatibility layer to handle discrepancies between Readest's CFI-based progress and KOReader's XPointer/page-based progress, primarily using the `percentage` field as a common ground.
Key additions include:
- A new settings panel under "KOReader Sync" for server configuration, authentication, and sync strategy management (e.g., prompt on conflict, always use latest).
- A conflict resolution dialog that appears when remote progress differs significantly from local progress, allowing the user to choose which version to keep.
- A client-side `useKOSync` hook to manage the entire synchronization lifecycle, including API calls, state management, and conflict resolution logic.
- A new API endpoint `/api/kosync` that acts as a secure proxy to the user-configured KOReader sync server, handling authentication and forwarding requests.
- Logic to differentiate between paginated (PDF/CBZ) and reflowable (EPUB) formats, using page numbers for paginated files where possible and falling back to percentage for reliability.
- Spanish translations for all UI elements related to the KOReader sync feature.
- Addition of `uuid` package to generate a unique `device_id` for sync purposes.
Refactor:
- The `debounce` utility has been improved to include `flush` and `cancel` methods, allowing for more precise control over debounced function execution, which is now used in the sync hook.
* fix(kosync): add support for converting between XPointer and CFI in progress synchronization
* fix(kosync): update navigation method to use select instead of goTo for paginated formats
* fix(kosync): refactor synchronization settings and improve conflict resolution handling
* fix(kosync): add event dispatcher for flushing KOReader synchronization
* fix(sync): handle xpointer in a different section, fix styling
* i18n: update translations
---------
Co-authored-by: Huang Xin <chrox.huang@gmail.com>
Huang Xin