akai-tools/readest
2
0
Fork 0
forked from akai/readest
Code Pull Requests Activity
2,129 Commits 4 Branches 102 Tags
a1279a65ce3f255b1fed91dc18a5c828d7c2ed68
Commit Graph

5 Commits

Author SHA1 Message Date
Huang Xin 1eae2af23e feat(sync): batch replica sync into one /api/sync/replicas request (#4109) 
Auto-sync triggers (boot non-settings, focus, visibilitychange, online,
periodic) used to fan out N parallel `GET /api/sync/replicas?kind=…`
requests, one per replica kind. With 5 kinds today and the focus path
firing on every foreground transition, that's 5x the Cloudflare Worker
invocations of what the work actually requires.

Server: extend POST /api/sync/replicas to accept a batched-pull body
(`{ cursors: [{kind, since}, …] }`) alongside the existing push
(`{ rows: […] }`). Per-kind queries fan out via Promise.all — Supabase
calls inside the Worker aren't billed as Cloudflare requests, so DB
load is unchanged while Worker invocations collapse from N to 1.

Client/manager: add `client.pullBatch` and `manager.pullMany` that
share the existing cursor/HLC machinery. The boot path's `since=null`
override carries over via `pullMany(kinds, { since: null })`.

Orchestrator: `triggerIncrementalPullAll` now does ONE pullMany call
then fans out per-kind apply via Promise.allSettled. Boot does the
same for non-settings kinds (settings stays a single call to preserve
its apply-first ordering invariant).

Foreground triggers: listen to BOTH `focus` and `visibilitychange`,
sharing one throttle. focus is fastest on iOS Tauri WKWebView (~T=0,
~400ms ahead of visibilitychange). visibilitychange is the only
signal that fires on browser tab switching — focus does not. Drops
the Supabase user-ref-change listener (was the slowest of the three
foreground signals; redundant with the DOM events).

Bonus: `useBooksSync` now serializes `handleAutoSync` against
`pullLibrary` via the shared `isPullingRef` gate. The two paths used
to fire two concurrent `/api/sync?type=books` requests on the same
`since` value at startup; now whichever runs first claims the gate
and the other skips (throttle's `emitLast` retries afterwards).

Per session: boot 5→2 Worker calls. Per foreground trigger: 5→1.
 2026-05-09 16:18:46 +02:00
Huang Xin 6e7c9d1395 feat(sync): bundled settings replica kind for cross-device prefs and credentials (#4094) 
* feat(sync): add bundled `settings` replica kind for cross-device prefs and credentials

Adds a single-row `settings` replica that syncs a whitelist of
`SystemSettings` fields across devices via per-field LWW (one entry
per dot-namespaced path). Plaintext for theme / highlight colour /
TTS configuration; encrypted (AES-GCM under the user's sync
passphrase) for kosync / Readwise / Hardcover credentials.

Highlights:
- Push-side diff against an in-memory snapshot for plaintext paths
  and a localStorage SHA-256 hash for encrypted paths, so a refresh
  doesn't re-publish or re-prompt for the passphrase.
- Pull-side cipher-fingerprint dedupe + per-row passphrase gate;
  decryption failures surface as toasts (wrong passphrase / orphan
  cipher) instead of silent drops.
- Auto-recovery for orphaned ciphers: when a row references a
  saltId no longer in `replica_keys`, clear the local hash and
  re-encrypt under the current salt on the next save.
- Single in-flight `/sync/replica-keys` fetch with a value cache
  to coalesce the boot-time burst of concurrent unlock callers.

* fix(sync): guard settings dot-path helpers against prototype-polluting keys

Reject `__proto__`, `constructor`, and `prototype` segments in the
settings adapter's `readPath` / `writePath`. Every caller currently
passes a constant from `SETTINGS_WHITELIST`, so the guard is purely
defensive — but it silences the CodeQL prototype-pollution warning
on PR #4094 and keeps the helpers safe if a future call site ever
forwards an untrusted path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 19:03:23 +02:00
Huang Xin 712d564e9d feat(sync): encrypted OPDS credentials + Tauri keychain (PR 4c + 4d) (#4090) 
* feat(sync): encrypt opds_catalog credentials end-to-end (TS path)

Wires encrypted-credential sync for opds_catalog via the CryptoSession
shipped in PR 4a (#4084) plus a new publish/pull crypto middleware.
TS-only — native still uses ephemeral storage (re-enter passphrase per
launch); PR 4d wires the OS keychain.

- ReplicaAdapter gains optional `encryptedFields: readonly string[]`.
  Adapters stay sync; the middleware handles the crypto round trip.
- replicaCryptoMiddleware.ts: encryptPackedFields drops the named
  fields from the push when the session is locked (no plaintext
  leak); decryptRowFields drops them on pull failure (local
  plaintext preserved by the store merge).
- replicaPublish / replicaPullAndApply invoke the middleware.
- OPDS adapter declares encryptedFields = [username, password] and
  now pack/unpack them as plaintext.
- passphraseGate.ts: ensurePassphraseUnlocked coalesces concurrent
  calls, prompts via the registered prompter with kind=setup|unlock,
  throws NO_PASSPHRASE on cancel.
- PassphrasePromptModal mounted at the Providers root; registers
  itself as the gate prompter.
- CryptoSession.forget() wipes server-side envelopes + salts.
- Migration 010 + replica_keys_forget RPC; DELETE
  /api/sync/replica-keys + client wrapper.
- SyncPassphraseSection on the user page: status / Set / Unlock /
  Lock / Forgot.
- CatalogManager pre-save: ensurePassphraseUnlocked when credentials
  are present; user cancel saves locally without sync.

Plan updated: PR 4 split documented as 4a/4b/4c/4d.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): persist sync passphrase via OS keychain (Tauri)

Replaces the EphemeralPassphraseStore stub on native with real
OS-keychain storage so users don't re-enter their sync passphrase
every launch. Web stays on the in-memory ephemeral store by design.

Native bridge plugin gains 4 commands wired across all platforms:

- Rust desktop (`keyring` crate): macOS Keychain on apple-native,
  Windows Credential Manager on windows-native, Linux libsecret/
  Secret Service on sync-secret-service. Per-target features so each
  platform compiles only the backend it needs.
- iOS Swift: Security framework Keychain (kSecClassGenericPassword,
  SecItemAdd / Copy / Delete).
- Android Kotlin: androidx.security EncryptedSharedPreferences
  (AndroidKeystore-derived AES-GCM master key,
  AES256_SIV / AES256_GCM key/value encryption).

TS layer:

- TauriPassphraseStore wraps the bridge calls. set is fail-loud
  (surfaces keychain rejection); get is fail-soft (returns null on
  any error so the gate prompts).
- createPassphraseStore returns ephemeral synchronously;
  upgradeToKeychainIfAvailable swaps the singleton to
  TauriPassphraseStore on Tauri after probing the bridge. CryptoSession
  resolves the store via createPassphraseStore() each touch so the
  swap is transparent.
- CryptoSession.tryRestoreFromStore: silent unlock at boot. Stale-
  entry recovery clears the store when the account has no salt
  server-side. unlock/setup persist; forget also clears the store.
- Providers boot effect: upgrade keychain → silent restore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(sync): make encrypted-credential pull actually decrypt + UX polish

PR 4c shipped the encrypt path but the pull side silently dropped
ciphers when locked, the modal was busy with double-rings, and web
re-prompted on every page refresh. This rolls up the post-test
fixes + UX polish:

Pull-side decrypt:
- decryptRowFields takes an `onLocked` callback the orchestrator
  wires to the passphrase gate; encountering a cipher field with a
  locked session now triggers the lazy-prompt path instead of
  dropping the field.
- replicaPullAndApply re-applies the unpacked row for metadata-only
  kinds even when a local copy exists, so the now-decrypted creds
  reach the store (the binary-kind skip-if-local optimization
  doesn't apply).
- Cipher fingerprint comparison: capture the row's `cipher.c` for
  each encrypted field, compare against the local record's
  lastSeenCipher. Same → skip prompt + decrypt entirely. Different
  (rotation / value change on another device) → prompt to
  re-decrypt. Fingerprint persists via OPDSCatalog.lastSeenCipher.

Web persistence:
- SessionStoragePassphraseStore: passphrase survives page refresh
  within the same tab, dies on tab close. Replaces
  EphemeralPassphraseStore as the default on web. Avoids
  localStorage / IndexedDB to keep the tab-scoped trust boundary.

UI:
- Renamed PassphrasePromptModal → PassphrasePrompt; modernized: filled
  input style with single subtle focus border, btn-primary +
  btn-ghost replaced with leaner custom buttons. eink-bordered +
  btn-primary classes give the dialog correct e-paper rendering.
- globals.css: suppress redundant outline/box-shadow on focused text
  inputs / textareas (the element's own border is the focus
  indicator).
- AGENTS.md: documents the e-ink convention (`eink-bordered`,
  `btn-primary` for inverted CTAs, etc.) so future widgets ship with
  e-paper support.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 13:22:49 +02:00
Huang Xin 35227ecd61 feat(sync): wire crypto session for encrypted-field sync (#4084) 
Adds the per-account PBKDF2 salt endpoint and an in-memory CryptoSession
that derives keys lazily per saltId. Lets future kinds (OPDS catalogs in
PR 4b) encrypt/decrypt fields without re-deriving on every operation.

- Migration 008: replica_keys_{create,list} RPCs round-trip the bytea
  salt as base64; SECURITY INVOKER, RLS-gated by the existing replica_keys
  policies.
- /api/sync/replica-keys GET/POST endpoint matches the dual app/pages
  shape used by /api/sync/replicas.
- ReplicaSyncClient.{listReplicaKeys,createReplicaKey} wraps the endpoint.
- CryptoSession.{unlock,setup,encryptField,decryptField,lock} caches
  derived keys per saltId; foreign envelopes trigger a lazy re-list +
  derive. Iterations injectable so tests run with PBKDF2 ITER=1000.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 20:05:04 +02:00
Huang Xin 3b348c8f35 feat(sync): CRDT replica sync foundation (#4075) 
* feat(sync): foundation for CRDT-based cross-device replica sync (Phase 1+2)

Adds the primitives and orchestration layer for syncing user-imported
assets (dictionaries, fonts, textures, OPDS catalogs, dict settings)
across devices via a polymorphic `replicas` table with field-level LWW
under HLC ordering. Phase 1 ships the foundation (CRDT, crypto, server
schemas, SQL migrations, push/pull endpoint); Phase 2 adds the adapter
registry, HTTP client, and sync manager. No modifications to existing
book sync — additive only.

Phase 1:
- src/libs/crdt.ts — HlcGenerator (monotonic + remote-absorption +
  clock-regression-safe), per-field LWW with deviceId tiebreak,
  remove-wins tombstones, reincarnation token revival.
- src/libs/crypto/{derive,encrypt,envelope,passphrase}.ts —
  PBKDF2-600k key derivation (OWASP 2024), AES-GCM round-trip,
  envelope {c,i,s,alg,h} with SHA-256 sidecar integrity check,
  passphrase storage abstraction (web ephemeral; Tauri keychain stub).
- src/libs/replica-schemas.ts — Zod-backed allowlist (dictionary only
  in PR 1), 64KiB row cap, 64-field cap, schemaVersion bounds,
  filename validator.
- src/libs/replica-sync-server.ts — push batch validation
  (auth + allowlist + schema + HLC ±60s skew clamp).
- src/pages/api/sync/replicas.ts — POST/GET endpoint wrapping the
  Postgres crdt_merge_replica function via RPC.
- docker/volumes/db/migrations/003_add_replicas.sql — replicas table
  + replica_keys table + RLS.
- docker/volumes/db/migrations/004_crdt_merge_replica_fn.sql — atomic
  per-field LWW merge function (forwards-compat preserves unknown
  fields).

Phase 2:
- src/services/sync/replicaRegistry.ts — adapter contract
  (core + optional BinaryCapability + LifecycleHooks per eng review).
- src/libs/replica-sync-client.ts — HTTP wrapper mapping status codes
  to typed SyncError codes.
- src/services/sync/replicaSyncManager.ts — 5s debounced push,
  immediate flush on visibilitychange/online, per-kind pull cursor,
  remote HLC absorption.

Tests: 125 new (crdt 26, crypto 32, schemas 21, server 16, client 12,
registry 6, manager 12). Full suite 3656 passing, lint clean. Existing
book/config/note sync paths untouched.

Plan: ~/.claude/plans/vivid-orbiting-thimble.md
CEO plan: ~/.gstack/projects/readest-readest/ceo-plans/2026-05-06-replica-sync-cathedral.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): add kind="replica" path through TransferManager (Phase 3)

Adds the replica branch to the existing book-shaped transfer
infrastructure so dictionary (and future kinds) bundles can flow through
the same queue, retry, and progress UI as book uploads.

Existing book transfer paths remain unchanged. The book-side regression
suite (37 tests in transfer-store.test.ts, 37 in transfer-manager.test.ts)
all stay green.

Store (src/store/transferStore.ts):
- TransferItem gains kind: 'book' | 'replica' (default 'book' on legacy
  persisted rows), replicaKind, replicaId, replicaFiles, replicaBase.
- New addReplicaTransfer(replicaKind, replicaId, displayTitle, type, opts)
  with files + base in opts; auto-computes totalBytes from file sizes.
- New getReplicaTransfer(replicaKind, replicaId, type) lookup.
- getTransferByBookHash filters to kind === 'book' (defensive against
  bookHash="" collisions on replica items).
- restoreTransfers fills kind: 'book' for legacy persisted rows.

Manager (src/services/transferManager.ts):
- queueReplicaUpload / queueReplicaDownload / queueReplicaDelete.
- executeTransfer dispatches by kind to the new executeReplicaTransfer
  (iterates files, calls appService.uploadReplicaFile per file with
  per-file progress aggregation) or the existing executeBookTransfer
  (refactored out, byte-identical behavior).
- Dispatches replica-transfer-complete event on success so stores can
  react (e.g., commit manifest_jsonb to the replica row).

Storage / cloud (src/libs/storage.ts, src/services/cloudService.ts):
- uploadReplicaFile bypasses the book-only File.name smuggling and
  takes an explicit cfp (cloud file path).
- uploadReplicaFileToCloud / downloadReplicaFileFromCloud /
  deleteReplicaBundleFromCloud orchestrate per-file operations under
  ${userId}/Readest/replicas/<kind>/<replicaId>/<filename>.
- replicaCloudKey() centralizes the path-construction rule.
- New CLOUD_REPLICAS_SUBDIR constant.

App service (src/services/appService.ts, src/types/system.ts):
- AppService gains uploadReplicaFile, downloadReplicaFile,
  deleteReplicaBundle (file-level operations; orchestration lives in
  TransferManager).

Tests: 18 new (12 in transfer-store.test.ts, 5 in transfer-manager.test.ts,
1 fixture). Full suite 3674 passing, lint clean. Existing book regression
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): dictionary replica adapter + bootstrap (Phase 4, partial)

Lands the safe-to-ship foundation of Phase 4 — adapter logic, registry
bootstrap, and SystemSettings hooks for replica sync. The on-disk
migration of legacy customDictionaries (bundleDir → content-hash id),
the live store wiring, and the Settings → Sync UI are deferred to a
follow-up PR so they can land with real-device QA.

Adapter (src/services/sync/adapters/dictionary.ts):
- dictionaryAdapter: kind='dictionary', schemaVersion=1.
- pack/unpack — only synced subset (name, kind, lang, addedAt,
  unsupported{,Reason}). bundleDir / files / unavailable / deletedAt
  stay per-device or are handled by the tombstone mechanism.
- BinaryCapability.enumerateFiles dispatches by bundle kind:
  - mdict:    mdx + mdd[] + css[]
  - stardict: ifo + idx + dict + syn (skips .idx.offsets / .syn.offsets
              sidecars — those are device-local indices)
  - dict:     dict + index
  - slob:     single .slob file
- primaryDictionaryFile() picks the anchor file per kind for
  partialMD5 hashing.
- computeDictionaryReplicaId(partialMd5, byteSize, sortedFilenames)
  produces a deterministic 32-hex content-hash id used at import time.
- 23 tests cover pack/unpack identity, kind dispatch, file enumeration,
  id determinism, and per-device-field exclusion.

Bootstrap (src/services/sync/replicaBootstrap.ts):
- bootstrapReplicaAdapters() registers all known adapters once at app
  start. Idempotent (safe to call multiple times). Wired into
  EnvContext.tsx so the registry populates on app mount.
- 3 tests cover registration, idempotency, and the PR-1 allowlist.

SystemSettings (src/types/settings.ts, src/services/constants.ts):
- +SyncCategory = 'book' | 'progress' | 'note' | 'dictionary' — typed
  union for the user-facing sync toggles. 'progress' gates the
  existing book-config sync (reading progress); 'note' gates
  annotations; 'book' gates book binaries + metadata; 'dictionary'
  gates the new replica sync. Future replica kinds extend the union.
- +SYNC_CATEGORIES readonly array for UI iteration.
- +syncCategories: Partial<Record<SyncCategory, boolean>> — per-
  category opt-in toggles in DEFAULT_SYSTEM_SETTINGS (default ON for
  all four). UI panel ships in the follow-up.
- +lastSyncedAtReplicas: Record<string, string> — per-kind HLC pull
  cursors (matches replicaSyncManager's CursorStore contract).

Registry type cleanup (src/services/sync/replicaRegistry.ts):
- BinaryCapability.enumerateFiles return shape: localRelPath → lfp
  to match the existing TransferStore.ReplicaTransferFile convention.

Tests: 28 new (23 dict + 3 bootstrap + 2 syncCategories defaults).
Full suite 3702 passing, lint clean. Existing book/config/note sync
paths untouched.

Deferred to PR 1 follow-up (with real-device QA):
- customDictionaryStore migration: rehash legacy uniqueId() bundleDir
  to content-hash id; preserve providerOrder mapping; staged
  .legacy/<old-id>/ backup.
- Wire customDictionaryStore mutations through replicaSyncManager
  (markDirty on add/rename/delete; pull on init).
- Settings → Sync panel: per-category toggles + last-sync timestamps.
- Sync passphrase modal: set / change / forgot flow (lazy first prompt
  on encrypted-field push/pull).
- <CloudReplicaRow> in CustomDictionaries.tsx for "Download from cloud
  (X MB)" affordance.
- Tauri keychain backend for sync passphrase storage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): replicaSync singleton + content-hash id at dict import (Phase 4b)

Two foundation pieces that everything UI-side will sit on top of, both
purely additive — no live store wiring, no behavior change for existing
dictionary imports.

replicaSync singleton (src/services/sync/replicaSync.ts):
- initReplicaSync({deviceId, cursorStore, hlcStore?, client?}) builds
  one ReplicaSyncManager backed by an HlcGenerator with persistence
  wrapped around .next()/.observe(). Idempotent (second init returns
  the existing instance).
- LocalStorageHlcStore (src/libs/hlc-store.ts) snapshots the HLC
  counter under 'readest_replica_hlc' so it survives restart. Falls
  back silently when localStorage is unavailable (private mode, SSR);
  client re-derives via the existing remote max(updated_at_ts) repair
  path. InMemoryHlcStore is the test backend.
- 16 tests (9 hlc-store, 7 replicaSync) covering snapshot persistence,
  restore-on-init, and idempotency.
- Wiring into EnvContext for production deferred to the follow-up that
  also adds the cursor store backed by useSettingsStore.

Content-hash id at dictionary import
(src/services/dictionaries/contentId.ts):
- computeDictionaryContentId(primaryFile, filenames) wraps
  computeDictionaryReplicaId(partialMd5(primary), byteSize,
  sortedFilenames) — the cross-device id used as the replica_id when
  the dict actually pushes/pulls.
- Wired into all four import paths in dictionaryService.ts:
  - stardict primary = .ifo (small text, partialMD5 ≈ full hash)
  - mdict primary    = .mdx (body)
  - dict primary     = .dict.dz (gzipped body)
  - slob primary     = .slob (single-file bundle)
  ImportedDictionary gains contentId?: string. Optional for backwards
  compat; legacy bundles without contentId are flagged as
  "needs rehash before sync" by the upcoming store-wiring follow-up.
- 6 tests cover identity determinism, byteSize sensitivity, filename-
  set sensitivity, and order-independence.

Tests: 22 new (9 + 7 + 6). Full suite 3724 passing, lint clean.
Existing dictionary import flow unchanged for users — contentId is an
additional field, not a replacement for the bundleDir-based id.

Deferred to follow-up (with on-device QA):
- Production cursor store backed by useSettingsStore +
  appService.saveSettings.
- EnvContext call to initReplicaSync after appService boot.
- customDictionaryStore mutation hooks → replicaSyncManager.markDirty.
- Legacy bundleDir → contentId migration with .legacy/ backup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(sync): land replica-sync design plan in repo

Moves the plan document that drove this PR's foundation work
(`~/.claude/plans/vivid-orbiting-thimble.md`) into the project tree at
`apps/readest-app/.claude/plans/` so reviewers and future contributors
can read it alongside the code without leaving the repo.

The plan went through three review passes — Codex (19 findings, all
absorbed), CEO/scope review (mode SCOPE EXPANSION; encrypted secrets
pulled forward to v1, "private-only forever" posture lock), and eng
review (FULL_REVIEW mode, 16 findings absorbed). The full review trail
lives in the file's `## GSTACK REVIEW REPORT` section at the bottom.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(crdt): point README at in-repo plan path

Now that vivid-orbiting-thimble.md lives at
apps/readest-app/.claude/plans/, the README link should point there
rather than at the home-dir copy that no longer exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(sync): rename src/libs files to camelCase per project convention

Test files renamed in lockstep. Imports + comment references updated
across cloudService, storage, transferManager, replicaSync,
replicaSyncManager, /api/sync/replicas, and all four test files. No
behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-06 15:50:15 +02:00