Files
readest/apps/readest-app/src/utils/serializer.ts
T
loveheaven 5c82351ab9 feat(integrations): add WebDAV sync to Reading Sync settings (#4204)
* feat(integrations): add WebDAV sync to Reading Sync settings

Adds a WebDAV entry under Settings -> Integrations -> Reading Sync with configure/browse UI, library-wide Sync now, and per-book sync of progress, annotations and (opt-in) book files + covers.

Reading progress and annotations are always synced when WebDAV is enabled; only Sync Book Files stays as a toggle since it's bandwidth-heavy.

* feat(webdav): add diagnostic sync history panel and document viewSettings invariant

Surface a per-run history for the WebDAV "Sync now" button so users can self-triage failures without rummaging through the dev console — a screenshot of the panel is now enough to file a useful bug report. The same change tightens the docs around viewSettings so the "device-local UI preferences" boundary is impossible to misread on the next refactor pass.

Sync history panel:

  * New WebDAVSettings.syncLog ring buffer (cap 10), persisted alongside the rest of settings so a screenshot survives across app restarts. WebDAVSyncLogEntry captures startedAt, finishedAt, status (success / partial / failure), trigger, the eight counters from SyncLibraryResult, the toast text, and an optional per-book failure list with a phase tag (download / upload-config / upload-file).

  * SyncLibraryResult gains a failedBooks: SyncFailureEntry[] field. The two existing failure points in syncLibrary (download catch, upload catch) now record per-book reason+phase via formatFailureReason(), which keeps the persisted blob small by stripping stacks/whitespace and capping length at 200 chars.

  * WebDAVForm.handleSyncNow now timestamps the run, builds an entry from the result on success/partial paths and from the caught error on failure paths, and appends through a fresh-read appendSyncLogEntry() so concurrent toggle changes can't clobber the log.

  * New SyncHistoryPanel + SyncStatusBadge + SyncHistoryDetails components render the log inline in the Settings page. The detail row groups counters into three semantic columns (activity, skipped, outcome) on a six-column grid so labels can wrap freely while numbers stay tabular and right-aligned. Per-book failures render as a separate stack below the counters.

viewSettings invariant:

  * buildRemotePayload and pullBookConfig already implement the right thing — only progress/location/xpointer/booknotes travel; viewSettings stays device-local. Comments now spell out the contract on both sides so future contributors don't reintroduce viewSettings on the wire by mistake.

* fix(webdav): preserve prior state across reconnect, drop stale closure in ensureDeviceId

Two bugs in the WebDAV sync flow surfaced during review:

1. WebDAVForm.handleConnect rebuilt the entire `webdav` settings block
   from the four credential fields the user just typed, dropping
   `deviceId`, `syncBooks`, `strategy`, `syncProgress`, `syncNotes`,
   `lastSyncedAt`, and `syncLog` on every reconnect. Most concerning is
   the deviceId rotation: a disconnect + reconnect made the next sync
   look like a brand-new device, defeating the cross-device clobber
   detection encoded in `RemoteBookConfig.writerDeviceId`. Extract a
   pure helper `buildWebDAVConnectSettings` that spreads the previous
   webdav object first so reconnect is non-destructive, matching the
   sibling pattern in KOSyncForm.

2. useWebDAVSync.ensureDeviceId merged the new deviceId into the closure
   variable `settings`, which can be stale when `pullNow → pushNow`
   fires back-to-back on book open or when the settings panel writes a
   sibling field concurrently. Read latest settings via
   `useSettingsStore.getState()` to match the pattern already used in
   `updateLastSyncedAt` and `persistWebdav`.

Adds three unit tests for the new helper, including the reconnect
preservation invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(webdav): address review observations on encodePath, pull skip, and remote GC

Three follow-ups from the review pass on top of 3f721d04. Each one was
called out as a smaller observation the reviewer noted but did not push:

* WebDAVClient.encodePath silently re-escaped literal % characters
  despite a comment claiming existing %-escapes are preserved. A caller
  that pre-encoded a space as %20 would see %20 become %2520 in the
  request URL, breaking any path that came in already escaped. Tokenise
  each segment into already-escaped %XX runs and everything-else, and
  only run encodeURIComponent on the latter. Add four unit tests
  exercising pure-unicode, pure-pre-escaped, mixed, and root-slash
  paths.

  Implementation note: two regexes are needed because a /g RegExp.test
  is stateful and would skip every other token in this map; the split
  regex has /g for the iteration, the classifier regex is anchored
  without /g for the per-token check.

* OPEN_PULL_SKIP_MS doc-comment claimed it catches the
  close-then-reopen flow, but useWebDAVSync unmounts on reader close so
  lastPulledAtRef resets to 0 — the new instance always passes the
  cooldown check on remount. The guard actually only fires on
  re-invocations of the open-book effect inside one hook lifetime
  (book-to-book navigation, double-render before hasPulledOnce flips).
  Rewrite both the constant's doc-comment and the call-site comment to
  match the real semantics.

* WebDAVSync push path doesn't DELETE the per-hash directory of a
  tombstoned book. The deletion *is* propagated through library.json
  so other devices hide the book, but storage on the WebDAV server
  grows monotonically. Add a TODO at the pushLibraryIndex call with a
  sketch of what a future garbage-collection sweep would need (a per-
  device acknowledgment field on RemoteLibraryIndex so we don't wipe
  data a peer hasn't seen the deletion for yet).

* refactor(webdav): extract WebDAVBrowsePane and SyncHistoryPanel from WebDAVForm

The WebDAV settings form was nearing 1500 lines and hosted three
loosely related surfaces — credential entry, sync controls + manual
trigger, and the in-app file browser — that didn't share much state.
Reviewer flagged it as a refactor candidate; this commit does the
actual split.

* WebDAVBrowsePane (new, 534 lines): owns currentPath, the directory
  listing, per-entry download status, the navigation handlers and the
  per-file icon / filename helpers. Reads credentials from the
  settings prop and otherwise reaches for envConfig / useLibraryStore
  / useAuth itself rather than threading them through props (matches
  how the rest of the integrations panels are wired).

* SyncHistoryPanel (new, 293 lines): the diagnostic history surface
  plus its three private helpers (SyncStatusBadge, formatSyncSummary
  Line, formatSyncTimestamp, SyncHistoryDetails). Moved verbatim from
  the inline definitions at the bottom of WebDAVForm — the component
  was already presentation-only and accepting the translation fn as a
  prop, so no API change.

* WebDAVForm (676 lines, down from 1456): keeps the mode switch
  (configured vs. not), the credential form, the sync sub-controls
  (Upload Book Files / Sync Strategy / Sync now button), and the
  large handleSyncNow effect — those last two are intrinsically tied
  to the settings store and would have just been pushed back up the
  prop chain by any extraction. The standalone SyncHistoryPanel and
  WebDAVBrowsePane are now mounted as siblings inside the configured
  branch.

No behavioural change — both new files run the same effects, build
the same JSX, and read/write the same store fields as before. All
existing webdav-related unit tests still pass.

Resolves the last of the reviewer's smaller observations on
3f721d04 (file length).

* fix(webdav): stream book uploads to avoid renderer OOM on large files

Both syncLibrary (manual Sync now in WebDAVForm) and useWebDAVSync (per-book auto/manual sync triggered on book open) materialised the full book binary as an ArrayBuffer in the V8 heap before PUTting it. With multi-hundred-megabyte PDFs / scanned books, the renderer either accumulates buffers across sequential pushes (library sync) or blows its heap ceiling on a single book (per-book sync), surfacing as a blank white screen on desktop and a binder-OOM kill of the WebView on Android.

Add a BookFileStreamingLoader option to pushBookFile that, on Tauri targets, hands the file path off to tauriUpload's Rust-side streamer so bytes never enter JS. The HEAD short-circuit is shared across both paths, so steady-state syncs still cost a single round-trip per book. Web targets keep the buffered fallback (no streaming HTTP primitive available there).

Wire the streaming loader through SyncLibraryOptions.loadBookFileStreaming for the library Sync now path, and inline it in useWebDAVSync.pushBookFileNow for the per-book path. Covers stay on the buffered loader — they're capped at a few hundred KB and don't justify widening the API.

* fix(webdav): keep Sync now state alive across Settings navigation/close

WebDAVForm tracked the library-wide Sync now run in component state, so any navigation that unmounted the form (drilling back to the Integrations list, or closing the SettingsDialog entirely) destroyed the in-flight indicator while syncLibrary's promise kept running off-thread. On return the user saw a re-enabled button with no progress affordance, an empty Sync History (until the run finally finished), and could trigger a second concurrent syncLibrary against the server.

Hoist isSyncing / progressLabel into a process-local zustand store (webdavSyncStore) and consume it from WebDAVForm. The store outlives any single mount, so re-mounting the form picks up the running sync's state on first render — button stays disabled, progress label keeps ticking, and the re-entrancy gate (now reading the live store rather than a stale closure) blocks duplicate clicks. Also surface 'Syncing…' in the IntegrationsPanel row so users get the cue without drilling into the sub-page.

Not persisted: the store dies with the renderer, which is the right semantic — a sync killed by app exit shouldn't look like it's still going on next launch.

* feat(webdav): cleanup mode for orphan book directories on the server

WebDAV pushes set Book.deletedAt as a tombstone but never DELETE the per-hash directory on the server, so the remote Readest/books/ tree accumulates dead entries from books the user deleted long ago. Add a dedicated cleanup mode in the WebDAV browser to evict them in batch.

Cleanup mode is reached via a new sweep button next to Refresh. Entering it pins the listing to Readest/books/, filters down to directories whose local Book carries deletedAt, and replaces the per-row icon with a checkbox. The footer carries a single right-aligned Delete from server action; selecting one or more rows and clicking it sends a confirm dialog (appService.ask, so it actually blocks on Tauri) and then runs sequential DELETEs against the server. Each row splices out of the listing the moment its DELETE returns, so the listing itself is the progress indicator; the button keeps a stable width by always reserving space for the spinner via the invisible class.

The local library is left untouched. Book.deletedAt is the authoritative deletion signal in readest's sync model — clearing or rewriting it here would cause sibling devices to either resurrect the book or lose the deletion event. Restore is therefore not offered: the per-entry download button already provides full recovery (tauriDownload + ingestFile streams the file back, ingestFile clears deletedAt as a side-effect, and the next sync round-trip merges remote progress and notes), and a metadata-only restore would leave users staring at unopenable shelf rows whenever the bytes had been GCed off local disk.

Browse mode is friendlier too. Per-hash subdirectory rows under Readest/books/ resolve their hash to the local library's title and short-form hash for skimmability; soft-deleted entries get a folder-off icon plus a 60% dimmed title (a redundant signal for touch platforms where the desktop-only hover tooltip doesn't fire). Cleanup runs are persisted into the existing sync history with a kind: 'cleanup' discriminator and a booksDeleted counter, so destructive batch operations are auditable alongside regular Sync now runs without polluting the common case (the new counter is zero-suppressed on plain sync entries).

* test(webdav): cover deleteDirectory and deleteRemoteBookDir

Pin the contract of the cleanup-mode delete plumbing: HTTP method, Depth: infinity header, Authorization header and target URL on the low-level deleteDirectory; success/failure/auth-failure routing and per-hash path construction on the high-level deleteRemoteBookDir. Status-code semantics are exercised end to end (200/204 ok, 404 idempotent, 401/403 AUTH_FAILED, 5xx generic, network throw NETWORK), so a future refactor can't silently drop the explicit Depth header or merge the auth-failure path into the per-book result struct without tripping a regression.

---------

Co-authored-by: Huang Xin <chrox.huang@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 18:55:12 +02:00

79 lines
2.9 KiB
TypeScript

import {
BOOK_CONFIG_SCHEMA_VERSION,
BookConfig,
BookSearchConfig,
ViewSettings,
} from '@/types/book';
export const stampBookConfigSchema = <T extends Partial<BookConfig>>(config: T): T => {
return { ...config, schemaVersion: BOOK_CONFIG_SCHEMA_VERSION };
};
export const serializeRawConfig = (config: Partial<BookConfig>): string => {
return JSON.stringify(stampBookConfigSchema(config));
};
export const serializeConfig = (
config: BookConfig,
globalViewSettings: ViewSettings,
defaultSearchConfig: BookSearchConfig,
): string => {
config = JSON.parse(JSON.stringify(config));
// Tolerate configs that arrive without these fields. Two real-world
// call sites can produce that shape:
// 1. A freshly-initialised config (`INIT_BOOK_CONFIG`) that has
// never been touched by the reader yet.
// 2. The WebDAV sync download path, which merges `{ updatedAt: 0,
// booknotes: [] }` with a remote `compressConfig` payload — the
// latter omits viewSettings/searchConfig entirely when they
// match global defaults.
// Treating null/undefined as `{}` is semantically identical to "no
// overrides vs global", so the reduce below correctly emits an empty
// object that downstream `deserializeConfig` re-hydrates from globals.
const viewSettings = (config.viewSettings ?? {}) as Partial<ViewSettings>;
const searchConfig = (config.searchConfig ?? {}) as Partial<BookSearchConfig>;
config.viewSettings = Object.entries(viewSettings).reduce(
(acc: Partial<Record<keyof ViewSettings, unknown>>, [key, value]) => {
if (globalViewSettings[key as keyof ViewSettings] !== value) {
acc[key as keyof ViewSettings] = value;
}
return acc;
},
{} as Partial<Record<keyof ViewSettings, unknown>>,
) as Partial<ViewSettings>;
config.searchConfig = Object.entries(searchConfig).reduce(
(acc: Partial<Record<keyof BookSearchConfig, unknown>>, [key, value]) => {
if (defaultSearchConfig[key as keyof BookSearchConfig] !== value) {
acc[key as keyof BookSearchConfig] = value;
}
return acc;
},
{} as Partial<BookSearchConfig>,
) as Partial<BookSearchConfig>;
config.schemaVersion = BOOK_CONFIG_SCHEMA_VERSION;
return JSON.stringify(config);
};
export const deserializeConfig = (
str: string,
globalViewSettings: ViewSettings,
defaultSearchConfig: BookSearchConfig,
): BookConfig => {
const config = JSON.parse(str) as BookConfig;
const { viewSettings, searchConfig } = config;
config.viewSettings = { ...globalViewSettings, ...viewSettings };
config.searchConfig = { ...defaultSearchConfig, ...searchConfig };
config.schemaVersion ??= BOOK_CONFIG_SCHEMA_VERSION;
config.updatedAt ??= Date.now();
return config;
};
export const compressConfig = (
config: BookConfig,
globalViewSettings: ViewSettings,
defaultSearchConfig: BookSearchConfig,
): string => {
return JSON.parse(serializeConfig(config, globalViewSettings, defaultSearchConfig));
};