Huang Xin
ccb937015d
* fix(sync): record remote-present files for no-source books in the upload cursor With "Upload Book Files" on, a device that holds no local copy of a book (e.g. the web app with a cloud-only library) HEAD-probed the remote for every book on every sync: pushBookFile returned 'no-source' and the hash was never recorded in library.json's uploadedHashes, so needsFilePush stayed true for the whole library and each run (tab focus, Sync Now, library change) issued one Drive/WebDAV request per book, 646 requests per sweep in the reported case. The HEAD probe already answers whether the file is on the remote. Carry that in PushBookFileResult.remoteExists and record the hash when a no-source book's file is already mirrored, so the next incremental sync skips it and stays O(changed). Books absent both locally and remotely stay unrecorded so a device that has the bytes can upload them later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): reach the no-source verdict without probing the remote A book file sync with "Upload Book Files" on probed the remote for every book before checking whether this device even holds the bytes. On a device with a cloud-only library (the web app), none of the books have a local source, so every sync run (tab focus, Sync Now, library change) issued one name-lookup request per book against Google Drive, a full per-book request storm that never converged: with no local file there is nothing to upload and nothing to record, so the next run repeated it. Resolve the local source first and return 'no-source' from local state alone; the remote head probe now runs only when there is a local file to compare or upload. The probe keeps its non-NETWORK rethrow semantics so the auth-failure latch (#4981) still stops a run on an expired session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sync): incremental file sync and per-book transfers for the active provider Cut the redundant remote work a file-sync run does and route explicit per-book uploads/downloads to the active third-party provider (WebDAV / Google Drive) instead of the gated Readest Cloud transfer queue. Engine (services/sync/file/engine.ts, wire.ts): - etag change-probe: one HEAD on library.json, cached per provider for the session. When the etag matches the last successful pull, reuse the cached index and skip both the index download and the discovery scan. An AUTH failure on the probe aborts the run like the full pull does. - emptyDirs memo carried in the index: dirs found to hold no book file are recorded so clients stop re-listing them every run. Re-checked when the file arrives (uploadedHashes), on Full Sync, or when a legacy client drops the record; pruned only against a listing that actually ran. - skip the index re-push when the rebuilt index is semantically identical to the pulled one, so a restamped byte-copy no longer churns the remote and invalidates peers' etag change detection. - downloadBookFile() for the explicit per-book Download action. Provider reuse (services/sync/file/providerRegistry.ts): - memoise one provider per connection key, shared by every surface (reader per-book sync, library auto-sync, Sync now / pull to refresh). Reuses the Drive path->id cache instead of re-resolving /Readest, books/ and library.json by name query on every engine build. Drive connect/disconnect resets the cache since its token source changes identity with no key input changing. Google Drive (services/sync/providers/gdrive/GoogleDriveProvider.ts): - write fast-path: PATCH a path whose id is cached in place with no lookup, falling back to a full resolve on a 404 (stale id). Removes a files.list per PUT in the steady state. - dev-only request diagnostics: one line per provider op and per HTTP attempt so a run's request budget can be attributed from the console. Per-book transfers (services/sync/file/runLibrarySync.ts): - runActiveFileBookUpload / runActiveFileBookDownload build the active provider's engine and push/pull a single book, stamping downloadedAt like the native path. Wired into the reader/library book actions with toasts. UI, status, and i18n: - Readest Cloud sub-page: drop the quota stats and wrap the "Account and Storage" row in the BoxedList primitive so it aligns to the design system. - Shorten provider status/toast copy ("Active", "Google Drive session expired", "Library sync via {{provider}}", "KOReader") and translate the new keys across all locales. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3.3 KiB
name, description, metadata
| name | description | metadata | ||||||
|---|---|---|---|---|---|---|---|---|
| koplugin-library-open-mosaic-cache-4954 | koplugin Library slow open on large libraries — group-cover mosaics recomposed every paint; fixed by availability-keyed cache + async compose |
|
Issue #4954 (PR #4974, MERGED 2026-07-07): opening the KOReader plugin Library was slow on large libraries (~1000 books) while navigation stayed fast.
Root cause (measured, not guessed). Added open-path timing instrumentation
(ui/time + elapsed_ms helper) to library/librarywidget.lua (initial
build_item_table, lightScan, post-scan refresh, total synchronous open,
cloud-sync elapsed) and a step breakdown in library/localscanner.lua. On a
685-book library the synchronous open was ~300ms, dominated by a 254ms
post-scan refresh = library/group_covers.lua recomposing each folder's 2x2
cover mosaic from scratch on every paint (up to 4 MuPDF decodes+scales per
cell), with no cache, and again on the post-sync refresh. build_item_table
(7ms) and lightScan (28ms, only 16 sidecar reads) were NOT the bottleneck —
my initial hypotheses (defer lightScan / incremental history) were refuted by
the log. Why "slow to load, fast to navigate": the root Groups view is mosaics;
drilling into a group shows single covers (cheap, BIM-cached). Soft-scales with
size (fuller groups → 4 covers/mosaic vs 1). Data-side pagination is NOT
possible (KOReader Menu derives page count from #item_table).
Fix (mirror cloud_covers async pattern in group_covers).
- Cache composed master bb per group, keyed by
mosaic_cache_key= ordered child hashes + a per-child cover-availability bit (child_cover_available→cloud_covers.cover_exists(hash)or local file stat). Servecopy_bbon hit. The availability bit fixes the historical "partial composite served forever" bug that killed the prior on-disk cache: a late cover flips the key and recomposes once. - Cache the
nilresult too (critical): a coverless group whose children aren't downloaded makescomposereturn nil; if not cached it re-enqueues +schedule_refreshon every refresh → infinite recompose/refresh loop (eink flashing). Caught this in the second emulator log (group_nameLanguagegridmissing every refresh). Cache nil under the availability key → placeholder served, no re-enqueue. - Compose off first-paint: miss enqueues a single-slot background job (one
mosaic per UI
nextTick,_pump_scheduledcoalesces), returns nil so the cell paints its FakeCover placeholder; completions coalesce into one refresh. clear_cache()on Library close (vialibraryitem.set_visible_hashes(nil)) frees masters (~0.7MB each).
Result: synchronous open 300ms→151ms, post-scan refresh 254ms→89ms (now just the 4 visible single cloud-book cover decodes + placeholders, mosaics deferred).
Left out (follow-ups noted in PR): single cloud-book covers
(cloud_covers.load_cover_bb) still re-decode from disk each refresh (~89ms/4)
— same copy-on-serve cache could apply; deferred cloud sync uses synchronous
HTTP that briefly freezes UI after the menu appears (elapsed 1.6-6.8s, network
variance). Instrumentation kept intentionally (Library open is infrequent).
See koplugin-stats-duplicate-book-rows-4861, koplugin-library-stale-synced-cursor-4934.