forked from akai/readest
23d1ef6f1328d7417fab6beffc178a00a4ed458b
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Huang Xin
|
7810981417 |
fix(koplugin): repair reading-stats sync push/pull (#4666)
* fix(koplugin): repair reading-stats sync push/pull
Reading statistics (statistics.sqlite3 page events) never reached the
Readest sync server. Three stacked defects in the koplugin stats path:
1. push/pull called settings:readSetting/saveSetting on self.settings,
which is the plain readest_sync data table (not a LuaSettings object),
so auto-sync crashed on every book open
("attempt to call method 'readSetting' (a nil value)").
2. pushChanges declares books/notes/configs as required_params, but the
stats push sent only statBooks/statPages, so Spore rejected the request
client-side ("books is required for method pushChanges").
3. statBooks/statPages were listed only under the spec's payload, not as
optional_params. Spore's expected-param set is
required_params + optional_params, so it rejected them too
("statBooks is not expected for method pushChanges").
Fixes:
- Read/persist the cursor as a plain field and save the whole table via
G_reader_settings:saveSetting, mirroring readest_syncauth.
- Send empty books/notes/configs alongside statBooks/statPages; the server
defaults each to [] and processes the stat arrays independently.
- Declare statBooks/statPages as optional_params in readest-sync-api.json.
Also add ReadestStats debug logging across pushBookStats/pullBookStats and
push/pull (cursor, collected counts, dispatch, response status, cursor
advance), and surface the push failure status/body that was previously
swallowed on non-interactive syncs.
Tests: cover push/pull cursor handling, the pushChanges required_params
contract, and a spec-level check that every stats payload key is an
expected pushChanges param.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(koplugin): add interactive Push/Pull stats now menu items
Add "Push stats now" and "Pull stats now" entries to the Readest sync
menu, placed after "Readest library" and before "Push books now" and
gated on being signed in (access_token + user_id) like the other
account-level items. They call pushBookStats(true) / pullBookStats(true)
for a manual interactive sync of reading statistics (the whole
statistics.sqlite3 delta), complementing the automatic pull-on-open /
push-on-close.
Interactive push/pull now also confirm their result (pushed / pulled /
up to date) to match the sibling "now" items, since a silent success
would look like a no-op.
Extract the five new strings into the koplugin .po catalogs (empty
msgstr, English fallback at runtime).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(koplugin): translate reading-stats sync strings (33 locales)
Fill the previously empty msgstr entries for the reading-statistics sync
strings across all koplugin locale catalogs: the two new "Push/Pull stats
now" menu items, the pushed / pulled / up-to-date confirmations, and the
push/pull failure messages. Each locale mirrors its existing
Push/Pull/Failed-to verbs and "reading" terminology for consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Huang Xin
|
72233e1c6a |
feat: sync reading status across devices and with KOReader (#4634) (#4656)
* docs(sync): design spec for syncing reading status (#4634) Field-level LWW for reading_status (dedicated reading_status_updated_at), a new 'abandoned' status in the Readest UI, and a koplugin bridge to KOReader's native summary.status (whole-library apply + capture). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(sync): implementation plan for syncing reading status (#4634) Bite-sized TDD tasks across 3 parts: A) cloud field-level LWW (reading_status_updated_at on server upsert + client pull-merge), B) 'abandoned'/On-hold status in the Readest UI, C) koplugin bridge to KOReader summary.status (mapping + reconcile + whole-library apply/capture). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): add reading_status_updated_at for field-level status LWW (#4634) * feat(sync): stamp readingStatusUpdatedAt on status change in updateBookProgress Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): stamp status timestamp on explicit library status edits * feat(sync): resolve reading status by its own timestamp in client pull-merge * feat(sync): resolve reading status by its own timestamp in server upsert (#4634) * fix(sync): tighten reading-status merge typing + strengthen test (A5 review) Replace as-unknown-as double-casts at read sites with typed locals (clientBook/serverBook); retain a single as-unknown-as only at the server-wins construction site where the static type is too narrow. Strengthen test 3 to assert both fields with toEqual. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(library): render the 'On hold' (abandoned) status badge * feat(library): add 'Mark as On hold' actions + i18n for abandoned status Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * i18n: translate 'On hold' and 'Mark as On hold' for the abandoned status (#4634) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(koplugin): add reading-status mapping + reconcile between Readest and KOReader * feat(koplugin): persist + sync reading_status_updated_at in LibraryStore Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(koplugin): bridge reading status to KOReader summary.status on library sync (#4634) * test(sync): cover koplugin v1->v2 migration + tighten status-sync tests (final review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(sync): redesign KOReader first-sync (decisive-only + bootstrap) (#4634) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(koplugin): safe first-sync of reading status (decisive-only + bootstrap) (#4634) KOReader auto-sets summary.status='reading' on first open, and legacy Readest statuses have reading_status_updated_at=0, so pure timestamp LWW let opening a finished book downgrade it. Restrict sync to deliberate statuses (finished/ complete, abandoned/on-hold, unread->clear); never capture KO 'reading'/'New'. On the unsynced baseline (Readest ts=0) conflicts resolve Readest-authoritative, then stamp now_ms to exit bootstrap into steady-state LWW. reconcile now returns write_ko/write_store flags; statussync captures now_ms once and equalizes both sides (convergent, idempotent, resumable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(koplugin): cover remaining first-sync graph cells + document sort effect (review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
9d0c5dc524 |
fix(koplugin): sync note deletions to Readest via tombstones (#4632)
Deleting a highlight or bookmark in the koplugin never reached the server. SyncAnnotations:push builds its payload from getAnnotations, which walks only the live ui.annotation.annotations list, so a deleted note can never appear in a push — the server kept the row and the next pull resurrected it. The pull direction (server deletions → koplugin) was already handled by removeDeletedAnnotations (#4119); this is the missing push direction. Capture a tombstone at deletion time instead. onAnnotationsModified detects a removal (negative index_modified, with the deleted item at items[1]) and records a deletedAt-stamped descriptor in the per-book sidecar (readest_sync.deleted_notes). push folds those tombstones into the payload and clears them only once the server accepts them, so a failed push retries. Extracted buildNoteDescriptor so the deletion path derives a note's id identically to the push walk. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
35b02c4efc |
feat(statistics): KOReader-compatible reading stats with cross-device sync (#4606)
Supersedes #3156. Adds a reading-statistics system whose canonical data model is KOReader's own (book + page_stat_data), so stats round-trip losslessly between Readest and KOReader. - Storage: a cross-platform Turso statistics.db in KOReader's exact schema (web/Workers, desktop, iOS, Android) — replacing #3156's Node-only better-sqlite3 + statistics.json. - Tracking: per-page reading events (time-on-page, idle-capped) flushed on page-change/idle/hide/close — the KOReader model — not session aggregates. - Sync: legacy /api/sync extended with a stats type backed by self-contained Supabase tables (stat_books, stat_pages); union/longer-duration-wins merge keyed on book_hash. apps/readest.koplugin syncs through the same endpoint. - Scale & robustness: per-tab singleton connection (avoids OPFS lock conflicts) + explicit WAL checkpoint; transactional bulk apply; chunked resumable push; client-driven paged pull with trailing-ms completion; paginated/scoped server merge. Verified: 5668 unit tests, 155 koplugin busted tests, biome+tsgo + luacheck all green; web OPFS DB verified live. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
a92fe0ce2e |
fix(koplugin): upload local book cover so synced books show a cover (#4374) (#4385)
uploadBook only attached a cover.png when one was already cached under readest_covers/<hash>.png from a prior cloud download. Books that originated locally in KOReader were never downloaded, so the cover step was silently skipped and they synced to Readest with no cover. Add extractLocalCover, which renders the book's embedded cover via coverbrowser's FileManagerBookInfo:getCoverImage(nil, file_path) and writes it as PNG. uploadBook now falls back to it when no cached cover exists, caching the result under covers_dir so the Library view reuses it like a downloaded cover. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
f2a2d96938 |
fix(koplugin): honor remote annotation deletions, closes #4119 (#4194)
Pull skipped notes carrying a deleted_at tombstone but never removed the matching local annotation. A highlight deleted on Readest therefore lingered in KOReader, and a later push (notably a full sync) re-uploaded it, resurrecting the note on the server and making it reappear on every device. Add removeDeletedAnnotations, invoked at the start of the pull callback, to drop local annotations the server has tombstoned. Tombstones are matched by stored id, by the hash-derived id for native KOReader highlights, or by position/page xpointer. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
ab2def32dd |
fix(koplugin): stop sync from wiping cloud book fields + Library polish (#4166)
* fix(koplugin): pull before push so sync doesn't wipe cloud book fields, closes #4138 The Library sync pushed a touched book row before pulling, so a row still missing the cloud's uploaded_at / metadata / group_id (e.g. one created by lightScan, not yet merged from a cloud pull) was sent with those fields nil. The server's transformBookToDB explicit-nulls uploaded_at and metadata for any field absent from the wire payload, wiping the cloud copy — after which every device that pulled lost the book's upload state. syncBooks("both") now pulls first, then pushes, and takes a before_push callback. syncBooksLibrary passes touchOpenBook through it so the open book's updated_at bump lands after the pull has refreshed the local row, letting the push carry the preserved cloud fields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(koplugin): hide Library books with neither an uploaded nor a local file The Library showed any row with cloud_present = 1, but a bare cloud *record* whose file was never uploaded (uploaded_at NULL) has no cover and can't be opened — showing it is meaningless. Tighten the visibility predicate to (uploaded_at IS NOT NULL OR local_present = 1) across listBooks, getGroups, listBookshelfGroups and listBooksInGroup. This mirrors Readest, which only adds a synced book to the library when uploadedAt is set and keeps locally-imported books that carry a downloadedAt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(koplugin): close the Library widget when opening a book Opening a book from the Library called ReaderUI:showReader without closing the Library Menu, so it stayed in the UIManager widget stack with M._menu still set. A later background M.refresh() — a cloud-sync or cover-download completion — then repainted that ghost Library over the reader, making it flash on screen for a few seconds. Add M.close(); route the title-bar X, M.reopen() and both handleTap book-open paths through it. A wrapped onCloseWidget clears M._menu on every close path, so M.refresh() no-ops once the Menu is gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
15c0a7a2f2 |
fix(koplugin): render group cover previews in Library (#4064)
Group cells in the Readest Library view rendered as FakeCover even when their child books had perfectly good cloud covers — the queue that fetches <hash>.png covers was only primed for cloud-only book entries on the visible page, so children of group entries were never requested. A later partial composite (3/4 covers) was also written to a content-fingerprinted disk cache and kept serving forever, since the fingerprint stayed the same after the 4th cover landed. Two fixes wired together: - libraryitem.set_visible_hashes now expands visible group entries to include their first-N children's hashes, so trigger_download's visibility filter no longer rejects them. - group_covers.child_cover_bb queues a cloud-cover download when the fallback path misses for a cloud-present book. Capped at 4 per group by the existing cells_for(shape) limit. Disk caching of composites is dropped entirely; mosaics are recomposed in memory each paint. New spec/library/group_covers_spec.lua locks the contract for URI round-trip, child_cover_bb's missing-cover branches, and libraryitem's group-children expansion. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Huang Xin
|
ed8956e9ed |
feat(koplugin): Readest Library view in KOReader (#4056)
* feat(koplugin/library): data layer + busted harness + design doc - LibraryStore: per-user SQLite index merging cloud + local books by partial-md5 hash. listBooks/listBookshelfBooks/listBookshelfGroups, upsertBook with cloud_present/local_present OR-merge + _force/clear sentinels, parseSyncRow, getChangedBooks for tombstone push. - EXTS table mirroring web's document.ts. - busted harness with KOReader stubs (G_reader_settings, DataStorage, lua-ljsqlite3 against :memory:); spec_helper, exts_spec, smoke_spec, librarystore_spec covering schema, sort, group nesting, dedupe. - Library design doc + spec README. - pnpm test:lua wired through root + app package.json; lint-koplugin recurses into library/ + spec/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): cloud sync — push, pull, upload, delete, downloads - Spore methods: pullBooks (incremental /sync), getDownloadUrl, getUploadUrl, listFiles, deleteFile. - syncbooks.lua: pushBook + pushChangedBooks (advances watermark to max(updated_at, deleted_at)), syncBooks(opts, mode) for push/pull/both, downloadBook + downloadCover (sync socket.http with file sink; cover download via fork+poll with single-slot queue + visible-page filter + coalesced refresh), uploadBook (presigned-PUT flow + best-effort cover), deleteCloudFiles (list-then-delete-each, mirrors cloudService.deleteBook). - SyncAuth.withFreshToken wrapper resolves the ensureClient race; 401/403 unified across syncconfig + syncannotations. - Cloud + local book covers shared by partial-md5 hash; cover.png cached at <settings>/readest_covers/<hash>.png with sentinel for known 404s. - syncbooks_spec covers row-to-wire conversion + file_key shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): local discovery — sidecar scanner + cover provider - localscanner.lightScan iterates ReadHistory entries, reads partial_md5_checksum from .sdr sidecars, and upserts local rows. Slow filesystem walks deferred to fullSidecarWalk (24h-gated). - coverprovider wraps BookInfoManager:getBookInfo for local books with graceful FakeCover fallback when coverbrowser is absent. - localscanner_spec + coverprovider_spec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): UI — widget, item, view menu, FileManager hooks - librarywidget: full-screen Menu mixed in with CoverMenu + Mosaic/List per zen_ui's group_view pattern. Title-bar tap → view menu, search via left icon, drill-in/back for grouping. Async cloud sync deferred via scheduleIn so the menu paints before HTTP fires. - libraryitem: BIM patch for cloud-only (readest-cloud://) and group (readest-group://) URIs; group-cover composer (2x2 mosaic for grid, same in list) with cache key derived from the actual first-N hashes for content-based invalidation; ListMenuItem update + paintTo patches for wider list-mode cover strip and cloud-up/down icon overlay. - libraryviewmenu: ButtonDialog with View/Group by/Sort by/Actions. Default Group by = Groups (parity with web), values authors/groups. - librarypaint: partial-page e-ink repaint shim adapted from zen_ui. - main.lua: Library menu entry, dispatcher actions (Open Library / Push / Pull as general; progress + annotations as reader-only), "Add to Readest" button in FileManager's long-press file dialog (dedupe by partial_md5; bumps updated_at when present, inserts a fresh local-only row otherwise; un-tombstones via _clear_fields). Push books on Library open when auto_sync is on, pull-only otherwise. - Long-press action sheet with Readest BookDetailView parity: Remove from Cloud & Device / Cloud Only / Device Only, Upload to Cloud, Download Book / Cover / All. - Cloud-down + cloud-up SVG icons (LiaCloudDownloadAltSolid / LiaCloudUploadAltSolid) painted in the right-side wpageinfo slot. - i18n catalog updated for new strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(koplugin/library): split libraryitem into focused modules libraryitem.lua had grown to 1018 lines mixing five unrelated concerns. Split along the natural seams: cloud_covers — readest-cloud:// URI scheme, on-disk <hash>.png cache, single-slot async download queue, visible-page filter group_covers — readest-group:// URI scheme, 2x2 mosaic composer with content-derived cache key (first-N hashes), cell layout table cloud_icons — bundled cloud-up/cloud-down SVG loader, IconWidget cache, paint-overlay positioning list_strip — list-mode group row builder (4-cover wider strip replacing ListMenu's square cover slot) bim_patch — BookInfoManager:getBookInfo router (cloud / group / local) + ListMenuItem update + paintTo patches; owns the _library_local_paths set and orig BIM reference libraryitem.lua is now 141 lines: just the entry-table constructors (entry_from_row, entry_from_group, entry_back) plus thin install / set_visible_hashes delegates. Each new module is 88-216 lines. No behavior change — same 113 specs pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(koplugin): co-locate dev tooling, ship-zip exclusions, CI job split - apps/readest.koplugin/scripts/build-koplugin.mjs: local sideloading build with the same exclusions the release workflow uses. - Move lint-koplugin + test-koplugin from apps/readest-app/scripts/ to apps/readest.koplugin/scripts/. All koplugin dev tooling now lives with the koplugin and is excluded from the published release zip. - Rename to .mjs so Node treats them as ESM without the reparse warning (the i18n CommonJS scripts stay .js). - Release workflow: zip -r exclusions for scripts/, docs/, spec/, .busted so dev artifacts don't ship to end users. - PR workflow: split build_web_app into build_web_app + test_web_app for parallelism. The test job installs luarocks + busted + lsqlite3complete and runs pnpm test:lua. test-koplugin.mjs now hard-fails (instead of soft-skipping) when CI=true and a tool is missing — a broken CI toolchain previously exited 0 silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |