The stats pull keyed the statistics book table by md5 alone, while
KOReader's native statistics plugin keys rows by exact (title, authors,
md5). When the two parsers extract slightly different metadata for the
same file, the first native open creates a second, zeroed book row that
the KOReader UI reads, and the reading time synced from Readest stays
stranded on the sync-created row.
applyRemote now inserts a book row only for an md5 the DB has never
seen, attaches pulled events to the row the native plugin reads (native
rows always set pages and last_open; sync-created rows leave pages
NULL), and folds never-adopted duplicate rows into the surviving row on
every pull, so existing databases heal and the stranded time reappears.
Adopted rows and the live session's cached book id are never deleted,
and the totals recompute no longer regresses last_open below a real
native open timestamp.
Fixes#4861
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(koplugin): repair reading-stats sync push/pull
Reading statistics (statistics.sqlite3 page events) never reached the
Readest sync server. Three stacked defects in the koplugin stats path:
1. push/pull called settings:readSetting/saveSetting on self.settings,
which is the plain readest_sync data table (not a LuaSettings object),
so auto-sync crashed on every book open
("attempt to call method 'readSetting' (a nil value)").
2. pushChanges declares books/notes/configs as required_params, but the
stats push sent only statBooks/statPages, so Spore rejected the request
client-side ("books is required for method pushChanges").
3. statBooks/statPages were listed only under the spec's payload, not as
optional_params. Spore's expected-param set is
required_params + optional_params, so it rejected them too
("statBooks is not expected for method pushChanges").
Fixes:
- Read/persist the cursor as a plain field and save the whole table via
G_reader_settings:saveSetting, mirroring readest_syncauth.
- Send empty books/notes/configs alongside statBooks/statPages; the server
defaults each to [] and processes the stat arrays independently.
- Declare statBooks/statPages as optional_params in readest-sync-api.json.
Also add ReadestStats debug logging across pushBookStats/pullBookStats and
push/pull (cursor, collected counts, dispatch, response status, cursor
advance), and surface the push failure status/body that was previously
swallowed on non-interactive syncs.
Tests: cover push/pull cursor handling, the pushChanges required_params
contract, and a spec-level check that every stats payload key is an
expected pushChanges param.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(koplugin): add interactive Push/Pull stats now menu items
Add "Push stats now" and "Pull stats now" entries to the Readest sync
menu, placed after "Readest library" and before "Push books now" and
gated on being signed in (access_token + user_id) like the other
account-level items. They call pushBookStats(true) / pullBookStats(true)
for a manual interactive sync of reading statistics (the whole
statistics.sqlite3 delta), complementing the automatic pull-on-open /
push-on-close.
Interactive push/pull now also confirm their result (pushed / pulled /
up to date) to match the sibling "now" items, since a silent success
would look like a no-op.
Extract the five new strings into the koplugin .po catalogs (empty
msgstr, English fallback at runtime).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(koplugin): translate reading-stats sync strings (33 locales)
Fill the previously empty msgstr entries for the reading-statistics sync
strings across all koplugin locale catalogs: the two new "Push/Pull stats
now" menu items, the pushed / pulled / up-to-date confirmations, and the
push/pull failure messages. Each locale mirrors its existing
Push/Pull/Failed-to verbs and "reading" terminology for consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deleting a highlight or bookmark in the koplugin never reached the
server. SyncAnnotations:push builds its payload from getAnnotations,
which walks only the live ui.annotation.annotations list, so a deleted
note can never appear in a push — the server kept the row and the next
pull resurrected it. The pull direction (server deletions → koplugin)
was already handled by removeDeletedAnnotations (#4119); this is the
missing push direction.
Capture a tombstone at deletion time instead. onAnnotationsModified
detects a removal (negative index_modified, with the deleted item at
items[1]) and records a deletedAt-stamped descriptor in the per-book
sidecar (readest_sync.deleted_notes). push folds those tombstones into
the payload and clears them only once the server accepts them, so a
failed push retries. Extracted buildNoteDescriptor so the deletion path
derives a note's id identically to the push walk.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes #3156. Adds a reading-statistics system whose canonical data model
is KOReader's own (book + page_stat_data), so stats round-trip losslessly
between Readest and KOReader.
- Storage: a cross-platform Turso statistics.db in KOReader's exact schema
(web/Workers, desktop, iOS, Android) — replacing #3156's Node-only
better-sqlite3 + statistics.json.
- Tracking: per-page reading events (time-on-page, idle-capped) flushed on
page-change/idle/hide/close — the KOReader model — not session aggregates.
- Sync: legacy /api/sync extended with a stats type backed by self-contained
Supabase tables (stat_books, stat_pages); union/longer-duration-wins merge
keyed on book_hash. apps/readest.koplugin syncs through the same endpoint.
- Scale & robustness: per-tab singleton connection (avoids OPFS lock
conflicts) + explicit WAL checkpoint; transactional bulk apply; chunked
resumable push; client-driven paged pull with trailing-ms completion;
paginated/scoped server merge.
Verified: 5668 unit tests, 155 koplugin busted tests, biome+tsgo + luacheck
all green; web OPFS DB verified live.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(koplugin): pull before push so sync doesn't wipe cloud book fields, closes#4138
The Library sync pushed a touched book row before pulling, so a row
still missing the cloud's uploaded_at / metadata / group_id (e.g. one
created by lightScan, not yet merged from a cloud pull) was sent with
those fields nil. The server's transformBookToDB explicit-nulls
uploaded_at and metadata for any field absent from the wire payload,
wiping the cloud copy — after which every device that pulled lost the
book's upload state.
syncBooks("both") now pulls first, then pushes, and takes a before_push
callback. syncBooksLibrary passes touchOpenBook through it so the
open book's updated_at bump lands after the pull has refreshed the
local row, letting the push carry the preserved cloud fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(koplugin): hide Library books with neither an uploaded nor a local file
The Library showed any row with cloud_present = 1, but a bare cloud
*record* whose file was never uploaded (uploaded_at NULL) has no cover
and can't be opened — showing it is meaningless. Tighten the visibility
predicate to (uploaded_at IS NOT NULL OR local_present = 1) across
listBooks, getGroups, listBookshelfGroups and listBooksInGroup.
This mirrors Readest, which only adds a synced book to the library when
uploadedAt is set and keeps locally-imported books that carry a
downloadedAt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(koplugin): close the Library widget when opening a book
Opening a book from the Library called ReaderUI:showReader without
closing the Library Menu, so it stayed in the UIManager widget stack
with M._menu still set. A later background M.refresh() — a cloud-sync
or cover-download completion — then repainted that ghost Library over
the reader, making it flash on screen for a few seconds.
Add M.close(); route the title-bar X, M.reopen() and both handleTap
book-open paths through it. A wrapped onCloseWidget clears M._menu on
every close path, so M.refresh() no-ops once the Menu is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(koplugin/library): data layer + busted harness + design doc
- LibraryStore: per-user SQLite index merging cloud + local books by
partial-md5 hash. listBooks/listBookshelfBooks/listBookshelfGroups,
upsertBook with cloud_present/local_present OR-merge + _force/clear
sentinels, parseSyncRow, getChangedBooks for tombstone push.
- EXTS table mirroring web's document.ts.
- busted harness with KOReader stubs (G_reader_settings, DataStorage,
lua-ljsqlite3 against :memory:); spec_helper, exts_spec, smoke_spec,
librarystore_spec covering schema, sort, group nesting, dedupe.
- Library design doc + spec README.
- pnpm test:lua wired through root + app package.json; lint-koplugin
recurses into library/ + spec/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(koplugin/library): cloud sync — push, pull, upload, delete, downloads
- Spore methods: pullBooks (incremental /sync), getDownloadUrl,
getUploadUrl, listFiles, deleteFile.
- syncbooks.lua: pushBook + pushChangedBooks (advances watermark to
max(updated_at, deleted_at)), syncBooks(opts, mode) for push/pull/both,
downloadBook + downloadCover (sync socket.http with file sink; cover
download via fork+poll with single-slot queue + visible-page filter +
coalesced refresh), uploadBook (presigned-PUT flow + best-effort
cover), deleteCloudFiles (list-then-delete-each, mirrors
cloudService.deleteBook).
- SyncAuth.withFreshToken wrapper resolves the ensureClient race; 401/403
unified across syncconfig + syncannotations.
- Cloud + local book covers shared by partial-md5 hash; cover.png cached
at <settings>/readest_covers/<hash>.png with sentinel for known 404s.
- syncbooks_spec covers row-to-wire conversion + file_key shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(koplugin/library): local discovery — sidecar scanner + cover provider
- localscanner.lightScan iterates ReadHistory entries, reads
partial_md5_checksum from .sdr sidecars, and upserts local rows.
Slow filesystem walks deferred to fullSidecarWalk (24h-gated).
- coverprovider wraps BookInfoManager:getBookInfo for local books with
graceful FakeCover fallback when coverbrowser is absent.
- localscanner_spec + coverprovider_spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(koplugin/library): UI — widget, item, view menu, FileManager hooks
- librarywidget: full-screen Menu mixed in with CoverMenu + Mosaic/List
per zen_ui's group_view pattern. Title-bar tap → view menu, search via
left icon, drill-in/back for grouping. Async cloud sync deferred via
scheduleIn so the menu paints before HTTP fires.
- libraryitem: BIM patch for cloud-only (readest-cloud://) and group
(readest-group://) URIs; group-cover composer (2x2 mosaic for grid,
same in list) with cache key derived from the actual first-N hashes
for content-based invalidation; ListMenuItem update + paintTo patches
for wider list-mode cover strip and cloud-up/down icon overlay.
- libraryviewmenu: ButtonDialog with View/Group by/Sort by/Actions.
Default Group by = Groups (parity with web), values authors/groups.
- librarypaint: partial-page e-ink repaint shim adapted from zen_ui.
- main.lua: Library menu entry, dispatcher actions (Open Library / Push
/ Pull as general; progress + annotations as reader-only),
"Add to Readest" button in FileManager's long-press file dialog
(dedupe by partial_md5; bumps updated_at when present, inserts a
fresh local-only row otherwise; un-tombstones via _clear_fields).
Push books on Library open when auto_sync is on, pull-only otherwise.
- Long-press action sheet with Readest BookDetailView parity:
Remove from Cloud & Device / Cloud Only / Device Only,
Upload to Cloud, Download Book / Cover / All.
- Cloud-down + cloud-up SVG icons (LiaCloudDownloadAltSolid /
LiaCloudUploadAltSolid) painted in the right-side wpageinfo slot.
- i18n catalog updated for new strings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(koplugin/library): split libraryitem into focused modules
libraryitem.lua had grown to 1018 lines mixing five unrelated
concerns. Split along the natural seams:
cloud_covers — readest-cloud:// URI scheme, on-disk <hash>.png
cache, single-slot async download queue, visible-page
filter
group_covers — readest-group:// URI scheme, 2x2 mosaic composer with
content-derived cache key (first-N hashes), cell
layout table
cloud_icons — bundled cloud-up/cloud-down SVG loader, IconWidget
cache, paint-overlay positioning
list_strip — list-mode group row builder (4-cover wider strip
replacing ListMenu's square cover slot)
bim_patch — BookInfoManager:getBookInfo router (cloud / group /
local) + ListMenuItem update + paintTo patches; owns
the _library_local_paths set and orig BIM reference
libraryitem.lua is now 141 lines: just the entry-table constructors
(entry_from_row, entry_from_group, entry_back) plus thin install /
set_visible_hashes delegates. Each new module is 88-216 lines.
No behavior change — same 113 specs pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(koplugin): co-locate dev tooling, ship-zip exclusions, CI job split
- apps/readest.koplugin/scripts/build-koplugin.mjs: local sideloading
build with the same exclusions the release workflow uses.
- Move lint-koplugin + test-koplugin from apps/readest-app/scripts/ to
apps/readest.koplugin/scripts/. All koplugin dev tooling now lives
with the koplugin and is excluded from the published release zip.
- Rename to .mjs so Node treats them as ESM without the reparse warning
(the i18n CommonJS scripts stay .js).
- Release workflow: zip -r exclusions for scripts/, docs/, spec/,
.busted so dev artifacts don't ship to end users.
- PR workflow: split build_web_app into build_web_app + test_web_app
for parallelism. The test job installs luarocks + busted +
lsqlite3complete and runs pnpm test:lua. test-koplugin.mjs now
hard-fails (instead of soft-skipping) when CI=true and a tool is
missing — a broken CI toolchain previously exited 0 silently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- i18n loader at apps/readest.koplugin/i18n.lua: isolated callable,
falls back to KOReader's gettext when a string is untranslated
- Translation catalog at locales/<i18next-code>/translation.po for 31
languages, mirroring apps/readest-app/public/locales/
- scripts/extract-i18n.js (scan _("...") and _([[...]]); preserve
existing, drop obsolete, add new) and scripts/apply-translations.js
(bulk import from /tmp/koplugin-translations/<lang>.json)
- Mirror apps/readest-app SyncInfoDialog: rename showMetaHashInfo to
showSyncInfo, dialog title "Sync Info", new Last Synced row computed
as max(last_synced_at_config, last_synced_at_notes) from doc_settings
- syncconfig.lua / syncannotations.lua mark per-book sync timestamps
on push/pull success
- Rename "Meta Hash" -> "Book Fingerprint" in koplugin and
apps/readest-app SyncInfoDialog.tsx; translations propagated to
all readest-app locales
- "book config" -> "reading progress" wording across user-facing
strings (matches QiuYukang fork terminology)
- Replace "Log out as " / "Login failed: " concat prefixes with
T(_("...%1..."), arg) placeholder pattern (RTL / verb-final friendly)
- pnpm lint:lua: luajit -b syntax check across koplugin .lua files;
soft-skips when luajit is missing locally; CI installs luajit and
runs the check unconditionally
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the hook introduced by koreader/koreader#15240, called when
the user deletes the plugin via the plugin manager with "Also delete
plugin settings" checked. Removes the readest_sync entry from
G_reader_settings (auth tokens, user info, auto_sync flag, last_sync_at)
and resets the in-memory copy to defaults.
Closes#4039
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add "Full sync all annotations" menu item that pushes and pulls all
annotations regardless of the last sync timestamp, enabling users to
sync old highlights that were created before the plugin was installed.
Changes:
- Add full_sync parameter to push/pull that bypasses timestamp filter
and uses since=0 for pulling all server annotations
- Deduplicate by annotation ID alongside position-based dedup
- Store server ID on pulled annotations and reuse it when pushing
- Parse ISO 8601 timestamps from server to preserve original
created/updated dates instead of using current time
- Resolve KOReader page numbers from xpointers via getPageFromXPointer
- Resolve Readest page numbers from CFI via getCFIProgress on pull
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add bidirectional annotation/highlight sync between KOReader and Readest:
- Add xpointer0/xpointer1 fields to BookNote and DBBookNote types for
KOReader XPointer positions alongside Readest's CFI format
- Extend transform layer to pass through xpointer fields to/from DB
- Convert CFI→XPointer on push and XPointer→CFI on pull in useNotesSync,
discarding notes that fail conversion
- Support KOReader's text()[K].N indexed text node format in xcfi.ts for
paragraphs with inline elements (e.g. <a> page anchors)
- Generate KOReader-compatible XPointers: text().N for single text nodes,
text()[K].N only when multiple direct text nodes exist
- Skip cfi-inert elements (injected by Readest at runtime) in XPointer
path building and resolution
- Map highlight colors between KOReader and Readest color systems
- Implement KOReader plugin annotation push/pull with deterministic IDs,
auto-sync on document open/close, and UIManager refresh on pull
- Refactor koplugin into focused modules: syncauth, syncconfig,
syncannotations, selfupdate
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Huang Xin