* feat(sync): decouple the incremental-pull cursor from updated_at (#4678)
`books.updated_at` was overloaded as both the incremental-pull cursor
(`GET /api/sync?since=…` filters `updated_at > since`, devices keep one
global `max(updated_at)` watermark) and the library "date read" sort key.
A server-resolved reading-status merge had to be written with a timestamp
greater than every peer's global cursor to propagate, which forced
`updated_at = now()` and reordered the date-read library by sync-processing
time (the #4677 symptom).
Introduce a server-assigned `synced_at` column on `books`, stamped by a
`BEFORE INSERT OR UPDATE` trigger on every write, used only as the pull
cursor. `updated_at` stays pure client event time used only for sorting.
- Migration 016 + baseline schema: add `synced_at` (NOT NULL DEFAULT now()),
index `(user_id, synced_at)`, trigger `set_books_synced_at`. Backfill
`synced_at = updated_at` before creating the trigger so existing devices'
cursors hand over without a re-sync storm.
- GET: books filters/orders on `synced_at > since` (a delete bumps synced_at,
so the deleted_at clause is dropped); configs/notes stay on updated_at.
- POST: extract `buildStatusPropagationRow` and drop the `updated_at = now()`
bump — the trigger advances synced_at so peers re-pull the status change
while updated_at (the sort key) stays put.
- Client `computeMaxTimestamp` keys on synced_at, falling back to
updated_at/deleted_at for pre-migration servers and configs/notes.
Backward-compatible: `synced_at >= updated_at` always, so `synced_at > since`
is a strict superset of `updated_at > since` — old web clients and the
koplugin keep working with no data loss (at worst a redundant idempotent
re-pull of rare server-merged rows). The koplugin's shared pull/push cursor
is left untouched; a proper split is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sync): make the books synced_at backfill safe for large live tables (#4678)
The single `UPDATE … WHERE synced_at IS NULL` deadlocked on a 3.8M-row
production `books` table: it rewrites every row in one transaction while the
live /api/sync push path upserts books rows, and the two lock rows in opposite
orders. `ALTER COLUMN … SET NOT NULL` (full-table ACCESS EXCLUSIVE scan) and a
plain CREATE INDEX (write-blocking SHARE lock) compounded it.
Rework migration 016 as an online migration (run via psql, not in a wrapping
transaction):
- backfill in small autocommitted batches via a procedure, using
FOR UPDATE SKIP LOCKED so it never waits on an app-locked row;
- CREATE INDEX CONCURRENTLY instead of a blocking build;
- install the trigger last (so it can't clobber the updated_at backfill);
- drop the hard SET NOT NULL (the default + trigger + backfill keep the column
populated and the client falls back to updated_at); a NOT VALID CHECK +
VALIDATE alternative is included, commented, for operators who want it.
The baseline schema.sql (fresh, empty installs) keeps the simple inline
NOT NULL DEFAULT now() + trigger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): design spec for syncing reading status (#4634)
Field-level LWW for reading_status (dedicated reading_status_updated_at),
a new 'abandoned' status in the Readest UI, and a koplugin bridge to
KOReader's native summary.status (whole-library apply + capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): implementation plan for syncing reading status (#4634)
Bite-sized TDD tasks across 3 parts: A) cloud field-level LWW
(reading_status_updated_at on server upsert + client pull-merge),
B) 'abandoned'/On-hold status in the Readest UI, C) koplugin bridge
to KOReader summary.status (mapping + reconcile + whole-library
apply/capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): add reading_status_updated_at for field-level status LWW (#4634)
* feat(sync): stamp readingStatusUpdatedAt on status change in updateBookProgress
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): stamp status timestamp on explicit library status edits
* feat(sync): resolve reading status by its own timestamp in client pull-merge
* feat(sync): resolve reading status by its own timestamp in server upsert (#4634)
* fix(sync): tighten reading-status merge typing + strengthen test (A5 review)
Replace as-unknown-as double-casts at read sites with typed locals
(clientBook/serverBook); retain a single as-unknown-as only at the
server-wins construction site where the static type is too narrow.
Strengthen test 3 to assert both fields with toEqual.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(library): render the 'On hold' (abandoned) status badge
* feat(library): add 'Mark as On hold' actions + i18n for abandoned status
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* i18n: translate 'On hold' and 'Mark as On hold' for the abandoned status (#4634)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): add reading-status mapping + reconcile between Readest and KOReader
* feat(koplugin): persist + sync reading_status_updated_at in LibraryStore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): bridge reading status to KOReader summary.status on library sync (#4634)
* test(sync): cover koplugin v1->v2 migration + tighten status-sync tests (final review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): redesign KOReader first-sync (decisive-only + bootstrap) (#4634)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(koplugin): safe first-sync of reading status (decisive-only + bootstrap) (#4634)
KOReader auto-sets summary.status='reading' on first open, and legacy Readest
statuses have reading_status_updated_at=0, so pure timestamp LWW let opening a
finished book downgrade it. Restrict sync to deliberate statuses (finished/
complete, abandoned/on-hold, unread->clear); never capture KO 'reading'/'New'.
On the unsynced baseline (Readest ts=0) conflicts resolve Readest-authoritative,
then stamp now_ms to exit bootstrap into steady-state LWW. reconcile now returns
write_ko/write_store flags; statussync captures now_ms once and equalizes both
sides (convergent, idempotent, resumable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(koplugin): cover remaining first-sync graph cells + document sort effect (review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduces a 'global' annotation flag so a highlight/note created on one occurrence of a phrase is automatically applied to every matching occurrence in the book (and stays applied across reloads). Renders these expansions as transient overlays without creating duplicate persisted notes. This flag will not show when the book is fixed layout like PDF or CBZ.
- types: add 'global?: boolean' to BookNote and DBBookNote; transform layer round-trips the field, with regression coverage ensuring older clients do not clobber it on write-back.
- db: new migration 013_add_book_notes_global.sql adds nullable 'global' column to public.book_notes; init schema.sql updated to match.
- annotator: new utils/globalAnnotations.ts handles cfi expansion / text-match search across the spine and overlay synthesis. Annotator.tsx fans out global notes on load and on overlay creation; AnnotationPopup and HighlightOptions expose a toggle to mark a highlight as global.
- sync path is transparent: a global note created on another device is fanned out locally on next render with no extra UI required.
Add bidirectional annotation/highlight sync between KOReader and Readest:
- Add xpointer0/xpointer1 fields to BookNote and DBBookNote types for
KOReader XPointer positions alongside Readest's CFI format
- Extend transform layer to pass through xpointer fields to/from DB
- Convert CFI→XPointer on push and XPointer→CFI on pull in useNotesSync,
discarding notes that fail conversion
- Support KOReader's text()[K].N indexed text node format in xcfi.ts for
paragraphs with inline elements (e.g. <a> page anchors)
- Generate KOReader-compatible XPointers: text().N for single text nodes,
text()[K].N only when multiple direct text nodes exist
- Skip cfi-inert elements (injected by Readest at runtime) in XPointer
path building and resolution
- Map highlight colors between KOReader and Readest color systems
- Implement KOReader plugin annotation push/pull with deterministic IDs,
auto-sync on document open/close, and UIManager refresh on pull
- Refactor koplugin into focused modules: syncauth, syncconfig,
syncannotations, selfupdate
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In #2636, FileRecord was defined to have updated_at field
which is used when it is accessed from the database. But the
local dev setup is missing this field.
This diff adds an updated_at column to match the expectation
* initial files
* added testing files
* removed unused files
* cleaned additional mounts
* fixed sql init
* removed more unused files
* moved to docker folder
* revert package.json
* gitignore update
* env example comments and compose necessary healthcheck
* ghcr package impl
* updated dockerfile steps for layer caching
* added development-stage to dockerfile to dev environment
* added documentation on how to use dockerfile and compose.yml
* fixed prettier issues
* fixed image tag
* removed workflow for later
Huang Xin