* feat(sync): decouple the incremental-pull cursor from updated_at (#4678)
`books.updated_at` was overloaded as both the incremental-pull cursor
(`GET /api/sync?since=…` filters `updated_at > since`, devices keep one
global `max(updated_at)` watermark) and the library "date read" sort key.
A server-resolved reading-status merge had to be written with a timestamp
greater than every peer's global cursor to propagate, which forced
`updated_at = now()` and reordered the date-read library by sync-processing
time (the #4677 symptom).
Introduce a server-assigned `synced_at` column on `books`, stamped by a
`BEFORE INSERT OR UPDATE` trigger on every write, used only as the pull
cursor. `updated_at` stays pure client event time used only for sorting.
- Migration 016 + baseline schema: add `synced_at` (NOT NULL DEFAULT now()),
index `(user_id, synced_at)`, trigger `set_books_synced_at`. Backfill
`synced_at = updated_at` before creating the trigger so existing devices'
cursors hand over without a re-sync storm.
- GET: books filters/orders on `synced_at > since` (a delete bumps synced_at,
so the deleted_at clause is dropped); configs/notes stay on updated_at.
- POST: extract `buildStatusPropagationRow` and drop the `updated_at = now()`
bump — the trigger advances synced_at so peers re-pull the status change
while updated_at (the sort key) stays put.
- Client `computeMaxTimestamp` keys on synced_at, falling back to
updated_at/deleted_at for pre-migration servers and configs/notes.
Backward-compatible: `synced_at >= updated_at` always, so `synced_at > since`
is a strict superset of `updated_at > since` — old web clients and the
koplugin keep working with no data loss (at worst a redundant idempotent
re-pull of rare server-merged rows). The koplugin's shared pull/push cursor
is left untouched; a proper split is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sync): make the books synced_at backfill safe for large live tables (#4678)
The single `UPDATE … WHERE synced_at IS NULL` deadlocked on a 3.8M-row
production `books` table: it rewrites every row in one transaction while the
live /api/sync push path upserts books rows, and the two lock rows in opposite
orders. `ALTER COLUMN … SET NOT NULL` (full-table ACCESS EXCLUSIVE scan) and a
plain CREATE INDEX (write-blocking SHARE lock) compounded it.
Rework migration 016 as an online migration (run via psql, not in a wrapping
transaction):
- backfill in small autocommitted batches via a procedure, using
FOR UPDATE SKIP LOCKED so it never waits on an app-locked row;
- CREATE INDEX CONCURRENTLY instead of a blocking build;
- install the trigger last (so it can't clobber the updated_at backfill);
- drop the hard SET NOT NULL (the default + trigger + backfill keep the column
populated and the client falls back to updated_at); a NOT VALID CHECK +
VALIDATE alternative is included, commented, for operators who want it.
The baseline schema.sql (fresh, empty installs) keeps the simple inline
NOT NULL DEFAULT now() + trigger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync POST handler rewrote books.updated_at = now() whenever a pushed
book's resolved reading_status differed from the server row's. A book that
was imported locally and never given a status sends reading_status:
undefined, while the server stores null, so `undefined !== null` reported a
spurious status change. The 1-day re-sync window re-pushes every recently
touched book on each sync, so the server stamped those books with a fresh,
batch-identical timestamp every cycle, floating them to the top of the
date-sorted library (and above a book the user had just read).
Normalize nullish reading_status values before comparing so a statusless
book never registers as a status change. Verified on-device via CDP:
PUSH_SENT carried the old timestamp while PUSH_RETURNED came back with a
fresh now() for exactly the statusless books.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): design spec for syncing reading status (#4634)
Field-level LWW for reading_status (dedicated reading_status_updated_at),
a new 'abandoned' status in the Readest UI, and a koplugin bridge to
KOReader's native summary.status (whole-library apply + capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): implementation plan for syncing reading status (#4634)
Bite-sized TDD tasks across 3 parts: A) cloud field-level LWW
(reading_status_updated_at on server upsert + client pull-merge),
B) 'abandoned'/On-hold status in the Readest UI, C) koplugin bridge
to KOReader summary.status (mapping + reconcile + whole-library
apply/capture).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): add reading_status_updated_at for field-level status LWW (#4634)
* feat(sync): stamp readingStatusUpdatedAt on status change in updateBookProgress
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): stamp status timestamp on explicit library status edits
* feat(sync): resolve reading status by its own timestamp in client pull-merge
* feat(sync): resolve reading status by its own timestamp in server upsert (#4634)
* fix(sync): tighten reading-status merge typing + strengthen test (A5 review)
Replace as-unknown-as double-casts at read sites with typed locals
(clientBook/serverBook); retain a single as-unknown-as only at the
server-wins construction site where the static type is too narrow.
Strengthen test 3 to assert both fields with toEqual.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(library): render the 'On hold' (abandoned) status badge
* feat(library): add 'Mark as On hold' actions + i18n for abandoned status
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* i18n: translate 'On hold' and 'Mark as On hold' for the abandoned status (#4634)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): add reading-status mapping + reconcile between Readest and KOReader
* feat(koplugin): persist + sync reading_status_updated_at in LibraryStore
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(koplugin): bridge reading status to KOReader summary.status on library sync (#4634)
* test(sync): cover koplugin v1->v2 migration + tighten status-sync tests (final review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sync): redesign KOReader first-sync (decisive-only + bootstrap) (#4634)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(koplugin): safe first-sync of reading status (decisive-only + bootstrap) (#4634)
KOReader auto-sets summary.status='reading' on first open, and legacy Readest
statuses have reading_status_updated_at=0, so pure timestamp LWW let opening a
finished book downgrade it. Restrict sync to deliberate statuses (finished/
complete, abandoned/on-hold, unread->clear); never capture KO 'reading'/'New'.
On the unsynced baseline (Readest ts=0) conflicts resolve Readest-authoritative,
then stamp now_ms to exit bootstrap into steady-state LWW. reconcile now returns
write_ko/write_store flags; statussync captures now_ms once and equalizes both
sides (convergent, idempotent, resumable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(koplugin): cover remaining first-sync graph cells + document sort effect (review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes #3156. Adds a reading-statistics system whose canonical data model
is KOReader's own (book + page_stat_data), so stats round-trip losslessly
between Readest and KOReader.
- Storage: a cross-platform Turso statistics.db in KOReader's exact schema
(web/Workers, desktop, iOS, Android) — replacing #3156's Node-only
better-sqlite3 + statistics.json.
- Tracking: per-page reading events (time-on-page, idle-capped) flushed on
page-change/idle/hide/close — the KOReader model — not session aggregates.
- Sync: legacy /api/sync extended with a stats type backed by self-contained
Supabase tables (stat_books, stat_pages); union/longer-duration-wins merge
keyed on book_hash. apps/readest.koplugin syncs through the same endpoint.
- Scale & robustness: per-tab singleton connection (avoids OPFS lock
conflicts) + explicit WAL checkpoint; transactional bulk apply; chunked
resumable push; client-driven paged pull with trailing-ms completion;
paginated/scoped server merge.
Verified: 5668 unit tests, 155 koplugin busted tests, biome+tsgo + luacheck
all green; web OPFS DB verified live.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#4222.
Three changes that together stabilize Readest sync across devices:
**Stop the artificial updatedAt bump in useProgressAutoSave**
saveConfig unconditionally bumps config.updatedAt = Date.now(), and
useProgressAutoSave used to fire saveConfig on the very first relocate
after book open — even though that relocate just reflects the position
loaded from disk, not user action. The stale local config then looked
"newer" than a fresher server-side push, so the next auto-push
overwrote the other device's real progress via last-writer-wins. The
hook now snapshots the loaded location and skips saveConfig when the
in-memory location still matches it.
**Retry the first config pull with backoff + release the gate**
useProgressSync gates pushes behind a successful pull (so a brand-new
import can't clobber the server's real progress). But handleAutoSync
only re-arms on progress.location changes, so a single failed pull
(Android cold-start contention, Wi-Fi/LTE handoff, captive portal)
used to block every push for the whole reader session. The new
pullWithRetry retries on backoff (1500/4000/10000 ms) and releases
the gate after exhaustion — server-side last-writer-wins still
protects the cross-device case (a stale local push with an older
updated_at loses to a fresher server record). sync-book-progress
events reset the chain so manual pull-to-refresh recovers cleanly.
Fetch timeout bumped from 8s to 15s to better tolerate slow networks
in that cold-start window.
**Server piggybacks books.progress off configs push**
/api/sync POST now updates books.progress + books.updated_at for each
upserted config, gated by .lt('updated_at') so a concurrent newer
books push is never downgraded and a missing row is a silent no-op
(useBooksSync still seeds new rows from the library page). The
in-reader syncBooks round-trip is dropped — the reader now sends one
POST per auto-save instead of two, and the books row stays consistent
with config pushes even while a reader stays open (#4198).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Huang Xin