Huang Xin 3b348c8f35 feat(sync): CRDT replica sync foundation (#4075)
* feat(sync): foundation for CRDT-based cross-device replica sync (Phase 1+2)

Adds the primitives and orchestration layer for syncing user-imported
assets (dictionaries, fonts, textures, OPDS catalogs, dict settings)
across devices via a polymorphic `replicas` table with field-level LWW
under HLC ordering. Phase 1 ships the foundation (CRDT, crypto, server
schemas, SQL migrations, push/pull endpoint); Phase 2 adds the adapter
registry, HTTP client, and sync manager. No modifications to existing
book sync — additive only.

Phase 1:
- src/libs/crdt.ts — HlcGenerator (monotonic + remote-absorption +
  clock-regression-safe), per-field LWW with deviceId tiebreak,
  remove-wins tombstones, reincarnation token revival.
- src/libs/crypto/{derive,encrypt,envelope,passphrase}.ts —
  PBKDF2-600k key derivation (OWASP 2024), AES-GCM round-trip,
  envelope {c,i,s,alg,h} with SHA-256 sidecar integrity check,
  passphrase storage abstraction (web ephemeral; Tauri keychain stub).
- src/libs/replica-schemas.ts — Zod-backed allowlist (dictionary only
  in PR 1), 64KiB row cap, 64-field cap, schemaVersion bounds,
  filename validator.
- src/libs/replica-sync-server.ts — push batch validation
  (auth + allowlist + schema + HLC ±60s skew clamp).
- src/pages/api/sync/replicas.ts — POST/GET endpoint wrapping the
  Postgres crdt_merge_replica function via RPC.
- docker/volumes/db/migrations/003_add_replicas.sql — replicas table
  + replica_keys table + RLS.
- docker/volumes/db/migrations/004_crdt_merge_replica_fn.sql — atomic
  per-field LWW merge function (forwards-compat preserves unknown
  fields).

Phase 2:
- src/services/sync/replicaRegistry.ts — adapter contract
  (core + optional BinaryCapability + LifecycleHooks per eng review).
- src/libs/replica-sync-client.ts — HTTP wrapper mapping status codes
  to typed SyncError codes.
- src/services/sync/replicaSyncManager.ts — 5s debounced push,
  immediate flush on visibilitychange/online, per-kind pull cursor,
  remote HLC absorption.

Tests: 125 new (crdt 26, crypto 32, schemas 21, server 16, client 12,
registry 6, manager 12). Full suite 3656 passing, lint clean. Existing
book/config/note sync paths untouched.

Plan: ~/.claude/plans/vivid-orbiting-thimble.md
CEO plan: ~/.gstack/projects/readest-readest/ceo-plans/2026-05-06-replica-sync-cathedral.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): add kind="replica" path through TransferManager (Phase 3)

Adds the replica branch to the existing book-shaped transfer
infrastructure so dictionary (and future kinds) bundles can flow through
the same queue, retry, and progress UI as book uploads.

Existing book transfer paths remain unchanged. The book-side regression
suite (37 tests in transfer-store.test.ts, 37 in transfer-manager.test.ts)
all stay green.

Store (src/store/transferStore.ts):
- TransferItem gains kind: 'book' | 'replica' (default 'book' on legacy
  persisted rows), replicaKind, replicaId, replicaFiles, replicaBase.
- New addReplicaTransfer(replicaKind, replicaId, displayTitle, type, opts)
  with files + base in opts; auto-computes totalBytes from file sizes.
- New getReplicaTransfer(replicaKind, replicaId, type) lookup.
- getTransferByBookHash filters to kind === 'book' (defensive against
  bookHash="" collisions on replica items).
- restoreTransfers fills kind: 'book' for legacy persisted rows.

Manager (src/services/transferManager.ts):
- queueReplicaUpload / queueReplicaDownload / queueReplicaDelete.
- executeTransfer dispatches by kind to the new executeReplicaTransfer
  (iterates files, calls appService.uploadReplicaFile per file with
  per-file progress aggregation) or the existing executeBookTransfer
  (refactored out, byte-identical behavior).
- Dispatches replica-transfer-complete event on success so stores can
  react (e.g., commit manifest_jsonb to the replica row).

Storage / cloud (src/libs/storage.ts, src/services/cloudService.ts):
- uploadReplicaFile bypasses the book-only File.name smuggling and
  takes an explicit cfp (cloud file path).
- uploadReplicaFileToCloud / downloadReplicaFileFromCloud /
  deleteReplicaBundleFromCloud orchestrate per-file operations under
  ${userId}/Readest/replicas/<kind>/<replicaId>/<filename>.
- replicaCloudKey() centralizes the path-construction rule.
- New CLOUD_REPLICAS_SUBDIR constant.

App service (src/services/appService.ts, src/types/system.ts):
- AppService gains uploadReplicaFile, downloadReplicaFile,
  deleteReplicaBundle (file-level operations; orchestration lives in
  TransferManager).

Tests: 18 new (12 in transfer-store.test.ts, 5 in transfer-manager.test.ts,
1 fixture). Full suite 3674 passing, lint clean. Existing book regression
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): dictionary replica adapter + bootstrap (Phase 4, partial)

Lands the safe-to-ship foundation of Phase 4 — adapter logic, registry
bootstrap, and SystemSettings hooks for replica sync. The on-disk
migration of legacy customDictionaries (bundleDir → content-hash id),
the live store wiring, and the Settings → Sync UI are deferred to a
follow-up PR so they can land with real-device QA.

Adapter (src/services/sync/adapters/dictionary.ts):
- dictionaryAdapter: kind='dictionary', schemaVersion=1.
- pack/unpack — only synced subset (name, kind, lang, addedAt,
  unsupported{,Reason}). bundleDir / files / unavailable / deletedAt
  stay per-device or are handled by the tombstone mechanism.
- BinaryCapability.enumerateFiles dispatches by bundle kind:
  - mdict:    mdx + mdd[] + css[]
  - stardict: ifo + idx + dict + syn (skips .idx.offsets / .syn.offsets
              sidecars — those are device-local indices)
  - dict:     dict + index
  - slob:     single .slob file
- primaryDictionaryFile() picks the anchor file per kind for
  partialMD5 hashing.
- computeDictionaryReplicaId(partialMd5, byteSize, sortedFilenames)
  produces a deterministic 32-hex content-hash id used at import time.
- 23 tests cover pack/unpack identity, kind dispatch, file enumeration,
  id determinism, and per-device-field exclusion.

Bootstrap (src/services/sync/replicaBootstrap.ts):
- bootstrapReplicaAdapters() registers all known adapters once at app
  start. Idempotent (safe to call multiple times). Wired into
  EnvContext.tsx so the registry populates on app mount.
- 3 tests cover registration, idempotency, and the PR-1 allowlist.

SystemSettings (src/types/settings.ts, src/services/constants.ts):
- +SyncCategory = 'book' | 'progress' | 'note' | 'dictionary' — typed
  union for the user-facing sync toggles. 'progress' gates the
  existing book-config sync (reading progress); 'note' gates
  annotations; 'book' gates book binaries + metadata; 'dictionary'
  gates the new replica sync. Future replica kinds extend the union.
- +SYNC_CATEGORIES readonly array for UI iteration.
- +syncCategories: Partial<Record<SyncCategory, boolean>> — per-
  category opt-in toggles in DEFAULT_SYSTEM_SETTINGS (default ON for
  all four). UI panel ships in the follow-up.
- +lastSyncedAtReplicas: Record<string, string> — per-kind HLC pull
  cursors (matches replicaSyncManager's CursorStore contract).

Registry type cleanup (src/services/sync/replicaRegistry.ts):
- BinaryCapability.enumerateFiles return shape: localRelPath → lfp
  to match the existing TransferStore.ReplicaTransferFile convention.

Tests: 28 new (23 dict + 3 bootstrap + 2 syncCategories defaults).
Full suite 3702 passing, lint clean. Existing book/config/note sync
paths untouched.

Deferred to PR 1 follow-up (with real-device QA):
- customDictionaryStore migration: rehash legacy uniqueId() bundleDir
  to content-hash id; preserve providerOrder mapping; staged
  .legacy/<old-id>/ backup.
- Wire customDictionaryStore mutations through replicaSyncManager
  (markDirty on add/rename/delete; pull on init).
- Settings → Sync panel: per-category toggles + last-sync timestamps.
- Sync passphrase modal: set / change / forgot flow (lazy first prompt
  on encrypted-field push/pull).
- <CloudReplicaRow> in CustomDictionaries.tsx for "Download from cloud
  (X MB)" affordance.
- Tauri keychain backend for sync passphrase storage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): replicaSync singleton + content-hash id at dict import (Phase 4b)

Two foundation pieces that everything UI-side will sit on top of, both
purely additive — no live store wiring, no behavior change for existing
dictionary imports.

replicaSync singleton (src/services/sync/replicaSync.ts):
- initReplicaSync({deviceId, cursorStore, hlcStore?, client?}) builds
  one ReplicaSyncManager backed by an HlcGenerator with persistence
  wrapped around .next()/.observe(). Idempotent (second init returns
  the existing instance).
- LocalStorageHlcStore (src/libs/hlc-store.ts) snapshots the HLC
  counter under 'readest_replica_hlc' so it survives restart. Falls
  back silently when localStorage is unavailable (private mode, SSR);
  client re-derives via the existing remote max(updated_at_ts) repair
  path. InMemoryHlcStore is the test backend.
- 16 tests (9 hlc-store, 7 replicaSync) covering snapshot persistence,
  restore-on-init, and idempotency.
- Wiring into EnvContext for production deferred to the follow-up that
  also adds the cursor store backed by useSettingsStore.

Content-hash id at dictionary import
(src/services/dictionaries/contentId.ts):
- computeDictionaryContentId(primaryFile, filenames) wraps
  computeDictionaryReplicaId(partialMd5(primary), byteSize,
  sortedFilenames) — the cross-device id used as the replica_id when
  the dict actually pushes/pulls.
- Wired into all four import paths in dictionaryService.ts:
  - stardict primary = .ifo (small text, partialMD5 ≈ full hash)
  - mdict primary    = .mdx (body)
  - dict primary     = .dict.dz (gzipped body)
  - slob primary     = .slob (single-file bundle)
  ImportedDictionary gains contentId?: string. Optional for backwards
  compat; legacy bundles without contentId are flagged as
  "needs rehash before sync" by the upcoming store-wiring follow-up.
- 6 tests cover identity determinism, byteSize sensitivity, filename-
  set sensitivity, and order-independence.

Tests: 22 new (9 + 7 + 6). Full suite 3724 passing, lint clean.
Existing dictionary import flow unchanged for users — contentId is an
additional field, not a replacement for the bundleDir-based id.

Deferred to follow-up (with on-device QA):
- Production cursor store backed by useSettingsStore +
  appService.saveSettings.
- EnvContext call to initReplicaSync after appService boot.
- customDictionaryStore mutation hooks → replicaSyncManager.markDirty.
- Legacy bundleDir → contentId migration with .legacy/ backup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(sync): land replica-sync design plan in repo

Moves the plan document that drove this PR's foundation work
(`~/.claude/plans/vivid-orbiting-thimble.md`) into the project tree at
`apps/readest-app/.claude/plans/` so reviewers and future contributors
can read it alongside the code without leaving the repo.

The plan went through three review passes — Codex (19 findings, all
absorbed), CEO/scope review (mode SCOPE EXPANSION; encrypted secrets
pulled forward to v1, "private-only forever" posture lock), and eng
review (FULL_REVIEW mode, 16 findings absorbed). The full review trail
lives in the file's `## GSTACK REVIEW REPORT` section at the bottom.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(crdt): point README at in-repo plan path

Now that vivid-orbiting-thimble.md lives at
apps/readest-app/.claude/plans/, the README link should point there
rather than at the home-dir copy that no longer exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(sync): rename src/libs files to camelCase per project convention

Test files renamed in lockstep. Imports + comment references updated
across cloudService, storage, transferManager, replicaSync,
replicaSyncManager, /api/sync/replicas, and all four test files. No
behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:50:15 +02:00
2025-01-21 07:18:00 +01:00
2024-11-11 21:25:22 +01:00

Readest Logo

Readest


Readest is an open-source ebook reader designed for immersive and deep reading experiences. Built as a modern rewrite of Foliate, it leverages Next.js 16 and Tauri v2 to deliver a smooth, cross-platform experience across macOS, Windows, Linux, Android, iOS, and the Web.

Website Web App OS
Discord Reddit AGPL Licence Language Coverage Donate Latest release Last commit Commits Ask DeepWiki

FeaturesPlanned FeaturesScreenshotsDownloadsGetting StartedTroubleshootingSupportLicense

Features

Implemented
Feature Description Status
Multi-Format Support Support EPUB, MOBI, KF8 (AZW3), FB2, CBZ, TXT, PDF
Scroll/Page View Modes Switch between scrolling or paginated reading modes.
Full-Text Search Search across the entire book to find relevant sections.
Annotations and Highlighting Add highlights, bookmarks, and notes to enhance your reading experience and use instant mode for quicker interactions.
Dictionary/Wikipedia Lookup Instantly look up words and terms when reading.
Parallel Read Read two books or documents simultaneously in a split-screen view.
Customize Font and Layout Adjust font, layout, theme mode, and theme colors for a personalized experience.
Code Syntax Highlighting Read software manuals with rich coloring of code examples.
File Association and Open With Quickly open files in Readest in your file browser with one-click.
Library Management Organize, sort, and manage your entire ebook library.
OPDS/Calibre Integration Integrate OPDS/Calibre to access online libraries and catalogs.
Translate with DeepL and Yandex From a single sentence to the entire book—translate instantly.
Text-to-Speech (TTS) Support Enjoy smooth, multilingual narration—even within a single book.
Sync across Platforms Synchronize book files, reading progress, notes, and bookmarks across all supported platforms.
Sync with Koreader Synchronize reading progress, notes, and bookmarks with Koreader devices.
Accessibility Provides full keyboard navigation and supports for screen readers such as VoiceOver, TalkBack, NVDA, and Orca.
Visual & Focus Aids Reading ruler, paragraph-by-paragraph reading mode, and speed reading features.

Planned Features

🛠 Building
🔄 Planned
Feature Description Priority
AI-Powered Summarization Generate summaries of books or chapters using AI for quick insights. 🛠
Advanced Reading Stats Track reading time, pages read, and more for detailed insights. 🛠
Audiobook Support Extend functionality to play and manage audiobooks. 🔄
Handwriting Annotations Add support for handwriting annotations using a pen on compatible devices. 🔄
In-Library Full-Text Search Search across your entire ebook library to find topics and quotes. 🔄

Stay tuned for continuous improvements and updates! Contributions and suggestions are always welcome—let's build the ultimate reading experience together. 😊

Screenshots

Annotations

TTS

DeepL

Footnote

Wikipedia

Theming Dark Mode


Downloads

Mobile Apps

Download on the App Store     Get it on Google Play

Platform-Specific Downloads

Requirements

  • Node.js and pnpm for Next.js development
  • Rust and Cargo for Tauri development

For the best experience to build Readest for yourself, use a recent version of Node.js and Rust. Refer to the Tauri documentation for details on setting up the development environment prerequisites on different platforms.

nvm install v24
nvm use v24
npm install -g pnpm
rustup update

Getting Started

To get started with Readest, follow these steps to clone and build the project.

1. Clone the Repository

git clone https://github.com/readest/readest.git
cd readest

2. Install Dependencies

# might need to rerun this when code is updated
git submodule update --init --recursive
pnpm install
# copy vendors dist libs to public directory
pnpm --filter @readest/readest-app setup-vendors

3. Verify Dependencies Installation

To confirm that all dependencies are correctly installed, run the following command:

pnpm tauri info

This command will display information about the installed Tauri dependencies and configuration on your platform. Note that the output may vary depending on the operating system and environment setup. Please review the output specific to your platform for any potential issues.

For Windows targets, “Build Tools for Visual Studio 2022” (or a higher edition of Visual Studio) and the “Desktop development with C++” workflow must be installed. For Windows ARM64 targets, the “VS 2022 C++ ARM64 build tools” and "C++ Clang Compiler for Windows" components must be installed. And make sure clang can be found in the path by adding C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\Llvm\x64\bin for example in the environment variable Path.

4. Build for Development

# Start development for the Tauri app
pnpm tauri dev
# or start development for the Web app
pnpm dev-web
# preview with OpenNext build for the Web app
pnpm preview

For Android:

# Initialize the Android environment (run once)
rm apps/readest-app/src-tauri/gen/android
pnpm tauri android init
pnpm tauri icon ../../data/icons/readest-book.png
git checkout apps/readest-app/src-tauri/gen/android

pnpm tauri android dev
# or if you want to dev on a real device
pnpm tauri android dev --host

For iOS:

# Set up the iOS environment (run once)
pnpm tauri ios init
pnpm tauri icon ../../data/icons/readest-book.png

pnpm tauri ios dev
# or if you want to dev on a real device
pnpm tauri ios dev --host

5. Build for Production

pnpm tauri build
pnpm tauri android build
pnpm tauri ios build

Please refer to our release script if you experience any issues: https://github.com/readest/readest/blob/main/.github/workflows/release.yml

6. Setup dev environment with Nix

If you have Nix installed, you can leverage flake to enter a development shell with all the necessary dependencies:

nix develop ./ops  # enter a dev shell for the web app
nix develop ./ops#ios # enter a dev shell for the ios app
nix develop ./ops#android # enter a dev shell for the android app

7. More information

Please check the wiki of this project for more information on development.

Troubleshooting

1. Readest Wont Launch on Windows (Missing Edge WebView2 Runtime)

Symptom

  • When you double-click readest.exe, nothing happens. No window appears, and Task Manager does not show the process.
  • This can affect both the standard installer and the portable version.

Cause

  • Microsoft Edge WebView2 Runtime is either missing, outdated, or improperly installed on your system. Readest depends on WebView2 to render the interface on Windows.

How to Fix

  1. Check if WebView2 is installed
    • Open “Add or Remove Programs” (a.k.a. Apps & features) on Windows. Look for “Microsoft Edge WebView2 Runtime.”
  2. Install or Update WebView2
    • Download the WebView2 Runtime directly from Microsoft: link.
    • If you prefer an offline installer, download the offline package and run it as an Administrator.
  3. Re-run Readest
    • After installing/updating WebView2, launch readest.exe again.
    • If you still encounter problems, reboot your PC and try again.

Additional Tips

  • If reinstalling once doesnt work, uninstall Edge WebView2 completely, then reinstall it with Administrator privileges.
  • Verify your Windows installation has the latest updates from Microsoft.

Still Stuck?

  • See Issue readest/readest#358 for further details, or head over to our Discord server and open a support discussion with detailed logs of your environment and the steps youve taken.

2. AppImage Launches but Only Shows a Taskbar Icon

On some Arch Linux systems—especially those using Wayland—the Readest AppImage may briefly show an icon in the taskbar and then exit without opening a window.

You might see logs such as:

Could not create default EGL display: EGL_BAD_PARAMETER. Aborting...

This behavior is usually caused by compatibility issues between the bundled AppImage libraries and the systems EGL / Wayland environment.

Workaround 1: Launch with LD_PRELOAD (recommended)

You can preload the system Wayland client library before launching the AppImage:

LD_PRELOAD=/usr/lib/libwayland-client.so /path/to/Readest.AppImage

This workaround has been confirmed to resolve the issue on affected systems.

Workaround 2: Use the Flatpak Version

If you prefer a more reliable out-of-the-box experience on Arch Linux, consider using the Flatpak build on Flathub instead. The Flatpak runtime helps avoid system library mismatches and tends to behave more consistently across different Wayland and X11 setups.

Contributors

Readest is open-source, and contributions are welcome! Feel free to open issues, suggest features, or submit pull requests. Please review our contributing guidelines before you start. We also welcome you to join our Discord community for either support or contributing guidance.

A table of avatars from the project's contributors

Support

If Readest has been useful to you, consider supporting its development. You can become a sponsor on GitHub, donate via Stripe, or donate with crypto. Your contribution helps us squash bugs faster, improve performance, and keep building great features.

Sponsors

License

Readest is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See the LICENSE file for details.

The following libraries and frameworks are used in this software:

  • foliate-js, which is MIT licensed.
  • zip.js, which is licensed under the BSD-3-Clause license.
  • fflate, which is MIT licensed.
  • PDF.js, which is licensed under Apache License 2.0.
  • daisyUI, which is MIT licensed.
  • marked, which is MIT licensed.
  • next.js, which is MIT licensed.
  • react-icons, which has various open-source licenses.
  • react, which is MIT licensed.
  • tauri, which is MIT licensed.

The following fonts are utilized in this software, either bundled within the application or provided through web fonts:

Bitter, Fira Code, Inter, Literata, Merriweather, Noto Sans, Roboto, LXGW WenKai, MiSans, Source Han, WenQuanYi Micro Hei

We would also like to thank the Web Chinese Fonts Plan for offering open-source tools that enable the use of Chinese fonts on the web.


Happy reading with Readest!
S
Description
Local mirror of readest/readest for EPUB review editor migration
Readme 160 MiB
Languages
TypeScript 78.3%
MDX 10.1%
Lua 3.8%
Rust 3.2%
Kotlin 1.3%
Other 3.1%