forked from akai/readest
3b348c8f35
* feat(sync): foundation for CRDT-based cross-device replica sync (Phase 1+2)
Adds the primitives and orchestration layer for syncing user-imported
assets (dictionaries, fonts, textures, OPDS catalogs, dict settings)
across devices via a polymorphic `replicas` table with field-level LWW
under HLC ordering. Phase 1 ships the foundation (CRDT, crypto, server
schemas, SQL migrations, push/pull endpoint); Phase 2 adds the adapter
registry, HTTP client, and sync manager. No modifications to existing
book sync — additive only.
Phase 1:
- src/libs/crdt.ts — HlcGenerator (monotonic + remote-absorption +
clock-regression-safe), per-field LWW with deviceId tiebreak,
remove-wins tombstones, reincarnation token revival.
- src/libs/crypto/{derive,encrypt,envelope,passphrase}.ts —
PBKDF2-600k key derivation (OWASP 2024), AES-GCM round-trip,
envelope {c,i,s,alg,h} with SHA-256 sidecar integrity check,
passphrase storage abstraction (web ephemeral; Tauri keychain stub).
- src/libs/replica-schemas.ts — Zod-backed allowlist (dictionary only
in PR 1), 64KiB row cap, 64-field cap, schemaVersion bounds,
filename validator.
- src/libs/replica-sync-server.ts — push batch validation
(auth + allowlist + schema + HLC ±60s skew clamp).
- src/pages/api/sync/replicas.ts — POST/GET endpoint wrapping the
Postgres crdt_merge_replica function via RPC.
- docker/volumes/db/migrations/003_add_replicas.sql — replicas table
+ replica_keys table + RLS.
- docker/volumes/db/migrations/004_crdt_merge_replica_fn.sql — atomic
per-field LWW merge function (forwards-compat preserves unknown
fields).
Phase 2:
- src/services/sync/replicaRegistry.ts — adapter contract
(core + optional BinaryCapability + LifecycleHooks per eng review).
- src/libs/replica-sync-client.ts — HTTP wrapper mapping status codes
to typed SyncError codes.
- src/services/sync/replicaSyncManager.ts — 5s debounced push,
immediate flush on visibilitychange/online, per-kind pull cursor,
remote HLC absorption.
Tests: 125 new (crdt 26, crypto 32, schemas 21, server 16, client 12,
registry 6, manager 12). Full suite 3656 passing, lint clean. Existing
book/config/note sync paths untouched.
Plan: ~/.claude/plans/vivid-orbiting-thimble.md
CEO plan: ~/.gstack/projects/readest-readest/ceo-plans/2026-05-06-replica-sync-cathedral.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): add kind="replica" path through TransferManager (Phase 3)
Adds the replica branch to the existing book-shaped transfer
infrastructure so dictionary (and future kinds) bundles can flow through
the same queue, retry, and progress UI as book uploads.
Existing book transfer paths remain unchanged. The book-side regression
suite (37 tests in transfer-store.test.ts, 37 in transfer-manager.test.ts)
all stay green.
Store (src/store/transferStore.ts):
- TransferItem gains kind: 'book' | 'replica' (default 'book' on legacy
persisted rows), replicaKind, replicaId, replicaFiles, replicaBase.
- New addReplicaTransfer(replicaKind, replicaId, displayTitle, type, opts)
with files + base in opts; auto-computes totalBytes from file sizes.
- New getReplicaTransfer(replicaKind, replicaId, type) lookup.
- getTransferByBookHash filters to kind === 'book' (defensive against
bookHash="" collisions on replica items).
- restoreTransfers fills kind: 'book' for legacy persisted rows.
Manager (src/services/transferManager.ts):
- queueReplicaUpload / queueReplicaDownload / queueReplicaDelete.
- executeTransfer dispatches by kind to the new executeReplicaTransfer
(iterates files, calls appService.uploadReplicaFile per file with
per-file progress aggregation) or the existing executeBookTransfer
(refactored out, byte-identical behavior).
- Dispatches replica-transfer-complete event on success so stores can
react (e.g., commit manifest_jsonb to the replica row).
Storage / cloud (src/libs/storage.ts, src/services/cloudService.ts):
- uploadReplicaFile bypasses the book-only File.name smuggling and
takes an explicit cfp (cloud file path).
- uploadReplicaFileToCloud / downloadReplicaFileFromCloud /
deleteReplicaBundleFromCloud orchestrate per-file operations under
${userId}/Readest/replicas/<kind>/<replicaId>/<filename>.
- replicaCloudKey() centralizes the path-construction rule.
- New CLOUD_REPLICAS_SUBDIR constant.
App service (src/services/appService.ts, src/types/system.ts):
- AppService gains uploadReplicaFile, downloadReplicaFile,
deleteReplicaBundle (file-level operations; orchestration lives in
TransferManager).
Tests: 18 new (12 in transfer-store.test.ts, 5 in transfer-manager.test.ts,
1 fixture). Full suite 3674 passing, lint clean. Existing book regression
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): dictionary replica adapter + bootstrap (Phase 4, partial)
Lands the safe-to-ship foundation of Phase 4 — adapter logic, registry
bootstrap, and SystemSettings hooks for replica sync. The on-disk
migration of legacy customDictionaries (bundleDir → content-hash id),
the live store wiring, and the Settings → Sync UI are deferred to a
follow-up PR so they can land with real-device QA.
Adapter (src/services/sync/adapters/dictionary.ts):
- dictionaryAdapter: kind='dictionary', schemaVersion=1.
- pack/unpack — only synced subset (name, kind, lang, addedAt,
unsupported{,Reason}). bundleDir / files / unavailable / deletedAt
stay per-device or are handled by the tombstone mechanism.
- BinaryCapability.enumerateFiles dispatches by bundle kind:
- mdict: mdx + mdd[] + css[]
- stardict: ifo + idx + dict + syn (skips .idx.offsets / .syn.offsets
sidecars — those are device-local indices)
- dict: dict + index
- slob: single .slob file
- primaryDictionaryFile() picks the anchor file per kind for
partialMD5 hashing.
- computeDictionaryReplicaId(partialMd5, byteSize, sortedFilenames)
produces a deterministic 32-hex content-hash id used at import time.
- 23 tests cover pack/unpack identity, kind dispatch, file enumeration,
id determinism, and per-device-field exclusion.
Bootstrap (src/services/sync/replicaBootstrap.ts):
- bootstrapReplicaAdapters() registers all known adapters once at app
start. Idempotent (safe to call multiple times). Wired into
EnvContext.tsx so the registry populates on app mount.
- 3 tests cover registration, idempotency, and the PR-1 allowlist.
SystemSettings (src/types/settings.ts, src/services/constants.ts):
- +SyncCategory = 'book' | 'progress' | 'note' | 'dictionary' — typed
union for the user-facing sync toggles. 'progress' gates the
existing book-config sync (reading progress); 'note' gates
annotations; 'book' gates book binaries + metadata; 'dictionary'
gates the new replica sync. Future replica kinds extend the union.
- +SYNC_CATEGORIES readonly array for UI iteration.
- +syncCategories: Partial<Record<SyncCategory, boolean>> — per-
category opt-in toggles in DEFAULT_SYSTEM_SETTINGS (default ON for
all four). UI panel ships in the follow-up.
- +lastSyncedAtReplicas: Record<string, string> — per-kind HLC pull
cursors (matches replicaSyncManager's CursorStore contract).
Registry type cleanup (src/services/sync/replicaRegistry.ts):
- BinaryCapability.enumerateFiles return shape: localRelPath → lfp
to match the existing TransferStore.ReplicaTransferFile convention.
Tests: 28 new (23 dict + 3 bootstrap + 2 syncCategories defaults).
Full suite 3702 passing, lint clean. Existing book/config/note sync
paths untouched.
Deferred to PR 1 follow-up (with real-device QA):
- customDictionaryStore migration: rehash legacy uniqueId() bundleDir
to content-hash id; preserve providerOrder mapping; staged
.legacy/<old-id>/ backup.
- Wire customDictionaryStore mutations through replicaSyncManager
(markDirty on add/rename/delete; pull on init).
- Settings → Sync panel: per-category toggles + last-sync timestamps.
- Sync passphrase modal: set / change / forgot flow (lazy first prompt
on encrypted-field push/pull).
- <CloudReplicaRow> in CustomDictionaries.tsx for "Download from cloud
(X MB)" affordance.
- Tauri keychain backend for sync passphrase storage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(sync): replicaSync singleton + content-hash id at dict import (Phase 4b)
Two foundation pieces that everything UI-side will sit on top of, both
purely additive — no live store wiring, no behavior change for existing
dictionary imports.
replicaSync singleton (src/services/sync/replicaSync.ts):
- initReplicaSync({deviceId, cursorStore, hlcStore?, client?}) builds
one ReplicaSyncManager backed by an HlcGenerator with persistence
wrapped around .next()/.observe(). Idempotent (second init returns
the existing instance).
- LocalStorageHlcStore (src/libs/hlc-store.ts) snapshots the HLC
counter under 'readest_replica_hlc' so it survives restart. Falls
back silently when localStorage is unavailable (private mode, SSR);
client re-derives via the existing remote max(updated_at_ts) repair
path. InMemoryHlcStore is the test backend.
- 16 tests (9 hlc-store, 7 replicaSync) covering snapshot persistence,
restore-on-init, and idempotency.
- Wiring into EnvContext for production deferred to the follow-up that
also adds the cursor store backed by useSettingsStore.
Content-hash id at dictionary import
(src/services/dictionaries/contentId.ts):
- computeDictionaryContentId(primaryFile, filenames) wraps
computeDictionaryReplicaId(partialMd5(primary), byteSize,
sortedFilenames) — the cross-device id used as the replica_id when
the dict actually pushes/pulls.
- Wired into all four import paths in dictionaryService.ts:
- stardict primary = .ifo (small text, partialMD5 ≈ full hash)
- mdict primary = .mdx (body)
- dict primary = .dict.dz (gzipped body)
- slob primary = .slob (single-file bundle)
ImportedDictionary gains contentId?: string. Optional for backwards
compat; legacy bundles without contentId are flagged as
"needs rehash before sync" by the upcoming store-wiring follow-up.
- 6 tests cover identity determinism, byteSize sensitivity, filename-
set sensitivity, and order-independence.
Tests: 22 new (9 + 7 + 6). Full suite 3724 passing, lint clean.
Existing dictionary import flow unchanged for users — contentId is an
additional field, not a replacement for the bundleDir-based id.
Deferred to follow-up (with on-device QA):
- Production cursor store backed by useSettingsStore +
appService.saveSettings.
- EnvContext call to initReplicaSync after appService boot.
- customDictionaryStore mutation hooks → replicaSyncManager.markDirty.
- Legacy bundleDir → contentId migration with .legacy/ backup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(sync): land replica-sync design plan in repo
Moves the plan document that drove this PR's foundation work
(`~/.claude/plans/vivid-orbiting-thimble.md`) into the project tree at
`apps/readest-app/.claude/plans/` so reviewers and future contributors
can read it alongside the code without leaving the repo.
The plan went through three review passes — Codex (19 findings, all
absorbed), CEO/scope review (mode SCOPE EXPANSION; encrypted secrets
pulled forward to v1, "private-only forever" posture lock), and eng
review (FULL_REVIEW mode, 16 findings absorbed). The full review trail
lives in the file's `## GSTACK REVIEW REPORT` section at the bottom.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(crdt): point README at in-repo plan path
Now that vivid-orbiting-thimble.md lives at
apps/readest-app/.claude/plans/, the README link should point there
rather than at the home-dir copy that no longer exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(sync): rename src/libs files to camelCase per project convention
Test files renamed in lockstep. Imports + comment references updated
across cloudService, storage, transferManager, replicaSync,
replicaSyncManager, /api/sync/replicas, and all four test files. No
behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
154 lines
5.4 KiB
TypeScript
154 lines
5.4 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
|
|
|
|
vi.mock('@/utils/access', () => ({
|
|
getAccessToken: vi.fn(async () => 'fake-token'),
|
|
}));
|
|
vi.mock('@/services/environment', () => ({
|
|
getAPIBaseUrl: () => 'https://example.test',
|
|
}));
|
|
|
|
import { ReplicaSyncClient } from '@/libs/replicaSyncClient';
|
|
import { hlcPack } from '@/libs/crdt';
|
|
import type { Hlc, ReplicaRow } from '@/types/replica';
|
|
import { SyncError } from '@/libs/errors';
|
|
|
|
const HLC = hlcPack(1_700_000_000_000, 0, 'd') as Hlc;
|
|
|
|
const sampleRow: ReplicaRow = {
|
|
user_id: 'u1',
|
|
kind: 'dictionary',
|
|
replica_id: 'r1',
|
|
fields_jsonb: { name: { v: 'Webster', t: HLC, s: 'd' } },
|
|
manifest_jsonb: null,
|
|
deleted_at_ts: null,
|
|
reincarnation: null,
|
|
updated_at_ts: HLC,
|
|
schema_version: 1,
|
|
};
|
|
|
|
const mockFetch = vi.fn();
|
|
|
|
beforeEach(() => {
|
|
mockFetch.mockReset();
|
|
globalThis.fetch = mockFetch as unknown as typeof fetch;
|
|
});
|
|
|
|
afterEach(() => {
|
|
vi.restoreAllMocks();
|
|
});
|
|
|
|
describe('ReplicaSyncClient.push', () => {
|
|
test('POSTs rows to /sync/replicas with bearer token', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ rows: [sampleRow] }), { status: 200 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
const result = await client.push([sampleRow]);
|
|
expect(mockFetch).toHaveBeenCalledOnce();
|
|
const [url, init] = mockFetch.mock.calls[0]!;
|
|
expect(url).toBe('https://example.test/sync/replicas');
|
|
expect(init.method).toBe('POST');
|
|
expect(init.headers.Authorization).toBe('Bearer fake-token');
|
|
expect(init.headers['Content-Type']).toBe('application/json');
|
|
expect(JSON.parse(init.body)).toEqual({ rows: [sampleRow] });
|
|
expect(result).toEqual([sampleRow]);
|
|
});
|
|
|
|
test('400 / VALIDATION → SyncError VALIDATION', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'bad', code: 'VALIDATION' }), { status: 400 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({
|
|
name: 'SyncError',
|
|
code: 'VALIDATION',
|
|
});
|
|
});
|
|
|
|
test('401 → SyncError AUTH', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'unauth', code: 'AUTH' }), { status: 401 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({
|
|
code: 'AUTH',
|
|
});
|
|
});
|
|
|
|
test('409 / CLOCK_SKEW → SyncError CLOCK_SKEW', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'skew', code: 'CLOCK_SKEW' }), { status: 409 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'CLOCK_SKEW' });
|
|
});
|
|
|
|
test('413 / batch too large → SyncError VALIDATION', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'batch', code: 'VALIDATION' }), { status: 413 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'VALIDATION' });
|
|
});
|
|
|
|
test('422 / UNKNOWN_KIND → SyncError UNKNOWN_KIND', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'unknown', code: 'UNKNOWN_KIND' }), { status: 422 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'UNKNOWN_KIND' });
|
|
});
|
|
|
|
test('5xx → SyncError SERVER', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ error: 'oops' }), { status: 500 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'SERVER' });
|
|
});
|
|
|
|
test('network error → SyncError TIMEOUT/SERVER', async () => {
|
|
mockFetch.mockRejectedValueOnce(new TypeError('Failed to fetch'));
|
|
const client = new ReplicaSyncClient();
|
|
await expect(client.push([sampleRow])).rejects.toBeInstanceOf(SyncError);
|
|
});
|
|
|
|
test('empty rows is a no-op (no fetch call)', async () => {
|
|
const client = new ReplicaSyncClient();
|
|
const result = await client.push([]);
|
|
expect(result).toEqual([]);
|
|
expect(mockFetch).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
describe('ReplicaSyncClient.pull', () => {
|
|
test('GETs with kind + since query params', async () => {
|
|
mockFetch.mockResolvedValueOnce(
|
|
new Response(JSON.stringify({ rows: [sampleRow] }), { status: 200 }),
|
|
);
|
|
const client = new ReplicaSyncClient();
|
|
const rows = await client.pull('dictionary', HLC);
|
|
const [url, init] = mockFetch.mock.calls[0]!;
|
|
expect(url).toBe(
|
|
`https://example.test/sync/replicas?kind=dictionary&since=${encodeURIComponent(HLC)}`,
|
|
);
|
|
expect(init.method).toBe('GET');
|
|
expect(rows).toEqual([sampleRow]);
|
|
});
|
|
|
|
test('GET without since cursor', async () => {
|
|
mockFetch.mockResolvedValueOnce(new Response(JSON.stringify({ rows: [] }), { status: 200 }));
|
|
const client = new ReplicaSyncClient();
|
|
await client.pull('dictionary', null);
|
|
const [url] = mockFetch.mock.calls[0]!;
|
|
expect(url).toBe('https://example.test/sync/replicas?kind=dictionary');
|
|
});
|
|
|
|
test('404 → empty array (server lacks /api/sync/replicas; old backend)', async () => {
|
|
mockFetch.mockResolvedValueOnce(new Response('not found', { status: 404 }));
|
|
const client = new ReplicaSyncClient();
|
|
const rows = await client.pull('dictionary', null);
|
|
expect(rows).toEqual([]);
|
|
});
|
|
});
|