Files
readest/apps/readest-app/src/__tests__/libs/replicaSyncClient.test.ts
T
Huang Xin 3b348c8f35 feat(sync): CRDT replica sync foundation (#4075)
* feat(sync): foundation for CRDT-based cross-device replica sync (Phase 1+2)

Adds the primitives and orchestration layer for syncing user-imported
assets (dictionaries, fonts, textures, OPDS catalogs, dict settings)
across devices via a polymorphic `replicas` table with field-level LWW
under HLC ordering. Phase 1 ships the foundation (CRDT, crypto, server
schemas, SQL migrations, push/pull endpoint); Phase 2 adds the adapter
registry, HTTP client, and sync manager. No modifications to existing
book sync — additive only.

Phase 1:
- src/libs/crdt.ts — HlcGenerator (monotonic + remote-absorption +
  clock-regression-safe), per-field LWW with deviceId tiebreak,
  remove-wins tombstones, reincarnation token revival.
- src/libs/crypto/{derive,encrypt,envelope,passphrase}.ts —
  PBKDF2-600k key derivation (OWASP 2024), AES-GCM round-trip,
  envelope {c,i,s,alg,h} with SHA-256 sidecar integrity check,
  passphrase storage abstraction (web ephemeral; Tauri keychain stub).
- src/libs/replica-schemas.ts — Zod-backed allowlist (dictionary only
  in PR 1), 64KiB row cap, 64-field cap, schemaVersion bounds,
  filename validator.
- src/libs/replica-sync-server.ts — push batch validation
  (auth + allowlist + schema + HLC ±60s skew clamp).
- src/pages/api/sync/replicas.ts — POST/GET endpoint wrapping the
  Postgres crdt_merge_replica function via RPC.
- docker/volumes/db/migrations/003_add_replicas.sql — replicas table
  + replica_keys table + RLS.
- docker/volumes/db/migrations/004_crdt_merge_replica_fn.sql — atomic
  per-field LWW merge function (forwards-compat preserves unknown
  fields).

Phase 2:
- src/services/sync/replicaRegistry.ts — adapter contract
  (core + optional BinaryCapability + LifecycleHooks per eng review).
- src/libs/replica-sync-client.ts — HTTP wrapper mapping status codes
  to typed SyncError codes.
- src/services/sync/replicaSyncManager.ts — 5s debounced push,
  immediate flush on visibilitychange/online, per-kind pull cursor,
  remote HLC absorption.

Tests: 125 new (crdt 26, crypto 32, schemas 21, server 16, client 12,
registry 6, manager 12). Full suite 3656 passing, lint clean. Existing
book/config/note sync paths untouched.

Plan: ~/.claude/plans/vivid-orbiting-thimble.md
CEO plan: ~/.gstack/projects/readest-readest/ceo-plans/2026-05-06-replica-sync-cathedral.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): add kind="replica" path through TransferManager (Phase 3)

Adds the replica branch to the existing book-shaped transfer
infrastructure so dictionary (and future kinds) bundles can flow through
the same queue, retry, and progress UI as book uploads.

Existing book transfer paths remain unchanged. The book-side regression
suite (37 tests in transfer-store.test.ts, 37 in transfer-manager.test.ts)
all stay green.

Store (src/store/transferStore.ts):
- TransferItem gains kind: 'book' | 'replica' (default 'book' on legacy
  persisted rows), replicaKind, replicaId, replicaFiles, replicaBase.
- New addReplicaTransfer(replicaKind, replicaId, displayTitle, type, opts)
  with files + base in opts; auto-computes totalBytes from file sizes.
- New getReplicaTransfer(replicaKind, replicaId, type) lookup.
- getTransferByBookHash filters to kind === 'book' (defensive against
  bookHash="" collisions on replica items).
- restoreTransfers fills kind: 'book' for legacy persisted rows.

Manager (src/services/transferManager.ts):
- queueReplicaUpload / queueReplicaDownload / queueReplicaDelete.
- executeTransfer dispatches by kind to the new executeReplicaTransfer
  (iterates files, calls appService.uploadReplicaFile per file with
  per-file progress aggregation) or the existing executeBookTransfer
  (refactored out, byte-identical behavior).
- Dispatches replica-transfer-complete event on success so stores can
  react (e.g., commit manifest_jsonb to the replica row).

Storage / cloud (src/libs/storage.ts, src/services/cloudService.ts):
- uploadReplicaFile bypasses the book-only File.name smuggling and
  takes an explicit cfp (cloud file path).
- uploadReplicaFileToCloud / downloadReplicaFileFromCloud /
  deleteReplicaBundleFromCloud orchestrate per-file operations under
  ${userId}/Readest/replicas/<kind>/<replicaId>/<filename>.
- replicaCloudKey() centralizes the path-construction rule.
- New CLOUD_REPLICAS_SUBDIR constant.

App service (src/services/appService.ts, src/types/system.ts):
- AppService gains uploadReplicaFile, downloadReplicaFile,
  deleteReplicaBundle (file-level operations; orchestration lives in
  TransferManager).

Tests: 18 new (12 in transfer-store.test.ts, 5 in transfer-manager.test.ts,
1 fixture). Full suite 3674 passing, lint clean. Existing book regression
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): dictionary replica adapter + bootstrap (Phase 4, partial)

Lands the safe-to-ship foundation of Phase 4 — adapter logic, registry
bootstrap, and SystemSettings hooks for replica sync. The on-disk
migration of legacy customDictionaries (bundleDir → content-hash id),
the live store wiring, and the Settings → Sync UI are deferred to a
follow-up PR so they can land with real-device QA.

Adapter (src/services/sync/adapters/dictionary.ts):
- dictionaryAdapter: kind='dictionary', schemaVersion=1.
- pack/unpack — only synced subset (name, kind, lang, addedAt,
  unsupported{,Reason}). bundleDir / files / unavailable / deletedAt
  stay per-device or are handled by the tombstone mechanism.
- BinaryCapability.enumerateFiles dispatches by bundle kind:
  - mdict:    mdx + mdd[] + css[]
  - stardict: ifo + idx + dict + syn (skips .idx.offsets / .syn.offsets
              sidecars — those are device-local indices)
  - dict:     dict + index
  - slob:     single .slob file
- primaryDictionaryFile() picks the anchor file per kind for
  partialMD5 hashing.
- computeDictionaryReplicaId(partialMd5, byteSize, sortedFilenames)
  produces a deterministic 32-hex content-hash id used at import time.
- 23 tests cover pack/unpack identity, kind dispatch, file enumeration,
  id determinism, and per-device-field exclusion.

Bootstrap (src/services/sync/replicaBootstrap.ts):
- bootstrapReplicaAdapters() registers all known adapters once at app
  start. Idempotent (safe to call multiple times). Wired into
  EnvContext.tsx so the registry populates on app mount.
- 3 tests cover registration, idempotency, and the PR-1 allowlist.

SystemSettings (src/types/settings.ts, src/services/constants.ts):
- +SyncCategory = 'book' | 'progress' | 'note' | 'dictionary' — typed
  union for the user-facing sync toggles. 'progress' gates the
  existing book-config sync (reading progress); 'note' gates
  annotations; 'book' gates book binaries + metadata; 'dictionary'
  gates the new replica sync. Future replica kinds extend the union.
- +SYNC_CATEGORIES readonly array for UI iteration.
- +syncCategories: Partial<Record<SyncCategory, boolean>> — per-
  category opt-in toggles in DEFAULT_SYSTEM_SETTINGS (default ON for
  all four). UI panel ships in the follow-up.
- +lastSyncedAtReplicas: Record<string, string> — per-kind HLC pull
  cursors (matches replicaSyncManager's CursorStore contract).

Registry type cleanup (src/services/sync/replicaRegistry.ts):
- BinaryCapability.enumerateFiles return shape: localRelPath → lfp
  to match the existing TransferStore.ReplicaTransferFile convention.

Tests: 28 new (23 dict + 3 bootstrap + 2 syncCategories defaults).
Full suite 3702 passing, lint clean. Existing book/config/note sync
paths untouched.

Deferred to PR 1 follow-up (with real-device QA):
- customDictionaryStore migration: rehash legacy uniqueId() bundleDir
  to content-hash id; preserve providerOrder mapping; staged
  .legacy/<old-id>/ backup.
- Wire customDictionaryStore mutations through replicaSyncManager
  (markDirty on add/rename/delete; pull on init).
- Settings → Sync panel: per-category toggles + last-sync timestamps.
- Sync passphrase modal: set / change / forgot flow (lazy first prompt
  on encrypted-field push/pull).
- <CloudReplicaRow> in CustomDictionaries.tsx for "Download from cloud
  (X MB)" affordance.
- Tauri keychain backend for sync passphrase storage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(sync): replicaSync singleton + content-hash id at dict import (Phase 4b)

Two foundation pieces that everything UI-side will sit on top of, both
purely additive — no live store wiring, no behavior change for existing
dictionary imports.

replicaSync singleton (src/services/sync/replicaSync.ts):
- initReplicaSync({deviceId, cursorStore, hlcStore?, client?}) builds
  one ReplicaSyncManager backed by an HlcGenerator with persistence
  wrapped around .next()/.observe(). Idempotent (second init returns
  the existing instance).
- LocalStorageHlcStore (src/libs/hlc-store.ts) snapshots the HLC
  counter under 'readest_replica_hlc' so it survives restart. Falls
  back silently when localStorage is unavailable (private mode, SSR);
  client re-derives via the existing remote max(updated_at_ts) repair
  path. InMemoryHlcStore is the test backend.
- 16 tests (9 hlc-store, 7 replicaSync) covering snapshot persistence,
  restore-on-init, and idempotency.
- Wiring into EnvContext for production deferred to the follow-up that
  also adds the cursor store backed by useSettingsStore.

Content-hash id at dictionary import
(src/services/dictionaries/contentId.ts):
- computeDictionaryContentId(primaryFile, filenames) wraps
  computeDictionaryReplicaId(partialMd5(primary), byteSize,
  sortedFilenames) — the cross-device id used as the replica_id when
  the dict actually pushes/pulls.
- Wired into all four import paths in dictionaryService.ts:
  - stardict primary = .ifo (small text, partialMD5 ≈ full hash)
  - mdict primary    = .mdx (body)
  - dict primary     = .dict.dz (gzipped body)
  - slob primary     = .slob (single-file bundle)
  ImportedDictionary gains contentId?: string. Optional for backwards
  compat; legacy bundles without contentId are flagged as
  "needs rehash before sync" by the upcoming store-wiring follow-up.
- 6 tests cover identity determinism, byteSize sensitivity, filename-
  set sensitivity, and order-independence.

Tests: 22 new (9 + 7 + 6). Full suite 3724 passing, lint clean.
Existing dictionary import flow unchanged for users — contentId is an
additional field, not a replacement for the bundleDir-based id.

Deferred to follow-up (with on-device QA):
- Production cursor store backed by useSettingsStore +
  appService.saveSettings.
- EnvContext call to initReplicaSync after appService boot.
- customDictionaryStore mutation hooks → replicaSyncManager.markDirty.
- Legacy bundleDir → contentId migration with .legacy/ backup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(sync): land replica-sync design plan in repo

Moves the plan document that drove this PR's foundation work
(`~/.claude/plans/vivid-orbiting-thimble.md`) into the project tree at
`apps/readest-app/.claude/plans/` so reviewers and future contributors
can read it alongside the code without leaving the repo.

The plan went through three review passes — Codex (19 findings, all
absorbed), CEO/scope review (mode SCOPE EXPANSION; encrypted secrets
pulled forward to v1, "private-only forever" posture lock), and eng
review (FULL_REVIEW mode, 16 findings absorbed). The full review trail
lives in the file's `## GSTACK REVIEW REPORT` section at the bottom.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(crdt): point README at in-repo plan path

Now that vivid-orbiting-thimble.md lives at
apps/readest-app/.claude/plans/, the README link should point there
rather than at the home-dir copy that no longer exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(sync): rename src/libs files to camelCase per project convention

Test files renamed in lockstep. Imports + comment references updated
across cloudService, storage, transferManager, replicaSync,
replicaSyncManager, /api/sync/replicas, and all four test files. No
behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:50:15 +02:00

154 lines
5.4 KiB
TypeScript

import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
vi.mock('@/utils/access', () => ({
getAccessToken: vi.fn(async () => 'fake-token'),
}));
vi.mock('@/services/environment', () => ({
getAPIBaseUrl: () => 'https://example.test',
}));
import { ReplicaSyncClient } from '@/libs/replicaSyncClient';
import { hlcPack } from '@/libs/crdt';
import type { Hlc, ReplicaRow } from '@/types/replica';
import { SyncError } from '@/libs/errors';
const HLC = hlcPack(1_700_000_000_000, 0, 'd') as Hlc;
const sampleRow: ReplicaRow = {
user_id: 'u1',
kind: 'dictionary',
replica_id: 'r1',
fields_jsonb: { name: { v: 'Webster', t: HLC, s: 'd' } },
manifest_jsonb: null,
deleted_at_ts: null,
reincarnation: null,
updated_at_ts: HLC,
schema_version: 1,
};
const mockFetch = vi.fn();
beforeEach(() => {
mockFetch.mockReset();
globalThis.fetch = mockFetch as unknown as typeof fetch;
});
afterEach(() => {
vi.restoreAllMocks();
});
describe('ReplicaSyncClient.push', () => {
test('POSTs rows to /sync/replicas with bearer token', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ rows: [sampleRow] }), { status: 200 }),
);
const client = new ReplicaSyncClient();
const result = await client.push([sampleRow]);
expect(mockFetch).toHaveBeenCalledOnce();
const [url, init] = mockFetch.mock.calls[0]!;
expect(url).toBe('https://example.test/sync/replicas');
expect(init.method).toBe('POST');
expect(init.headers.Authorization).toBe('Bearer fake-token');
expect(init.headers['Content-Type']).toBe('application/json');
expect(JSON.parse(init.body)).toEqual({ rows: [sampleRow] });
expect(result).toEqual([sampleRow]);
});
test('400 / VALIDATION → SyncError VALIDATION', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'bad', code: 'VALIDATION' }), { status: 400 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({
name: 'SyncError',
code: 'VALIDATION',
});
});
test('401 → SyncError AUTH', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'unauth', code: 'AUTH' }), { status: 401 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({
code: 'AUTH',
});
});
test('409 / CLOCK_SKEW → SyncError CLOCK_SKEW', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'skew', code: 'CLOCK_SKEW' }), { status: 409 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'CLOCK_SKEW' });
});
test('413 / batch too large → SyncError VALIDATION', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'batch', code: 'VALIDATION' }), { status: 413 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'VALIDATION' });
});
test('422 / UNKNOWN_KIND → SyncError UNKNOWN_KIND', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'unknown', code: 'UNKNOWN_KIND' }), { status: 422 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'UNKNOWN_KIND' });
});
test('5xx → SyncError SERVER', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ error: 'oops' }), { status: 500 }),
);
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toMatchObject({ code: 'SERVER' });
});
test('network error → SyncError TIMEOUT/SERVER', async () => {
mockFetch.mockRejectedValueOnce(new TypeError('Failed to fetch'));
const client = new ReplicaSyncClient();
await expect(client.push([sampleRow])).rejects.toBeInstanceOf(SyncError);
});
test('empty rows is a no-op (no fetch call)', async () => {
const client = new ReplicaSyncClient();
const result = await client.push([]);
expect(result).toEqual([]);
expect(mockFetch).not.toHaveBeenCalled();
});
});
describe('ReplicaSyncClient.pull', () => {
test('GETs with kind + since query params', async () => {
mockFetch.mockResolvedValueOnce(
new Response(JSON.stringify({ rows: [sampleRow] }), { status: 200 }),
);
const client = new ReplicaSyncClient();
const rows = await client.pull('dictionary', HLC);
const [url, init] = mockFetch.mock.calls[0]!;
expect(url).toBe(
`https://example.test/sync/replicas?kind=dictionary&since=${encodeURIComponent(HLC)}`,
);
expect(init.method).toBe('GET');
expect(rows).toEqual([sampleRow]);
});
test('GET without since cursor', async () => {
mockFetch.mockResolvedValueOnce(new Response(JSON.stringify({ rows: [] }), { status: 200 }));
const client = new ReplicaSyncClient();
await client.pull('dictionary', null);
const [url] = mockFetch.mock.calls[0]!;
expect(url).toBe('https://example.test/sync/replicas?kind=dictionary');
});
test('404 → empty array (server lacks /api/sync/replicas; old backend)', async () => {
mockFetch.mockResolvedValueOnce(new Response('not found', { status: 404 }));
const client = new ReplicaSyncClient();
const rows = await client.pull('dictionary', null);
expect(rows).toEqual([]);
});
});