Files
readest/apps/readest-app/src/components/settings/integrations/FileSyncForm.tsx
T
Huang Xin ccb937015d feat(sync): incremental file sync and per-book transfers for the active provider (#4982)
* fix(sync): record remote-present files for no-source books in the upload cursor

With "Upload Book Files" on, a device that holds no local copy of a book
(e.g. the web app with a cloud-only library) HEAD-probed the remote for
every book on every sync: pushBookFile returned 'no-source' and the hash
was never recorded in library.json's uploadedHashes, so needsFilePush
stayed true for the whole library and each run (tab focus, Sync Now,
library change) issued one Drive/WebDAV request per book, 646 requests
per sweep in the reported case.

The HEAD probe already answers whether the file is on the remote. Carry
that in PushBookFileResult.remoteExists and record the hash when a
no-source book's file is already mirrored, so the next incremental sync
skips it and stays O(changed). Books absent both locally and remotely
stay unrecorded so a device that has the bytes can upload them later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): reach the no-source verdict without probing the remote

A book file sync with "Upload Book Files" on probed the remote for every
book before checking whether this device even holds the bytes. On a
device with a cloud-only library (the web app), none of the books have a
local source, so every sync run (tab focus, Sync Now, library change)
issued one name-lookup request per book against Google Drive, a full
per-book request storm that never converged: with no local file there is
nothing to upload and nothing to record, so the next run repeated it.

Resolve the local source first and return 'no-source' from local state
alone; the remote head probe now runs only when there is a local file to
compare or upload. The probe keeps its non-NETWORK rethrow semantics so
the auth-failure latch (#4981) still stops a run on an expired session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(sync): incremental file sync and per-book transfers for the active provider

Cut the redundant remote work a file-sync run does and route explicit
per-book uploads/downloads to the active third-party provider (WebDAV /
Google Drive) instead of the gated Readest Cloud transfer queue.

Engine (services/sync/file/engine.ts, wire.ts):
- etag change-probe: one HEAD on library.json, cached per provider for the
  session. When the etag matches the last successful pull, reuse the cached
  index and skip both the index download and the discovery scan. An AUTH
  failure on the probe aborts the run like the full pull does.
- emptyDirs memo carried in the index: dirs found to hold no book file are
  recorded so clients stop re-listing them every run. Re-checked when the
  file arrives (uploadedHashes), on Full Sync, or when a legacy client drops
  the record; pruned only against a listing that actually ran.
- skip the index re-push when the rebuilt index is semantically identical to
  the pulled one, so a restamped byte-copy no longer churns the remote and
  invalidates peers' etag change detection.
- downloadBookFile() for the explicit per-book Download action.

Provider reuse (services/sync/file/providerRegistry.ts):
- memoise one provider per connection key, shared by every surface (reader
  per-book sync, library auto-sync, Sync now / pull to refresh). Reuses the
  Drive path->id cache instead of re-resolving /Readest, books/ and
  library.json by name query on every engine build. Drive connect/disconnect
  resets the cache since its token source changes identity with no key input
  changing.

Google Drive (services/sync/providers/gdrive/GoogleDriveProvider.ts):
- write fast-path: PATCH a path whose id is cached in place with no lookup,
  falling back to a full resolve on a 404 (stale id). Removes a files.list
  per PUT in the steady state.
- dev-only request diagnostics: one line per provider op and per HTTP attempt
  so a run's request budget can be attributed from the console.

Per-book transfers (services/sync/file/runLibrarySync.ts):
- runActiveFileBookUpload / runActiveFileBookDownload build the active
  provider's engine and push/pull a single book, stamping downloadedAt like
  the native path. Wired into the reader/library book actions with toasts.

UI, status, and i18n:
- Readest Cloud sub-page: drop the quota stats and wrap the "Account and
  Storage" row in the BoxedList primitive so it aligns to the design system.
- Shorten provider status/toast copy ("Active", "Google Drive session
  expired", "Library sync via {{provider}}", "KOReader") and translate the
  new keys across all locales.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 22:49:35 +02:00

249 lines
9.5 KiB
TypeScript

import clsx from 'clsx';
import React from 'react';
import { MdCloudSync } from 'react-icons/md';
import { v4 as uuidv4 } from 'uuid';
import { useEnv } from '@/context/EnvContext';
import { useTranslation, type TranslationFunc } from '@/hooks/useTranslation';
import { useSettingsStore } from '@/store/settingsStore';
import { useLibraryStore } from '@/store/libraryStore';
import { useFileSyncStore } from '@/store/fileSyncStore';
import { eventDispatcher } from '@/utils/event';
import { FileSyncEngine } from '@/services/sync/file/engine';
import { FileSyncError } from '@/services/sync/file/provider';
import { createAppLocalStore } from '@/services/sync/file/appLocalStore';
import {
createFileSyncProvider,
type FileSyncBackendKind,
} from '@/services/sync/file/providerRegistry';
import type { KOSyncStrategy } from '@/types/settings';
import { BoxedList, SettingsRow, SettingsSelect, SettingsSwitchRow } from '../primitives';
/** The settings fields the shared sync controls read/write (WebDAV + Drive share these). */
export interface FileSyncFormSettings {
enabled?: boolean;
syncBooks?: boolean;
fullSync?: boolean;
strategy?: KOSyncStrategy;
deviceId?: string;
lastSyncedAt?: number;
}
interface FileSyncFormProps {
/** Which backend these controls drive (also keys the progress store + mutex). */
kind: FileSyncBackendKind;
/** This backend's settings slice. */
stored: FileSyncFormSettings;
/** Persist a patch into this backend's settings slice (must merge store-latest). */
persist: (patch: Partial<FileSyncFormSettings>) => Promise<void>;
/**
* Disable the "Sync now" button — set when the connection needs attention
* (e.g. an expired web Google Drive session) so a manual sync that would just
* fail isn't offered. The parent panel shows the reconnect affordance.
*/
syncNowDisabled?: boolean;
}
/**
* Translate a sync-time error into a user-facing string. Backend-neutral: the
* provider maps every failure to a {@link FileSyncError} with a normalised `code`
* so we never show a raw English `e.message`.
*/
const formatSyncError = (_: TranslationFunc, e: unknown): string => {
if (e instanceof FileSyncError) {
switch (e.code) {
case 'AUTH_FAILED':
return _('Authentication failed. Reconnect in Settings.');
case 'NOT_FOUND':
return _('Remote resource not found');
case 'NETWORK':
return _('Network error');
}
if (typeof e.status === 'number') {
return _('Sync failed (status {{status}})', { status: e.status });
}
}
return _('Sync failed.');
};
/**
* The provider-agnostic sync controls shared by every file-sync backend's
* settings form: the sub-category toggles, the conflict strategy, and a manual
* "Sync now" button with progress + result toast. The backend-specific connect
* panel (WebDAV URL/credentials, the Drive Connect button) lives in the parent
* form; everything below the connect line is identical across backends, so it
* lives here once and is parameterised by {@link FileSyncFormProps.kind}.
*/
const FileSyncForm: React.FC<FileSyncFormProps> = ({
kind,
stored,
persist,
syncNowDisabled = false,
}) => {
const _ = useTranslation();
const { settings } = useSettingsStore();
const { envConfig } = useEnv();
const isSyncing = useFileSyncStore((s) => s.byKind[kind]?.isSyncing ?? false);
const syncProgressLabel = useFileSyncStore((s) => s.byKind[kind]?.progressLabel ?? null);
const syncProgressDetail = useFileSyncStore((s) => s.byKind[kind]?.progressDetail ?? null);
const beginSync = useFileSyncStore((s) => s.beginSync);
const updateProgress = useFileSyncStore((s) => s.updateProgress);
const endSync = useFileSyncStore((s) => s.endSync);
const handleToggleSyncBooks = () => persist({ syncBooks: !(stored.syncBooks ?? false) });
const handleToggleFullSync = () => persist({ fullSync: !(stored.fullSync ?? false) });
const handleStrategyChange = async (e: React.ChangeEvent<HTMLSelectElement>) => {
await persist({ strategy: e.target.value as KOSyncStrategy });
};
/**
* Manual "Sync now" — reconcile the local library with the remote over a
* bounded-concurrency pool. Incremental by default (only books whose local
* copy differs from the shared index); "Full Sync" re-checks every book. The
* provider is built by kind through the registry so this stays backend-neutral.
*/
const handleSyncNow = async () => {
if (syncNowDisabled) return;
if (useFileSyncStore.getState().byKind[kind]?.isSyncing) return;
if (!stored.enabled) return;
const { libraryLoaded, library } = useLibraryStore.getState();
const appService = await envConfig.getAppService();
let currentLibrary = library ?? [];
if (!libraryLoaded && appService) {
currentLibrary = await appService.loadLibraryBooks();
// Hydrate the store before syncing so the engine's addBookToLibrary /
// updateBookMetadata merge against the real library, not an empty one.
useLibraryStore.getState().setLibrary(currentLibrary);
}
// Count only live books for the progress label, but sync the FULL library
// (including soft-deleted books): the engine tombstones deleted books in
// library.json so deletions propagate, and keeping them in the input set
// stops the discovery pass from re-downloading a book the user just deleted.
const liveBookCount = currentLibrary.filter((b) => !b.deletedAt).length;
// Lazily ensure a deviceId so the first cross-device sync attributes its
// rows correctly (the reader hook also touches this on first push).
let deviceId = stored.deviceId;
if (!deviceId) {
deviceId = uuidv4();
await persist({ deviceId });
}
// Acquire the global library-sync mutex; bail if another backend's Sync now
// is already mutating the local library.
if (!beginSync(kind, _('Syncing {{n}} / {{total}}', { n: 0, total: liveBookCount }))) {
return;
}
try {
const provider = await createFileSyncProvider(kind, settings);
if (!provider) {
throw new FileSyncError('Sync backend is not available on this device', 'UNKNOWN');
}
const store = createAppLocalStore({ appService, settings, envConfig });
const engine = new FileSyncEngine(provider, store);
const result = await engine.syncLibrary(currentLibrary, {
strategy: stored.strategy === 'prompt' ? 'silent' : stored.strategy,
syncBooks: stored.syncBooks ?? false,
fullSync: stored.fullSync ?? false,
deviceId: deviceId as string,
onProgress: ({ book, index, total, action }) => {
const actionStr = action === 'downloading' ? _('Downloading') : _('Uploading');
updateProgress(
kind,
_('{{action}} {{n}} / {{total}}', { action: actionStr, n: index + 1, total }),
book.title || book.hash.slice(0, 8),
);
},
});
await persist({ lastSyncedAt: Date.now() });
if (result.failures > 0) {
eventDispatcher.dispatch('toast', {
type: 'warning',
message: _('Sync finished with {{failed}} failure(s). {{ok}} ok.', {
failed: result.failures,
ok: Math.max(0, result.totalBooks - result.failures),
}),
});
} else {
eventDispatcher.dispatch('toast', {
type: 'info',
message: _('{{count}} book(s) synced', { count: result.booksSynced }),
});
}
} catch (e) {
eventDispatcher.dispatch('toast', { type: 'error', message: formatSyncError(_, e) });
} finally {
endSync(kind);
}
};
return (
<BoxedList>
<SettingsSwitchRow
label={_('Upload Book Files')}
description={_('Uploads book files to your other devices')}
checked={stored.syncBooks ?? false}
onChange={handleToggleSyncBooks}
/>
<SettingsSwitchRow
label={_('Full Sync')}
description={_('Re-check every book instead of only changed ones')}
checked={stored.fullSync ?? false}
onChange={handleToggleFullSync}
/>
<SettingsRow label={_('Sync Strategy')}>
<SettingsSelect
value={stored.strategy ?? 'silent'}
onChange={handleStrategyChange}
ariaLabel={_('Sync Strategy')}
options={[
{ value: 'silent', label: _('Send and receive') },
{ value: 'send', label: _('Send only') },
{ value: 'receive', label: _('Receive only') },
]}
/>
</SettingsRow>
<SettingsRow
label={
syncProgressLabel
? syncProgressLabel
: stored.lastSyncedAt
? _('Last synced {{when}}', { when: new Date(stored.lastSyncedAt).toLocaleString() })
: _('Never synced')
}
description={
syncProgressDetail ? (
<span className='line-clamp-1'>{syncProgressDetail}</span>
) : undefined
}
>
<button
type='button'
onClick={handleSyncNow}
disabled={isSyncing || syncNowDisabled}
className={clsx(
'btn btn-ghost btn-sm h-8 min-h-8 gap-1 px-2',
(isSyncing || syncNowDisabled) && 'opacity-60',
)}
title={_('Sync now')}
aria-label={_('Sync now')}
>
{isSyncing ? (
<span className='loading loading-spinner loading-xs' />
) : (
<MdCloudSync className='h-4 w-4' />
)}
{_('Sync now')}
</button>
</SettingsRow>
</BoxedList>
);
};
export default FileSyncForm;