forked from akai/readest
bfb85c2f68
* docs(reader): TTS-sync design spec for paragraph mode + RSVP (#3235) Hardened via brainstorming + /autoplan (CEO/Design/Eng dual-voice review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(tts): emit canonical tts-position event from TTSController (#3235) Controller emits { cfi, kind, sectionIndex, sequence } alongside the existing tts-highlight-mark/-word events. Monotonic sequence lets downstream consumers (paragraph mode, RSVP — later slices) drop out-of-order positions. Additive; existing events untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): containment+cursor CFI->index mappers for TTS sync (#3235) RSVPController.syncToCfi + setExternallyDriven: containment match (fixes mid-token skip), monotonic cursor + binary search (avoids O(N)-per-word jank, no per-word getCFI), -1/no-op on no match (no silent jump to word 0), timer suspension while externally driven. ParagraphIterator.findIndexByRange: hinted + binary-search containment mapper returning -1 on no match (never first()). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(tts): forward tts-position + tts-playback-state onto the app bus (#3235) useTTSControl republishes the controller's canonical tts-position (tagged with bookKey) via a dedicated listener — NOT inside the suppression-gated highlight handlers, so page-follow suppression can't silently desync the modes. Adds tts-playback-state (playing/paused/stopped) so RSVP can track playback without the hook-local isPlaying. Verified by extending the real-foliate-view browser harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): paragraph mode follows TTS playback (#3235) When paragraph mode + TTS are both active, the focused paragraph follows the spoken position (sentence granularity, all engines). Section-generation contract (stash cross-section position, apply after the iterator re-inits); sync-focus path that does NOT arm isFocusingRef (avoids the relocate-eaten wrong-section paragraph-0 bug); stale-sequence drop; decouple on manual nav, re-engage on next playing. Start-alignment + visible indicator deferred to later slices. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(rsvp): speed reader follows TTS playback (#3235) Edge word-boundary voices: RSVP shows the spoken word via syncToCfi. Non-Edge (sentence-only) voices: sentence-paced estimator (clamp 60..600 wpm from voice rate, hold at +60 words cap, snap to first word on each new sentence mark). RSVP auto-advance suspended while TTS-driven. Decouple on manual nav via a rsvp-manual-nav signal; re-engage on next playing. Cross-section positions re-extract then apply. Pure decideRsvpTtsPosition helper unit-tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): fixed-layout gate + ttsSyncStatus for TTS sync (#3235) Gate sync to reflowable books (D7): fixed-layout reports 'unsupported' and never engages. Both modes expose ttsSyncStatus (idle/following/syncing/ decoupled/unsupported) as the data source for the upcoming indicator. RSVPControl now forwardRef-exposes the status via an imperative handle. Cross-bookKey events ignored (regression-tested). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): 'following audio' indicator for TTS sync (#3235) 5-state pill (following/syncing/decoupled, idle+unsupported render null) shown top-center in the paragraph overlay and as a status row in the RSVP overlay. Decoupled state is the tap-to-resume control; first decouple fires a one-time toast. eink-bordered, glyph+text (no color-only), RTL logical props, touch targets, safe-area top inset. RSVP 'plain' variant matches its themed surface; non-Edge shows '· estimated'. New i18n keys need extraction before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(rsvp): in-overlay TTS toggle + audio-paced speed control (#3235) Voice-glyph audio toggle in the RSVP control row (trailing, by the gear) starts/ stops read-along from inside the full-screen overlay, start-aligned to the current word (range validated against the live doc). While TTS-driven, the WPM control shows a locked 'Audio pace' affordance that opens a compact rate picker; rate changes go through a new tts-set-rate bus event reusing the existing throttled setRate path. Pure buildRsvpTtsSpeakDetail helper unit-tested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(reader): e2e paragraph mode follows TTS across a section boundary (#3235) Real <foliate-view> browser e2e: with paragraph mode active, the focused paragraph follows the TTS walk and re-targets to the new section after a Ch4->Ch5 boundary (proves no stuck wrong-section paragraph-0 / isFocusingRef trap). Asserts on the owning section of the current range. Test-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * i18n(reader): translate TTS-sync strings across 33 locales (#3235) Following audio / · estimated / Resume audio / Stopped following audio / Play audio / Pause audio / Audio pace / Speed follows audio. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reader): resolve TTS CFI anchors across iframe realms (#3235) RSVP and paragraph follow silently failed to track the spoken word: the CFI anchor from view.resolveCFI(...).anchor(doc) is a Range created in the book iframe's realm, so 'anchor instanceof Range' (top realm) was always false (cross-realm instanceof) -> resolveCfiToRange/applySyncCfi returned null -> syncToCfi never advanced. Add isRangeLike() duck-type (cloneRange is unique to Range) and use it at all 4 CFI-resolution sites. Confirmed live via CDP: before = syncToCfi false (frozen); after = exact word map + RSVP follows Edge TTS at ~171 wpm (audio pace). Unit tests reproduce the cross-realm anchor (jsdom is single-realm so the old code passed there but died in the app). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rsvp): stop estimator/word fight + map transport to TTS play/pause (#3235) Two read-along refinements (verified live via CDP with Edge TTS): 1. No more jump-ahead-then-snap-back flashing. Word-boundary engines (Edge) emit BOTH sentence marks and word boundaries; RSVP was routing sentence -> the estimator (self-paces ~190xrate, up to +60 words ahead) while word positions snapped it back. Now once a word position is seen, sentence positions are ignored and any running estimator is stopped, so words alone drive RSVP. 2. The RSVP transport (center play/pause, Space, center-tap) maps to TTS play/pause while read-along is engaged (tts-toggle-play), instead of RSVP's own suspended timer. Pausing TTS keeps RSVP suspended (no runaway); a full stop releases it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rsvp): keep indicator on pause + reach dict management from RSVP (#3235) - Pausing read-along no longer dismisses the 'following audio' indicator / 'Audio pace' lock (layout shift). New 'paused' sync status keeps the indicator row and WPM lock present while TTS is engaged-but-paused; only a full stop clears them. Verified live via CDP: pause keeps the layout, no shift. - Dict management is reachable from RSVP: the settings dialog is z-50, far below the full-screen RSVP overlay (z-[10000]), so it opened invisibly behind it. handleManageDictionary now exits RSVP first (position saved/resumable) so management shows over the reader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rsvp): show dict management over RSVP instead of exiting it (#3235) Per feedback: opening dictionary management from the RSVP lookup popup no longer closes the speed reader. The settings dialog is raised above the full-screen RSVP overlay (z-[10000] -> SettingsDialog !z-[10050]) so it shows on top, and RSVP's capture-phase keyboard handler bails while the settings dialog is open so its inputs accept Space and Escape closes settings (not RSVP). Verified live via CDP: management opens over RSVP, RSVP stays active behind it, Escape returns to it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dict): only apply drag-handle margin compensation when the handle shows (#3235) The dictionary sheet header used -mt-4 to compensate for Dialog's drag handle, but that handle is sm:hidden (shown only below sm). On sm+ the handle is display:none, so -mt-4 pulled the header up into the top edge (broken layout when the lookup renders as a sheet on a short/wide window). Mirror the handle's breakpoint: -mt-4 sm:mt-0. Verified live via CDP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): in-mode TTS audio toggle for paragraph mode (#3235) Paragraph mode already follows TTS, but there was no way to start read-along from inside it. Add an audio toggle to the ParagraphBar (mirroring RSVP's): it starts TTS start-aligned to the focused paragraph (range validated live, + section index) and stops it. Track session-active vs playing so a pause keeps the indicator ('paused' status) instead of collapsing to idle. Pure buildParagraphTtsSpeakDetail helper unit-tested. Verified live via CDP: tapping the icon starts audio from the focused paragraph and the focus follows speech. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reader): highlight current TTS word/sentence in paragraph mode (#3235) Paragraph mode follows TTS by advancing the focused paragraph, but the spoken word wasn't highlighted within it like normal mode. The overlay renders a CLONE of the paragraph, so the iframe's TTS highlight isn't visible there — reproduce it on the clone with the CSS Custom Highlight API (no DOM mutation, spans inline boundaries natively, leaves the fade-in animation untouched). - TTSController already tags tts-position with kind word|sentence. The hook decides granularity: word boundaries (Edge) drive a per-word highlight; once seen, the coarse sentence event is skipped so the whole sentence doesn't flicker over the current word. Engines without word boundaries (WebSpeech/Native) fall back to the sentence highlight. - Offsets are computed relative to the paragraph start (so they map 1:1 onto the clone's text) and tagged with the paragraph index so a stale highlight never paints the wrong paragraph. Cleared on stop / section change / disabled. - The ::highlight() style mirrors the user's ttsHighlightOptions color+style. Pure helpers (offset math, word/sentence decision, css builder) unit-tested. Verified live via CDP: word highlight tracks Edge word-by-word and follows across paragraph boundaries (news -> ... -> ladies), matching the TTS color. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
864 lines
31 KiB
TypeScript
864 lines
31 KiB
TypeScript
import { useState, useRef, useEffect, useCallback, useMemo } from 'react';
|
|
import { useEnv } from '@/context/EnvContext';
|
|
import { useAuth } from '@/context/AuthContext';
|
|
import { useThemeStore } from '@/store/themeStore';
|
|
import { useBookDataStore } from '@/store/bookDataStore';
|
|
import { useReaderStore } from '@/store/readerStore';
|
|
import { useBookProgress } from '@/store/readerProgressStore';
|
|
import { useProofreadStore } from '@/store/proofreadStore';
|
|
import { TransformContext } from '@/services/transformers/types';
|
|
import { proofreadTransformer } from '@/services/transformers/proofread';
|
|
import { useTranslation } from '@/hooks/useTranslation';
|
|
import { TTSController, TTSMark, TTSHighlightOptions, TTSVoicesGroup } from '@/services/tts';
|
|
import { TauriMediaSession } from '@/libs/mediaSession';
|
|
import { eventDispatcher } from '@/utils/event';
|
|
import { genSSMLRaw, parseSSMLLang } from '@/utils/ssml';
|
|
import { throttle } from '@/utils/throttle';
|
|
import { isCfiInLocation } from '@/utils/cfi';
|
|
import { getLocale } from '@/utils/misc';
|
|
import { buildTTSMediaMetadata } from '@/utils/ttsMetadata';
|
|
import { invokeUseBackgroundAudio } from '@/utils/bridge';
|
|
import { estimateTTSTime } from '@/utils/ttsTime';
|
|
import { useTTSMediaSession } from './useTTSMediaSession';
|
|
|
|
interface UseTTSControlProps {
|
|
bookKey: string;
|
|
onRequestHidePanel?: () => void;
|
|
}
|
|
|
|
export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProps) => {
|
|
const _ = useTranslation();
|
|
const { appService } = useEnv();
|
|
const { user } = useAuth();
|
|
const { isDarkMode } = useThemeStore();
|
|
const getBookData = useBookDataStore((s) => s.getBookData);
|
|
const getView = useReaderStore((s) => s.getView);
|
|
const getProgress = useReaderStore((s) => s.getProgress);
|
|
const getViewSettings = useReaderStore((s) => s.getViewSettings);
|
|
const setViewSettings = useReaderStore((s) => s.setViewSettings);
|
|
const setTTSEnabled = useReaderStore((s) => s.setTTSEnabled);
|
|
const { getMergedRules } = useProofreadStore();
|
|
|
|
const [ttsLang, setTtsLang] = useState<string>('en');
|
|
const [isPlaying, setIsPlaying] = useState(false);
|
|
const [isPaused, setIsPaused] = useState(false);
|
|
const [showIndicator, setShowIndicator] = useState(false);
|
|
const [showTTSBar, setShowTTSBar] = useState(() => !!getViewSettings(bookKey)?.showTTSBar);
|
|
const [showBackToCurrentTTSLocation, setShowBackToCurrentTTSLocation] = useState(false);
|
|
|
|
const [timeoutOption, setTimeoutOption] = useState(0);
|
|
const [timeoutTimestamp, setTimeoutTimestamp] = useState(0);
|
|
const [timeoutFunc, setTimeoutFunc] = useState<ReturnType<typeof setTimeout> | null>(null);
|
|
|
|
const followingTTSLocationRef = useRef(true);
|
|
const sectionChangingTimestampRef = useRef(0);
|
|
const previousSectionLabelRef = useRef<string | undefined>(undefined);
|
|
const ttsControllerRef = useRef<TTSController | null>(null);
|
|
const isStartingTTSRef = useRef(false);
|
|
const [ttsController, setTtsController] = useState<TTSController | null>(null);
|
|
const [ttsClientsInited, setTtsClientsInitialized] = useState(false);
|
|
|
|
const {
|
|
mediaSessionRef,
|
|
unblockAudio,
|
|
releaseUnblockAudio,
|
|
initMediaSession,
|
|
deinitMediaSession,
|
|
} = useTTSMediaSession({ bookKey });
|
|
|
|
// Broadcast playback transitions on the app-wide bus so consumers that
|
|
// can't read the hook-local isPlaying flag (RSVP, paragraph mode) can react.
|
|
const emitPlaybackState = (state: 'playing' | 'paused' | 'stopped') => {
|
|
eventDispatcher.dispatch('tts-playback-state', { bookKey, state });
|
|
};
|
|
|
|
const handleTTSForward = async (event: CustomEvent) => {
|
|
const detail = event.detail as { bookKey: string; byMark?: boolean } | undefined;
|
|
if (detail?.bookKey !== bookKey) return;
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
await ttsController.forward(detail?.byMark ?? false);
|
|
}
|
|
};
|
|
|
|
const handleTTSBackward = async (event: CustomEvent) => {
|
|
const detail = event.detail as { bookKey: string; byMark?: boolean } | undefined;
|
|
if (detail?.bookKey !== bookKey) return;
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
await ttsController.backward(detail?.byMark ?? false);
|
|
}
|
|
};
|
|
|
|
const handleTTSHighlightSentence = (event: CustomEvent) => {
|
|
const detail = event.detail as { bookKey: string } | undefined;
|
|
if (detail?.bookKey !== bookKey) return;
|
|
const sentence = ttsControllerRef.current?.getSpokenSentence();
|
|
if (!sentence) return;
|
|
eventDispatcher.dispatch('create-tts-highlight', { bookKey, ...sentence });
|
|
};
|
|
|
|
// Set the TTS rate from the app bus. The RSVP overlay is full-screen, so its
|
|
// rate picker can't reach the TTS panel; it dispatches `tts-set-rate` and we
|
|
// reuse the same controller rate-change path the panel uses (handleSetRate,
|
|
// defined below — stop→setRate→start while playing, throttled). Also persists
|
|
// the value to viewSettings so it survives like a panel change.
|
|
const handleTTSSetRate = (event: CustomEvent) => {
|
|
const detail = event.detail as { bookKey: string; rate?: number } | undefined;
|
|
if (detail?.bookKey !== bookKey || typeof detail.rate !== 'number') return;
|
|
const viewSettings = getViewSettings(bookKey);
|
|
if (viewSettings) {
|
|
viewSettings.ttsRate = detail.rate;
|
|
setViewSettings(bookKey, viewSettings);
|
|
}
|
|
handleSetRate(detail.rate);
|
|
};
|
|
|
|
const handleTTSTogglePlay = async (event: CustomEvent) => {
|
|
const detail = event.detail as { bookKey: string } | undefined;
|
|
if (detail?.bookKey !== bookKey) return;
|
|
const ttsController = ttsControllerRef.current;
|
|
if (!ttsController) return;
|
|
if (ttsController.state === 'playing') {
|
|
setIsPlaying(false);
|
|
setIsPaused(true);
|
|
emitPlaybackState('paused');
|
|
await ttsController.pause();
|
|
} else {
|
|
setIsPlaying(true);
|
|
setIsPaused(false);
|
|
emitPlaybackState('playing');
|
|
if (ttsController.state === 'paused') {
|
|
await ttsController.resume();
|
|
} else {
|
|
await ttsController.start();
|
|
}
|
|
}
|
|
};
|
|
|
|
useEffect(() => {
|
|
eventDispatcher.on('tts-speak', handleTTSSpeak);
|
|
eventDispatcher.on('tts-stop', handleTTSStop);
|
|
eventDispatcher.on('tts-forward', handleTTSForward);
|
|
eventDispatcher.on('tts-backward', handleTTSBackward);
|
|
eventDispatcher.on('tts-toggle-play', handleTTSTogglePlay);
|
|
eventDispatcher.on('tts-set-rate', handleTTSSetRate);
|
|
eventDispatcher.on('tts-highlight-sentence', handleTTSHighlightSentence);
|
|
return () => {
|
|
eventDispatcher.off('tts-speak', handleTTSSpeak);
|
|
eventDispatcher.off('tts-stop', handleTTSStop);
|
|
eventDispatcher.off('tts-forward', handleTTSForward);
|
|
eventDispatcher.off('tts-backward', handleTTSBackward);
|
|
eventDispatcher.off('tts-toggle-play', handleTTSTogglePlay);
|
|
eventDispatcher.off('tts-set-rate', handleTTSSetRate);
|
|
eventDispatcher.off('tts-highlight-sentence', handleTTSHighlightSentence);
|
|
if (ttsControllerRef.current) {
|
|
ttsControllerRef.current.shutdown();
|
|
ttsControllerRef.current = null;
|
|
}
|
|
};
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, []);
|
|
|
|
// Controller event listeners (re-registered when ttsController changes)
|
|
useEffect(() => {
|
|
if (!ttsController || !bookKey) return;
|
|
const bookData = getBookData(bookKey);
|
|
if (!bookData || !bookData.book) return;
|
|
const { title, author, coverImageUrl } = bookData.book;
|
|
|
|
const handleNeedAuth = () => {
|
|
eventDispatcher.dispatch('toast', {
|
|
message: _('Please log in to use advanced TTS features'),
|
|
type: 'error',
|
|
timeout: 5000,
|
|
});
|
|
};
|
|
|
|
const handleSpeakMark = (e: Event) => {
|
|
const progress = getProgress(bookKey);
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const { sectionLabel } = progress || {};
|
|
const mark = (e as CustomEvent<TTSMark>).detail;
|
|
const ttsMediaMetadata = viewSettings?.ttsMediaMetadata ?? 'sentence';
|
|
|
|
const metadata = buildTTSMediaMetadata({
|
|
markText: mark?.text || '',
|
|
markName: mark?.name || '',
|
|
sectionLabel: sectionLabel || '',
|
|
title,
|
|
author,
|
|
ttsMediaMetadata,
|
|
previousSectionLabel: previousSectionLabelRef.current,
|
|
});
|
|
|
|
if (ttsMediaMetadata === 'chapter') {
|
|
previousSectionLabelRef.current = sectionLabel;
|
|
}
|
|
|
|
if (metadata.shouldUpdate && mediaSessionRef.current) {
|
|
const mediaSession = mediaSessionRef.current;
|
|
if (mediaSession instanceof TauriMediaSession) {
|
|
mediaSession.updateMetadata({
|
|
title: metadata.title,
|
|
artist: metadata.artist,
|
|
album: metadata.album,
|
|
artwork: '',
|
|
});
|
|
} else {
|
|
mediaSession.metadata = new MediaMetadata({
|
|
title: metadata.title,
|
|
artist: metadata.artist,
|
|
album: metadata.album,
|
|
artwork: [{ src: coverImageUrl || '/icon.png', sizes: '512x512', type: 'image/png' }],
|
|
});
|
|
}
|
|
}
|
|
};
|
|
|
|
const handleHighlightMark = (e: Event) => {
|
|
const { cfi } = (e as CustomEvent<{ cfi: string }>).detail;
|
|
const view = getView(bookKey);
|
|
const progress = getProgress(bookKey);
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const { location } = progress || {};
|
|
if (!cfi || !view || !location || !viewSettings) return;
|
|
|
|
viewSettings.ttsLocation = cfi;
|
|
setViewSettings(bookKey, viewSettings);
|
|
|
|
const hlContents = view.renderer.getContents();
|
|
const hlPrimaryIdx = view.renderer.primaryIndex;
|
|
const { doc, index: viewSectionIndex } = (hlContents.find((x) => x.index === hlPrimaryIdx) ??
|
|
hlContents[0]) as {
|
|
doc: Document;
|
|
index?: number;
|
|
};
|
|
|
|
const { anchor, index: ttsSectionIndex } = view.resolveCFI(cfi);
|
|
if (viewSectionIndex !== ttsSectionIndex) {
|
|
// TTS crossed into a new section before the view caught up. The
|
|
// `await onSectionChange` path in TTSController fires renderer.goTo
|
|
// via handleSectionChange, but the new paginator's #goTo can resolve
|
|
// before the visible page actually flips when the target section is
|
|
// already preloaded as an adjacent view — leaving the user stuck on
|
|
// the last page of the previous chapter while audio continues. Drive
|
|
// navigation from the highlight cfi directly, stamping the timestamp
|
|
// so the "back-to-TTS" button stays suppressed while progress.location
|
|
// catches up. Skip only when the user is actively selecting text.
|
|
if (hlContents.some(({ doc }) => (doc.getSelection()?.toString().length ?? 0) > 0)) {
|
|
return;
|
|
}
|
|
sectionChangingTimestampRef.current = Date.now();
|
|
followingTTSLocationRef.current = true;
|
|
view.goTo?.(cfi);
|
|
return;
|
|
}
|
|
|
|
if (!followingTTSLocationRef.current) return;
|
|
|
|
if (hlContents.some(({ doc }) => (doc.getSelection()?.toString().length ?? 0) > 0)) {
|
|
return;
|
|
}
|
|
|
|
const range = anchor(doc);
|
|
if (!view.renderer.scrolled) {
|
|
view.renderer.scrollToAnchor?.(range);
|
|
} else {
|
|
const rect = range.getBoundingClientRect();
|
|
const { start, end, sideProp } = view.renderer;
|
|
const rangeTop = rect[sideProp === 'height' ? 'y' : 'x'];
|
|
const rangeBottom = rangeTop + rect[sideProp === 'height' ? 'height' : 'width'];
|
|
|
|
const showHeader = viewSettings.showHeader;
|
|
const showFooter = viewSettings.showFooter;
|
|
const headerScrollOverlap = showHeader ? viewSettings.marginTopPx : 0;
|
|
const footerScrollOverlap = showFooter ? viewSettings.marginBottomPx : 0;
|
|
const scrollingOverlap = viewSettings.scrollingOverlap;
|
|
const outOfView =
|
|
rangeBottom > end - footerScrollOverlap - scrollingOverlap ||
|
|
rangeTop < start + headerScrollOverlap + scrollingOverlap;
|
|
if (outOfView) {
|
|
view.renderer.scrollToAnchor?.(range);
|
|
}
|
|
}
|
|
};
|
|
|
|
// Word-level page following: turn the page as soon as the spoken word
|
|
// moves off the visible page, instead of waiting for the next sentence's
|
|
// mark. Only navigates when the word is outside the visible range, so
|
|
// on-page words don't trigger relocations.
|
|
const handleHighlightWord = (e: Event) => {
|
|
const { cfi } = (e as CustomEvent<{ cfi: string }>).detail;
|
|
const view = getView(bookKey);
|
|
if (!cfi || !view || !followingTTSLocationRef.current) return;
|
|
|
|
const hlContents = view.renderer.getContents();
|
|
const hlPrimaryIdx = view.renderer.primaryIndex;
|
|
const { doc, index: viewSectionIndex } = (hlContents.find((x) => x.index === hlPrimaryIdx) ??
|
|
hlContents[0]) as { doc: Document; index?: number };
|
|
|
|
const { anchor, index: ttsSectionIndex } = view.resolveCFI(cfi);
|
|
// Cross-section navigation is driven by the sentence-level mark handler.
|
|
if (viewSectionIndex !== ttsSectionIndex) return;
|
|
if (hlContents.some(({ doc }) => (doc.getSelection()?.toString().length ?? 0) > 0)) return;
|
|
|
|
const wordRange = anchor(doc);
|
|
const visibleRange = getProgress(bookKey)?.range as Range | undefined;
|
|
if (!wordRange || !visibleRange) return;
|
|
|
|
try {
|
|
const ahead = wordRange.compareBoundaryPoints(Range.END_TO_START, visibleRange) > 0;
|
|
const behind = wordRange.compareBoundaryPoints(Range.START_TO_END, visibleRange) < 0;
|
|
if (ahead || behind) {
|
|
view.renderer.scrollToAnchor?.(wordRange);
|
|
}
|
|
} catch {
|
|
// Ranges may briefly belong to different documents during a section
|
|
// change; the mark handler takes over in that case.
|
|
}
|
|
};
|
|
|
|
// Republish the controller's canonical position signal onto the app-wide
|
|
// bus so paragraph mode + RSVP can follow TTS without touching the
|
|
// controller. This MUST be its own listener: handleHighlightMark /
|
|
// handleHighlightWord early-return on following-suppression and text
|
|
// selection, which would silently stop the modes from following. The
|
|
// forward fires on every controller 'tts-position', gated only by the
|
|
// listener's lifecycle (it exists only while the controller does).
|
|
const handlePosition = (e: Event) => {
|
|
eventDispatcher.dispatch('tts-position', {
|
|
bookKey,
|
|
...(e as CustomEvent).detail,
|
|
});
|
|
};
|
|
|
|
ttsController.addEventListener('tts-need-auth', handleNeedAuth);
|
|
ttsController.addEventListener('tts-speak-mark', handleSpeakMark);
|
|
ttsController.addEventListener('tts-highlight-mark', handleHighlightMark);
|
|
ttsController.addEventListener('tts-highlight-word', handleHighlightWord);
|
|
ttsController.addEventListener('tts-position', handlePosition);
|
|
return () => {
|
|
ttsController.removeEventListener('tts-need-auth', handleNeedAuth);
|
|
ttsController.removeEventListener('tts-speak-mark', handleSpeakMark);
|
|
ttsController.removeEventListener('tts-highlight-mark', handleHighlightMark);
|
|
ttsController.removeEventListener('tts-highlight-word', handleHighlightWord);
|
|
ttsController.removeEventListener('tts-position', handlePosition);
|
|
};
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, [ttsController, bookKey]);
|
|
|
|
// Location tracking — re-highlight when progress changes.
|
|
// Reactive subscription via readerProgressStore so the effect below
|
|
// re-runs on page turns without dragging in the whole readerStore.
|
|
const progress = useBookProgress(bookKey);
|
|
useEffect(() => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (!ttsController) return;
|
|
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const ttsLocation = viewSettings?.ttsLocation;
|
|
const { location } = progress || {};
|
|
if (!location || !ttsLocation) return;
|
|
|
|
// Check the actual highlighted position against the view. During
|
|
// word-by-word playback the word can sit on a different page than the
|
|
// sentence's ttsLocation (a sentence spanning a page break), so the word
|
|
// position is the correct reference — otherwise the back-to-TTS button
|
|
// wrongly appears after the view follows the word onto the next page.
|
|
const highlightCfi = ttsController.getCurrentHighlightCfi() ?? ttsLocation;
|
|
if (isCfiInLocation(highlightCfi, location)) {
|
|
setShowBackToCurrentTTSLocation(false);
|
|
// Word-aware re-apply: re-draws the current word during word-by-word
|
|
// playback instead of redrawing the whole sentence over it.
|
|
ttsController.reapplyCurrentHighlight();
|
|
} else {
|
|
const msSinceSectionChange = Date.now() - sectionChangingTimestampRef.current;
|
|
if (msSinceSectionChange < 2000) return;
|
|
setShowBackToCurrentTTSLocation(true);
|
|
}
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, [progress]);
|
|
|
|
// Location tracking — keep followingTTSLocationRef in sync with showBackToCurrentTTSLocation
|
|
useEffect(() => {
|
|
if (showBackToCurrentTTSLocation) {
|
|
followingTTSLocationRef.current = false;
|
|
} else {
|
|
followingTTSLocationRef.current = true;
|
|
}
|
|
}, [showBackToCurrentTTSLocation]);
|
|
|
|
// Location tracking — handleBackToCurrentTTSLocation
|
|
const handleBackToCurrentTTSLocation = () => {
|
|
const view = getView(bookKey);
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const ttsLocation = viewSettings?.ttsLocation;
|
|
if (!view || !ttsLocation) return;
|
|
|
|
const resolved = view.resolveNavigation(ttsLocation);
|
|
view.renderer.goTo?.(resolved);
|
|
};
|
|
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const bookData = getBookData(bookKey);
|
|
const ttsTime = useMemo(() => {
|
|
const rate = viewSettings?.ttsRate ?? 1;
|
|
return estimateTTSTime(progress, rate);
|
|
}, [progress, viewSettings?.ttsRate]);
|
|
|
|
const getTTSTargetLang = useCallback((): string | null => {
|
|
const vs = getViewSettings(bookKey);
|
|
const ttsReadAloudText = vs?.ttsReadAloudText;
|
|
if (vs?.translationEnabled && ttsReadAloudText === 'translated') {
|
|
return vs?.translateTargetLang || getLocale();
|
|
} else if (vs?.translationEnabled && ttsReadAloudText === 'source') {
|
|
return bookData?.book?.primaryLanguage || '';
|
|
}
|
|
return null;
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, [
|
|
bookKey,
|
|
getBookData,
|
|
getViewSettings,
|
|
viewSettings?.translationEnabled,
|
|
viewSettings?.ttsReadAloudText,
|
|
viewSettings?.translateTargetLang,
|
|
]);
|
|
|
|
useEffect(() => {
|
|
ttsControllerRef.current?.setTargetLang(getTTSTargetLang() || '');
|
|
}, [getTTSTargetLang]);
|
|
|
|
// SSML preprocessing
|
|
const transformCtx: TransformContext = useMemo(
|
|
() => ({
|
|
bookKey,
|
|
viewSettings: getViewSettings(bookKey)!,
|
|
userLocale: getLocale(),
|
|
isFixedLayout: bookData?.isFixedLayout || false,
|
|
content: '',
|
|
transformers: [],
|
|
reversePunctuationTransform: true,
|
|
}),
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
[],
|
|
);
|
|
|
|
const preprocessSSMLForTTS = useCallback(
|
|
async (ssml: string) => {
|
|
const rules = getMergedRules(bookKey);
|
|
const viewSettings = getViewSettings(bookKey)!;
|
|
const ttsOnlyRules = rules.filter(
|
|
(rule) =>
|
|
rule.enabled && rule.onlyForTTS && (rule.scope === 'book' || rule.scope === 'library'),
|
|
);
|
|
if (ttsOnlyRules.length === 0) return ssml;
|
|
|
|
transformCtx['content'] = ssml;
|
|
transformCtx['viewSettings'] = viewSettings;
|
|
ssml = await proofreadTransformer.transform(transformCtx, {
|
|
docType: 'text/xml',
|
|
onlyForTTS: true,
|
|
});
|
|
return ssml;
|
|
},
|
|
[bookKey, getMergedRules, getViewSettings, transformCtx],
|
|
);
|
|
|
|
// Section change callback
|
|
const handleSectionChange = useCallback(
|
|
async (sectionIndex: number) => {
|
|
if (!followingTTSLocationRef.current) return;
|
|
const view = getView(bookKey);
|
|
const sections = view?.book.sections;
|
|
if (!sections || sectionIndex < 0 || sectionIndex >= sections.length) return;
|
|
sectionChangingTimestampRef.current = Date.now();
|
|
const resolved = view.resolveNavigation(sectionIndex);
|
|
// Await so TTSController's `await onSectionChange` doesn't proceed to
|
|
// speak the new section before the view has finished navigating to it.
|
|
await view.renderer.goTo?.(resolved);
|
|
},
|
|
[bookKey, getView],
|
|
);
|
|
|
|
// TTS highlight options
|
|
const getTTSHighlightOptions = useCallback(
|
|
(ttsHighlightOptions: TTSHighlightOptions, isEink: boolean) => {
|
|
const einkBgColor = isDarkMode ? '#000000' : '#ffffff';
|
|
const color = isEink ? einkBgColor : ttsHighlightOptions.color;
|
|
return {
|
|
...ttsHighlightOptions,
|
|
color,
|
|
};
|
|
},
|
|
[isDarkMode],
|
|
);
|
|
|
|
useEffect(() => {
|
|
const ttsHighlightOptions = viewSettings?.ttsHighlightOptions;
|
|
if (ttsControllerRef.current && ttsHighlightOptions) {
|
|
ttsControllerRef.current.updateHighlightOptions(
|
|
getTTSHighlightOptions(ttsHighlightOptions, viewSettings!.isEink),
|
|
);
|
|
}
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, [viewSettings?.ttsHighlightOptions, viewSettings?.isEink, getTTSHighlightOptions]);
|
|
|
|
// handleStop (defined before handleTTSSpeak/handleTTSStop which reference it)
|
|
const handleStop = useCallback(
|
|
async (bookKey: string) => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
await ttsController.shutdown();
|
|
ttsControllerRef.current = null;
|
|
setTtsController(null);
|
|
getView(bookKey)?.deselect();
|
|
setIsPlaying(false);
|
|
emitPlaybackState('stopped');
|
|
onRequestHidePanel?.();
|
|
setShowIndicator(false);
|
|
setShowBackToCurrentTTSLocation(false);
|
|
}
|
|
previousSectionLabelRef.current = undefined;
|
|
if (appService?.isIOSApp) {
|
|
await invokeUseBackgroundAudio({ enabled: false });
|
|
}
|
|
if (appService?.isMobile) {
|
|
releaseUnblockAudio();
|
|
}
|
|
await deinitMediaSession();
|
|
setTTSEnabled(bookKey, false);
|
|
},
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
[appService],
|
|
);
|
|
|
|
// handleTTSSpeak / handleTTSStop (plain functions, registered once at mount via closure)
|
|
const handleTTSSpeak = async (event: CustomEvent) => {
|
|
const { bookKey: ttsBookKey, range, index, oneTime = false } = event.detail;
|
|
if (bookKey !== ttsBookKey) return;
|
|
// Guard against concurrent starts (e.g. rapid double-clicks on the TTS
|
|
// icon). Without this, both invocations race past the `await`s below and
|
|
// end up creating two TTSController instances that speak simultaneously.
|
|
if (isStartingTTSRef.current) return;
|
|
isStartingTTSRef.current = true;
|
|
|
|
try {
|
|
const view = getView(bookKey);
|
|
const progress = getProgress(bookKey);
|
|
const viewSettings = getViewSettings(bookKey);
|
|
const bookData = getBookData(bookKey);
|
|
const { location } = progress || {};
|
|
if (!view || !progress || !viewSettings || !bookData || !bookData.book) return;
|
|
const ttsSpeakRange = range as Range | null;
|
|
let ttsFromRange = ttsSpeakRange;
|
|
let ttsFromIndex = typeof index === 'number' ? index : null;
|
|
if (!ttsFromRange && viewSettings.ttsLocation) {
|
|
const ttsCfi = viewSettings.ttsLocation;
|
|
if (isCfiInLocation(ttsCfi, location)) {
|
|
const { index, anchor } = view.resolveCFI(ttsCfi);
|
|
const { doc } = view.renderer.getContents().find((x) => x.index === index) || {};
|
|
if (doc) {
|
|
ttsFromRange = anchor(doc);
|
|
ttsFromIndex = index;
|
|
}
|
|
}
|
|
}
|
|
|
|
if (!ttsFromIndex) {
|
|
ttsFromIndex = progress.index;
|
|
}
|
|
|
|
if (!ttsFromRange && !bookData.isFixedLayout) {
|
|
ttsFromRange = progress.range;
|
|
}
|
|
|
|
const currentSection = view.renderer.getContents().find((x) => x.index === ttsFromIndex);
|
|
if (ttsFromRange && currentSection) {
|
|
const ttsLocation = view.getCFI(currentSection?.index || 0, ttsFromRange);
|
|
viewSettings.ttsLocation = ttsLocation;
|
|
setViewSettings(bookKey, viewSettings);
|
|
if (isCfiInLocation(ttsLocation, location)) {
|
|
setShowBackToCurrentTTSLocation(false);
|
|
}
|
|
}
|
|
|
|
const primaryLang = bookData.book.primaryLanguage;
|
|
|
|
if (ttsControllerRef.current) {
|
|
ttsControllerRef.current.stop();
|
|
ttsControllerRef.current = null;
|
|
}
|
|
|
|
try {
|
|
if (appService?.isIOSApp) {
|
|
await invokeUseBackgroundAudio({ enabled: true });
|
|
}
|
|
if (appService?.isMobile) {
|
|
unblockAudio();
|
|
}
|
|
await initMediaSession();
|
|
setTtsClientsInitialized(false);
|
|
|
|
setShowIndicator(true);
|
|
const ttsController = new TTSController(
|
|
appService,
|
|
view,
|
|
!!user?.id,
|
|
preprocessSSMLForTTS,
|
|
handleSectionChange,
|
|
);
|
|
ttsControllerRef.current = ttsController;
|
|
setTtsController(ttsController);
|
|
|
|
await ttsController.init();
|
|
await ttsController.initViewTTS(ttsFromIndex);
|
|
ttsController.updateHighlightOptions(
|
|
getTTSHighlightOptions(viewSettings.ttsHighlightOptions, viewSettings.isEink),
|
|
);
|
|
const ssml =
|
|
oneTime && ttsSpeakRange
|
|
? genSSMLRaw(ttsSpeakRange.toString().trim())
|
|
: ttsFromRange
|
|
? view.tts?.from(ttsFromRange)
|
|
: view.tts?.start();
|
|
if (ssml) {
|
|
const lang = parseSSMLLang(ssml, primaryLang) || 'en';
|
|
setIsPlaying(true);
|
|
emitPlaybackState('playing');
|
|
setTtsLang(lang);
|
|
|
|
ttsController.setLang(lang);
|
|
ttsController.setRate(viewSettings.ttsRate);
|
|
ttsController.speak(ssml, oneTime, () => handleStop(bookKey));
|
|
ttsController.setTargetLang(getTTSTargetLang() || '');
|
|
}
|
|
setTtsClientsInitialized(true);
|
|
setTTSEnabled(bookKey, true);
|
|
} catch (error) {
|
|
eventDispatcher.dispatch('toast', {
|
|
message: _('TTS not supported for this document'),
|
|
type: 'error',
|
|
});
|
|
console.error(error);
|
|
}
|
|
} finally {
|
|
isStartingTTSRef.current = false;
|
|
}
|
|
};
|
|
|
|
const handleTTSStop = async (event: CustomEvent) => {
|
|
const { bookKey: ttsBookKey } = event.detail;
|
|
if (ttsControllerRef.current && bookKey === ttsBookKey) {
|
|
handleStop(bookKey);
|
|
}
|
|
};
|
|
|
|
// Playback callbacks
|
|
const handleTogglePlay = useCallback(async () => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (!ttsController) return;
|
|
|
|
if (isPlaying) {
|
|
setIsPlaying(false);
|
|
setIsPaused(true);
|
|
emitPlaybackState('paused');
|
|
await ttsController.pause();
|
|
} else if (isPaused) {
|
|
setIsPlaying(true);
|
|
setIsPaused(false);
|
|
emitPlaybackState('playing');
|
|
// start for forward/backward/setvoice-paused
|
|
// set rate don't pause the tts
|
|
if (ttsController.state === 'paused') {
|
|
await ttsController.resume();
|
|
} else {
|
|
await ttsController.start();
|
|
}
|
|
}
|
|
|
|
if (mediaSessionRef.current) {
|
|
const mediaSession = mediaSessionRef.current;
|
|
if (mediaSession instanceof TauriMediaSession) {
|
|
await mediaSession.updatePlaybackState({ playing: !isPlaying });
|
|
} else {
|
|
mediaSession.playbackState = isPlaying ? 'paused' : 'playing';
|
|
}
|
|
}
|
|
}, [isPlaying, isPaused, mediaSessionRef]);
|
|
|
|
const handleBackward = useCallback(async (byMark = false) => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
await ttsController.backward(byMark);
|
|
}
|
|
}, []);
|
|
|
|
const handleForward = useCallback(async (byMark = false) => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
await ttsController.forward(byMark);
|
|
}
|
|
}, []);
|
|
|
|
const handlePause = useCallback(async () => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
setIsPlaying(false);
|
|
setIsPaused(true);
|
|
emitPlaybackState('paused');
|
|
await ttsController.pause();
|
|
}
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
}, []);
|
|
|
|
// Rate/voice/timeout/bar controls
|
|
// rate range: 0.5 - 3, 1.0 is normal speed
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
const handleSetRate = useCallback(
|
|
throttle(async (rate: number) => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
if (ttsController.state === 'playing') {
|
|
await ttsController.stop();
|
|
await ttsController.setRate(rate);
|
|
await ttsController.start();
|
|
} else {
|
|
await ttsController.setRate(rate);
|
|
}
|
|
}
|
|
}, 3000),
|
|
[],
|
|
);
|
|
|
|
// eslint-disable-next-line react-hooks/exhaustive-deps
|
|
const handleSetVoice = useCallback(
|
|
throttle(async (voice: string, lang: string) => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
if (ttsController.state === 'playing') {
|
|
await ttsController.stop();
|
|
await ttsController.setVoice(voice, lang);
|
|
await ttsController.start();
|
|
} else {
|
|
await ttsController.setVoice(voice, lang);
|
|
}
|
|
}
|
|
}, 3000),
|
|
[],
|
|
);
|
|
|
|
const handleGetVoices = async (lang: string): Promise<TTSVoicesGroup[]> => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
return ttsController.getVoices(lang);
|
|
}
|
|
return [];
|
|
};
|
|
|
|
const handleGetVoiceId = () => {
|
|
const ttsController = ttsControllerRef.current;
|
|
if (ttsController) {
|
|
return ttsController.getVoiceId();
|
|
}
|
|
return '';
|
|
};
|
|
|
|
const handleSelectTimeout = (bookKey: string, value: number) => {
|
|
setTimeoutOption(value);
|
|
if (timeoutFunc) {
|
|
clearTimeout(timeoutFunc);
|
|
}
|
|
if (value > 0) {
|
|
setTimeoutFunc(
|
|
setTimeout(() => {
|
|
handleStop(bookKey);
|
|
}, value * 1000),
|
|
);
|
|
setTimeoutTimestamp(Date.now() + value * 1000);
|
|
} else {
|
|
setTimeoutTimestamp(0);
|
|
}
|
|
};
|
|
|
|
const handleToggleTTSBar = () => {
|
|
const viewSettings = getViewSettings(bookKey)!;
|
|
viewSettings.showTTSBar = !viewSettings.showTTSBar;
|
|
setShowTTSBar(viewSettings.showTTSBar);
|
|
if (viewSettings.showTTSBar) {
|
|
onRequestHidePanel?.();
|
|
}
|
|
setViewSettings(bookKey, viewSettings);
|
|
};
|
|
|
|
const refreshTtsLang = useCallback(() => {
|
|
const speakingLang = ttsControllerRef.current?.getSpeakingLang();
|
|
if (speakingLang) {
|
|
setTtsLang(speakingLang);
|
|
}
|
|
}, []);
|
|
|
|
// Media session action handler effect
|
|
useEffect(() => {
|
|
const { current: mediaSession } = mediaSessionRef;
|
|
if (mediaSession) {
|
|
mediaSession.setActionHandler('play', () => {
|
|
handleTogglePlay();
|
|
});
|
|
|
|
mediaSession.setActionHandler('pause', () => {
|
|
handleTogglePlay();
|
|
});
|
|
|
|
mediaSession.setActionHandler('stop', () => {
|
|
handlePause();
|
|
});
|
|
|
|
mediaSession.setActionHandler('seekforward', () => {
|
|
handleForward(true);
|
|
});
|
|
|
|
mediaSession.setActionHandler('seekbackward', () => {
|
|
handleBackward(true);
|
|
});
|
|
|
|
mediaSession.setActionHandler('nexttrack', () => {
|
|
handleForward();
|
|
});
|
|
|
|
mediaSession.setActionHandler('previoustrack', () => {
|
|
handleBackward();
|
|
});
|
|
}
|
|
}, [handleTogglePlay, handlePause, handleForward, handleBackward, mediaSessionRef]);
|
|
|
|
return {
|
|
isPlaying,
|
|
isPaused,
|
|
ttsLang,
|
|
ttsClientsInited,
|
|
isTTSActive: ttsController !== null,
|
|
showIndicator,
|
|
showTTSBar,
|
|
showBackToCurrentTTSLocation,
|
|
timeoutOption,
|
|
timeoutTimestamp,
|
|
chapterRemainingSec: ttsTime.chapterRemainingSec,
|
|
bookRemainingSec: ttsTime.bookRemainingSec,
|
|
finishAtTimestamp: ttsTime.finishAtTimestamp,
|
|
handleTogglePlay,
|
|
handleBackward,
|
|
handleForward,
|
|
handlePause,
|
|
handleSetRate,
|
|
handleSetVoice,
|
|
handleGetVoices,
|
|
handleGetVoiceId,
|
|
handleSelectTimeout,
|
|
handleToggleTTSBar,
|
|
handleBackToCurrentTTSLocation,
|
|
refreshTtsLang,
|
|
};
|
|
};
|