Implement on-device iOS text-to-speech using AVSpeechSynthesizer,
mirroring the Android native TextToSpeech plugin so the shared
NativeTTSClient drives both platforms through the same command and
tts_events contract.
- Swift NativeTTSPlugin: speak/stop/pause/resume/rate/pitch/voice and
voice enumeration, with region-disambiguated duplicate voice names and
a small preUtteranceDelay to avoid first-word clipping.
- Enable the native TTS client on iOS in TTSController.
- Make TTS teardown resilient: reset UI state up front and tear down the
controller, media session, and background audio in parallel so a slow
native shutdown can never leave the TTS icon or lock-screen session
stuck on.
- Keep iOS on navigator.mediaSession for the lock screen (Android uses
the native foreground service), which restores the Edge TTS cover and
current-sentence metadata.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switching into Paragraph or RSVP mode while TTS is already playing now
syncs to the live session instead of forcing a stop + restart inside the
mode. Bundles several related TTS fixes uncovered along the way.
Session reuse (enter from normal mode):
- TTSController.redispatchPosition() re-emits the current position on the
canonical tts-position signal with a fresh sequence.
- useTTSControl answers a new tts-sync-request by replaying the current
position then playback state (position-first so RSVP's paused handler
can't discard it).
- Paragraph & RSVP engage following on entry and dispatch the request;
no-op when no session exists.
RSVP refinements:
- Reusing a session skips the start dialog and the get-ready countdown
(starts externally driven); gated on a live tts-playback-state signal
so the countdown can't flash.
- Stopping TTS now pauses RSVP instead of resuming its own pacing.
Word-sync fixes:
- rangeTextExcludingInert honours the range offsets inside a single text
node, fixing word-highlight drift on middle sentences of single-<span>
paragraphs (Edge word highlighting).
- foliate-js TTS.from() starts at the sentence containing the selection,
not the next one (submodule bump).
- Selecting a word and starting TTS now clears the selection.
- Dev-only [TTS] word-sync trace (stripped from production builds).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(reader): TTS-sync design spec for paragraph mode + RSVP (#3235)
Hardened via brainstorming + /autoplan (CEO/Design/Eng dual-voice review).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(tts): emit canonical tts-position event from TTSController (#3235)
Controller emits { cfi, kind, sectionIndex, sequence } alongside the existing
tts-highlight-mark/-word events. Monotonic sequence lets downstream consumers
(paragraph mode, RSVP — later slices) drop out-of-order positions. Additive;
existing events untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): containment+cursor CFI->index mappers for TTS sync (#3235)
RSVPController.syncToCfi + setExternallyDriven: containment match (fixes
mid-token skip), monotonic cursor + binary search (avoids O(N)-per-word jank,
no per-word getCFI), -1/no-op on no match (no silent jump to word 0), timer
suspension while externally driven.
ParagraphIterator.findIndexByRange: hinted + binary-search containment mapper
returning -1 on no match (never first()).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(tts): forward tts-position + tts-playback-state onto the app bus (#3235)
useTTSControl republishes the controller's canonical tts-position (tagged with
bookKey) via a dedicated listener — NOT inside the suppression-gated highlight
handlers, so page-follow suppression can't silently desync the modes. Adds
tts-playback-state (playing/paused/stopped) so RSVP can track playback without
the hook-local isPlaying. Verified by extending the real-foliate-view browser
harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): paragraph mode follows TTS playback (#3235)
When paragraph mode + TTS are both active, the focused paragraph follows the
spoken position (sentence granularity, all engines). Section-generation contract
(stash cross-section position, apply after the iterator re-inits); sync-focus
path that does NOT arm isFocusingRef (avoids the relocate-eaten wrong-section
paragraph-0 bug); stale-sequence drop; decouple on manual nav, re-engage on next
playing. Start-alignment + visible indicator deferred to later slices.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(rsvp): speed reader follows TTS playback (#3235)
Edge word-boundary voices: RSVP shows the spoken word via syncToCfi. Non-Edge
(sentence-only) voices: sentence-paced estimator (clamp 60..600 wpm from voice
rate, hold at +60 words cap, snap to first word on each new sentence mark).
RSVP auto-advance suspended while TTS-driven. Decouple on manual nav via a
rsvp-manual-nav signal; re-engage on next playing. Cross-section positions
re-extract then apply. Pure decideRsvpTtsPosition helper unit-tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): fixed-layout gate + ttsSyncStatus for TTS sync (#3235)
Gate sync to reflowable books (D7): fixed-layout reports 'unsupported' and
never engages. Both modes expose ttsSyncStatus (idle/following/syncing/
decoupled/unsupported) as the data source for the upcoming indicator. RSVPControl
now forwardRef-exposes the status via an imperative handle. Cross-bookKey events
ignored (regression-tested).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): 'following audio' indicator for TTS sync (#3235)
5-state pill (following/syncing/decoupled, idle+unsupported render null) shown
top-center in the paragraph overlay and as a status row in the RSVP overlay.
Decoupled state is the tap-to-resume control; first decouple fires a one-time
toast. eink-bordered, glyph+text (no color-only), RTL logical props, touch
targets, safe-area top inset. RSVP 'plain' variant matches its themed surface;
non-Edge shows '· estimated'. New i18n keys need extraction before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(rsvp): in-overlay TTS toggle + audio-paced speed control (#3235)
Voice-glyph audio toggle in the RSVP control row (trailing, by the gear) starts/
stops read-along from inside the full-screen overlay, start-aligned to the current
word (range validated against the live doc). While TTS-driven, the WPM control
shows a locked 'Audio pace' affordance that opens a compact rate picker; rate
changes go through a new tts-set-rate bus event reusing the existing throttled
setRate path. Pure buildRsvpTtsSpeakDetail helper unit-tested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(reader): e2e paragraph mode follows TTS across a section boundary (#3235)
Real <foliate-view> browser e2e: with paragraph mode active, the focused
paragraph follows the TTS walk and re-targets to the new section after a Ch4->Ch5
boundary (proves no stuck wrong-section paragraph-0 / isFocusingRef trap).
Asserts on the owning section of the current range. Test-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* i18n(reader): translate TTS-sync strings across 33 locales (#3235)
Following audio / · estimated / Resume audio / Stopped following audio /
Play audio / Pause audio / Audio pace / Speed follows audio.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reader): resolve TTS CFI anchors across iframe realms (#3235)
RSVP and paragraph follow silently failed to track the spoken word: the CFI
anchor from view.resolveCFI(...).anchor(doc) is a Range created in the book
iframe's realm, so 'anchor instanceof Range' (top realm) was always false
(cross-realm instanceof) -> resolveCfiToRange/applySyncCfi returned null ->
syncToCfi never advanced. Add isRangeLike() duck-type (cloneRange is unique to
Range) and use it at all 4 CFI-resolution sites. Confirmed live via CDP: before
= syncToCfi false (frozen); after = exact word map + RSVP follows Edge TTS at
~171 wpm (audio pace). Unit tests reproduce the cross-realm anchor (jsdom is
single-realm so the old code passed there but died in the app).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(rsvp): stop estimator/word fight + map transport to TTS play/pause (#3235)
Two read-along refinements (verified live via CDP with Edge TTS):
1. No more jump-ahead-then-snap-back flashing. Word-boundary engines (Edge) emit
BOTH sentence marks and word boundaries; RSVP was routing sentence -> the
estimator (self-paces ~190xrate, up to +60 words ahead) while word positions
snapped it back. Now once a word position is seen, sentence positions are
ignored and any running estimator is stopped, so words alone drive RSVP.
2. The RSVP transport (center play/pause, Space, center-tap) maps to TTS
play/pause while read-along is engaged (tts-toggle-play), instead of RSVP's
own suspended timer. Pausing TTS keeps RSVP suspended (no runaway); a full
stop releases it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(rsvp): keep indicator on pause + reach dict management from RSVP (#3235)
- Pausing read-along no longer dismisses the 'following audio' indicator / 'Audio
pace' lock (layout shift). New 'paused' sync status keeps the indicator row and
WPM lock present while TTS is engaged-but-paused; only a full stop clears them.
Verified live via CDP: pause keeps the layout, no shift.
- Dict management is reachable from RSVP: the settings dialog is z-50, far below
the full-screen RSVP overlay (z-[10000]), so it opened invisibly behind it.
handleManageDictionary now exits RSVP first (position saved/resumable) so
management shows over the reader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(rsvp): show dict management over RSVP instead of exiting it (#3235)
Per feedback: opening dictionary management from the RSVP lookup popup no longer
closes the speed reader. The settings dialog is raised above the full-screen RSVP
overlay (z-[10000] -> SettingsDialog !z-[10050]) so it shows on top, and RSVP's
capture-phase keyboard handler bails while the settings dialog is open so its
inputs accept Space and Escape closes settings (not RSVP). Verified live via CDP:
management opens over RSVP, RSVP stays active behind it, Escape returns to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(dict): only apply drag-handle margin compensation when the handle shows (#3235)
The dictionary sheet header used -mt-4 to compensate for Dialog's drag handle,
but that handle is sm:hidden (shown only below sm). On sm+ the handle is
display:none, so -mt-4 pulled the header up into the top edge (broken layout
when the lookup renders as a sheet on a short/wide window). Mirror the handle's
breakpoint: -mt-4 sm:mt-0. Verified live via CDP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): in-mode TTS audio toggle for paragraph mode (#3235)
Paragraph mode already follows TTS, but there was no way to start read-along
from inside it. Add an audio toggle to the ParagraphBar (mirroring RSVP's): it
starts TTS start-aligned to the focused paragraph (range validated live, +
section index) and stops it. Track session-active vs playing so a pause keeps
the indicator ('paused' status) instead of collapsing to idle. Pure
buildParagraphTtsSpeakDetail helper unit-tested. Verified live via CDP: tapping
the icon starts audio from the focused paragraph and the focus follows speech.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reader): highlight current TTS word/sentence in paragraph mode (#3235)
Paragraph mode follows TTS by advancing the focused paragraph, but the spoken
word wasn't highlighted within it like normal mode. The overlay renders a CLONE
of the paragraph, so the iframe's TTS highlight isn't visible there — reproduce
it on the clone with the CSS Custom Highlight API (no DOM mutation, spans inline
boundaries natively, leaves the fade-in animation untouched).
- TTSController already tags tts-position with kind word|sentence. The hook
decides granularity: word boundaries (Edge) drive a per-word highlight; once
seen, the coarse sentence event is skipped so the whole sentence doesn't
flicker over the current word. Engines without word boundaries
(WebSpeech/Native) fall back to the sentence highlight.
- Offsets are computed relative to the paragraph start (so they map 1:1 onto the
clone's text) and tagged with the paragraph index so a stale highlight never
paints the wrong paragraph. Cleared on stop / section change / disabled.
- The ::highlight() style mirrors the user's ttsHighlightOptions color+style.
Pure helpers (offset math, word/sentence decision, css builder) unit-tested.
Verified live via CDP: word highlight tracks Edge word-by-word and follows
across paragraph boundaries (news -> ... -> ladies), matching the TTS color.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Highlight each word as it is spoken (Edge TTS only) instead of keeping the
whole sentence highlighted, and keep the view tracking the spoken word
across page boundaries.
Word boundaries
- Capture Edge's audio.metadata WordBoundary frames (offset/duration in
100ns ticks plus the verbatim input-text span) in the Tauri, browser, and
Cloudflare-Workers WebSocket transports.
- Carry boundaries through the authenticated HTTPS proxy route via an
X-TTS-Word-Boundaries response header (percent-encoded JSON, ASCII-safe),
so word highlighting works on the web where the browser cannot open the
wss connection directly. Cache them alongside the audio blob URL.
Highlighting
- Sync a requestAnimationFrame loop to audio.currentTime against the
boundary table and highlight the word sub-range within the spoken
sentence. Synthesis stays sentence-level (natural prosody); only the
visual highlight is word-level.
- Suppress the sentence highlight when the active client reports word
boundaries and draw the first word immediately, so the whole sentence
never flashes before the first word. Fall back to the sentence highlight
when a chunk has no boundaries (other engines, empty metadata).
- Re-apply the current word (not the sentence) when the view relocates.
Page following
- Turn the page as soon as the spoken word crosses a page boundary (a
tts-highlight-word event scrolls only when the word is outside the visible
range), instead of waiting for the next sentence.
- Check the word's position for the "back to TTS location" badge so it no
longer appears while the view follows the word onto the next page.
Also fixes a pre-existing bug where the browser WebSocket was constructed
with an options object (valid only for the Node ws package), which threw in
browsers and made the wss path unusable on the web.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
setProgress was called multiple times per swipe burst, each call writing
into readerStore.viewStates[key].progress. ~65 places in the reader
subtree subscribed to useReaderStore() without a selector, so every
setProgress fan-out re-rendered all of them -- even the 51 that didn't
care about progress. On Android release builds this showed up as
Layout = 9.8% and Function Call = 9.6% of main-thread self time in
Chrome DevTools' Bottom-Up profile during a reading session.
Fix:
- New tiny store store/readerProgressStore.ts holds the per-book
BookProgress map. setBookProgress only fires its own subscribers.
- readerStore.setProgress now writes progress to the new store and only
touches bookDataStore for the primary view (secondary parallel views
shouldn't overwrite the shared config).
- readerStore.getProgress is kept as a delegating facade so existing
imperative call sites don't break.
- Components / hooks that genuinely need to react to progress changes
subscribe via the new useBookProgress(bookKey) hook. The handful of
call sites that just want a one-shot read use getBookProgress(key) so
they don't subscribe at all.
- readerStore.clearViewState calls clearBookProgress so the map doesn't
grow unbounded across book opens/closes.
See store/readerProgressStore.ts header for the full rationale.
Add a "Highlight Current Sentence" keyboard action (default Shift+M, in the
Text to Speech shortcut section) that persists the sentence TTS is reading
aloud as a normal highlight using the user's default style/color — no text
selection, eyes-off, silent, and idempotent (a repeat press on the same
sentence is a no-op rather than a duplicate).
Flow: the shortcut handler in useBookShortcuts dispatches tts-highlight-sentence
→ useTTSControl (which owns the TTSController) resolves the current sentence via
the new TTSController.getSpokenSentence() and relays create-tts-highlight
→ Annotator builds the BookNote with the pure, unit-tested buildTTSSentenceHighlight
helper and persists/renders it like any other highlight.
Closes#4085
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the redundant "Apply also in Scrolled Mode" options for bars and
margins so scrolled mode renders the header/footer consistently with
paginated mode: transparent, fixed in position, and not obscuring content.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When TTS playback crosses a section boundary, the page would stay on
the last page of the previous chapter while audio continued reading
the next chapter — leaving the user stuck behind the "back-to-TTS"
button.
Two compounding issues since the paginator adjacent-section preloading
landed:
1. `handleSectionChange` called `view.renderer.goTo(resolved)` without
awaiting. `TTSController.#initTTSForSection` does
`await this.onSectionChange?.(sectionIndex)` precisely so the view
can finish navigating before audio of the new section starts, but
the missing await defeated that contract.
2. `handleHighlightMark` returned silently on a cross-section
mismatch (`viewSectionIndex !== ttsSectionIndex`), so when the
renderer.goTo above completed only partially — which can happen on
the new paginator when the target section is already loaded as an
adjacent view and the post-goTo state appears reused without a
visible page flip — there was no second chance to drag the view to
the TTS cfi.
Fix:
- Await `view.renderer.goTo` in `handleSectionChange`.
- In `handleHighlightMark`, run the cross-section branch *before* the
`followingTTSLocationRef` check and call `view.goTo(cfi)` directly,
stamping `sectionChangingTimestampRef` so the back-to-TTS button
stays suppressed while progress.location catches up. Skip only when
the user is actively selecting text.
Adds unit tests covering both the cross-section navigation path and
the in-section scrollToAnchor path.
Guard handleTTSSpeak with a single-flight ref so a second tts-speak
event that arrives while the first is still inside its initial awaits
(initMediaSession / backgroundAudio / TTSController.init) is ignored
instead of racing ahead to construct a second TTSController. Without
this, rapid clicks produced two concurrent speakers talking over each
other because viewState.ttsEnabled is only set at the end of the first
invocation, so the footer bar would dispatch tts-speak twice.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a new TTS tab in settings with media metadata update frequency control
(sentence/paragraph/chapter) to reduce Bluetooth notification spam, and move
TTS highlight settings from Color tab to the new TTS tab. Also add a highlight
opacity setting with live preview in the Color tab.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Huang Xin