akai-tools/readest
2
0
Fork 0
forked from akai/readest
Code Pull Requests Activity

feat(reader): sync paragraph mode & speed reader with TTS read-along (#3235) (#4576)

Browse Source 
* docs(reader): TTS-sync design spec for paragraph mode + RSVP (#3235)

Hardened via brainstorming + /autoplan (CEO/Design/Eng dual-voice review).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(tts): emit canonical tts-position event from TTSController (#3235)

Controller emits { cfi, kind, sectionIndex, sequence } alongside the existing
tts-highlight-mark/-word events. Monotonic sequence lets downstream consumers
(paragraph mode, RSVP — later slices) drop out-of-order positions. Additive;
existing events untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): containment+cursor CFI->index mappers for TTS sync (#3235)

RSVPController.syncToCfi + setExternallyDriven: containment match (fixes
mid-token skip), monotonic cursor + binary search (avoids O(N)-per-word jank,
no per-word getCFI), -1/no-op on no match (no silent jump to word 0), timer
suspension while externally driven.

ParagraphIterator.findIndexByRange: hinted + binary-search containment mapper
returning -1 on no match (never first()).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(tts): forward tts-position + tts-playback-state onto the app bus (#3235)

useTTSControl republishes the controller's canonical tts-position (tagged with
bookKey) via a dedicated listener — NOT inside the suppression-gated highlight
handlers, so page-follow suppression can't silently desync the modes. Adds
tts-playback-state (playing/paused/stopped) so RSVP can track playback without
the hook-local isPlaying. Verified by extending the real-foliate-view browser
harness.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): paragraph mode follows TTS playback (#3235)

When paragraph mode + TTS are both active, the focused paragraph follows the
spoken position (sentence granularity, all engines). Section-generation contract
(stash cross-section position, apply after the iterator re-inits); sync-focus
path that does NOT arm isFocusingRef (avoids the relocate-eaten wrong-section
paragraph-0 bug); stale-sequence drop; decouple on manual nav, re-engage on next
playing. Start-alignment + visible indicator deferred to later slices.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(rsvp): speed reader follows TTS playback (#3235)

Edge word-boundary voices: RSVP shows the spoken word via syncToCfi. Non-Edge
(sentence-only) voices: sentence-paced estimator (clamp 60..600 wpm from voice
rate, hold at +60 words cap, snap to first word on each new sentence mark).
RSVP auto-advance suspended while TTS-driven. Decouple on manual nav via a
rsvp-manual-nav signal; re-engage on next playing. Cross-section positions
re-extract then apply. Pure decideRsvpTtsPosition helper unit-tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): fixed-layout gate + ttsSyncStatus for TTS sync (#3235)

Gate sync to reflowable books (D7): fixed-layout reports 'unsupported' and
never engages. Both modes expose ttsSyncStatus (idle/following/syncing/
decoupled/unsupported) as the data source for the upcoming indicator. RSVPControl
now forwardRef-exposes the status via an imperative handle. Cross-bookKey events
ignored (regression-tested).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): 'following audio' indicator for TTS sync (#3235)

5-state pill (following/syncing/decoupled, idle+unsupported render null) shown
top-center in the paragraph overlay and as a status row in the RSVP overlay.
Decoupled state is the tap-to-resume control; first decouple fires a one-time
toast. eink-bordered, glyph+text (no color-only), RTL logical props, touch
targets, safe-area top inset. RSVP 'plain' variant matches its themed surface;
non-Edge shows '· estimated'. New i18n keys need extraction before merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(rsvp): in-overlay TTS toggle + audio-paced speed control (#3235)

Voice-glyph audio toggle in the RSVP control row (trailing, by the gear) starts/
stops read-along from inside the full-screen overlay, start-aligned to the current
word (range validated against the live doc). While TTS-driven, the WPM control
shows a locked 'Audio pace' affordance that opens a compact rate picker; rate
changes go through a new tts-set-rate bus event reusing the existing throttled
setRate path. Pure buildRsvpTtsSpeakDetail helper unit-tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(reader): e2e paragraph mode follows TTS across a section boundary (#3235)

Real <foliate-view> browser e2e: with paragraph mode active, the focused
paragraph follows the TTS walk and re-targets to the new section after a Ch4->Ch5
boundary (proves no stuck wrong-section paragraph-0 / isFocusingRef trap).
Asserts on the owning section of the current range. Test-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* i18n(reader): translate TTS-sync strings across 33 locales (#3235)

Following audio / · estimated / Resume audio / Stopped following audio /
Play audio / Pause audio / Audio pace / Speed follows audio.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reader): resolve TTS CFI anchors across iframe realms (#3235)

RSVP and paragraph follow silently failed to track the spoken word: the CFI
anchor from view.resolveCFI(...).anchor(doc) is a Range created in the book
iframe's realm, so 'anchor instanceof Range' (top realm) was always false
(cross-realm instanceof) -> resolveCfiToRange/applySyncCfi returned null ->
syncToCfi never advanced. Add isRangeLike() duck-type (cloneRange is unique to
Range) and use it at all 4 CFI-resolution sites. Confirmed live via CDP: before
= syncToCfi false (frozen); after = exact word map + RSVP follows Edge TTS at
~171 wpm (audio pace). Unit tests reproduce the cross-realm anchor (jsdom is
single-realm so the old code passed there but died in the app).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rsvp): stop estimator/word fight + map transport to TTS play/pause (#3235)

Two read-along refinements (verified live via CDP with Edge TTS):

1. No more jump-ahead-then-snap-back flashing. Word-boundary engines (Edge) emit
   BOTH sentence marks and word boundaries; RSVP was routing sentence -> the
   estimator (self-paces ~190xrate, up to +60 words ahead) while word positions
   snapped it back. Now once a word position is seen, sentence positions are
   ignored and any running estimator is stopped, so words alone drive RSVP.

2. The RSVP transport (center play/pause, Space, center-tap) maps to TTS
   play/pause while read-along is engaged (tts-toggle-play), instead of RSVP's
   own suspended timer. Pausing TTS keeps RSVP suspended (no runaway); a full
   stop releases it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rsvp): keep indicator on pause + reach dict management from RSVP (#3235)

- Pausing read-along no longer dismisses the 'following audio' indicator / 'Audio
  pace' lock (layout shift). New 'paused' sync status keeps the indicator row and
  WPM lock present while TTS is engaged-but-paused; only a full stop clears them.
  Verified live via CDP: pause keeps the layout, no shift.
- Dict management is reachable from RSVP: the settings dialog is z-50, far below
  the full-screen RSVP overlay (z-[10000]), so it opened invisibly behind it.
  handleManageDictionary now exits RSVP first (position saved/resumable) so
  management shows over the reader.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rsvp): show dict management over RSVP instead of exiting it (#3235)

Per feedback: opening dictionary management from the RSVP lookup popup no longer
closes the speed reader. The settings dialog is raised above the full-screen RSVP
overlay (z-[10000] -> SettingsDialog !z-[10050]) so it shows on top, and RSVP's
capture-phase keyboard handler bails while the settings dialog is open so its
inputs accept Space and Escape closes settings (not RSVP). Verified live via CDP:
management opens over RSVP, RSVP stays active behind it, Escape returns to it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dict): only apply drag-handle margin compensation when the handle shows (#3235)

The dictionary sheet header used -mt-4 to compensate for Dialog's drag handle,
but that handle is sm:hidden (shown only below sm). On sm+ the handle is
display:none, so -mt-4 pulled the header up into the top edge (broken layout
when the lookup renders as a sheet on a short/wide window). Mirror the handle's
breakpoint: -mt-4 sm:mt-0. Verified live via CDP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): in-mode TTS audio toggle for paragraph mode (#3235)

Paragraph mode already follows TTS, but there was no way to start read-along
from inside it. Add an audio toggle to the ParagraphBar (mirroring RSVP's): it
starts TTS start-aligned to the focused paragraph (range validated live, +
section index) and stops it. Track session-active vs playing so a pause keeps
the indicator ('paused' status) instead of collapsing to idle. Pure
buildParagraphTtsSpeakDetail helper unit-tested. Verified live via CDP: tapping
the icon starts audio from the focused paragraph and the focus follows speech.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reader): highlight current TTS word/sentence in paragraph mode (#3235)

Paragraph mode follows TTS by advancing the focused paragraph, but the spoken
word wasn't highlighted within it like normal mode. The overlay renders a CLONE
of the paragraph, so the iframe's TTS highlight isn't visible there — reproduce
it on the clone with the CSS Custom Highlight API (no DOM mutation, spans inline
boundaries natively, leaves the fade-in animation untouched).

- TTSController already tags tts-position with kind word|sentence. The hook
  decides granularity: word boundaries (Edge) drive a per-word highlight; once
  seen, the coarse sentence event is skipped so the whole sentence doesn't
  flicker over the current word. Engines without word boundaries
  (WebSpeech/Native) fall back to the sentence highlight.
- Offsets are computed relative to the paragraph start (so they map 1:1 onto the
  clone's text) and tagged with the paragraph index so a stale highlight never
  paints the wrong paragraph. Cleared on stop / section change / disabled.
- The ::highlight() style mirrors the user's ttsHighlightOptions color+style.

Pure helpers (offset math, word/sentence decision, css builder) unit-tested.
Verified live via CDP: word highlight tracks Edge word-by-word and follows
across paragraph boundaries (news -> ... -> ladies), matching the TTS color.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Huang Xin
2026-06-14 15:46:18 +08:00
committed by GitHub
parent cc618b8739
commit bfb85c2f68
63 changed files with 4941 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/readest-app/src/utils/paragraph.ts
+76
View File
@@ -212,6 +212,82 @@ export class ParagraphIterator {
return this.first();
}
/**
* Synchronous, containment-based mapper for TTS-driven sync.
*
* Returns the index of the block whose range CONTAINS the target's start
* position. If the target start falls in a gap between blocks, returns the
* nearest FOLLOWING block's index. Returns -1 when the target is after the
* last block, unresolvable, or there are no blocks.
*
* Unlike `findByRangeAsync`, this NEVER falls back to `first()` (index 0) on
* a no-match — a no-match yields -1. `#blocks` are in document order, so this
* uses a binary search; when `hintIndex >= 0`, the hint and its immediate
* neighbours are checked first (forward word-by-word sync usually stays in the
* same or next block).
*/
findIndexByRange(targetRange: Range, hintIndex = -1): number {
const n = this.#blocks.length;
if (n === 0 || !targetRange) return -1;
const point = { node: targetRange.startContainer, offset: targetRange.startOffset };
// Classify a block against the target's start point via Range.comparePoint:
// 1 the point is AFTER this block's end -> block is before the target
// 0 the point is WITHIN this block -> containment (a direct hit)
// -1 the point is BEFORE this block's start -> block follows the target
// Returns null on a comparison error so callers can skip the block.
const classify = (index: number): number | null => {
const block = this.#blocks[index];
if (!block) return null;
try {
return block.comparePoint(point.node, point.offset);
} catch {
return null;
}
};
// Fast path: most word-by-word syncs stay in the hinted block or the next.
if (hintIndex >= 0 && hintIndex < n) {
for (const i of [hintIndex, hintIndex + 1, hintIndex - 1]) {
if (i < 0 || i >= n) continue;
if (classify(i) === 0) return i;
}
}
// Binary search over document-ordered blocks for the first block that does
// NOT sit entirely before the target start (i.e. classify(mid) <= 0). That
// block is either the container (0) or the nearest following block (-1).
let lo = 0;
let hi = n - 1;
let candidate = -1;
while (lo <= hi) {
const mid = (lo + hi) >> 1;
const cmp = classify(mid);
if (cmp === null) {
// Comparison failed for this block; fall back to a linear scan from lo.
for (let i = lo; i <= hi; i++) {
const c = classify(i);
if (c !== null && c <= 0) return i;
}
return -1;
}
if (cmp > 0) {
// Block ends before the target start — search the right half.
lo = mid + 1;
} else {
// Block contains (0) or follows (-1) the target start — remember it and
// search the left half for an earlier match.
candidate = mid;
hi = mid - 1;
}
}
// `candidate` is the first block that contains or follows the target start;
// -1 means every block ended before it (target is after the last block).
return candidate;
}
async findByRangeAsync(targetRange: Range | null, batchSize = 50): Promise<Range | null> {
if (!targetRange) return this.first();
apps/readest-app/src/utils/range.ts
+15
View File
@@ -0,0 +1,15 @@
/**
* Realm-safe check for a DOM Range.
*
* A Range produced by `view.resolveCFI(cfi).anchor(doc)` is created inside the
* book iframe's realm. App-bundle code runs in the top realm, so
* `anchor instanceof Range` compares against the *top* window's `Range`
* constructor and is ALWAYS false for an iframe-realm Range (cross-realm
* instanceof). That silently turns CFI resolution into `null`.
*
* Duck-type on `cloneRange` instead — it is unique to `Range` (a `Node` does not
* have it), so this distinguishes a Range from a Node anchor without depending
* on which realm created it. See the iframe-cross-realm-instanceof learning.
*/
export const isRangeLike = (value: unknown): value is Range =>
typeof value === 'object' && value !== null && typeof (value as Range).cloneRange === 'function';